| |
Main
Date: 06 Aug 2006 20:27:40
From: Dave (from the UK)
Subject: Script to classify ICC games as Bullet, Blitz or Standard
|
I've written a small UNIX/Linux shell script that reads from a PGN file
containing one or more chess games and adds to the PGN file either:
[GameTypeOnICC "Bullet"]
[GameTypeOnICC "Blitz"]
[GameTypeOnICC "Standard"]
for each game played. So you can search games in a database based on the
type of game.
The script does this based on the 'TimeControl' header added by ICC in
the PGN file. In other words it extracts the initial time and increment,
then determines whether it is Bullet, Blitz or Standard according to the
formula used on ICC.
So instead of the PGN file having this one line:
[TimeControl "180+1"]
it will now have an additional line:
[GameTypeOnICC "Blitz"]
[TimeControl "180+1"]
It should make it easier split up your games and perhaps search them by
type in a database.
It is a bit slow, doing only 20 lines/s on my 450 MHz machine, so is
perhaps not ideally suited to crunch though a million game PGN file !!
It can be dowloaded from
http://www.drkirkby.co.uk/chess/tools/rating
It reads from a file and prints to standard output.
$ rating mygames.pgn > mygames-with-type.pgn
--
Dave (from the UK)
Please note my email address changes periodically to avoid spam.
It is always of the form: [email protected]
Hitting reply will work for a few months only - later set it manually.
http://witm.sourceforge.net/ (Web based Mathematica front end)
|
|
| |
Date: 06 Aug 2006 16:10:46
From: Chris F.A. Johnson
Subject: Re: Script to classify ICC games as Bullet, Blitz or Standard
|
On 2006-08-06, Dave (from the UK) wrote:
> I've written a small UNIX/Linux shell script that reads from a PGN file
> containing one or more chess games and adds to the PGN file either:
>
> [GameTypeOnICC "Bullet"]
> [GameTypeOnICC "Blitz"]
> [GameTypeOnICC "Standard"]
>
> for each game played. So you can search games in a database based on the
> type of game.
>
> The script does this based on the 'TimeControl' header added by ICC in
> the PGN file. In other words it extracts the initial time and increment,
> then determines whether it is Bullet, Blitz or Standard according to the
> formula used on ICC.
>
> So instead of the PGN file having this one line:
>
> [TimeControl "180+1"]
>
> it will now have an additional line:
>
> [GameTypeOnICC "Blitz"]
> [TimeControl "180+1"]
>
> It should make it easier split up your games and perhaps search them by
> type in a database.
>
> It is a bit slow, doing only 20 lines/s on my 450 MHz machine, so is
> perhaps not ideally suited to crunch though a million game PGN file !!
It's slow because you call an external command, grep, for every
line of the file, and then call more external commands when the
line matches.
The greatest speed-up will come if you replace these lines:
str=`echo $f
|
| | |
Date: 07 Aug 2006 06:55:07
From: Dave (from the UK)
Subject: Re: Script to classify ICC games as Bullet, Blitz or Standard
|
Chris F.A. Johnson wrote:
> It's slow because you call an external command, grep, for every
> line of the file, and then call more external commands when the
> line matches.
>
> The greatest speed-up will come if you replace these lines:
>
> str=`echo $f
|
| | | |
Date: 07 Aug 2006 02:27:32
From: Chris F.A. Johnson
Subject: Re: Script to classify ICC games as Bullet, Blitz or Standard
|
Much faster than anything else will be to use awk for the whole
script:
awk -F '"' '/TimeControl.*[0-9]+[0-9]/ {
split( $2, t, "+" )
etime = t[1] + t[2] * 40
if ( etime > 900 )
type = "Standard"
else if ( etime > 180 )
type = "Blitz"
else
type = "Bullet"
printf "[GameTypeOnICC \"%s\"]\n", type
}
{ print $0 }' "$@"
--
Chris F.A. Johnson <http://cfaj.freeshell.org >
===================================================================
Author:
Shell Scripting Recipes: A Problem-Solution Approach (2005, Apress)
|
| | | | |
Date: 07 Aug 2006 12:40:01
From: Dave (from the UK)
Subject: Re: Script to classify ICC games as Bullet, Blitz or Standard
|
Chris F.A. Johnson wrote:
> Much faster than anything else will be to use awk for the whole
> script:
>
> awk -F '"' '/TimeControl.*[0-9]+[0-9]/ {
> split( $2, t, "+" )
> etime = t[1] + t[2] * 40
> if ( etime > 900 )
> type = "Standard"
> else if ( etime > 180 )
> type = "Blitz"
> else
> type = "Bullet"
> printf "[GameTypeOnICC \"%s\"]\n", type
> }
> { print $0 }' "$@"
>
Yes, that is *much* faster - it went from processing about 50
lines/second to processing about 30,000 lines/second, so is more than
5000x faster.
However, it will not work with the default (/usr/bin/awk) version of awk
on Solaris 10. It will work with /usr/xpg4/bin/awk or nawn or gawk, but
is perhaps a bit less portable than other methods. Sun's awk throws a
wobbler:
% rating 1k.pgn
awk: can't open /TimeControl.*[0-9]+[0-9]/ {
split( $2, t, "+" )
etime = t[1] + t[2] * 40
if ( etime > 900 )
type = "Standard"
else if ( etime > 180 )
type = "Blitz"
else
type = "Bullet"
printf "[GameTypeOnICC \"%s\"]\n", type
}
{ print $0 }
--
Dave (from the UK)
Please note my email address changes periodically to avoid spam.
It is always of the form: [email protected]
Hitting reply will work for a few months only - later set it manually.
http://witm.sourceforge.net/ (Web based Mathematica front end)
|
| | | | | |
Date: 07 Aug 2006 12:48:25
From: Chris F.A. Johnson
Subject: Re: Script to classify ICC games as Bullet, Blitz or Standard
|
On 2006-08-07, Dave (from the UK) wrote:
> Chris F.A. Johnson wrote:
>> Much faster than anything else will be to use awk for the whole
>> script:
>>
>> awk -F '"' '/TimeControl.*[0-9]+[0-9]/ {
>> split( $2, t, "+" )
>> etime = t[1] + t[2] * 40
>> if ( etime > 900 )
>> type = "Standard"
>> else if ( etime > 180 )
>> type = "Blitz"
>> else
>> type = "Bullet"
>> printf "[GameTypeOnICC \"%s\"]\n", type
>> }
>> { print $0 }' "$@"
>
> Yes, that is *much* faster - it went from processing about 50
> lines/second to processing about 30,000 lines/second, so is more than
> 5000x faster.
>
> However, it will not work with the default (/usr/bin/awk) version of awk
> on Solaris 10.
Solaris is, AFAIK, the only system that still ships the original,
pre-1987 version of awk. But, as you note, it does have more modern
versions. The script will work with all other versions of awk
(including nawk, mawk, gawk, Kernighan's "One True AWK", tawk).
It should also work with the default Sun awk if you change the
beginning to:
awk 'BEGIN { FS = "\"" }
/TimeControl.*[0-9]+[0-9]/ {
> It will work with /usr/xpg4/bin/awk or nawn or gawk, but
> is perhaps a bit less portable than other methods. Sun's awk throws a
> wobbler:
>
>
>% rating 1k.pgn
> awk: can't open /TimeControl.*[0-9]+[0-9]/ {
> split( $2, t, "+" )
> etime = t[1] + t[2] * 40
> if ( etime > 900 )
> type = "Standard"
> else if ( etime > 180 )
> type = "Blitz"
> else
> type = "Bullet"
> printf "[GameTypeOnICC \"%s\"]\n", type
> }
> { print $0 }
--
Chris F.A. Johnson <http://cfaj.freeshell.org >
===================================================================
Author:
Shell Scripting Recipes: A Problem-Solution Approach (2005, Apress)
|
| | | |
Date: 07 Aug 2006 02:12:36
From: Chris F.A. Johnson
Subject: Re: Script to classify ICC games as Bullet, Blitz or Standard
|
On 2006-08-07, Dave (from the UK) wrote:
> Chris F.A. Johnson wrote:
>
>> It's slow because you call an external command, grep, for every
>> line of the file, and then call more external commands when the
>> line matches.
>>
>> The greatest speed-up will come if you replace these lines:
>>
>> str=`echo $f
|
| | |
Date: 06 Aug 2006 22:00:26
From: Dave (from the UK)
Subject: Re: Script to classify ICC games as Bullet, Blitz or Standard
|
Chris F.A. Johnson wrote:
> On 2006-08-06, Dave (from the UK) wrote:
>>It is a bit slow, doing only 20 lines/s on my 450 MHz machine, so is
>>perhaps not ideally suited to crunch though a million game PGN file !!
>
>
> It's slow because you call an external command, grep, for every
> line of the file, and then call more external commands when the
> line matches.
Thanks for that. I did realise why it was slow, but did not do anything
about it. Had I thought about it more from the beginning I should have
done it differently.
I guess like many things, I started it off to do a quick job, then
thought I might as well make it available.
But I must admit it is slower than I was expecting. Still, it is still
usable for the number of games any one person will play on ICC in their
lifetime. It is far less usable if you have a huge PGN file of a million
+ games.
> The greatest speed-up will come if you replace these lines:
>
> str=`echo $f
|
| | | |
Date: 06 Aug 2006 17:59:57
From: Chris F.A. Johnson
Subject: Re: Script to classify ICC games as Bullet, Blitz or Standard
|
On 2006-08-06, Dave (from the UK) wrote:
> Chris F.A. Johnson wrote:
>> On 2006-08-06, Dave (from the UK) wrote:
>
>>>It is a bit slow, doing only 20 lines/s on my 450 MHz machine, so is
>>>perhaps not ideally suited to crunch though a million game PGN file !!
>>
>> It's slow because you call an external command, grep, for every
>> line of the file, and then call more external commands when the
>> line matches.
>
>
> Thanks for that. I did realise why it was slow, but did not do anything
> about it. Had I thought about it more from the beginning I should have
> done it differently.
>
> I guess like many things, I started it off to do a quick job, then
> thought I might as well make it available.
>
> But I must admit it is slower than I was expecting.
Even if you use a single call to grep, instead of three, it would
be considerably faster:
str=`echo $f
|
| | | | |
Date: 07 Aug 2006 01:09:08
From: Dave (from the UK)
Subject: Re: Script to classify ICC games as Bullet, Blitz or Standard
|
Chris F.A. Johnson wrote:
> On 2006-08-06, Dave (from the UK) wrote:
>
>>Chris F.A. Johnson wrote:
>>
>>>On 2006-08-06, Dave (from the UK) wrote:
>>
>>>>It is a bit slow, doing only 20 lines/s on my 450 MHz machine, so is
>>>>perhaps not ideally suited to crunch though a million game PGN file !!
>>>
>>> It's slow because you call an external command, grep, for every
>>> line of the file, and then call more external commands when the
>>> line matches.
>>
>>
>>Thanks for that. I did realise why it was slow, but did not do anything
>>about it. Had I thought about it more from the beginning I should have
>>done it differently.
>>
>>I guess like many things, I started it off to do a quick job, then
>>thought I might as well make it available.
>>
>>But I must admit it is slower than I was expecting.
>
>
> Even if you use a single call to grep, instead of three, it would
> be considerably faster:
>
> str=`echo $f
|
| | | | | |
Date: 06 Aug 2006 22:53:34
From: Chris F.A. Johnson
Subject: Re: Script to classify ICC games as Bullet, Blitz or Standard
|
On 2006-08-07, Dave (from the UK) wrote:
> Chris F.A. Johnson wrote:
>> On 2006-08-06, Dave (from the UK) wrote:
>>
>>>Chris F.A. Johnson wrote:
>>>
>>>>On 2006-08-06, Dave (from the UK) wrote:
>>>
>>>>>It is a bit slow, doing only 20 lines/s on my 450 MHz machine, so is
>>>>>perhaps not ideally suited to crunch though a million game PGN file !!
>>>>
>>>> It's slow because you call an external command, grep, for every
>>>> line of the file, and then call more external commands when the
>>>> line matches.
>>>
>>>
>>>Thanks for that. I did realise why it was slow, but did not do anything
>>>about it. Had I thought about it more from the beginning I should have
>>>done it differently.
>>>
>>>I guess like many things, I started it off to do a quick job, then
>>>thought I might as well make it available.
>>>
>>>But I must admit it is slower than I was expecting.
>>
>> Even if you use a single call to grep, instead of three, it would
>> be considerably faster:
>>
>> str=`echo $f
|
| |
Date: 06 Aug 2006 19:47:46
From: John J.
Subject: Re: Script to classify ICC games as Bullet, Blitz or Standard
|
Hey Dave, have you been able to connect to the Chessbase server with Pocket
Fritz?
"Dave (from the UK)" <[email protected] >
wrote in message news:[email protected]...
> I've written a small UNIX/Linux shell script that reads from a PGN file
> containing one or more chess games and adds to the PGN file either:
>
> [GameTypeOnICC "Bullet"]
> [GameTypeOnICC "Blitz"]
> [GameTypeOnICC "Standard"]
>
> for each game played. So you can search games in a database based on the
> type of game.
>
> The script does this based on the 'TimeControl' header added by ICC in the
> PGN file. In other words it extracts the initial time and increment, then
> determines whether it is Bullet, Blitz or Standard according to the
> formula used on ICC.
>
> So instead of the PGN file having this one line:
>
> [TimeControl "180+1"]
>
> it will now have an additional line:
>
> [GameTypeOnICC "Blitz"]
> [TimeControl "180+1"]
>
> It should make it easier split up your games and perhaps search them by
> type in a database.
>
> It is a bit slow, doing only 20 lines/s on my 450 MHz machine, so is
> perhaps not ideally suited to crunch though a million game PGN file !!
>
> It can be dowloaded from
>
> http://www.drkirkby.co.uk/chess/tools/rating
>
> It reads from a file and prints to standard output.
>
> $ rating mygames.pgn > mygames-with-type.pgn
>
> --
> Dave (from the UK)
>
> Please note my email address changes periodically to avoid spam.
> It is always of the form: [email protected]
> Hitting reply will work for a few months only - later set it manually.
>
> http://witm.sourceforge.net/ (Web based Mathematica front end)
|
| | |
Date: 06 Aug 2006 22:32:45
From: Dave (from the UK)
Subject: Re: Script to classify ICC games as Bullet, Blitz or Standard
|
John J. wrote:
> Hey Dave, have you been able to connect to the Chessbase server with Pocket
> Fritz?
I found the problem. It must be a subscription issue, as I changed the
date on my Pocket PC back to the year 2003 and it works!!!
Now is it too much to ask for a programmer to put an error message like
"Sorry, your subscription to use the database has lapsed. Please ..."
rather than the very confusing
"Position not found in the online database!"
Is it me, or their error message a bit confusing?
I'll see if I can get an update - I don't want to have to have my
machines date wrong by 3 years.
--
Dave (from the UK)
Please note my email address changes periodically to avoid spam.
It is always of the form: [email protected]
Hitting reply will work for a few months only - later set it manually.
http://witm.sourceforge.net/ (Web based Mathematica front end)
|
| | |
Date: 06 Aug 2006 22:20:03
From: Dave (from the UK)
Subject: Re: Script to classify ICC games as Bullet, Blitz or Standard
|
John J. wrote:
> Hey Dave, have you been able to connect to the Chessbase server with Pocket
> Fritz?
No. I am still stuck. I've got no idea what the problem can be. Someone
(was it you) said you need to pay a subscription to use it after 1 year.
But I've no idea how I could enter any code into PocketFritz to indicate
I've paid and I'm sure it was working more than a year after I got the
program. So I'm a bit puzzled.
In any case, if it is a subscription issue (which chessbase have failed
to bother answering), the error message could be a bit more useful than
"Position not found in the online database!"
It does not exactly tell you anything useful, as clearly the position
after the move 1. e4 will be in loads of the games.
I've looked at the firewall logs and keep seeing an outgoing connection
on port 80 (http) to the IP address 80.237.188.72. I can't verify if
that is Chessbases, but I can't see any packets coming back.
Yours, puzzled and a bit disappointed in chessbase's attitude to the
software, which seems to be to no longer develop it and give no support.
--
Dave (from the UK)
Please note my email address changes periodically to avoid spam.
It is always of the form: [email protected]
Hitting reply will work for a few months only - later set it manually.
http://witm.sourceforge.net/ (Web based Mathematica front end)
|
| | | |
Date: 06 Aug 2006 23:45:22
From: John J.
Subject: Re: Script to classify ICC games as Bullet, Blitz or Standard
|
Do you get the message when you are searching for statistics or for games?
The statistics search hasn't worked for me for many months. I wonder of they
discontinued it.
I simply plugged the IP address you posted into my web browser and Chessbase
came up. So yes, it's a chessbase address.
"Dave (from the UK)" <[email protected] >
wrote in message news:[email protected]...
> John J. wrote:
>> Hey Dave, have you been able to connect to the Chessbase server with
>> Pocket Fritz?
>
> No. I am still stuck. I've got no idea what the problem can be. Someone
> (was it you) said you need to pay a subscription to use it after 1 year.
> But I've no idea how I could enter any code into PocketFritz to indicate
> I've paid and I'm sure it was working more than a year after I got the
> program. So I'm a bit puzzled.
>
> In any case, if it is a subscription issue (which chessbase have failed to
> bother answering), the error message could be a bit more useful than
>
>
> "Position not found in the online database!"
>
> It does not exactly tell you anything useful, as clearly the position
> after the move 1. e4 will be in loads of the games.
>
> I've looked at the firewall logs and keep seeing an outgoing connection on
> port 80 (http) to the IP address 80.237.188.72. I can't verify if that is
> Chessbases, but I can't see any packets coming back.
>
> Yours, puzzled and a bit disappointed in chessbase's attitude to the
> software, which seems to be to no longer develop it and give no support.
>
> --
> Dave (from the UK)
>
> Please note my email address changes periodically to avoid spam.
> It is always of the form: [email protected]
> Hitting reply will work for a few months only - later set it manually.
>
> http://witm.sourceforge.net/ (Web based Mathematica front end)
|
| | | | |
Date: 07 Aug 2006 01:36:38
From: Dave (from the UK)
Subject: Re: Script to classify ICC games as Bullet, Blitz or Standard
|
John J. wrote:
> Do you get the message when you are searching for statistics or for games?
It was the statistics. I never used it to looks for games (much anyway),
but that seems to work.
But a couple of hours ago I did get the statistics working, but it is
unstable. I reset the date on my PocketPC to 3 years ago, to see if it
was date related. At first it seemed to have cured the problem, but now
it has not. I think something is not quite stable at their end.
> The statistics search hasn't worked for me for many months. I wonder of they
> discontinued it.
I wonder if they know they have discontinued it? I suspect it is a
programming error on their web server.
One possibilty is that someoen has made some sort of tool that extracts
data from their web site. So they changed the format slightly so their
tool would not work. But it also breaks their own software.
Just a thought!
The service pack 2 fixed a problem with the online database. I'm
wondering if they change the format slightly now and again, to stop
someone writing their own appliation to interface to the chessbase server.
> I simply plugged the IP address you posted into my web browser and Chessbase
> came up. So yes, it's a chessbase address.
Now why did I not do that? I used nslookup and it was unable to get me
any useful information.
--
Dave (from the UK)
Please note my email address changes periodically to avoid spam.
It is always of the form: [email protected]
Hitting reply will work for a few months only - later set it manually.
http://witm.sourceforge.net/ (Web based Mathematica front end)
|
|
