Opening Book Development II
Wednesday, December 17, 2008 at 9:18PM 
Before I go any farther let me address this issue of stealing lines. (I used this term in the last post) It is a fact that some players out there seem to think that if they personally worked out a line or a refutation of one; that they somehow own this series of moves. I find this thinking hilariously funny. It brings to my mind Bobby Fischer jumping up from the table and saying you can’t play that move it is mine, you stole it from my game in the interzonals last year… I call this concept of stealing lines. Fischer thinking. “Look Bobby, once you play the line in open play it belongs to all of us.” This is how, in esoteric disciplines, we collectively learn and advance human knowledge. It is by the possess of peer review that the lines you submit in open play are accepted or rejected. So there is no real stealing of anything only finding and using good ideas. We are all standing on the shoulders of geniuses to be where we are.
Now that you have made your seed book and played some games with it, you are ready to start its development by adding games. Adding games to a book (both your own and kibitz ones) is much different than hand tuning. I think it would be accurate to say that adding games increases the width of the book where as, tuning deals with depth and direction.
It is always best to add games to the book that are played by the book that you are adding them to. This technique has a tendency to reinforce and expand winning lines. However if you find good opening lines while kibitzing do not hesitate to include them into your book.
Lets look at the method of adding the games. First never add games directly to an existing book. Add the games to the database that the book was made from and then make a fresh book from the updated game bases. Importing games into an existing book is probably the most conmen mistake that is made in book making.
Make sure that if you add a game to the learn game base you also add it the main game base.
I recommend that you add all games that you won in online play to the main game base and a selected few that beat you solidly within the first 50 moves. Be more selective as to the games you add to the learn base. Look carefully at your played games; making sure that advantage in the game came from the opening or early in the middle game. (Never add draws, only wins and losses)
Continue to add the games that you play with the seed book back into the book using this technique. Every day review your games, update the game bases (both main base and learn) and remake the book for the next time at online play. This process will slowly but surly expand the books practical repertoire and adjust it to your particular system engine combination. You will begin to notice games that your engine struggled to win now being won easily and good winning lines that were used against you now being played by your book.
At least 2000 games or so should be added to the seed book before you start the hand tuning process. Once you start hand tuning you can no longer add games as red-green marks would be lost.
As always; I am sure that there are many impotent details I have inadvertently forgot to mention. If you get stuck or have a problem please do not hesitate to ask me specific question via email.
Reader Comments (6)
I really have to disagree with you on stealing lines! The reasons are very clear, 1st if I steal a line from B.Fischer :) I have to play the whole game and do the thinking for myself. Whereas if I steal lines from a very strong engine player I do not have to do any thinking, just let the engine do it for me! Granted any player using there "special lines" for elo hunting have no one to blame but themself. Its not so much that others get lines from played games as it is the professional game Stakers that are stealing them online! This is very unfair of any server that allows this pratice.
You make a good point. However I think you may be surprised at who these strong engine players are. It may be the players with the top Elo are the very players that are using the Macro-Programs to auto-Kibitz games.
I do agree with you that no chess server should allow secret nicks to run programs to collect games. Notice, it is games that they are stealing not just the specific lines or refutations of strong positions. (which I believe after being played in public are fair game)
I have spoken out, both publicly and in communications with the chess server you are referring to, about the elite group of players being allowed to grab all games 24-7 being played using auto-kibitzing software. (seemingly against server policy) But make no mistake, the players doing this are the elite top dogs on this server. Many have connections with the server or other commercial chess software publishers.
Right now there are two competing opening book development theories, hand tuning and statistical probability. One requires a certain knowledge of chess and the other does not. It mirrors the two ways engine players come into the hobby. Some approach it from the world of chess and others from the world of computers.
These players updating their opening books using the statistical approach are some times using private software that is designed for this purposes and is more advanced than what is available in commercial chess software. There is nothing inherently wrong with this approach, except that it requires large quantities of played games to search for better lines.
Kevin Frayer:
Thanks for nice article.
Here you are talking about stealing just few lines from the good games.
But, I found many people stealing whole book like Heretic 2.8 or SicilianRush completely and importing whole book into their own books.
Anyway there is no point in discussing this in details.
Thanks for putting lots of information on your nice web-site.
Thanks n regards
ChessKnight
ChessKnight:
Thanks for your nice words, here and on the Rybka forum.
I have always tried to impress the satisfaction a computer player gets from developing their own opening books. If done right, it can be compared to a formula 1 driver getting into the cockpit of his high tech vehicle, revving the engine and heading out on the track to compete. As advanced as F1 is, the driver is still needed. Book making may be analogues to only turning the wheel left and right, but on a road course this is an important thing. The driver must also control the engines throttle, brake biases and help tweak the aerodynamics. In my view the player will still hold an important place in computer chess for some time to come.
As for merging books; I realize many players do this all the time. Personally, I have not found it to be as effective as developing smaller more focused books from databases. However, it may have its place in some book making theories. I do not mind if someone wishes to combine my books with others, or just add games to them. Its just the resulting book will not be as sharp or as easy to control.
Hello Frayer,
Thank you for your usefull comments about book making. I have a question about this part II.
I already have my 2 bases, the MainBase (11000 games) and the LearnBase (1200 gamess). After playing online, I added games as you mentionned to these 2 bases. Now, for making a new fresh CTG book, must I :
1) Follow Part I, I mean : export the MainBase to my CTG up to 100 moves and then from the CTG "Learn from LearnBase" or
2) From my CTG "learn from LearnBase", without exporting the "MainBase" first ?
Hi Mr Frayer,
first up, many thanks for all the useful information gathered here and for taking the trouble to do this.
My question is, why do we have the step of deleting the drawn games from the opening book database ?
If a sharp gambit has a forced drawn continuation (for example a line I found in the Botvinnik Slav), then I would have to add this by hand if the drawn games were deleted ?
Thanks in advance!