The (U)ltimate (S)hort (B)oook
Project
For the first 10 moves
INTRODUCTION
The goal of this project is to create a Polyglot opening book [called usb.bin] for the first 10 moves only that roughly will ensure:
- Book moves for the first 5 moves with a 90+% security;
- Book moves for the first 10 moves with a 50+% security.
If these numbers are feasible has to be seen, we are going to try anyway.
The USB Polyglot book can be used in two ways:
- To merge into an existing Polyglot book, instructions below, no program change needed;
- To consult after the regular book search has failed to produce a move, a program changed is needed then.
______________________________________________________________________________________________
Phase ONE
For this reason we created a PGN database of 10+ million games combining several databases, 5 million human and 5 million computer games for maximum opening variety. From these 10 million games we extract the first 10 moves (20 plies) and store each unique position into an EPD record to be analyzed with SOMU Analyze EPD ++ using Stockfish 10 running 40 threads. In total about 13 million EPD positions to analyze, first run at 100ms resulting in usb-100.bin (taking 9 hours to complete) and then on 1000ms resulting in usb-1000.bin (taking almost 4 days), then play ProDeo matches at 40/20 and see what's happening.
Table 1, ProDeo self-play, USB-100ms
Testset | Score | ELO | Book Depth |
54.5% | +31 | 7.16 | |
55.9% | +41 | 7.46 | |
54.8% | +32 | 7.41 | |
54.9% | +33 | 7.37 | |
54.0% | +28 | 7.91 | |
52.9% | +19 | 8.39 |
Table-1, - Score is the match percentage, ELO the corresponding ELO gain, Book Depth the average book depth. For 2moves.pgn this means it stayed 7.16 - 2.00 = 5.16 moves longer iin book, for 3moves.pgn 7.46 - 3.00 = 4.46 and so on, measuring the effectiveness of the system.
Table 2, ProDeo self-play, USB-1000ms
Testset | Score | ELO | Book Depth |
2moves.pgn | 59.1% | +64 | 7.31 |
3moves.pgn | 58.6% | +60 | 7.61 |
4moves.pgn | 56.9% | +47 | 7.51 |
5moves.pgn | 56.1% | +42 | 7.40 |
6moves.pgn | 54.5% | +30 | 7.89 |
7moves.pgn | 50.6% | +3 | 8.40 |
Table-2, A massive ELO increase due to a longer Stockfish analysis time. Also a slightly higher book depth and that's what we are going to work on in phase two, adding more games with short opening llines.
_______________________________________________________________________________________________
Table 3, Fruit self-play, USB-100ms
Opening Book | Score | ELO | Book Depth |
Fruit big | 55.0% | +35 | 8.24 |
Fruit default | 45.0% | -35 | 5.80 |
Table-3 - Looking at the first initial good results we are going to try if it make sense to simply merge the created Polyglot usb-100.bin into an existing Polyglot opening book.
We took the good old Fruit 2.1 book and merged usb-100.bin into it, then started a match between Fruit 2.1 (defaultl) vs Fruit 2.1 (book-big) with excellent results.
Table 4, Brainfish self-play 1000ms
Testset | Score | ELO | Book Depth |
2moves.pgn | 55.5% | +38 | 8.38 |
3moves.pgn | 54.8% | +33 | 8.43 |
4moves.pgn | 54.4% | +30 | 8.63 |
5moves.pgn | 54.6% | +32 | 8.19 |
6moves.pgn | 50.5% | +3 | 8.38 |
7moves.pgn | 51.5% | +10 | 8.75 |
Table-4 - Can a top program also benefit from an ultra short book? For this reason we took Brainfish (Stockfish plus Polyglot book support) and the answer indicates a yes.
_______________________________________________________________________________________________
Phase TWO
Testset | Score | ELO | Book Depth |
2moves.pgn | 64.0% | +101 | 8.43 |
3moves.pgn | 60.5% | +74 | 8.67 |
4moves.pgn | 57.3% | +51 | 8.64 |
5moves.pgn | 57.5% | +52 | 8.80 |
6moves.pgn | 56.1% | +42 | 9.19 |
7moves.pgn | 56.8% | +47 | 9.54 |
8moves.pgn | 54.3% | +39 | 9.41 |
9moves.pgn | 51.9% | +12 | 9.69 |
We have extended the base database of 10 million games by adding 990,000 games with short opening llines from various sources in order to increase the Book Depth because after all that is the main goal of this project:
- Book moves for the first 5 moves with a 90+% security;
- Book moves for the first 10 moves with a 50+% security.
We play the ProDeo matches again and also produce a statistic if the goal of the project (see above) is in reach.
Book Depth split on move number
Testset | Games | Move 3 | Move 4 | Move 5 | Move 6 | Move 7 | Move 8 | Move 9 | Move 10 | Book Moves |
2moves.pgn | 400 | 95.7% | 94.5% | 93.2% | 88.5% | 83.2% | 67.2% | 52.2% | 22.8% | 2.570 |
3moves.pgn | 1000 | --- | 94.9% | 94.3% | 89.9% | 83.2% | 72.6% | 60.3% | 24.2% | 5.665 |
4moves.pgn | 1000 | --- | --- | 93.5% | 91.4% | 82.6% | 69.1% | 55.1% | 24.3% | 4.640 |
5moves.pgn | 1000 | --- | --- | --- | 96.1% | 86.6% | 70.3% | 53.4% | 23.3% | 2.552 |
6moves.pgn | 1000 | --- | --- | --- | --- | 95.5% | 83.1% | 66.3% | 26.3% | 3.194 |
7moves.pgn | 1000 | --- | --- | --- | --- | --- | 95.2% | 78.0% | 33.5% | 2.543 |
8moves.pgn | 1000 | --- | --- | --- | --- | --- | --- | 72.0% | 22.6% | 1.413 |
9moves.pgn | 1000 | --- | --- | --- | --- | --- | --- | --- | 48.6% | 687 |
Aside of the excellent results it seems we managed our goal.
GREEN - if the percentage matches the green goal of 90+%
BLUE - if the percentage matches the blue goal of 50+%
RED - if the percentage fails the green or blue goal.
USB.BIN [1000 ms]
Actually we think the results are too good to be realistic. 2moves.pgn and 3moves.pgn are from Jeroen Noomen and 4moves.pgn ... 9moves.pgn were created from a download from Stefan Pohl's site and apparently a lot of games from these testsets ended up in our database.
_____________________________________________________________________________________________
Phase THREE
Arrived here it would be advisable to increase the analysis time of the 14 million positions, the last step. Good advice is invaluable. 14 million positions at 1 second already took 4 days. An alternative would to analyze them with a fixed depth. We choose for the latter, since contrary to fixed time, fixed depth finishes the iteration. We analyze the 14 million positions with depth=16 plies.
Alternatively we offer the 14 million EPD positions ifor download so you can analyze them yourself for an even better result and produce an even stronger Polyglot USB (Ultra Short Book). Drop me a note via the contact form how to do that. Or, if you want to contribute new <=10 moves opening positions to the database you can also use the contact form and I will provide you with the necessary software and instructions.
USB.BIN [depth=16]
ALL.EPD
the 14 million EPD positions
___________________________________________________________________________________________
Technical and credits
Source code of the Polyglot format, here and here.
Download Polyglot by Fabien Letouzy (and others).
Documentation Polyglot and how to merge Polyglot books.
Stockfish by T. Romstad, M. Costalba, J. Kiiski.
2moves.pgn and 3moves.pgn by Jeroen Noomen.
4moves.pgn ... 9moves.pgn created from Stefan Pohl's site.