Opening Knowledge Engines


Opening Knowledge Engines


OKE is a research project how well nowadays (mosty top) chess engines play the early opening, the first 6 moves in particular. Since the arrival of the first serious rating list in 1979 -- the SSDF -- engines competed with opening books allowed, later in 2006 when CEGT and CCRL took over the role as leading rating lists that changed. Books were not allowed and engines were tested on pure engine strength. Yet engines for the most part were given the first 6-8-10-12 moves in advance to avoid doubles.


On this page we will show the early opening hasn't received the attention it deserved likely caused by the fact engines weren't challenged in this area. We will see engines rated hundreds elo points lower surpassing the stronger ones, in some extreme cases more than 500 elo points.


The message is that there is a lot room for improvement and we will offer a simple platform for engine programmers to improve in this area. The base is the program MEA from engine author Ferninand Mosca which allows engines to analyze all positions in an EPD file which contains a bonus for moves. Examples:

Opening Set

Positions

Maximum points

404

35.617

1.221

109.941

2.861

260.919

3.188

291.610

4.809

443.607

2, 3 and 4-moves.epd are created from 2600+ elo rated human players.


5 and 6-moves.epd are created from 2700+ elo rated human players.


Additionally larger test sets are avialable made from 2400+ and 2500+ elo rated human players for more wider testing, see the EPD folder.

Operation:

• Double click on the run-mea batch file.

• It will start the analysis of 2-moves.epd by Deuterium and Stockfish at 500ms per position.

• When finished double click on list-2-moves batch file.

• Your browser will show the result.


Technical:

• Modify the batch files to add or change engines.

• set MT=500 means the time control is set to 500ms.

• The system is able to handle 5 different freely defined MT= time controls.


Conclusions:


1. While the rating system is far from perfect (more below) the pattern is clear, if programs like Rodent III, Rebel Century (from 2000) and Mephisto Gideon (from 1993) outperform stronger engines with hundreds of elo points and in extreme cases with almost 1000 elo points then some fundamental opening knowledge is missing in modern top engines.


2. The big exception is Lc0, not only it tops on every test run its performamce becomes truly remarkable when you realize we used the slow CPU version in our testing doing only 30-35 NPS.


3. Although we believe the general pattern will not change, for a more reliable rating use the additional much larger opening sets made from 2400+ or 2500+ elo rated human players since volume increases accuracy.


4. Engines can use the EPD's also for eng-eng testing.

Credits

Ferdinand Mosca

for MEA

Tord Romstad, Marco Costalba and Joona Kiiski

Aprox 4½ Mb