Practical Engine Tuning
for starters
The PET package is a ready to use platform (everything pre-installed) for engine testing (tuning) and consist 4 parts.
Gold digging (hunting) for candidate improvements in a fast way;
The actual testing of those candidate improvements playing eng-eng matches;
Managing PET eng-eng matches.
2 well known engines (Fruit 2.1 and Stockfish 5) will serve as an explanation what PET can do. Fruit 2.1 and Stockfish 5 both offer some tunenable UCI settings, these are:
Fruit 2.1 | Stockfish 5 |
History Threshold = 60 | Mobility (Midgame) = 100 |
Futility Pruning = false | Mobility (Endgame) = 100 |
Futility Margin = 100 | Pawn Structure (Midgame) = 100 |
Delta Pruning = false | Pawn Structure (Endgame) = 100 |
Delta Margin = 50 | Passed Pawns (Midgame) = 100 |
Quiescence Check Plies = 1 | Passed Pawns (Endgame) = 100 |
Material = 100 | Space = 100 |
Piece Activity = 100 | Aggressiveness = 100 |
King Safety = 100 | Cowardice = 100 |
Pawn Structure = 100 | |
Passed Pawns = 100 |
For both engines we are going to tune its mobility
________________________________________________________________________________________________
Part-one : Gold Digging
We use the STS test suite that was developed during the years 2009-2010 by Dann Corbit and Swaminathan Natarajan containing 1500 theme based STS positions for the purpose of tuning engines and in particular the evaluation function. And with Fruit 2.1 we are going to explore its mobility. With MEA (pre-installed) we run STS at 5 different fast time controls and produce and overview.
Click picture to enlarge
The first table shows the sorted individual results of each setting for each time control used. The second table shows a sorted list of the overall rating by adding up the 5 gained ratings divided by 5.
We are now ready for step-2, the eng-eng testing part. And we are especially interested [mob=105] candidate improvement version and if it scores better against the official Fruit 2.1. Also if [mob=125] will do worse against the official Fruit 2.1 but let's test them all anyway.
______________________________________________________________________________________________
Part-two : eng-eng matches
We use cutechess-cli (pre-installed) and pitch the official Fruit 2.1 against the various mobility versions above at 40/10, 1000 games and run the new overview.
As one can see from the results not only [mob=105] on the STS part tops but also after playing 1000 bullet games.
__________________________________________________________________________________
Stockfish 5 mobility testing
As a wild guess we changed the Mobility (Midgame) = 100 parameter to 125 and ran 2 matches, 4000 games at 40/10 and 1000 games at 40/60. Note the surprising results.
Click picture to enlarge
______________________________________________________________________________________________
Part-three : Managing eng-eng matches
Keep track of your eng-eng match results. As one can see from the above picture PET offers you 2 time controls. Just store your match resullts in the tc-one.txt and tc-two.txt files, see pre-installed example. A more advanced (stand alone) system for match control based on eng-eng results we like to introduce now.
_____________________________________________________________________________________________
Part four : Match Control
LOS.EXE is a simple command line tool that will read the PGN of your matches and stores the overall information in a text-based database called LOS.TXT whereafter your browser will start and show you the results. When using the 6 Fruit and 2 Stockfish matches of this page we get:
LOS | Will produce the above web-page. |
LOS test.pgn | Will add test.pgn to the LOS.TXT database and show you the updated web-page. |
LOS test.pgn 100 | A third paramter will override the default setting of 1000 and produce the development of the match in steps of 100 games instead of 1000. It can very insightful to study the obstinacy how matches (which engines are so close in strength) may sometimes develop and why programmers play so many games. Consider an old ProDeo match as an example. It's only after 4500 games ProDeo X is starting to see the light. |
Format of the LOS.TXT database
The most easy way the maintain the database is after adding a match to open LOS.TXT with a text editor and modify the first [Match] and second [Parameter] items to your wishes.
Layout:
Fruit [mob=95] - Fruit 2.1,[Piece Activity = 95],1000,49.9%,46.8%,40/10,2018.06.20,mob-95.pgn
Just stick to the comma driven format.
While the LOS program is included in the PET download below you can download it seperately. Source code included for further development.
_____________________________________________________________________________________________
General remarks
1. Install the archieve in the root of your C-drive (obliged).
2. Operation goes via pre-installed batch files, see the introduction on the next page.
3. Chess engines that offer a lot of parameter tuning are Deuterium, Rodent III and of course ProDeo itself.
______________________________________________________________________________________________
Credits
Ferdinand Mosca | for MEA |
Tord Romstad, Marco Costalba and Joona Kiiski | for Stockfish 5 |
for Fruit 2.1 | |
Ilari Pihlajisto, Arto Jonsson | for cutechess-cli |
Dann Corbit, Swaminathan Natarajan | for STS |
Fabien Letouzey and others | fot Polyglot |