2.2. Use Sparkle for algorithm configuration¶
These steps can also be found as a Bash script in Examples/configuration_qualty.sh
2.2.1. Initialise the Sparkle platform¶
Commands/initialise.py
2.2.2. Add instances¶
Add train, and optionally test, instances (in this case for the VRP) in a given directory, without running solvers or feature extractors yet
Commands/add_instances.py Examples/Resources/CVRP/Instances/X-1-10/
Commands/add_instances.py Examples/Resources/CVRP/Instances/X-11-20/
2.2.3. Add a configurable solver¶
Add a configurable solver (here for vehicle routing) with a wrapper containing the executable name of the solver and a string of command line parameters, without running the solver yet
The solver directory should contain the solver executable, the sparkle_smac_wrapper.py
wrapper, and a .pcs
file describing the configurable parameters
Commands/add_solver.py --deterministic 0 Examples/Resources/CVRP/Solvers/VRP_SISRs/
If needed solvers can also include additional files or scripts in their directory, but keeping additional files to a minimum speeds up copying.
2.2.4. Configure the solver¶
Perform configuration on the solver to obtain a target configuration. For the VRP we measure the absolute quality performance by setting the --performance-measure
option, to avoid needing this for every command it can also be set in Settings/sparkle_settings.ini
.
Commands/configure_solver.py --solver Solvers/VRP_SISRs/ --instance-set-train Instances/X-1-10/ --performance-measure QUALITY_ABSOLUTE
2.2.5. Validate the configuration¶
To make sure configuration is completed before running validation you can use the sparkle_wait
command
Commands/sparkle_wait.py
Validate the performance of the best found parameter configuration. The test set is optional. We again set the performance measure to absolute quality.
Commands/validate_configured_vs_default.py --solver Solvers/VRP_SISRs/ --instance-set-train Instances/X-1-10/ --instance-set-test Instances/X-11-20/ --performance-measure QUALITY_ABSOLUTE
2.2.6. Generate a report¶
Wait for validation to be completed
Commands/sparkle_wait.py
Generate a report detailing the results on the training (and optionally testing) set. This includes the experimental procedure and performance information; this will be located in a Configuration_Reports/
subdirectory for the solver, training set, and optionally test set like VRP_SISRs_X-1-10_X-11-20/Sparkle-latex-generator-for-configuration/
. We again set the performance measure to absolute quality.
Commands/generate_report.py --performance-measure QUALITY_ABSOLUTE
By default the generate_report
command will create a report for the most recent solver and instance set(s). To generate a report for older solver-instance set combinations, the desired solver can be specified with --solver Solvers/VRP_SISRs/
, the training instance set with --instance-set-train Instances/X-1-10/
, and the testing instance set with --instance-set-test Instances/X-11-20/
.