2.6. Use Sparkle for algorithm selection with multi-file instances¶
2.6.1. Initialise the Sparkle platform¶
Commands/initialise.py
2.6.2. Add instances¶
Add instance files in a given directory, without running solvers or feature extractors yet. In addition to the instance files, the directory should contain a file sparkle_instance_list.txt where each line contains a space separated list of files that together form an instance.
Commands/add_instances.py Examples/Resources/CCAG/Instances/CCAG/
2.6.3. Add solvers¶
Add solvers (here for the constrained covering array generation (CCAG) problem) with a wrapper containing the executable name of the solver and a string of command line parameters, without running the solvers yet
Each solver directory should contain the solver executable and a wrapper
Commands/add_solver.py --deterministic 0 Examples/Resources/CCAG/Solvers/TCA/
Commands/add_solver.py --deterministic 0 Examples/Resources/CCAG/Solvers/FastCA/
2.6.4. Add feature extractor¶
Similarly, add a feature extractor, without immediately running it on the instances
Commands/add_feature_extractor.py Examples/Resources/CCAG/Extractors/CCAG-features_sparkle/
2.6.5. Compute features¶
Compute features for all the instances; add the --parallel option to run in parallel
Commands/compute_features.py
2.6.6. Run the solvers¶
Run the solvers on all instances; add the --parallel option to run in parallel. For the CCAG (Constrained Covering Array Generation) problem we measure the absolute quality performance by setting the --performance-measure option, to avoid needing this for every command it can also be set in Settings/sparkle_settings.ini.
Commands/run_solvers.py --performance-measure QUALITY_ABSOLUTE
2.6.7. Construct a portfolio selector¶
To make sure feature computation and solver performance computation are done before constructing the portfolio use the sparkle_wait command
Commands/sparkle_wait.py
Construct a portfolio selector, using the previously computed features and the results of running the solvers. We again set the performance measure to absolute quality.
Commands/construct_sparkle_portfolio_selector.py --performance-measure QUALITY_ABSOLUTE
2.6.8. [Coming soon] Generate a report¶
This is not yet implemented for quality performance
Generate an experimental report detailing the experimental procedure and performance information; this will be located at Components/Sparkle-latex-generator/Sparkle_Report.pdf. We again set the performance measure to absolute quality.
Commands/generate_report.py --performance-measure QUALITY_ABSOLUTE
2.6.9. [Coming soon] Run the portfolio selector (e.g. on the test set)¶
This is not yet implemented for quality performance
This is not yet implemented for multi-file instances
2.6.9.1. Run on a single instance¶
Run the portfolio selector on a single testing instance; the result will be printed to the command line. We again set the performance measure to absolute quality.
Commands/run_sparkle_portfolio_selector.py Examples/Resources/CCAG/Instances/CCAG2/Banking2.model Examples/Resources/CCAG/Instances/CCAG2/Banking2.constraints --performance-measure QUALITY_ABSOLUTE
2.6.9.2. Run on an instance set¶
Run the portfolio selector on a testing instance set. We again set the performance measure to absolute quality.
Commands/run_sparkle_portfolio_selector.py Examples/Resources/CCAG/Instances/CCAG2/ --performance-measure QUALITY_ABSOLUTE
2.6.10. Generate a report including results on the test set¶
Wait for the portfolio selector to be done running on the testing instance set
Commands/sparkle_wait.py
Generate an experimental report that includes the results on the test set, and as before the experimental procedure and performance information; this will be located at Components/Sparkle-latex-generator/Sparkle_Report_For_Test.pdf. We again set the performance measure to absolute quality.
Commands/generate_report.py --performance-measure QUALITY_ABSOLUTE
By default the generate_report command will create a report for the most recent instance set. To generate a report for an older instance set, the desired instance set can be specified with: --test-case-directory Test_Cases/CCAG2/