1. Sparkle user guide

1.1. Quick start

Note

Sparkle currently relies on Slurm, and does not work without it.

Follow these steps:

  1. Install Sparkle

  2. Prepare an algorithm configuration or an algorithm selection.

  3. Execute commands

1.2. Installing Sparkle

Note

The installation process use the conda command available in Anaconda or Miniconda to manage some dependencies.

1.2.1. Get a copy of Sparkle

To get a copy of Sparkle you can clone the repository.

If git is available on your system, this will clone the Sparkle repository and create a subdirectory named sparkle :

$ git clone https://bitbucket.org/sparkle-ai/sparkle.git

You can also download the stable version here: https://bitbucket.org/sparkle-ai/sparkle/get/main.zip

1.2.2. Install dependencies

Sparkle depends on Python 3.9+, swig 3.0, gnuplot, LaTeX, and multiple Python packages. An easy way to install most of what is needed is to use the conda package manager (https://docs.conda.io/en/latest/miniconda.html).

Note

LaTeX is used to create the reports and the documentation and must be installed manually on the system. If you don’t plan to use the reports or recreate the documentations, you can skip it.

You can install the base requirements with

$ conda env create -f environment.yml

This will create an environment named sparkle that contains everything needed to run Sparkle, except LaTeX that needs to be installed manually.

To activate the environment in the current shell, execute:

$ conda activate sparkle

Note

You will need to reactivate the environment every time you log in, before using Sparkle.

The file environment.yml contains a tested list of Python packages with fixed versions required to execute Sparkle. We recommended using it.

The file environment-dev.txt contains unpinned packages and the dependencies are not resolved. It is used for development and may cause problems.

The two environments can be created in parallel since one is named sparkle and the other sparkle-dev. If you want to update an environment it is better to do a clean installation by removing and recreating it. For example:

$ conda deactivate
$ conda env remove -n sparkle
$ conda env create -f environment.yml
$ conda activate sparkle

This should be fast as both conda and pip use local cache for the packages.

1.2.3. Configure Sparkle/Slurm

Before running Sparkle, you probably want to have a look at the settings described in Section 1.9. In particular, the default Slurm settings are set to work with the Grace cluster in Leiden University and should be adapted to your specific cluster.

1.3. Algorithm Configuration

Configuring an algorithm has the following minimal requirements for the algorithm (for an example of a solver directory see Section 1.6.2):

  • A working solver executable

  • An algorithm wrapper called sprakle_smac_wrapper.py

  • A PCS (parameter configuration space) file

  • The runsolver binary (e.g. from Examples/Resources/Solvers/PbO-CCSAT-Generic/)

Further, training and testing instance sets are needed (for an example of an instances directory see Section 1.6.1). For the purpose of testing whether your configuration setup works with Sparkle, it is advised to primarily use instances that are solved (relatively) quickly even with the default parameters.

Note

See the example page for a walk-through on how to perform configuration with Sparkle.

1.3.1. Creating a wrapper for your algorithm

A template for the wrapper that connects your algorithm with Sparkle is available at Examples/Resources/Solvers/template/sparkle_smac_wrapper.py. Within this template a number of TODOs are indicated where you are likely to need to make changes for your specific algorithm. You can also compare the different example solvers to get an idea for what kind of changes are needed.

1.3.2. Parameter configuration space (PCS) file

The PCS (parameter configuration space) format [1] is used to pass the possible parameter ranges of an algorithm to Sparkle in a .pcs file. For an example see e.g. Examples/Resources/Solvers/PbO-CCSAT-Generic/PbO-CCSAT-params_test.pcs.

In this file you should enter all configurable parameters of your algorithm. Note that parameters such as the random seed used by the algorithm should not be configured and therefore should also not be included in the PCS file.

1.4. Algorithm Selection

Creating a portfolio selector requires multiple algorithms with the following minimal requirements (for an example of a solver directory see Section 1.6.3):

  • A working solver executable

  • An algorithm wrapper called sparkle_run_default_wrapper.py

Further, training and testing instance sets are needed (for an example of an instances directory see Section 1.6.1). For the purpose of testing whether your selection setup works with Sparkle, it is advised to primarily use instances that are solved (relatively) quickly.

Note

See the example page for a walk-through on how to perform selection with Sparkle.

1.4.1. Creating a wrapper for your algorithm

A template for the wrapper that connects your algorithm with Sparkle is available at Examples/Resources/Solvers/template/sparkle_run_default_wrapper.py. Within this template a number of TODOs are indicated where you are likely to need to make changes for your specific algorithm. You can also compare the different example solvers to get an idea for what kind of changes are needed.

1.5. Executing commands

Executing commands in Sparkle is as simple as running them in the top directory of Sparkle, for example:

Commands/initialise.py

Do note that when running on a cluster additional arguments may be needed, for instance under the Slurm workload manager the above command would change to something like:

srun -N1 -n1 -c1 Commands/initialise.py

In the Examples/ directory a number of common command sequences are given. For instance, for configuration with specified training and testing sets see e.g. Examples/configuration.md for an example of a sequence of commands to execute. Note that some command run in the background and need time to complete before the next command is executed. To see whether a command is still running the Slurm command squeue can be used.

In the Output/ directory paths to generated scripts and logs are gathered per executed command.

1.6. File structure

1.6.1. A typical instance directory

An instance directory should look something like this:

Instances/
  Example_Instance_Set/
    instance_a.cnf
    instance_b.cnf
    ...        ...
    instance_z.cnf

This directory simply contains a collection of instances, as example here SAT instances in the CNF format are given.

For instances consisting of multiple files one additional file called sparkle_instance_list.txt should be included in the Example_Instance_Set directory, describing which files together form an instance. The format is a single instance per line with each file separated by a space, as shown below.

instance_a_part_one.abc instance_a_part_two.xyz
instance_b_part_one.abc instance_b_part_two.xyz
...                     ...
instance_z_part_one.abc instance_z_part_two.xyz

1.6.2. A typical solver directory (configuration)

A solver directory should look something like this:

Solver/
  Example_Solver/
    solver
    sparkle_smac_wrapper.py
    parameters.pcs
    runsolver

Here solver is a binary executable of the solver that is to be configured. The sprakle_smac_wrapper.py is a wrapper that Sparkle should call to run the solver with specific settings, and then returns a result for the configurator. In parameters.pcs the configurable parameters are described in the PCS format. Finally, runsolver is a binary executable of the runsolver tool. This allows Sparkle to make fair time measurements for all configuration experiments.

Note

Currently the runsolver binary has to be in every solver directory, it can be found in the Examples/Resources/Solvers/PbO-CCSAT-Generic/ directory.

1.6.3. A typical solver directory (selection)

A solver directory should look something like this:

Solver/
  Example_Solver/
    solver
    sparkle_run_default_wrapper.py

Here solver is a binary executable of a solver that is to be included in a portfolio selector. The sprakle_run_default_wrapper.py is a wrapper that Sparkle should call to run the solver on a specific instance.

1.6.4. The output directory

Note

This section describes a desirable behaviour but has not been implemented fully yet.

The output directory is located at the root of the Sparkle directory. Its structure is as follows:

Output/
  Logs/
  Common/
    Raw_Data/
    Analysis/
  Configuration/
    Raw_Data/
      run_<alias>/
        related files
    Analysis/
  Parallel_Portfolio/
    Raw_Data/
      run_<alias>/
        related files
    Analysis/
  Selection/
    Raw_Data/
      run_<alias>/
        related files
    Analysis/

The alias is based on the command and a timestamp.

The Logs directory should contain the history of commands and their output such that one can easily know what has been done in which order and find enough pointers to debug unwanted behaviour.

Other directories are cut into two subdirectories: Raw_Data contains the data produced by the main command, often time consuming to generate, handle with care; Analysis contains information extracted from the raw data, easy to generate, plots and reports.

For each type of task run by Sparkle, the related files differ. The aim is always to have all required files for reproducibility. A copy of the sparkle configuration file at the time of the run and of all files relevant to the run, a copy of any log or error file that could help with debugging or a link to it, and the output of the executed task.

For configuration the configuration trajectory if available, the training and testing sets, the default configuration and the final found configuration. The performance of those will be in the Analysis folder.

For parallel portfolio the resulting portfolio and its components. The performance of the portfolio will be in the Analysis folder.

For selection the algorithms and their performance on the training set, the model(s) generated if available and the resulting selector. The performance evaluation of the selector will be in the Analysis folder.

For analysis a link to the folder on which the analysis was performed (configuration, portfolio or selection), the performance evaluation from it and the report if it was generated.

1.7. Wrappers

1.7.1. sparkle_run_default_wrapper.py

The sparkle_run_default_wrapper.py has two functions that need to be implemented for each algorithm:

  • print_command(instance_file, seed_str: str, cutoff_time_str: str)

  • print_output(terminal_output_file: str)

print_command(...) should print a command line call that Sparkle can use to run the algorithm on a given instance file. Ideally, for reproducibility purposes, the seed provided by Sparkle should also be passed to the algorithm. If the algorithm requires this, the cutoff time can also be passed to the algorithm. However, in this case the cutoff time should be made very large. For instance by multiplying by ten with: cutoff_time_str = str(int(cutoff_time_str) * 10). This is necessary to ensure Sparkle stops the algorithm after the cutoff time, rather than the algorithm itself. By doing this it is ensured runtime measurements are always done by Sparkle, and thus consistent between algorithms that might measure time differently.

print_output(...) should process the algorithm output. If the performance measure is RUNTIME, this function only needs to output the algorithm status. For all QUALITY performance measures both the algorithm status and the solution quality have to be given. Sparkle internally measures RUNTIME, while it can be overwritten by the user if desired, for consistent runtime measurements between solvers this is not recommended. The output should be printed and formatted as in the example below.

quality 8734
status SUCCESS

Status can hold the following values {SUCCESS, TIMEOUT, CRASHED}. If the status is not known, reporting SUCCESS will allow Sparkle to continue, but may mean that Sparkle does not know when the algorithm crashed, and continues with faulty results.

1.8. Commands

Currently the commands below are available in Sparkle (listed alphabetically). Every command can be called with the –help option to get a description of the required arguments and other options.

Note

Arguments in [square brackets] are optional, arguments without brackets are mandatory. Input in <chevrons> indicate required text input, {curly brackets} indicate a set of inputs to choose from.

1.8.1. about.py

usage: about.py [-h]
-h, --help

show this help message and exit

1.8.2. add_feature_extractor.py

usage: add_feature_extractor.py [-h]
                                [--run-extractor-now | --run-extractor-later]
                                [--nickname NICKNAME] [--parallel]
                                extractor-path
extractor-path

path to the feature extractor

-h, --help

show this help message and exit

--run-extractor-now

immediately run the newly added feature extractor on the existing instances

--run-extractor-later

do not immediately run the newly added feature extractor on the existing instances (default)

--nickname <nickname>

set a nickname for the feature extractor

--parallel

run the feature extractor on multiple instances in parallel

1.8.3. add_instances.py

usage: add_instances.py [-h] [--run-extractor-now | --run-extractor-later]
                        [--run-solver-now | --run-solver-later]
                        [--nickname NICKNAME] [--parallel]
                        instances-path
instances-path

path to the instance set

-h, --help

show this help message and exit

--run-extractor-now

immediately run the feature extractor(s) on the newly added instances

--run-extractor-later

do not immediately run the feature extractor(s) on the newly added instances (default)

--run-solver-now

immediately run the solver(s) on the newly added instances

--run-solver-later

do not immediately run the solver(s) on the newly added instances (default)

--nickname <nickname>

set a nickname for the instance set

--parallel

run the solvers and feature extractor on multiple instances in parallel

1.8.4. add_solver.py

Add a solver to the Sparkle platform.

usage: add_solver.py [-h] --deterministic {0,1}
                     [--run-solver-now | --run-solver-later]
                     [--nickname NICKNAME] [--parallel]
                     [--solver-variations SOLVER_VARIATIONS]
                     solver-path
solver-path

path to the solver

-h, --help

show this help message and exit

--deterministic {0,1}

indicate whether the solver is deterministic or not

--run-solver-now

immediately run the newly added solver on all instances

--run-solver-later

do not immediately run the newly added solver on all instances (default)

--nickname <nickname>

set a nickname for the solver

--parallel

run the solver on multiple instances in parallel

--solver-variations <solver_variations>

Use this option to add multiple variations of the solver by using a different random seed for each varation.

1.8.5. cleanup_temporary_files.py

usage: cleanup_temporary_files.py [-h]
-h, --help

show this help message and exit

1.8.6. compute_features.py

usage: compute_features.py [-h] [--recompute] [--parallel]
                           [--settings-file SETTINGS_FILE] [--run-on RUN_ON]
-h, --help

show this help message and exit

--recompute

Re-run feature extractor for instances with previously computed features

--parallel

Run the feature extractor on multiple instances in parallel

--settings-file

Specify the settings file to use in case you want to use one other than the default

--run-on <run_on>

On which computer or cluster environment to execute the calculation.Available: Local, Slurm. Default: Slurm

1.8.7. compute_marginal_contribution.py

usage: compute_marginal_contribution.py [-h] (--perfect | --actual)
                                        [--recompute]
                                        [--performance-measure {RUNTIME,QUALITY_ABSOLUTE,QUALITY,QUALITY_ABSOLUTE_MINIMISATION,QUALITY_ABSOLUTE_MAXIMISATION}]
                                        [--settings-file SETTINGS_FILE]
-h, --help

show this help message and exit

--perfect

compute the marginal contribution for the perfect selector

--actual

compute the marginal contribution for the actual selector

--recompute

force marginal contribution to be recomputed even when it already exists in file for for the current selector

--performance-measure

the performance measure, e.g. runtime

--settings-file

specify the settings file to use in case you want to use one other than the default

1.8.8. configure_solver.py

Configure a solver in the Sparkle platform.

usage: configure_solver.py [-h] [--configurator CONFIGURATOR] --solver SOLVER
                           --instance-set-train INSTANCE_SET_TRAIN
                           [--instance-set-test INSTANCE_SET_TEST]
                           [--performance-measure {RUNTIME,QUALITY_ABSOLUTE,QUALITY,QUALITY_ABSOLUTE_MINIMISATION,QUALITY_ABSOLUTE_MAXIMISATION}]
                           [--target-cutoff-time TARGET_CUTOFF_TIME]
                           [--budget-per-run BUDGET_PER_RUN]
                           [--number-of-runs NUMBER_OF_RUNS]
                           [--settings-file SETTINGS_FILE] [--use-features]
                           [--validate] [--ablation] [--run-on RUN_ON]
-h, --help

show this help message and exit

--configurator <configurator>

path to configurator

--solver <solver>

path to solver

--instance-set-train <instance_set_train>

path to training instance set

--instance-set-test <instance_set_test>

path to testing instance set (only for validating)

--performance-measure {RUNTIME,QUALITY_ABSOLUTE,QUALITY,QUALITY_ABSOLUTE_MINIMISATION,QUALITY_ABSOLUTE_MAXIMISATION}

the performance measure, e.g. runtime

--target-cutoff-time <target_cutoff_time>

cutoff time per target algorithm run in seconds

--budget-per-run <budget_per_run>

configuration budget per configurator run in seconds

--number-of-runs <number_of_runs>

number of configuration runs to execute

--settings-file <settings_file>

specify the settings file to use instead of the default

--use-features

use the training set’s features for configuration

--validate

validate after configuration

--ablation

run ablation after configuration

--run-on <run_on>

On which computer or cluster environment to execute the calculation.Available: local, slurm. Default: slurm

Note that the test instance set is only used if the --ablation or --validation flags are given

1.8.9. construct_sparkle_parallel_portfolio.py

usage: construct_sparkle_parallel_portfolio.py [-h] [--nickname NICKNAME]
                                               [--solver SOLVER [SOLVER ...]]
                                               [--overwrite OVERWRITE]
                                               [--settings-file SETTINGS_FILE]
-h, --help

show this help message and exit

--nickname <nickname>

Give a nickname to the portfolio. (default: sparkle_parallel_portfolio)

--solver <solver>

Specify the list of solvers, add “,<#solver_variations>” to the end of a path to add multiple instances of a single solver. For example –solver Solver/PbO-CCSAT-Generic,25 to construct a portfolio containing 25 variations of PbO-CCSAT-Generic.

--overwrite <overwrite>

When set to True an existing parallel portfolio with the same name will be overwritten, when False an error will be thrown instead. (default: False)

--settings-file <settings_file>

Specify the settings file to use in case you want to use one other than the default (default: Settings/sparkle_settings.ini

1.8.10. construct_sparkle_portfolio_selector.py

usage: construct_sparkle_portfolio_selector.py [-h]
                                               [--recompute-portfolio-selector]
                                               [--recompute-marginal-contribution]
                                               [--performance-measure {RUNTIME,QUALITY_ABSOLUTE,QUALITY,QUALITY_ABSOLUTE_MINIMISATION,QUALITY_ABSOLUTE_MAXIMISATION}]
-h, --help

show this help message and exit

--recompute-portfolio-selector

force the construction of a new portfolio selector even when it already exists for the current feature and performance data. NOTE: This will also result in the computation of the marginal contributions of solvers to the new portfolio selector.

--recompute-marginal-contribution

force marginal contribution to be recomputed even when it already exists in file for the current selector

--performance-measure

the performance measure, e.g. runtime

1.8.11. generate_report.py

Without any arguments a report for the most recent algorithm selection or algorithm configuration procedure is generated.

usage: generate_report.py [-h] [--solver SOLVER]
                          [--instance-set-train INSTANCE_SET_TRAIN]
                          [--instance-set-test INSTANCE_SET_TEST]
                          [--no-ablation [FLAG_ABLATION]] [--selection]
                          [--test-case-directory TEST_CASE_DIRECTORY]
                          [--performance-measure {RUNTIME,QUALITY_ABSOLUTE,QUALITY,QUALITY_ABSOLUTE_MINIMISATION,QUALITY_ABSOLUTE_MAXIMISATION}]
                          [--settings-file SETTINGS_FILE]
-h, --help

show this help message and exit

--solver <solver>

path to solver for an algorithm configuration report

--instance-set-train <instance_set_train>

path to training instance set included in Sparkle for an algorithm configuration report

--instance-set-test <instance_set_test>

path to testing instance set included in Sparkle for an algorithm configuration report

--no-ablation <flag_ablation>

turn off reporting on ablation for an algorithm configuration report

--selection

set to generate a normal selection report

--test-case-directory <test_case_directory>

Path to test case directory of an instance set for a selection report

--performance-measure

the performance measure, e.g. runtime

--settings-file

specify the settings file to use in case you want to use one other than the default

Note that if a test instance set is given, the training instance set must also be given.

1.8.12. initialise.py

Initialise the Sparkle platform, this command does not have any arguments.

usage: initialise.py [-h]
-h, --help

show this help message and exit

1.8.13. load_snapshot.py

usage: load_snapshot.py [-h] snapshot-file-path
snapshot-file-path

path to the snapshot file

-h, --help

show this help message and exit

1.8.14. remove_feature_extractor.py

usage: remove_feature_extractor.py [-h] [--nickname] extractor-path
extractor-path

path to or nickname of the feature extractor

-h, --help

show this help message and exit

--nickname

if set to True extractor_path is used as a nickname for the feature extractor

1.8.15. remove_instances.py

usage: remove_instances.py [-h] [--nickname] instances-path
instances-path

path to or nickname of the instance set

-h, --help

show this help message and exit

--nickname

if given instances_path is used as a nickname for the instance set

1.8.16. remove_solver.py

usage: remove_solver.py [-h] [--nickname] solver-path
solver-path

path to or nickname of the solver

-h, --help

show this help message and exit

--nickname

if set to True solver_path is used as a nickname for the solver

1.8.17. run_ablation.py

Runs parameter importance between the default and configured parameters with ablation. This command requires a finished configuration for the solver instance pair.

usage: run_ablation.py [-h] [--solver SOLVER]
                       [--instance-set-train INSTANCE_SET_TRAIN]
                       [--instance-set-test INSTANCE_SET_TEST]
                       [--ablation-settings-help]
                       [--performance-measure {RUNTIME,QUALITY_ABSOLUTE,QUALITY,QUALITY_ABSOLUTE_MINIMISATION,QUALITY_ABSOLUTE_MAXIMISATION}]
                       [--target-cutoff-time TARGET_CUTOFF_TIME]
                       [--budget-per-run BUDGET_PER_RUN]
                       [--number-of-runs NUMBER_OF_RUNS] [--racing RACING]
                       [--settings-file SETTINGS_FILE] [--run-on RUN_ON]
-h, --help

show this help message and exit

--solver <solver>

path to solver

--instance-set-train <instance_set_train>

Path to training instance set

--instance-set-test <instance_set_test>

Path to testing instance set

--ablation-settings-help

Prints a list of setting that can be used for the ablation analysis

--performance-measure

The performance measure, e.g. runtime

--target-cutoff-time

Cutoff time per target algorithm run in seconds

--budget-per-run

Configuration budget per configurator run in seconds

--number-of-runs

Number of configuration runs to execute

--racing

Performs abaltion analysis with racing

--settings-file

Specify the settings file to use in case you want to use one other than the default

--run-on <run_on>

On which computer or cluster environment to execute the calculation.Available: local, slurm. Default: slurm

Note that if no test instance set is given, the validation is performed on the training set.

1.8.18. run_configured_solver.py

usage: run_configured_solver.py [-h] [--settings-file SETTINGS_FILE]
                                [--performance-measure {RUNTIME,QUALITY_ABSOLUTE,QUALITY,QUALITY_ABSOLUTE_MINIMISATION,QUALITY_ABSOLUTE_MAXIMISATION}]
                                [--parallel] [--run-on RUN_ON]
                                instance_path [instance_path ...]
instance_path

Path(s) to instance file(s) (when multiple files are given, it is assumed this is a multi-file instance) or instance directory.

-h, --help

show this help message and exit

--settings-file <settings_file>

settings file to use instead of the default (default: Settings/sparkle_settings.ini)

--performance-measure {RUNTIME,QUALITY_ABSOLUTE,QUALITY,QUALITY_ABSOLUTE_MINIMISATION,QUALITY_ABSOLUTE_MAXIMISATION}

the performance measure, e.g. runtime (default: RUNTIME)

--parallel

run the solver on multiple instances in parallel

--run-on <run_on>

On which computer or cluster environment to execute the calculation.Available: local, slurm. Default: slurm

1.8.19. run_solvers.py

usage: run_solvers.py [-h] [--recompute] [--parallel]
                      [--performance-measure {RUNTIME,QUALITY_ABSOLUTE,QUALITY,QUALITY_ABSOLUTE_MINIMISATION,QUALITY_ABSOLUTE_MAXIMISATION}]
                      [--target-cutoff-time TARGET_CUTOFF_TIME]
                      [--also-construct-selector-and-report]
                      [--verifier {NONE,SAT}] [--run-on RUN_ON]
                      [--settings-file SETTINGS_FILE]
-h, --help

show this help message and exit

--recompute

recompute the performance of all solvers on all instances

--parallel

run the solver on multiple instances in parallel

--performance-measure {RUNTIME,QUALITY_ABSOLUTE,QUALITY,QUALITY_ABSOLUTE_MINIMISATION,QUALITY_ABSOLUTE_MAXIMISATION}

the performance measure, e.g. runtime

--target-cutoff-time <target_cutoff_time>

cutoff time per target algorithm run in seconds

--also-construct-selector-and-report

after running the solvers also construct the selector and generate the report

--verifier {NONE,SAT}

problem specific verifier that should be used to verify solutions found by a target algorithm

--run-on <run_on>

On which computer or cluster environment to execute the calculation.Available: local, slurm. Default: slurm

--settings-file <settings_file>

specify the settings file to use in case you want to use one other than the default

1.8.20. run_sparkle_parallel_portfolio.py

usage: run_sparkle_parallel_portfolio.py [-h] --instance-paths PATH [PATH ...]
                                         [--portfolio-name PORTFOLIO_NAME]
                                         [--process-monitoring {REALISTIC,EXTENDED}]
                                         [--performance-measure {RUNTIME,QUALITY_ABSOLUTE,QUALITY,QUALITY_ABSOLUTE_MINIMISATION,QUALITY_ABSOLUTE_MAXIMISATION}]
                                         [--cutoff-time CUTOFF_TIME]
                                         [--run-on RUN_ON]
                                         [--settings-file SETTINGS_FILE]
-h, --help

show this help message and exit

--instance-paths <path>

Specify the instance_path(s) on which the portfolio will run. This can be a space-separated list of instances containing instance sets and/or singular instances. For example –instance-paths Instances/PTN/Ptn-7824-b01.cnf Instances/PTN2/

--portfolio-name <portfolio_name>

Specify the name of the portfolio. If the portfolio is not in the standard directory, use its full path, the default directory is Sparkle_Parallel_Portfolio. (default: use the latest constructed portfolio) (current latest: no scenario found, you have to construct a parallel portfolio first.)

--process-monitoring {REALISTIC,EXTENDED}

Specify whether the monitoring of the portfolio should cancel all solvers within a portfolio once a solver finishes (REALISTIC). Or allow all solvers within a portfolio to get an equal chance to have the shortest running time on an instance (EXTENDED), e.g., when this information is needed in an experiment. (default: REALISTIC) (current value: REALISTIC)

--performance-measure {RUNTIME,QUALITY_ABSOLUTE,QUALITY,QUALITY_ABSOLUTE_MINIMISATION,QUALITY_ABSOLUTE_MAXIMISATION}

The performance measure, e.g., RUNTIME (for decision algorithms) or QUALITY_ABSOLUTE (for optimisation algorithms) (default: RUNTIME) (current value: RUNTIME)

--cutoff-time <cutoff_time>

The duration the portfolio will run before the solvers within the portfolio will be stopped (default: 60) (current value: 60)

--run-on <run_on>

On which computer or cluster environment to execute the calculation.Available: local, slurm. Default: slurm

--settings-file <settings_file>

Specify the settings file to use instead of the default (default: Settings/sparkle_settings.ini

1.8.21. run_sparkle_portfolio_selector.py

usage: run_sparkle_portfolio_selector.py [-h] [--run-on RUN_ON]
                                         [--settings-file SETTINGS_FILE]
                                         [--performance-measure {RUNTIME,QUALITY_ABSOLUTE,QUALITY,QUALITY_ABSOLUTE_MINIMISATION,QUALITY_ABSOLUTE_MAXIMISATION}]
                                         instance_path [instance_path ...]
instance_path

Path to instance or instance directory

-h, --help

show this help message and exit

--run-on <run_on>

On which computer or cluster environment to execute the calculation.Available: local, slurm. Default: slurm

--settings-file

settings file to use instead of the default

--performance-measure

the performance measure, e.g. runtime

1.8.22. run_status.py

usage: run_status.py [-h] [--verbose]
-h, --help

show this help message and exit

--verbose, -v

output run status in verbose mode

1.8.23. save_snapshot.py

usage: save_snapshot.py [-h]
-h, --help

show this help message and exit

1.8.24. sparkle_wait.py

usage: sparkle_wait.py [-h]
                       [--job-id JOB_ID | --command {ABOUT,ADD_FEATURE_EXTRACTOR,ADD_INSTANCES,ADD_SOLVER,CLEANUP_CURRENT_SPARKLE_PLATFORM,CLEANUP_TEMPORARY_FILES,COMPUTE_FEATURES,COMPUTE_MARGINAL_CONTRIBUTION,CONFIGURE_SOLVER,CONSTRUCT_SPARKLE_PORTFOLIO_SELECTOR,GENERATE_REPORT,INITIALISE,LOAD_SNAPSHOT,REMOVE_FEATURE_EXTRACTOR,REMOVE_INSTANCES,REMOVE_SOLVER,RUN_ABLATION,RUN_SOLVERS,RUN_SPARKLE_PORTFOLIO_SELECTOR,RUN_STATUS,SAVE_SNAPSHOT,SPARKLE_WAIT,SYSTEM_STATUS,VALIDATE_CONFIGURED_VS_DEFAULT,RUN_CONFIGURED_SOLVER,CONSTRUCT_SPARKLE_PARALLEL_PORTFOLIO,RUN_SPARKLE_PARALLEL_PORTFOLIO}]
-h, --help

show this help message and exit

--job-id <job_id>

job ID to wait for

--command {ABOUT,ADD_FEATURE_EXTRACTOR,ADD_INSTANCES,ADD_SOLVER,CLEANUP_CURRENT_SPARKLE_PLATFORM,CLEANUP_TEMPORARY_FILES,COMPUTE_FEATURES,COMPUTE_MARGINAL_CONTRIBUTION,CONFIGURE_SOLVER,CONSTRUCT_SPARKLE_PORTFOLIO_SELECTOR,GENERATE_REPORT,INITIALISE,LOAD_SNAPSHOT,REMOVE_FEATURE_EXTRACTOR,REMOVE_INSTANCES,REMOVE_SOLVER,RUN_ABLATION,RUN_SOLVERS,RUN_SPARKLE_PORTFOLIO_SELECTOR,RUN_STATUS,SAVE_SNAPSHOT,SPARKLE_WAIT,SYSTEM_STATUS,VALIDATE_CONFIGURED_VS_DEFAULT,RUN_CONFIGURED_SOLVER,CONSTRUCT_SPARKLE_PARALLEL_PORTFOLIO,RUN_SPARKLE_PARALLEL_PORTFOLIO}

command you want to run. Sparkle will wait for the dependencies of this command to be completed

1.8.25. system_status.py

usage: system_status.py [-h] [--verbose]
-h, --help

show this help message and exit

--verbose, -v

output system status in verbose mode

1.8.26. validate_configured_vs_default.py

Test the performance of the configured solver and the default solver by doing validation experiments on the training and test sets.

usage: validate_configured_vs_default.py [-h] --solver SOLVER
                                         --instance-set-train
                                         INSTANCE_SET_TRAIN
                                         [--instance-set-test INSTANCE_SET_TEST]
                                         [--configurator CONFIGURATOR]
                                         [--performance-measure {RUNTIME,QUALITY_ABSOLUTE,QUALITY,QUALITY_ABSOLUTE_MINIMISATION,QUALITY_ABSOLUTE_MAXIMISATION}]
                                         [--target-cutoff-time TARGET_CUTOFF_TIME]
                                         [--settings-file SETTINGS_FILE]
                                         [--run-on RUN_ON]
-h, --help

show this help message and exit

--solver <solver>

path to solver

--instance-set-train <instance_set_train>

path to training instance set

--instance-set-test <instance_set_test>

path to testing instance set

--configurator <configurator>

path to configurator

--performance-measure

the performance measure, e.g. runtime

--target-cutoff-time

cutoff time per target algorithm run in seconds

--settings-file

specify the settings file to use instead of the default

--run-on <run_on>

On which computer or cluster environment to execute the calculation.Available: local, slurm. Default: slurm

1.9. Sparkle settings

Most settings can be controlled through Settings/sparkle_settings.ini. Possible settings are summarised per category in Section 1.9.2. For any settings that are not provided the defaults will be used. Meaning, in the extreme case, that if the settings file is empty (and nothing is set through the command line) everything will run with default values.

For convenience after every command Settings/latest.ini is written with the used settings. This can, for instance, be used to provide the same settings to the next command in a chain. E.g. for validate_configured_vs_default after configure_solver. The used settings are also recorded in the relevant Output/ subdirectory. Note that when writing settings Sparkle always uses the name, and not an alias.

1.9.1. Example sparkle_settings.ini

This is a short example to show the format, see the settings file in Settings/sparkle_settings.ini for more.

[general]
performance_measure = RUNTIME
target_cutoff_time = 60

[configuration]
number_of_runs = 25

[slurm]
number_of_runs_in_parallel = 25

1.9.2. Names and possible values

[general]

performance_measure

aliases: smac_run_obj

values: {RUNTIME, QUALITY_ABSOLUTE (also: QUALITY})

RUNTIME focuses on runtime the solver requires,

QUALITY_ABSOLUTE and QUALITY focuses on the average absolute improvements on the instances

description: The type of performance measure that sparkle uses.

target_cutoff_time

aliases: smac_each_run_cutoff_time, cutoff_time_each_performance_computation

values: integer

description: The time a solver is allowed to run before it is terminated.

extractor_cutoff_time

aliases: cutoff_time_each_feature_computation

values: integer

description: The time a feature extractor is allowed to run before it is terminated. In case of multiple feature extractors this budget is divided equally.

penalty_multiplier

aliases: penalty_number

values: integer

description: In case of not solving an instance within the cutoff time the runtime is set to be the penalty_multiplier * cutoff_time.

solution_verifier

aliases: N/A

values: {NONE, SAT}

note: Only available for SAT solving.

[configuration]

budget_per_run

aliases: smac_whole_time_budget

values: integer

description: The wallclock time one configuration run is allowed to use for finding configurations.

number_of_runs

aliases: num_of_smac_runs

values: integer

description: The number of separate configurations runs.

[smac]

target_cutoff_length

aliases: smac_each_run_cutoff_length

values: {max} (other values: whatever is allowed by SMAC)

[ablation]

racing

aliases: ablation_racing

values: boolean

description: Use racing when performing the ablation analysis between the default and configured parameters

[slurm]

number_of_runs_in_parallel

aliases: smac_run_obj

values: integer

description: The number of configuration runs that can run in parallel.

clis_per_node

aliases: N/A

values: integer

note: Not really a Slurm option, will likely be moved to another section.

description: The number of parallel processes that can be run on one compute node. In case a node has 32 cores and each solver uses 2 cores, the cli_per_node is at most 16.

1.9.3. Priorities

Sparkle has a large flexibility with passing along settings. Settings provided through different channels have different priorities as follows:

  • Default –- Default values will be overwritten if a value is given through any other mechanism;

  • File –- Settings form the Settings/sparkle_settings.ini overwrite default values, but are overwritten by settings given through the command line;

  • Command line file -– Settings files provided through the command line, overwrite default values and other settings files.

  • Command line –- Settings given through the command line overwrite all other settings, including settings files provided through the command line.

1.9.4. Slurm (focused on Grace)

Slurm settings can be specified in the Settings/sparkle_slurm_settings.txt file. Currently these settings are inserted as is in any srun or sbatch calls done by Sparkle. This means that any options exclusive to one or the other currently should not be used (see Section 1.9.4.2).

To overwrite the default settings specific to the cluster Grace in Leiden, you should set the option “–partition” with a valid value on your cluster. Also, you might have to adapt “–mem-per-cpu” to your system.

1.9.4.1. Tested options

Below a list of tested Slurm options for srun and sbatch is included. Most other options for these commands should also be safe to use (given they are valid), but have not been explicitly tested. Note that any options related to commands other than srun and sbatch should not be used with Sparkle, and should not be included in Settings/sparkle_slurm_settings.txt.

  • -–partition / -p

  • -–exclude

  • -–nodelist

1.9.4.2. Disallowed options

The options below are exclusive to sbatch and are thus disallowed:

  • -–array

  • -–clusters

  • -–wrap

The options below are exclusive to srun and are thus disallowed:

  • -–label

1.9.4.3. Nested srun calls

A number of Sparkle commands internally call the srun command, and for those commands the provided settings need to match the restrictions of your call to a Sparkle command. Take for instance the following command:

srun -N1 -n1 -p graceTST Commands/configure_solver.py --solver Solvers/PbO-CCSAT-Generic --instances-train Instances/PTN/

This call restricts itself to the graceTST partition (the graceTST partition only consists of node 22). So if the settings file contains the setting –exclude=ethnode22, all available nodes are excluded, and the command cannot execute any internal srun commands it may have.

Finally, Slurm ignores nested partition settings for srun, but not for sbatch. This means that if you specify the graceTST partition (as above) in your command, but the graceADA partition in the settings file, Slurm will still execute any nested srun commands on the graceTST partition only.

1.10. Required packages

1.10.1. Sparkle on Grace

Grace is the computing cluster of the ADA group [2] at LIACS, Leiden University. Since not all packages required by Sparkle are installed on the system, some have to be installed local to the user.

1.10.1.1. Making your algorithm run on Grace

Shell and Python scripts should work as is. If a compiled binary does not work, you may have to compile it on Grace and manually install packages on Grace that are needed by your algorithm.

1.10.1.2. General requirements

Other software used by Sparkle:

  • pdflatex

  • latex

  • bibtex

  • gnuplot

  • gnuplot-x11

To manually install gnuplot see for instance the instructions on their website http://www.gnuplot.info/development/

1.11. Installation and compilation of examples

1.11.1. Solvers

1.11.1.1. CSCCSat

CSCCSat can be recompiled as follows in the Examples/Resources/Solvers/CSCCSat/ directory:

unzip src.zip
cd src/CSCCSat_source_codes/
make
cp CSCCSat ../../

1.11.1.2. MiniSAT

MiniSAT can be recompiled as follows in the Examples/Resources/Solvers/MiniSAT/ directory:

unzip src.zip
cd minisat-master/
make
cp build/release/bin/minisat ../

1.11.1.3. PbO-CCSAT

PbO-CCSAT can be recompiled as follows in the Examples/Resources/Solvers/PbO-CCSAT-Generic/ directory:

unzip src.zip
cd PbO-CCSAT-master/PbO-CCSAT_process_oriented_version_source_code/
make
cp PbO-CCSAT ../../

1.11.1.4. TCA and FastCA

The TCA and FastCA solvers, require GLIBCXX_3.4.21. This library comes with GCC 5.1.0 (or greater). Following installation you may have to update environment variables such as LD_LIBRARY_PATH, LD_RUN_PATH, CPATH to point to your installation directory.

TCA can be recompiled as follows in the Examples/Resources/CCAG/Solvers/TCA/ directory:

unzip src.zip
cd TCA-master/
make clean
make
cp TCA ../

FastCA can be recompiled as follows in the Examples/Resources/CCAG/Solvers/FastCA/ directory:

unzip src.zip
cd fastca-master/fastCA/
make clean
make
cp FastCA ../../

1.11.1.5. VRP_SISRs

VRP_SISRs can be recompiled as follows in the Examples/Resources/CVRP/Solvers/VRP_SISRs/ directory:

unzip src.zip
cd src/
make
cp VRP_SISRs ../