Calls irace.main() using command-line options, maybe parsed from the
command line used to invoke R.
Usage
irace.cmdline(argv = commandArgs(trailingOnly = TRUE))Arguments
- argv
(
character())
The arguments provided on the R command line as a character vector, e.g.,c("--scenario", "scenario.txt", "-p", "parameters.txt"). Using the default value (not providing the parameter) is the easiest way to callirace.cmdline.
Value
(invisible(data.frame))
A data frame with the set of best algorithm configurations found by irace. The data frame has the following columns:
.ID.: Internal id of the candidate configuration.Parameter names: One column per parameter name inparameters..PARENT.: Internal id of the parent candidate configuration.
Additionally, this function saves an R data file containing an object called
iraceResults. The path of the file is indicated in scenario$logFile.
The iraceResults object is a list with the following structure:
scenarioThe scenario R object containing the irace options used for the execution. See
defaultScenariofor more information.parametersThe parameters R object containing the description of the target algorithm parameters. See
readParameters.allConfigurationsThe target algorithm configurations generated by irace. This object is a data frame, each row is a candidate configuration, the first column (
.ID.) indicates the internal identifier of the configuration, the following columns correspond to the parameter values, each column named as the parameter name specified in the parameter object. The final column (.PARENT.) is the identifier of the configuration from which model the actual configuration was sampled.allElitesA list that contains one element per iteration, each element contains the internal identifier of the elite candidate configurations of the corresponding iteration (identifiers correspond to
allConfigurations$.ID.).iterationElitesA vector containing the best candidate configuration internal identifier of each iteration. The best configuration found corresponds to the last one of this vector.
experimentsA matrix with configurations as columns and instances as rows. Column names correspond to the internal identifier of the configuration (
allConfigurations$.ID.).experimentLogA matrix with columns
iteration,instance,configuration,time. This matrix contains the log of all the experiments that irace performs during its execution. The instance column refers to the index of thescenario$instancesListdata frame. Time is saved ONLY when reported by thetargetRunner.softRestartA logical vector that indicates if a soft restart was performed on each iteration. If
FALSE, then no soft restart was performed.stateA list that contains the state of irace, the recovery is done using the information contained in this object.
testingA list that contains the testing results. The elements of this list are:
experimentsa matrix with the testing experiments of the selected configurations in the same format as the explained above andseedsa vector with the seeds used to execute each experiment.
Details
The function reads the parameters given on the command line
used to invoke R, finds the name of the scenario file,
initializes the scenario from the file (with the function
readScenario) and possibly from parameters passed in
the command line. It finally starts irace by calling
irace.main.
List of command-line options:
-h,--help Show this help.
-v,--version Show irace package version.
-c,--check Check scenario.
-i,--init Initialize the working directory with template config
files.
--only-test Only test the configurations given in the file passed
as argument.
-s,--scenario File that describes the configuration scenario setup
and other irace settings. Default: ./scenario.txt.
--exec-dir Directory where the programs will be run. Default: ./.
-p,--parameter-file File that contains the description of the parameters
of the target algorithm. Default: ./parameters.txt.
--configurations-file File that contains a table of initial configurations.
If empty or `NULL`, all initial configurations are
randomly generated.
-l,--log-file File to save tuning results as an R dataset, either
absolute path or relative to execDir. Default:
./irace.Rdata.
--recovery-file Previously saved log file to recover the execution of
`irace`, either absolute path or relative to the
current directory. If empty or `NULL`, recovery is
not performed.
--train-instances-dir Directory where training instances are located;
either absolute path or relative to current directory.
If no `trainInstancesFiles` is provided, all the files
in `trainInstancesDir` will be listed as instances.
Default: ./Instances.
--train-instances-file File that contains a list of training instances and
optionally additional parameters for them. If
`trainInstancesDir` is provided, `irace` will search
for the files in this folder.
--sample-instances Randomly sample the training instances or use them in
the order given. Default: 1.
--test-instances-dir Directory where testing instances are located, either
absolute or relative to current directory.
--test-instances-file File containing a list of test instances and
optionally additional parameters for them.
--test-num-elites Number of elite configurations returned by irace that
will be tested if test instances are provided.
Default: 1.
--test-iteration-elites Enable/disable testing the elite configurations
found at each iteration. Default: 0.
--test-type Statistical test used for elimination. The default
value selects `t-test` if `capping` is enabled or
`F-test`, otherwise. Valid values are: F-test
(Friedman test), t-test (pairwise t-tests with no
correction), t-test-bonferroni (t-test with
Bonferroni's correction for multiple comparisons),
t-test-holm (t-test with Holm's correction for
multiple comparisons).
--first-test Number of instances evaluated before the first
elimination test. It must be a multiple of `eachTest`.
Default: 5.
--block-size Number of training instances, that make up a `block'
in `trainInstancesFile`. Elimination of configurations
will only be performed after evaluating a complete
block and never in the middle of a block. Each block
typically contains one instance from each instance
class (type or family) and the block size is the
number of classes. Default: 1.
--each-test Number of instances evaluated between elimination
tests. Default: 1.
--target-runner Executable called for each configuration that executes
the target algorithm to be tuned. See the templates
and examples provided. Default: ./target-runner.
--target-runner-launcher Executable that will be used to launch the target
runner, when `targetRunner` cannot be executed
directly (e.g., a Python script in Windows).
--target-cmdline Command-line arguments provided to `targetRunner` (or
`targetRunnerLauncher` if defined). The substrings
`\{configurationID\}`, `\{instanceID\}`, `\{seed\}`,
`\{instance\}`, and `\{bound\}` will be replaced by
their corresponding values. The substring
`\{targetRunnerArgs\}` will be replaced by the
concatenation of the switch and value of all active
parameters of the particular configuration being
evaluated. The substring `\{targetRunner\}`, if
present, will be replaced by the value of
`targetRunner` (useful when using
`targetRunnerLauncher`). Default: {configurationID}
{instanceID} {seed} {instance} {bound}
{targetRunnerArgs}.
--target-runner-retries Number of times to retry a call to `targetRunner`
if the call failed. Default: 0.
--target-runner-timeout Timeout in seconds of any `targetRunner` call (only
applies to `target-runner` executables not to R
functions), ignored if 0. Default: 0.
--target-evaluator Optional script or R function that provides a numeric
value for each configuration. See
templates/target-evaluator.tmpl
--deterministic If the target algorithm is deterministic,
configurations will be evaluated only once per
instance. Default: 0.
--max-experiments Maximum number of runs (invocations of `targetRunner`)
that will be performed. It determines the maximum
budget of experiments for the tuning. Default: 0.
--min-experiments Minimum number of runs (invocations of `targetRunner`)
that will be performed. It determines the minimum
budget of experiments for the tuning. The actual
budget depends on the number of parameters and
`minSurvival`.
--max-time Maximum total execution time for the executions of
`targetRunner`. `targetRunner` must return two values:
cost and time. This value and the one returned by
`targetRunner` must use the same units (seconds,
minutes, iterations, evaluations, ...). Default: 0.
--budget-estimation Fraction (smaller than 1) of the budget used to
estimate the mean computation time of a configuration.
Only used when `maxTime` > 0 Default: 0.05.
--min-measurable-time Minimum time unit that is still (significantly)
measureable. Default: 0.01.
--parallel Number of calls to `targetRunner` to execute in
parallel. Values `0` or `1` mean no parallelization.
Default: 0.
--load-balancing Enable/disable load-balancing when executing
experiments in parallel. Load-balancing makes better
use of computing resources, but increases
communication overhead. If this overhead is large,
disabling load-balancing may be faster. Default: 1.
--mpi Enable/disable MPI. Use `Rmpi` to execute
`targetRunner` in parallel (parameter `parallel` is
the number of slaves). Default: 0.
--batchmode Specify how irace waits for jobs to finish when
`targetRunner` submits jobs to a batch cluster: sge,
pbs, torque, slurm or htcondor. `targetRunner` must
submit jobs to the cluster using, for example, `qsub`.
Default: 0.
--digits Maximum number of decimal places that are significant
for numerical (real) parameters. Default: 4.
-q,--quiet Reduce the output generated by irace to a minimum.
Default: 0.
--debug-level Debug level of the output of `irace`. Set this to 0 to
silence all debug messages. Higher values provide more
verbose debug messages. Default: 0.
--seed Seed of the random number generator (by default,
generate a random seed).
--soft-restart Enable/disable the soft restart strategy that avoids
premature convergence of the probabilistic model.
Default: 1.
--soft-restart-threshold Soft restart threshold value for numerical
parameters. If `NA`, `NULL` or `""`, it is computed as
`10^-digits`.
-e,--elitist Enable/disable elitist irace. Default: 1.
--elitist-new-instances Number of instances added to the execution list
before previous instances in elitist irace. Default:
1.
--elitist-limit In elitist irace, maximum number per race of
elimination tests that do not eliminate a
configuration. Use 0 for no limit. Default: 2.
--capping Enable the use of adaptive capping, a technique
designed for minimizing the computation time of
configurations. Capping is enabled by default if
`elitist` is active, `maxTime > 0` and `boundMax > 0`.
--capping-type Measure used to obtain the execution bound from the
performance of the elite configurations: median, mean,
worst, best. Default: median.
--bound-type Method to calculate the mean performance of elite
configurations: candidate or instance. Default:
candidate.
--bound-max Maximum execution bound for `targetRunner`. It must be
specified when capping is enabled. Default: 0.
--bound-digits Precision used for calculating the execution time. It
must be specified when capping is enabled. Default: 0.
--bound-par Penalization constant for timed out executions
(executions that reach `boundMax` execution time).
Default: 1.
--bound-as-timeout Replace the configuration cost of bounded executions
with `boundMax`. Default: 1.
--postselection Percentage of the configuration budget used to perform
a postselection race of the best configurations of
each iteration after the execution of irace. Default:
0.
--aclib Enable/disable AClib mode. This option enables
compatibility with GenericWrapper4AC as targetRunner
script. Default: 0.
--iterations Maximum number of iterations. Default: 0.
--experiments-per-iteration Number of runs of the target algorithm per
iteration. Default: 0.
--min-survival Minimum number of configurations needed to continue
the execution of each race (iteration). Default: 0.
--num-configurations Number of configurations to be sampled and evaluated
at each iteration. Default: 0.
--mu Parameter used to define the number of configurations
sampled and evaluated at each iteration. Default: 5.
--confidence Confidence level for the elimination test. Default:
0.95.See also
irace.main() to start irace with a given scenario.
Examples
irace.cmdline("--version")
#> #------------------------------------------------------------------------------
#> # irace: An implementation in R of (Elitist) Iterated Racing
#> # Version: 3.5.1.9000.036e23a-dirty
#> # Copyright (C) 2010-2020
#> # Manuel Lopez-Ibanez <manuel.lopez-ibanez@manchester.ac.uk>
#> # Jeremie Dubois-Lacoste
#> # Leslie Perez Caceres <leslie.perez.caceres@ulb.ac.be>
#> #
#> # This is free software, and you are welcome to redistribute it under certain
#> # conditions. See the GNU General Public License for details. There is NO
#> # WARRANTY; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
#> #
#> # irace builds upon previous code from the race package:
#> # race: Racing methods for the selection of the best
#> # Copyright (C) 2003 Mauro Birattari
#> #------------------------------------------------------------------------------
#> # installed at: /home/runner/work/_temp/Library/irace
#> # called with: --version
#> To cite package 'irace' in publications, please use
#>
#> Manuel López-Ibáñez, Jérémie Dubois-Lacoste, Leslie Pérez Cáceres,
#> Thomas Stützle, and Mauro Birattari. The irace package: Iterated
#> Racing for Automatic Algorithm Configuration. Operations Research
#> Perspectives, 3:43-58, 2016. doi:10.1016/j.orp.2016.09.002
#>
#> A BibTeX entry for LaTeX users is
#>
#> @Article{,
#> title = {The irace package: Iterated Racing for Automatic Algorithm Configuration},
#> author = {Manuel López-Ibáñez and Jérémie Dubois-Lacoste and Leslie {Pérez Cáceres} and Thomas Stützle and Mauro Birattari},
#> year = {2016},
#> journal = {Operations Research Perspectives},
#> doi = {10.1016/j.orp.2016.09.002},
#> volume = {3},
#> pages = {43--58},
#> }