Changelog • irace

irace 4.1

Major breaking changes

Recovery (--recover-file) is currently broken (issue #76).
The experiment list passed to the targetRunner R function (see target_runner_default()) does not contain an element switches. This element can be obtained from scenario$parameters$switches.

New features and improvements

Conditions that are always true because they depend on fixed parameters are replaced by a TRUE value to speed up their evaluation. This speed-up is only measurable with very large and complicated parameter space.
The options src and target of ablation() (or --src and --target of the command-line ablation tool) now accept a character string that points to a file containing a configuration that will be read with readConfigurationsFile(). This allows performing ablation between configurations not explored before.

Fixes

Fixed some typos in the user guide. (@sbomsdorf)
Fixed several bugs in post-selection including Error in if (any(left >= 0L)) { : missing value where TRUE/FALSE needed.
Fix bug in ablation() not calling stop_parallel().

irace 4.0

CRAN release: 2024-11-26

Major breaking changes

Requires R version >= 4.0
Logfiles *.Rdata use format version 3, which can only be read by R version >= 3.5.
The scenario options forbiddenFile and forbiddenExps have been removed and will give an error if present. Forbidden configurations are now specified in the parameter space description. See the example in readParameters().
The scenario option digits has been removed and will give an error if present. The number of digits for real-valued parameters is now specified in the parameter space description. See the example in readParameters().
The default value of the scenario option trainInstancesDir is now "". The previous default value of "./Instances" often caused confusion to users not using files as training instances.
The scenario object now includes the parameters object. Thus functions such as irace(), which previously took as arguments both scenario and parameters, now only take scenario. This also means that the log file irace.Rdata does not contain a separate parameters element since this element can now be found within scenario.
The default value of the scenario option softRestartThreshold is now 0.0001 and does not depend on digits.
The command-line executables irace and ablation (irace.exe and ablation.exe in Windows) will load the version of the irace package that is found in the same path where the executables are. In earlier versions, the executables will always load the version found via base::.libPaths(). This change allows installing multiple versions of the irace package in different locations and each executable will use its corresponding version. The correct location can be verified by looking at the line "installed at:" printed in the output.
Adaptive capping is now enabled by default if maxTime > 0 and boundMax > 0. It can be disabled with --capping 0 in the command-line options or capping=0 in the scenario options. See Pérez-Cáceres et al. (2017) for details. (Leslie Pérez Cáceres, Manuel López-Ibáñez)
The scenario option targetRunnerLauncherArgs, introduced in version 3.5, was removed and replaced by targetCmdline, which is more flexible (fixes #38). Please see the user-guide for details.
Command-line options in joined form, given as "--log-file= --check", without any argument after the '=' will be interpreted as an empty argument, equivalent to using logFile="" in scenario.txt.
irace will now give an error if you try to recover from a file generated by a different version of irace, since such attempts typically end up in errors that are difficult to understand.
irace warns about using '&&' and '||' instead of '&' and '|' in parameter conditions and forbidden expressions. A future version of irace will reject those uses as errors.
The internal function irace.reload.debug() has been removed. Use devtools::reload() instead.
The column "instance" of the instancesList data frame stored in the logFile has been renamed to "instanceID". This data frame should not be accessed directly. Instead use the new function get_instanceID_seed_pairs().
Using maxTime > 0 with elitist=0 now gives a clear error rather than fail later. (fix #65, reported by @DE0CH)
path_rel2abs() will not expand symlinks to avoid problems with Python’s venv. (fix #64, reported by @DE0CH)
Expansion of '~' in Windows now follows the definition of fs::path_expand() rather than base::path.expand().
irace is now more strict in enforcing runtime bounds given with scenario$boundMax and will stop with an error if the target-runner reports a runtime larger than the given bound.
All functions that contained a period ('.') in the name have been renamed to use '_' instead.
The periods ('.') in the arguments of scenario$targetRunnerParallel and scenario$targetEvaluator have also been replaced by '_'.
The environment .irace that was available in the log file under iraceResults$state$.irace is replaced directly by iraceResults$state. It contains similar information but some entries have been renamed. For example, the experimentLog data frame is now called experiment_log and it is a data.table.
The interface of psRace() has been simplified.
irace will automatically execute a post-selection race (psRace()) using any remaining budget (currently only when maxTime == 0). To disable this behavior, set the scenario option postselection (--postselection) to 0.

New features and improvements

sampleUniform() and sampleModel() are significantly faster thanks to using data.table.
Initial configurations are sampled using Sobol low-discrepancy sequences using spacefillr::generate_sobol_set(). This should provide a better initial distribution of parameter values.
Parameter spaces can be constructed programmatically using parametersNew(). See the documentation for details.
Ablation will report configurations that produced the same results, which indicates parameter values that have the same effect on the target algorithm, possibly indicating a bug in the target algorithm.
New option instancesFile of ablation() for using either the training instances, the test instances or reading instances from a given file.
New option nrep of ablation() specifies the number of replications per instance used in "full" ablation. It replaces the previous parameter n_instances, whose definition was more difficult to use correctly.
Matrix operations are faster thanks to matrixStats.
New scenario option blockSize for defining blocks of instances. Configurations are only eliminated after evaluating a complete block and never in the middle of a block. This is useful for scenarios when there are clearly defined instance classes and the best configuration should be balanced among them. In that case, trainInstancesFile should be written so that each block contains one instance of each class and blockSize is set to the number of classes.
New scenario option targetRunnerTimeout: Timeout in seconds of any targetRunner call (only applies to target-runner executables not to R functions).
plotAblation() has several new options:
- type='rank' to plot ranks per instance instead of raw cost values.
- n to limit the number of parameters shown in the plot.
- width replaces pdf.width.
- height sets the height of the plot in the PDF file.
The previously internal function check.output.target.runner is renamed to check_output_target_runner and exported to allow users who write their own targetRunnerParallel to check the output and repair it if possible. (Deyao Chen)
New functions read_ablogfile(), has_testing_data(), irace_summarise().
New functions get_random_seed(), set_random_seed(), restore_random_seed() useful for writing targetRunner functions in R.
New function get_instanceID_seed_pairs() to get the pairs of instanceID and random seed used during the races (and optionally the actual instances).
The parameters object now stores the number of digits (decimal places after the point) for each parameter of type r. As a result, the repairConfiguration function (see defaultScenario()) only needs two arguments: configuration and parameters. See examples in the user-guide.
readScenario() (and command-line irace) do not require a scenario.txt file. (Contributed by @DE0CH)
read_pcs_file() now supports forbidden configurations.
When testing, irace now prints the random seed used for each instance as an additional column.
The package provides a new executable target-runner-dummy (or target-runner-dummy.exe in Windows) for the purposes of testing. It may also be useful for understanding the typical setup of irace.
New scenario option minExperiments to set a minimum budget of runs. (proposed by @Saethox, fixes #58)
New function multi_irace() for executing multiple runs of irace with the same or different scenarios and parameters, possibly in parallel. (Contributed by @Saethox)

Fixes

ablation_cmdline() and plotAblation() no longer create an empty Rplots.pdf file when specifying an output PDF file.
Fix #66: when using maxTime > 0, irace estimates the time per run by executing 2 configurations on firstTest instances and adjusts boundMax to not go over budgetEstimation. This may result in a smaller boundMax than before. To reduce this impact, the default value of budgetEstimation is now 0.05 instead of 0.02. (Manuel López-Ibáñez, reported by @DE0CH)
Fix #55: Configurations provided may use <NA> in addition to NA to denote the missing value of a disabled parameter. (Manuel López-Ibáñez, reported by @TheIronBorn)
Fix #44: irace now will give an error if the domain of real-valued (r) parameters would change depending on the value of 'digits'. The solution is to increase the value of 'digits' or adjust the domain. (Manuel López-Ibáñez, reported by @mb706)
If scenario option targetRunnerParallel is set, irace no longer tries to initialize a parallel environment or setup MPI. It is now the responsibility of the user to do that before calling irace or within the function assigned to targetRunnerParallel.
irace no longer sets option(error=utils::recover()) in debug mode to avoid issues when calling irace from Python. The user can set this if desired.
Fix bug failing to restart with parameters that have dependent domains.
Fix bug with sampleInstances=FALSE that could re-evaluate the same (instance, seed) pair more than once.
Fix bug when using targetRunnerLauncher and targetRunner contain whitespace.
Fix bug in ablation_cmdline() about missing scenario object.
ablation() will now save and restore the previous random seed.
ablation() will detect if the logfile (e.g., irace.Rdata) is incomplete.
readConfigurationsFile() now handles parameters with dependent domains.
Fix #71: Ensure ".ID." is the first column in checkTargetFiles() (Manuel López-Ibáñez, reported by @ivan1arriola)

irace 3.5

CRAN release: 2022-10-23

New features and improvements

Handling of dependent parameter domains: These should be specified in the parameter domain definition and, for now, only numerical parameter can define dependent domains. A numerical domain can be dependent on one bound, e.g. (1, "param1*2"), where the dependent bound can include basic arithmetic operators. (Leslie Pérez Cáceres, Manuel López-Ibáñez)
The package now provides an ablation executable (ablation.exe in Windows) that makes easier to perform ablation analysis without having any R knowledge.
The interface to functions ablation() and plotAblation() has been simplified. The ablation() function now allows overriding scenario settings. The plotAblation() function will not create the plot if the ablation log does not contain a complete ablation. (Manuel López-Ibáñez)
The argument n.instances of ablation() has been renamed to n_instances and it is now a factor that multiplies scenario$firstTest. (Manuel López-Ibáñez)
New command-line option --quiet to run without producing any output except errors (also available as a scenario option). (Manuel López-Ibáñez)
New command-line option --init to initialize a scenario. (Deyao Chen)
Added support for HTCondor cluster framework to --batchmode. (Filippo Bistaffa)
--check now also check the contents of configurationsFile and runs configurations provided via initConfigurations. (Manuel López-Ibáñez, reported by Andreea Avramescu)
New scenario options targetRunnerLauncher and targetRunnerLauncherArgs to help in cases where the target-runner must be invoked via another software with particular options (such as python.exe in Windows). (Manuel López-Ibáñez)
New scenario option minMeasurableTime. (Manuel López-Ibáñez)
An error is produced if a variable set in the scenario file is not known to irace. If your scenario file contains R code, then use variable names beginning with a dot '.', which will be ignored by irace. (Manuel López-Ibáñez)
Plotting functions have been moved to the new package iraceplot. In particular, configurationsBoxplot() is replaced by iraceplot::boxplot_training() and iraceplot::boxplot_test(); parallelCoordinatesPlot() is replaced by iraceplot::parallel_cat() and iraceplot::parallel_coord(); and parameterFrequency() is replaced by iraceplot::sampling_frequency(). (Leslie Pérez Cáceres, Manuel López-Ibáñez)
The user-guide now contains a detailed section on “Hyper-parameter optimization of machine learning methods”. (Manuel López-Ibáñez)
When testType="F-test" and only two configurations remain, the elimination test now uses the pseudo-median estimated by the Wilcoxon signed-rank test to decide which configuration is the best one instead of comparing the median difference. (Manuel López-Ibáñez)
New functions testing_fromlog() and testing_fromfile() for independently executing the testing phase. The function testing.main() was removed as it is superseded by the new ones. (Manuel López-Ibáñez)
New function read_logfile() to easily read the log file produced by irace. (Manuel López-Ibáñez)
New function printParameters() that prints a parameters R object as a valid input text. (Manuel López-Ibáñez)
irace2pyimp moved to its own R package. (Manuel López-Ibáñez)
Generating the file irace.Rdata may be disabled by setting logFile="". (Manuel López-Ibáñez, reported by Johann Dreo)
path_rel2abs() and checkParameters() are now exported so that other packages may use them. (Manuel López-Ibáñez)
path_rel2abs() also searches in system paths. (Manuel López-Ibáñez)
readConfigurationsFile() will now detect duplicated configurations and error. (Manuel López-Ibáñez)
The interface to functions getFinalElites(), getConfigurationById() and getConfigurationByIteration() has been simplified.
The package provides a irace.sindef file that may be used for building a standalone container of irace using Singularity. See the README.md file for instructions. (Contributed by Johann Dreo)
New example examples/target-runner-python/target-runner-python-win.bat contributed by Levi Ribeiro.
New helper script in bin/parallel-irace-slurm to launch irace in SLURM computer clusters. (Manuel López-Ibáñez)
Rename scenario.update.paths() to scenario_update_paths(). The old name is deprecated. (Manuel López-Ibáñez)

Fixes

Correctly handle clear out-performance cases despite strong bi-modality. (Reported by Nguyen Dang, fixed by Manuel López-Ibáñez)
Fix error when recovering from a parallel run on Windows. (Manuel López-Ibáñez, reported by Tarek Gamal)
testNbElites now controls how many iteration elites are tested when testIterationElites=1. This is the documented behavior in the user guide. (Manuel López-Ibáñez, reported by Marcelo de Souza)
Fixes to the Matlab example. (Manuel López-Ibáñez)
The default of testType is now set to t-test when capping is enabled. (Manuel López-Ibáñez, reported by Jovana Radjenovic)
Fix various issues in the user guide. (Manuel López-Ibáñez, reported by Jovana Radjenovic)
Remove duplicated elites. (Manuel López-Ibáñez, reported by Federico Naldini)
Fix (#7): warnings with partial matched parameters. (Manuel López-Ibáñez, reported by Marc Becker)
Fix (#10): wrong assert with elitist=0. (Manuel López-Ibáñez)
Fix (#12): irace can be run with FastR.
Fix (#13): Maximum number configurations immediately rejected reached. (Manuel López-Ibáñez)
Fix: when setting the scenario file in the command-line, scenarioFile was not set correctly. The correct scenario was used, however, the debug output and the value stored in the log / recovery file was wrong. (Manuel López-Ibáñez, reported by Richard Schoonhoven)
With sampleInstances = FALSE, elitist irace does not change the order of instances already seen. However, if you want to make sure that the order of the instances is enforced, you also need to set elitistNewInstances=0.
The function irace.usage() was removed. It was not really useful for R users as the same result can be obtained by calling irace.cmdline("--help"). (Manuel López-Ibáñez)

irace 3.4.1 (31/03/2020)

CRAN release: 2020-03-31

NEWS converted to markdown.
Fix CRAN error on Solaris.

irace 3.4 (30/03/2020)

CRAN release: 2020-03-30

irace2pyimp function and executable (irace2pyimp.exe on Windows) to convert .Rdata files generated by irace to the input files required by the parameter importance analysis tool PyImp (https://github.com/automl/ParameterImportance). (Nguyen Dang, Manuel López-Ibáñez)
Initial configurations may also be provided directly in R using scenario$initConfigurations (Manuel López-Ibáñez)
Rdata files are saved in version 2 to keep compatibility with older R versions. (Manuel López-Ibáñez)
Fix invalid assert with ordered parameters: (Leslie Pérez Cáceres)
```
value >= 1L && value <= length(possibleValues) is not TRUE
```
The irace executable (irace.exe on Windows) is a compiled binary instead of a script. On Windows, irace.exe replaces irace.bat (Manuel López-Ibáñez)
inst/examples/Spear contains the Spear (SAT solver) configuration scenario. (Manuel López-Ibáñez)
Fixed bug when reporting minimum maxTime required. (Reported by Luciana Salete Buriol, fixed by Manuel López-Ibáñez)
Fixed bug detected by assert:
```
all(apply(!is.na(elite.data$experiments), 1, any)) is not TRUE
```
(Reported by Maxim Buzdalov, fixed by Manuel López-Ibáñez)

irace 3.3 (26/04/2019)

CRAN release: 2019-04-26

Fix buggy test that breaks CRAN. (Manuel López-Ibáñez)
Do not print “23:59:59” when wall-clock time is actually close to zero. (Manuel López-Ibáñez)

irace 3.2 (24/04/2019)

CRAN release: 2019-04-24

Fix irace --check --parallel 2 on Windows. (Manuel López-Ibáñez)
Values of real-valued parameter are now printed with sufficient precision to satisfy digits (up to digits=15). (Manuel López-Ibáñez)
It is possible to specify boundMax without capping. (Leslie Pérez Cáceres, Manuel López-Ibáñez)
irace --check will exit with code 1 if the check is unsuccessful (Manuel López-Ibáñez)
Print where irace is installed with --help. (Manuel López-Ibáñez)
irace will now complain if the output of target-runner or target-evaluator contains extra lines even if the first line of output is correct. This is to avoid parsing the wrong output. Unfortunately, this may break setups that relied on this behavior. The solution is to only print the output that irace expects. (Manuel López-Ibáñez)
Completely re-implement log parameters to fix several bugs. Domains that contain zero or negative values are now rejected. (Leslie Pérez Cáceres, Manuel López-Ibáñez)
New option aclib= (--aclib 1) enables compatibility with the GenericWrapper4AC (https://github.com/automl/GenericWrapper4AC/) used by AClib (http://aclib.net/). This is EXPERIMENTAL. --aclib 1 also sets digits to 15 for compatibility with AClib defaults. (Manuel López-Ibáñez)
Fix printing of output when capping is enabled. (Manuel López-Ibáñez)
checkTargetFiles() (--check) samples an instance unless sampleInstances is FALSE. (Manuel López-Ibáñez)
Fix symbol printed in elimination test. (Manuel López-Ibáñez)
Use dynGet() to find targetRunner and targetEvaluator. As a result, we now require R >= 3.2. (Manuel López-Ibáñez)
All tests now use testthat. (Manuel López-Ibáñez)
New function scenario.update.paths() (Manuel López-Ibáñez)
Fix assert failure that may happen when elitistNewInstances is larger than firstTest. Reported by Jose Riveaux. (Manuel López-Ibáñez)
Fix bug in checkTargetFiles() (--check) with capping. (Leslie Pérez Cáceres)
Clarify a few errors/warnings when maxTime > 0. (Manuel López-Ibáñez, suggested by Haroldo Gambini Santos)

irace 3.1 (12/07/2018)

CRAN release: 2018-07-12

Use testthat for unit testing. (Manuel López-Ibáñez)
Allow instances to be a list of arbitrary R objects (mlr bugfix). (Manuel López-Ibáñez)

irace 3.0 (05/07/2018)

CRAN release: 2018-07-05

irace now supports adaptive capping for computation time minimization. The default value of the testType option is t-test when adaptive capping is enabled. Please see the user-guide for details. (Leslie Pérez Cáceres, Manuel López-Ibáñez)
The package contains an ablation() function implementing the ablation method for parameter importance analysis by Fawcett and Hoos (2016). (Leslie Pérez Cáceres, Manuel López-Ibáñez)
New option postselection executes a post-selection race. (Leslie Pérez Cáceres)
At the end of each race, if the race stops before evaluating all instances seen in previous races, then the best overall may be different than the best of the race. We now print the best overall (best-so-far). Elites evaluated on more instances are considered better than those evaluated on fewer. (Manuel López-Ibáñez, Leslie Pérez Cáceres)
Last active parameter values of numerical parameters (i and r) are carried by the sampling model. When a value must be assigned and the parameter was previously not active, the sampling is performed around the last value. (Leslie Pérez Cáceres, Manuel López-Ibáñez)
R help pages are now generated with Roxygen2. (Leslie Pérez Cáceres, Manuel López-Ibáñez)
The user guide documents --version, --help, and --check. (Manuel López-Ibáñez)
A return value of Inf from targetRunner/targetEvaluation results in the immediate rejection of the configuration without any further evaluation. This is useful for handling unreliable or broken configurations that should not stop irace. (Manuel López-Ibáñez)
Numerical parameters may be sampled on a logarithmic scale using i,log or r,log. (Alberto Franzin)
New target-runner.bat for Windows contributed by André de Souza Andrade.
Fixed all shell scripts calling functions before defining them, which is not portable. (Manuel López-Ibáñez)
Fixed --parallel bug in Windows that resulted in Error in checkForRemoteErrors(val). (Manuel López-Ibáñez)
Improve error message when no training instances are given. (Manuel López-Ibáñez)

irace 2.4 (03/08/2017)

CRAN release: 2017-08-03

The output of irace now specifies in which order, if any, configurations are printed. (Manuel López-Ibáñez, suggested by Markus Wagner)
Several fixes for handling paths in Windows. (Manuel López-Ibáñez)
readConfigurationsFile() now has a text= argument, which allows reading configurations from a string. (Manuel López-Ibáñez)
User-provided functions (targetRunner, targetEvaluator and repairConfiguration) and user-provided conditions for forbidden configurations are now byte-compiled when read, which should make their evaluation noticeably faster. (Manuel López-Ibáñez)
The argument 'experiment' passed to the R function targetRunner does not contain anymore an element 'extra.params'. Similarly, the 'scenario' structure does not contain anymore the elements 'instances.extra.params' and 'testInstances.extra.params'. Any instance-specific parameters values now form part of the character string that defines an instance and it is up to the user-defined targetRunner to parse them appropriately. These changes make no difference when targetRunner is an external script, or when instances and instance-specific parameter values are read from a file. (Manuel López-Ibáñez)

irace 2.3

CRAN release: 2017-04-26

Fix bug that will cause iraceResults$experimentLog to count calls to targetEvaluator as experiments, even if no call to targetRunner was performed. This does not affect the computation of the budget consumed and, thus, it does not affect the termination criteria of irace. The bug triggers an assertion that terminates irace, thus no run that was successful with version 2.2 is affected. (Manuel López-Ibáñez)

irace 2.2

CRAN release: 2017-03-24

Command-line parameters are printed to stdout (useful for future replications). (Manuel López-Ibáñez, suggested by Markus Wagner)
Users may provide a function to repair configurations before being evaluated. See the scenario variable repairConfiguration. (Manuel López-Ibáñez)
The option --sge-cluster (sgeCluster) was removed and replaced by --batchmode (batchmode). It is now the responsibility of the target-runner to parse the output of the batch job submission command (e.g., qsub or squeue), and return just the job ID. Values supported are: “sge”, “torque”, “pbs” and “slurm”. (Manuel López-Ibáñez)
The option --parallel can now be combined with --batchmode to limit the number of jobs submitted by irace at once. This may be useful in batch clusters that have a small queue of jobs. (Manuel López-Ibáñez)
New examples under inst/examples/batchmode-cluster/. (Manuel López-Ibáñez)
It is now possible to include scenario definition files from other scenario files by using:
```
eval.parent(source("scenario-common.txt", chdir = TRUE, local = TRUE))
```
This feature is VERY experimental and the syntax is likely to change in the future. (Manuel López-Ibáñez)
Fix a bug that re-executed elite results under some circumstances. (Leslie Pérez Cáceres)
Restrict the number of maximum configurations per race to 1024. (Leslie Pérez Cáceres)
Do not warn if the last line in the instance file does not terminate with a newline. (Manuel López-Ibáñez)
Fix bug when deterministic == 1. (Manuel López-Ibáñez, Leslie Pérez Cáceres)
Update manual and vignette with details about the expected arguments and return value of targetRunner and targetEvaluator. (Manuel López-Ibáñez)
Many updates to the User Guide vignette. (Manuel López-Ibáñez)
Fix \dontrun example in irace-package.Rd (Manuel López-Ibáñez)
Fix bug: If testInstances contains duplicates, results of testing are not correctly saved in iraceResults$testing$experiments nor reported correctly at the end of a run. Now unique IDs of the form 1t, 2t, ... are used for each testing instance. These IDs are used for the rownames of iraceResults$testing$experiments and the names of the scenario$testInstances and iraceResults$testing$seeds vectors. (Manuel López-Ibáñez)
Fix bug where irace keeps retrying the target-runner call even if it succeeds. (Manuel López-Ibáñez)
New command-line parameter

       --only-test FILE

which just evaluates the configurations given in FILE on the testing instances defined by the scenario. Useful if you decide on the testing instances only after running irace. (Manuel López-Ibáñez)

Bugfix: When using maxTime != 0, the number of experiments performed may be miscounted in some cases. (Manuel López-Ibáñez)

irace 2.1

CRAN release: 2016-10-18

Fix CRAN errors in tests. (Manuel López-Ibáñez)
Avoid generating too many configurations at once if the initial time estimation is too small. (Manuel López-Ibáñez)

irace 2.0

CRAN release: 2016-10-08

Minimum R version is 2.15.
Elitist irace by default, it can be disabled with parameter --elitist 0. (Leslie Pérez Cáceres, Manuel López-Ibáñez)
The parameter --test-type gains two additional values: (Manuel López-Ibáñez)
- t-test-bonferroni (t-test with Bonferroni’s correction for multiple comparisons),
- t-test-holm (t-test with Holm’s correction for multiple comparisons)
MPI does not create log files with --debug-level 0. (Manuel López-Ibáñez)
For simplicity, the parallel-irace-* scripts do not use an auxiliary tune-main script. For customizing them, make a copy and edit them directly. (Manuel López-Ibáñez)
New parameters: (Manuel López-Ibáñez)

--target-runner-retries : Retry target-runner this many times in case of error.

We print diversity measures after evaluating on each instance: (Leslie Pérez Cáceres)
- Kendall’s W (also known as Kendall’s coefficient of concordance) If 1, all candidates have ranked in the same order in all instances. If 0, the ranking of each candidate on each instance is essentially random.
```
    W = Friedman / (m * (k-1))
```
- Spearman’s rho: average (Spearman) correlation coefficient computed on the ranks of all pairs of raters. If there are no repeated data values, a perfect Spearman correlation of +1 or -1 occurs when each of the variables is a perfect monotone function of the other.
Many internal and external interfaces have changed. For example, now we consistently use ‘scenario’ to denote the settings passed to irace and ‘configuration’ instead of ‘candidate’ to denote the parameter settings passed to the target algorithm. Other changes are:

   parameters$boundary -> parameters$domain
   hookRun             -> targetRunner
   hookEvaluate        -> targetEvaluator
   tune-conf           -> scenario.txt
   instanceDir         -> trainInstancesDir
   instanceFile        -> trainInstancesFile
   testInstanceDir     -> testInstancesDir
   testInstanceFile    -> testInstancesFile

Minimal example of configuring a MATLAB program (thanks to Esteban Diaz Leiva)
Paths to files or directories given in the scenario file are relative to the scenario file (except for --log-file, which is an output file and it is relative to --exec-dir). Paths given in the command-line are relative to the current working directory. Given

      $ cat scenario/scenario.txt
      targetRunner <- "./target-runner"
      $ irace -s scenario/scenario.txt

irace will search for "./scenario/target-runner", but given

      $ irace -s scenario/scenario.txt --target-runner ./target-runner

irace will search for "./target-runner". (Manuel López-Ibáñez)

New command-line wrapper for Windows installed at system.file("bin/irace.bat", package="irace") (thanks to Anthony Antoun)
Budget can be specified as maximum time (maxTime, --max-time) consumed by the target algorithm. See the documentation for the details about how this is handled. (Leslie Pérez Cáceres, Manuel López-Ibáñez)

irace 1.07

CRAN release: 2015-09-11

The best configurations found, either at the end or at each iteration of an irace run, can now be applied to a set of test instances different from the training instances. See options testInstanceDir, testInstanceFile, testNbElites, and testIterationElites. (Leslie Pérez Cáceres, Manuel López-Ibáñez)
The R interfaces of hookRun, hookEvaluate and hookRunParallel have changed. See help(hook.run.default) and help(hook.evaluate.default) for examples of the new interfaces.
Printing of race progress now reports the actual configuration and instance IDs, and numbers are printed in a more human-readable format. (Leslie Pérez Cáceres, Manuel López-Ibáñez)
Reduce memory use for very large values of maxExperiments. (Manuel López-Ibáñez, thanks to Federico Caselli for identifying the issue)
New option --load-balancing (loadBalancing) for disabling load-balancing when executing jobs in parallel. Load-balancing makes better use of computing resources, but increases communication overhead. If this overhead is large, disabling load-balancing may be faster. (Manuel López-Ibáñez, thanks to Federico Caselli for identifying the issue)
The option --parallel in Windows now uses load-balancing by default. (Manuel López-Ibáñez)
The wall-clock time after finishing each task is printed in the output. (Manuel López-Ibáñez, thanks to Federico Caselli for providing an initial patch)

irace 1.06

CRAN release: 2014-11-26

Fix bug that could introduce spurious whitespace when printing the final configurations. (Manuel López-Ibáñez)
Fix bug if there are more initial candidates than needed for the first race. (Leslie Pérez Cáceres, Manuel López-Ibáñez)
New configuration options, mainly for R users:
- hookRunParallel: Optional R function to provide custom parallelization of hook.run.
- hookRunData: Optional data passed to hookRun. This is ignored by the default hookRun function, but it may be used by custom hookRun R functions to pass persistent data around. (Manuel López-Ibáñez)

irace 1.05

CRAN release: 2014-07-10

New option --version. (Manuel López-Ibáñez)
Terminate early if there is no sufficient budget to run irace with the given settings. (Manuel López-Ibáñez)
The option --parallel (without --mpi) now works under Windows. (Manuel López-Ibáñez, thanks to Pablo Valledor Pellicer for testing it)
Improved error handling when running under Rmpi. Now irace will terminate as soon as the master node detects at least one failed slave node. This avoids irace reporting two times the same error. Also, irace will print all the unique errors returned by all slaves and not just the first one. (Manuel López-Ibáñez)
Forbidden configurations may be specified in terms of constraints on their values. Forbidden configurations will never be evaluated by irace. See --forbidden-file and inst/templates/forbidden.tmpl. (Manuel López-Ibáñez)
New option --recovery-file (recoveryFile) allows resuming a previous irace run. (Leslie Pérez Cáceres)
The confidence level for the elimination test is now configurable with parameter --confidence. (Leslie Pérez Cáceres)
Much more robust handling of relative/absolute paths. Improved support for Windows. (Leslie Pérez Cáceres, Manuel López-Ibáñez)
Provide better error messages for incorrect parameter descriptions. (Manuel López-Ibáñez) Examples:

    x "" i (0, 0)       # lower and upper bounds are the same
    x "" r (1e-4, 5e-4) # given digits=2, ditto
    x "" i (-1, -2)     # lower bound must be smaller than upper bound
    x "" c ("a", "a")   # duplicated values

Print elapsed time for calls to hook-run if debugLevel >=1. (Manuel López-Ibáñez)
examples/hook-run-python/hook-run: A multi-purpose hook-run written in Python. (Franco Mascia)
Parallel mode in an SGE cluster (--sge-cluster) is more robust. (Manuel López-Ibáñez)

irace 1.04

CRAN release: 2013-06-09

Replace obsolete package multicore by package parallel (requires R >= 2.14.0)
Use load-balancing (mc.preschedule = FALSE) in mclapply.

irace 1.03

CRAN release: 2013-01-15

Use reg.finalizer to finish Rmpi properly without clobbering .Last().
Remove uses of deprecated as.real().
Nicer error handling in readParameters().
Add hypervolume (multi-objective) example.
Fix several bugs in the computation of similar candidates.

irace 1.02

CRAN release: 2012-10-12

More concise output.
The parameters expName and expDescription are now useless and they were removed.
Faster computation of similar candidates (Jeremie Dubois-Lacoste and Leslie Pérez Cáceres).
Fix bug when saving instances in tunerResults$experiments.
irace.cmdline ("--help") does not try to quit R anymore.

irace 1.01

CRAN release: 2012-03-19

Fix bug caused by file.exists (and possibly other functions) not handling directory names with a trailing backslash or slash on Windows.
Fix bug using per-instance parameters (Leslie Pérez Cáceres).
Fix bug when reading initial candidates from a file.