Skip to contents

Ablation is a method for analyzing the differences between two configurations.

Usage

ablation(
  iraceResults,
  src = 1L,
  target = NULL,
  ab.params = NULL,
  type = c("full", "racing"),
  nrep = 1L,
  seed = 1234567,
  ablationLogFile = "log-ablation.Rdata",
  instancesFile = "train",
  ...
)

Arguments

iraceResults

(list()|character(1)) iraceResults object created by irace and typically saved in the log file irace.Rdata. If a character string is given, then it is interpreted as the path to the log file from which the iraceResults object will be loaded.

src, target

(integer(1)) Source and target configuration IDs. By default, the first configuration ever evaluated (ID 1) is used as src and the best configuration found by irace is used as target.

ab.params

Specific parameter names to be used for the ablation. They must be in parameters$names. By default, use all parameters.

type

Type of ablation to perform: "full" will execute each configuration on all n_instances to determine the best-performing one; "racing" will apply racing to find the best configurations.

nrep

(integer(1)) Number of replications per instance used in "full" ablation. When nrep > 1, each configuration will be executed nrep times on each instance with different random seeds.

seed

(integer(1)) Integer value to use as seed for the random number generation.

ablationLogFile

(character(1)) Log file to save the ablation log. If NULL, the results are not saved to a file.

instancesFile

(character(1)) Instances file used for ablation: 'train', 'test' or a filename containing the list of instances.

...

Further arguments to override scenario settings, e.g., debugLevel, parallel, etc.

Value

A list containing the following elements:

configurations

Configurations tested in the ablation.

instances

A matrix with the instances used in the experiments. First column has the instances IDs from iraceResults$scenario$instances, second column the seed assigned to the instance.

experiments

A matrix with the results of the experiments (columns are configurations, rows are instances).

scenario

Scenario object with the settings used for the experiments.

trajectory

IDs of the best configurations at each step of the ablation.

best

Best configuration found in the experiments.

References

C. Fawcett and H. H. Hoos. Analysing differences between algorithm configurations through ablation. Journal of Heuristics, 22(4):431–458, 2016.

Author

Leslie Pérez Cáceres and Manuel López-Ibáñez

Examples

# \donttest{
logfile <- system.file(package="irace", "exdata", "sann.rda")
# Execute ablation between the first and the best configuration found by irace.
ablog <- ablation(logfile, ablationLogFile = NULL)
#> # Using 'train' instances:
#> 0.894501768540685
#> 0.921318169444232
#> 0.9127270466508
#> 0.925070250885822
#> 0.900772186574571
#> 0.885437572813138
#> 0.914622617613828
#> 0.900573881957391
#> 0.9228579443943
#> 0.895961758961795
#> 0.891047757730183
#> 0.876912345548979
#> 0.892886015887865
#> 0.884941229399852
#> 0.917668138646215
#> 0.910891987746372
#> 0.881553298710373
#> 0.901092171585161
#> 0.902031803627614
#> 0.914085655706059
#> 0.896423595951403
#> 0.942569317483396
#> 0.892524837658347
#> 0.872552502804592
#> 0.915453495888186
#> 0.896802546983848
#> 0.911830208760366
#> 0.91047463540408
#> 0.916623563667357
#> 0.889239778737424
#> 0.895994337525062
#> 0.91090782640865
#> 0.918750204407407
#> 0.872484855172359
#> 0.888630377770892
#> 0.916126434762543
#> 0.884102798618619
#> 0.896637686385077
#> 0.910947548603319
#> 0.905946020433283
#> 0.916241571247595
#> 0.901204726044066
#> 0.851441094230539
#> 0.913533327256422
#> 0.906476050074882
#> 0.889735901018944
#> 0.899574041486192
#> 0.879203191352201
#> 0.903736401459846
#> 0.908164659055373
#> 0.871044038125979
#> 0.908964407424915
#> 0.899268157781966
#> 0.884274191222221
#> 0.877919537771623
#> 0.905312266434585
#> 0.910806141774629
#> 0.889188178918031
#> 0.931552309589065
#> 0.911839172687047
#> 0.914829022961218
#> 0.869643119109939
#> 0.908672231247637
#> 0.893659790862889
#> 0.909710508057249
#> 0.902008315260975
#> 0.906100388329975
#> 0.902885437923853
#> 0.904158012025015
#> 0.912963908565568
#> 0.888556177926922
#> 0.869826356281433
#> 0.900227756754038
#> 0.870759613727054
#> 0.912927679092212
#> 0.892307806055868
#> 0.954991078324949
#> 0.91208289633115
#> 0.948684366545179
#> 0.932256699244544
#> 0.930208505175288
#> 0.851852668316431
#> 0.907473145128439
#> 0.900122344899903
#> 0.884782261416131
#> 0.884120367390293
#> 0.880192618333673
#> 0.943225985188736
#> 0.922909785871304
#> 0.880364522127397
#> 0.892029330457589
#> 0.840359683493198
#> 0.926762421256025
#> 0.877630876042559
#> 0.858535147438306
#> 0.937992215954751
#> 0.907232515219819
#> 0.891451173618053
#> 0.877057825268654
#> 0.893056773371777
#> # 2023-01-21 23:37:06 UTC: Starting ablation from 1 to 131
#> # Seed: 1234567
#> # Source configuration (row number is ID):
#>   tmax    temp
#> 1   95 93.1484
#> # Target configuration (row number is ID):
#>     tmax    temp
#> 131 1839 35.5549
#> # 2023-01-21 23:37:06 UTC: Executing source and target configurations on the given instances * nrep (100)...
#> # Generating configurations (row number is ID): tmax temp 
#>   tmax    temp
#> 2 1839 93.1484
#> 3   95 35.5549
#> # 2023-01-21 23:37:12 UTC: Ablation (full) of 2 configurations on 100 instances.
#> # Markers:
#>      x No test is performed.
#>      c Configurations are discarded only due to capping.
#>      - The test is performed and some configurations are discarded.
#>      = The test is performed but no configuration is discarded.
#>      ! The test is performed and configurations could be discarded but elite configurations are preserved.
#>      . All alive configurations are elite and nothing is discarded
#> 
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> | |   Instance|      Alive|       Best|       Mean best| Exp so far|  W time|  rho|KenW|  Qvar|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> |x|          1|          2|          3|   0.04333580472|          2|00:00:00|   NA|  NA|    NA|
#> |x|          2|          2|          3|    0.1021911824|          4|00:00:00|+1.00|1.00|0.0000|
#> |x|          3|          2|          3|   0.08662852132|          6|00:00:00|+1.00|1.00|0.0000|
#> |x|          4|          2|          3|    0.1321507782|          8|00:00:00|+1.00|1.00|0.0000|
#> |x|          5|          2|          3|    0.1586145647|         10|00:00:00|+1.00|1.00|0.0000|
#> |x|          6|          2|          3|    0.1515579565|         12|00:00:00|+1.00|1.00|0.0000|
#> |x|          7|          2|          3|    0.1460237607|         14|00:00:00|+1.00|1.00|0.0000|
#> |x|          8|          2|          3|     2.028117223|         16|00:00:00|+0.50|0.56|0.2500|
#> |x|          9|          2|          3|     1.813206546|         18|00:00:00|+0.56|0.60|0.2222|
#> |x|         10|          2|          3|     1.637106795|         20|00:00:00|+0.60|0.64|0.2000|
#> |x|         11|          2|          3|     1.489877626|         22|00:00:00|+0.64|0.67|0.1818|
#> |x|         12|          2|          3|     1.607226214|         24|00:00:00|+0.39|0.44|0.3030|
#> |x|         13|          2|          3|     1.719058795|         26|00:00:00|+0.23|0.29|0.3846|
#> |x|         14|          2|          3|     1.602902468|         28|00:00:00|+0.27|0.33|0.3626|
#> |x|         15|          2|          3|     1.506645410|         30|00:00:00|+0.31|0.36|0.3429|
#> |x|         16|          2|          3|     1.422276696|         32|00:00:00|+0.20|0.25|0.4000|
#> |x|         17|          2|          3|     1.340430635|         34|00:00:00|+0.24|0.28|0.3824|
#> |x|         18|          2|          3|     1.271702194|         36|00:00:00|+0.15|0.20|0.4248|
#> |x|         19|          2|          3|     1.205092205|         38|00:00:00|+0.18|0.22|0.4094|
#> |x|         20|          2|          3|     1.149951901|         40|00:00:00|+0.21|0.25|0.3947|
#> |x|         21|          2|          3|     1.105985406|         42|00:00:00|+0.14|0.18|0.4286|
#> |x|         22|          2|          3|     1.059378233|         44|00:00:00|+0.17|0.21|0.4156|
#> |x|         23|          2|          3|     1.018417448|         46|00:00:00|+0.19|0.23|0.4032|
#> |x|         24|          2|          3|    0.9792428202|         48|00:00:00|+0.22|0.25|0.3913|
#> |x|         25|          2|          3|     1.381378481|         50|00:00:00|+0.16|0.19|0.4200|
#> |x|         26|          2|          3|     1.334540841|         52|00:00:00|+0.18|0.21|0.4092|
#> |x|         27|          2|          3|     1.290420952|         54|00:00:00|+0.20|0.23|0.3989|
#> |x|         28|          2|          3|     1.358871524|         56|00:00:00|+0.15|0.18|0.4233|
#> |x|         29|          2|          3|     1.316117711|         58|00:00:00|+0.17|0.20|0.4138|
#> |x|         30|          2|          3|     1.282241735|         60|00:00:00|+0.19|0.22|0.4046|
#> |x|         31|          2|          3|     1.335972765|         62|00:00:00|+0.21|0.23|0.3957|
#> |x|         32|          2|          3|     1.311449793|         64|00:00:00|+0.17|0.19|0.4173|
#> |x|         33|          2|          3|     1.392561366|         66|00:00:00|+0.13|0.16|0.4356|
#> |x|         34|          2|          3|     1.353914054|         68|00:00:00|+0.14|0.17|0.4278|
#> |x|         35|          2|          3|     1.316268540|         70|00:00:00|+0.16|0.18|0.4202|
#> |x|         36|          2|          3|     1.281549622|         72|00:00:00|+0.17|0.20|0.4127|
#> |x|         37|          2|          3|     1.332297343|         74|00:00:00|+0.14|0.16|0.4294|
#> |x|         38|          2|          3|     1.297820991|         76|00:00:00|+0.16|0.18|0.4225|
#> |x|         39|          2|          3|     1.267363142|         78|00:00:00|+0.17|0.19|0.4157|
#> |x|         40|          2|          3|     1.236738279|         80|00:00:00|+0.18|0.20|0.4090|
#> |x|         41|          2|          3|     1.207791606|         82|00:00:00|+0.20|0.21|0.4024|
#> |x|         42|          2|          3|     1.179182022|         84|00:00:00|+0.21|0.23|0.3961|
#> |x|         43|          2|          3|     1.151976396|         86|00:00:00|+0.22|0.24|0.3898|
#> |x|         44|          2|          3|     1.131526135|         88|00:00:00|+0.23|0.25|0.3837|
#> |x|         45|          2|          3|     1.117333811|         90|00:00:00|+0.24|0.26|0.3778|
#> |x|         46|          2|          3|     1.100296333|         92|00:00:00|+0.26|0.27|0.3720|
#> |x|         47|          2|          3|     1.141735069|         94|00:00:00|+0.27|0.28|0.3663|
#> |x|         48|          2|          3|     1.179765732|         96|00:00:00|+0.23|0.25|0.3830|
#> |x|         49|          2|          3|     1.160016379|         98|00:00:00|+0.24|0.26|0.3776|
#> |x|         50|          2|          3|     1.195139788|        100|00:00:00|+0.21|0.23|0.3927|
#> |x|         51|          2|          3|     1.171962330|        102|00:00:00|+0.23|0.24|0.3875|
#> |x|         52|          2|          3|     1.151006176|        104|00:00:00|+0.24|0.25|0.3824|
#> |x|         53|          2|          3|     1.130924126|        106|00:00:00|+0.25|0.26|0.3774|
#> |x|         54|          2|          3|     1.112310323|        108|00:00:00|+0.26|0.27|0.3725|
#> |x|         55|          2|          3|     1.094157199|        110|00:00:00|+0.26|0.28|0.3677|
#> |x|         56|          2|          3|     1.093339186|        112|00:00:00|+0.24|0.25|0.3818|
#> |x|         57|          2|          3|     1.076446177|        114|00:00:00|+0.25|0.26|0.3772|
#> |x|         58|          2|          3|     1.129874610|        116|00:00:00|+0.22|0.23|0.3902|
#> |x|         59|          2|          3|     1.112464268|        118|00:00:00|+0.23|0.24|0.3857|
#> |x|         60|          2|          3|     1.095213235|        120|00:00:00|+0.24|0.25|0.3814|
#> |x|         61|          2|          3|     1.128412592|        122|00:00:00|+0.21|0.23|0.3934|
#> |x|         62|          2|          3|     1.112073440|        124|00:00:00|+0.22|0.23|0.3892|
#> |x|         63|          2|          3|     1.099587043|        126|00:00:00|+0.20|0.21|0.4004|
#> |x|         64|          2|          3|     1.082668425|        128|00:00:00|+0.21|0.22|0.3963|
#> |x|         65|          2|          3|     1.110392658|        130|00:00:00|+0.19|0.20|0.4067|
#> |x|         66|          2|          3|     1.094658908|        132|00:00:00|+0.19|0.21|0.4028|
#> |x|         67|          2|          3|     1.080934558|        134|00:00:00|+0.20|0.21|0.3989|
#> |x|         68|          2|          3|     1.102144388|        136|00:00:00|+0.18|0.19|0.4087|
#> |x|         69|          2|          3|     1.087018761|        138|00:00:00|+0.19|0.20|0.4049|
#> |x|         70|          2|          3|     1.072328075|        140|00:00:00|+0.20|0.21|0.4012|
#> |x|         71|          2|          3|     1.101456529|        142|00:00:00|+0.18|0.19|0.4105|
#> |x|         72|          2|          3|     1.311554010|        144|00:00:00|+0.16|0.17|0.4190|
#> |x|         73|          2|          3|     1.294146365|        146|00:00:00|+0.17|0.18|0.4155|
#> |x|         74|          2|          3|     1.278135084|        148|00:00:00|+0.18|0.19|0.4121|
#> |x|         75|          2|          3|     1.315038849|        150|00:00:00|+0.18|0.19|0.4086|
#> |x|         76|          2|          3|     1.299486254|        152|00:00:00|+0.19|0.20|0.4053|
#> |x|         77|          2|          3|     1.293694049|        154|00:00:00|+0.20|0.21|0.4019|
#> |x|         78|          2|          3|     1.277131635|        156|00:00:00|+0.20|0.21|0.3986|
#> |x|         79|          2|          3|     1.299557525|        158|00:00:00|+0.19|0.20|0.4070|
#> |x|         80|          2|          3|     1.320645512|        160|00:00:00|+0.17|0.18|0.4149|
#> |x|         81|          2|          3|     1.306995772|        162|00:00:00|+0.16|0.17|0.4222|
#> |x|         82|          2|          3|     1.291682955|        164|00:00:00|+0.16|0.17|0.4192|
#> |x|         83|          2|          3|     1.311980899|        166|00:00:00|+0.15|0.16|0.4261|
#> |x|         84|          2|          3|     1.332779034|        168|00:00:00|+0.13|0.15|0.4326|
#> |x|         85|          2|          3|     1.329258574|        170|00:00:00|+0.14|0.15|0.4297|
#> |x|         86|          2|          3|     1.315279309|        172|00:00:00|+0.15|0.16|0.4268|
#> |x|         87|          2|          3|     1.302027620|        174|00:00:00|+0.15|0.16|0.4240|
#> |x|         88|          2|          3|     1.287823863|        176|00:00:00|+0.16|0.17|0.4211|
#> |x|         89|          2|          3|     1.276012392|        178|00:00:00|+0.16|0.17|0.4183|
#> |x|         90|          2|          3|     1.261927653|        180|00:00:00|+0.17|0.18|0.4155|
#> |x|         91|          2|          3|     1.248189192|        182|00:00:00|+0.17|0.18|0.4127|
#> |x|         92|          2|          3|     1.236557061|        184|00:00:00|+0.18|0.19|0.4099|
#> |x|         93|          2|          3|     1.254637605|        186|00:00:00|+0.17|0.18|0.4165|
#> |x|         94|          2|          3|     1.241301700|        188|00:00:00|+0.17|0.18|0.4139|
#> |x|         95|          2|          3|     1.230232450|        190|00:00:00|+0.18|0.19|0.4112|
#> |x|         96|          2|          3|     1.252351995|        192|00:00:00|+0.16|0.17|0.4175|
#> |x|         97|          2|          3|     1.341165168|        194|00:00:00|+0.15|0.16|0.4235|
#> |x|         98|          2|          3|     1.328377019|        196|00:00:00|+0.16|0.17|0.4210|
#> |x|         99|          2|          3|     1.345603337|        198|00:00:00|+0.15|0.16|0.4267|
#> |-|        100|          1|          3|     1.332873203|        200|00:00:00|   NA|  NA|    NA|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> Best-so-far configuration:           3    mean value:      1.332873203
#> Description of the best-so-far configuration:
#>   .ID. tmax    temp .PARENT.
#> 3    3   95 35.5549        1
#> 
#> # Best changed parameters:
#> # temp : 93.1484 -> 35.5549 
#> # 2023-01-21 23:37:18 UTC: Final best configuration:
#>   tmax    temp
#> 3   95 35.5549
plotAblation(ablog)

# Execute ablation between two selected configurations, and selecting only a
# subset of parameters, directly reading the setup from the irace log file.
ablog <- ablation(logfile, src = 1, target = 10,
                  ab.params = c("temp"), ablationLogFile = NULL)
#> # Using 'train' instances:
#> 0.894501768540685
#> 0.921318169444232
#> 0.9127270466508
#> 0.925070250885822
#> 0.900772186574571
#> 0.885437572813138
#> 0.914622617613828
#> 0.900573881957391
#> 0.9228579443943
#> 0.895961758961795
#> 0.891047757730183
#> 0.876912345548979
#> 0.892886015887865
#> 0.884941229399852
#> 0.917668138646215
#> 0.910891987746372
#> 0.881553298710373
#> 0.901092171585161
#> 0.902031803627614
#> 0.914085655706059
#> 0.896423595951403
#> 0.942569317483396
#> 0.892524837658347
#> 0.872552502804592
#> 0.915453495888186
#> 0.896802546983848
#> 0.911830208760366
#> 0.91047463540408
#> 0.916623563667357
#> 0.889239778737424
#> 0.895994337525062
#> 0.91090782640865
#> 0.918750204407407
#> 0.872484855172359
#> 0.888630377770892
#> 0.916126434762543
#> 0.884102798618619
#> 0.896637686385077
#> 0.910947548603319
#> 0.905946020433283
#> 0.916241571247595
#> 0.901204726044066
#> 0.851441094230539
#> 0.913533327256422
#> 0.906476050074882
#> 0.889735901018944
#> 0.899574041486192
#> 0.879203191352201
#> 0.903736401459846
#> 0.908164659055373
#> 0.871044038125979
#> 0.908964407424915
#> 0.899268157781966
#> 0.884274191222221
#> 0.877919537771623
#> 0.905312266434585
#> 0.910806141774629
#> 0.889188178918031
#> 0.931552309589065
#> 0.911839172687047
#> 0.914829022961218
#> 0.869643119109939
#> 0.908672231247637
#> 0.893659790862889
#> 0.909710508057249
#> 0.902008315260975
#> 0.906100388329975
#> 0.902885437923853
#> 0.904158012025015
#> 0.912963908565568
#> 0.888556177926922
#> 0.869826356281433
#> 0.900227756754038
#> 0.870759613727054
#> 0.912927679092212
#> 0.892307806055868
#> 0.954991078324949
#> 0.91208289633115
#> 0.948684366545179
#> 0.932256699244544
#> 0.930208505175288
#> 0.851852668316431
#> 0.907473145128439
#> 0.900122344899903
#> 0.884782261416131
#> 0.884120367390293
#> 0.880192618333673
#> 0.943225985188736
#> 0.922909785871304
#> 0.880364522127397
#> 0.892029330457589
#> 0.840359683493198
#> 0.926762421256025
#> 0.877630876042559
#> 0.858535147438306
#> 0.937992215954751
#> 0.907232515219819
#> 0.891451173618053
#> 0.877057825268654
#> 0.893056773371777
#> # 2023-01-21 23:37:19 UTC: Starting ablation from 1 to 10
#> # Seed: 1234567
#> # Source configuration (row number is ID):
#>   tmax    temp
#> 1   95 93.1484
#> # Target configuration (row number is ID):
#>    tmax    temp
#> 10 1737 40.7362
#> # 2023-01-21 23:37:19 UTC: Executing source and target configurations on the given instances * nrep (100)...
#> # 2023-01-21 23:37:24 UTC: Final best configuration:
#>   tmax    temp
#> 2 1737 40.7362
plotAblation(ablog)

# }