Performs ablation between two configurations (from source to target).
Source:R/ablation.R
ablation.RdAblation is a method for analyzing the differences between two configurations.
Usage
ablation(
iraceResults,
src = 1L,
target = NULL,
ab_params = NULL,
type = c("full", "racing"),
nrep = 1L,
seed = 1234567L,
ablationLogFile = "log-ablation.Rdata",
instancesFile = "train",
...
)Arguments
- iraceResults
(
list()|character(1))
Object created by irace and typically saved in the log fileirace.Rdata. If a character string is given, then it is interpreted as the path to the log file from which theiraceResultsobject will be loaded.- src, target
(
integer(1)) Source and target configuration IDs. By default, the first configuration ever evaluated (ID 1) is used assrcand the best configuration found by irace is used as target.- ab_params
Specific parameter names to be used for the ablation. They must be in
parameters$names. By default, use all parameters.- type
Type of ablation to perform:
"full"will execute each configuration on alln_instancesto determine the best-performing one;"racing"will apply racing to find the best configurations.- nrep
(
integer(1)) Number of replications per instance used in"full"ablation. Whennrep > 1, each configuration will be executednreptimes on each instance with different random seeds.- seed
(
integer(1)) Integer value to use as seed for the random number generation.- ablationLogFile
(
character(1)) Log file to save the ablation log. IfNULL, the results are not saved to a file.- instancesFile
(
character(1)) Instances file used for ablation:'train','test'or a filename containing the list of instances.- ...
Further arguments to override scenario settings, e.g.,
debugLevel,parallel, etc.
Value
A list containing the following elements:
- configurations
Configurations tested in the ablation.
- instances
A matrix with the instances used in the experiments. First column has the instances IDs from
iraceResults$scenario$instances, second column the seed assigned to the instance.- experiments
A matrix with the results of the experiments (columns are configurations, rows are instances).
- scenario
Scenario object with the settings used for the experiments.
- trajectory
IDs of the best configurations at each step of the ablation.
- best
Best configuration found in the experiments.
References
C. Fawcett and H. H. Hoos. Analysing differences between algorithm configurations through ablation. Journal of Heuristics, 22(4):431–458, 2016.
Examples
# \donttest{
logfile <- system.file(package="irace", "exdata", "sann.rda")
# Execute ablation between the first and the best configuration found by irace.
ablog <- ablation(logfile, ablationLogFile = NULL)
#> # Using 'train' instances:
#> 0.894501768540685
#> 0.921318169444232
#> 0.9127270466508
#> 0.925070250885822
#> 0.900772186574571
#> 0.885437572813138
#> 0.914622617613828
#> 0.900573881957391
#> 0.9228579443943
#> 0.895961758961795
#> 0.891047757730183
#> 0.876912345548979
#> 0.892886015887865
#> 0.884941229399852
#> 0.917668138646215
#> 0.910891987746372
#> 0.881553298710373
#> 0.901092171585161
#> 0.902031803627614
#> 0.914085655706059
#> 0.896423595951403
#> 0.942569317483396
#> 0.892524837658347
#> 0.872552502804592
#> 0.915453495888186
#> 0.896802546983848
#> 0.911830208760366
#> 0.91047463540408
#> 0.916623563667357
#> 0.889239778737424
#> 0.895994337525062
#> 0.91090782640865
#> 0.918750204407407
#> 0.872484855172359
#> 0.888630377770892
#> 0.916126434762543
#> 0.884102798618619
#> 0.896637686385077
#> 0.910947548603319
#> 0.905946020433283
#> 0.916241571247595
#> 0.901204726044066
#> 0.851441094230539
#> 0.913533327256422
#> 0.906476050074882
#> 0.889735901018944
#> 0.899574041486192
#> 0.879203191352201
#> 0.903736401459846
#> 0.908164659055373
#> 0.871044038125979
#> 0.908964407424915
#> 0.899268157781966
#> 0.884274191222221
#> 0.877919537771623
#> 0.905312266434585
#> 0.910806141774629
#> 0.889188178918031
#> 0.931552309589065
#> 0.911839172687047
#> 0.914829022961218
#> 0.869643119109939
#> 0.908672231247637
#> 0.893659790862889
#> 0.909710508057249
#> 0.902008315260975
#> 0.906100388329975
#> 0.902885437923853
#> 0.904158012025015
#> 0.912963908565568
#> 0.888556177926922
#> 0.869826356281433
#> 0.900227756754038
#> 0.870759613727054
#> 0.912927679092212
#> 0.892307806055868
#> 0.954991078324949
#> 0.91208289633115
#> 0.948684366545179
#> 0.932256699244544
#> 0.930208505175288
#> 0.851852668316431
#> 0.907473145128439
#> 0.900122344899903
#> 0.884782261416131
#> 0.884120367390293
#> 0.880192618333673
#> 0.943225985188736
#> 0.922909785871304
#> 0.880364522127397
#> 0.892029330457589
#> 0.840359683493198
#> 0.926762421256025
#> 0.877630876042559
#> 0.858535147438306
#> 0.937992215954751
#> 0.907232515219819
#> 0.891451173618053
#> 0.877057825268654
#> 0.893056773371777
#> # 2023-09-26 11:02:50 UTC: Starting ablation from 1 to 131
#> # Seed: 1234567
#> # Source configuration (row number is ID):
#> tmax temp
#> 1 95 93.1484
#> # Target configuration (row number is ID):
#> tmax temp
#> 131 1839 35.5549
#> # 2023-09-26 11:02:50 UTC: Executing source and target configurations on the given instances * nrep (100)...
#> # Generating configurations (row number is ID): tmax temp
#> tmax temp
#> 2 1839 93.1484
#> 3 95 35.5549
#> # 2023-09-26 11:02:56 UTC: Ablation (full) of 2 configurations on 100 instances.
#> # Markers:
#> x No test is performed.
#> c Configurations are discarded only due to capping.
#> - The test is performed and some configurations are discarded.
#> = The test is performed but no configuration is discarded.
#> ! The test is performed and configurations could be discarded but elite configurations are preserved.
#> . All alive configurations are elite and nothing is discarded.
#>
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> | | Instance| Alive| Best| Mean best| Exp so far| W time| rho|KenW| Qvar|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> |x| 1| 2| 3| 0.04333580472| 2|00:00:00| NA| NA| NA|
#> |x| 2| 2| 3| 0.1021911824| 4|00:00:00|+1.00|1.00|0.0000|
#> |x| 3| 2| 3| 0.08662852132| 6|00:00:00|+1.00|1.00|0.0000|
#> |x| 4| 2| 3| 0.1321507782| 8|00:00:00|+1.00|1.00|0.0000|
#> |x| 5| 2| 3| 0.1586145647| 10|00:00:00|+1.00|1.00|0.0000|
#> |x| 6| 2| 3| 0.1515579565| 12|00:00:00|+1.00|1.00|0.0000|
#> |x| 7| 2| 3| 0.1460237607| 14|00:00:00|+1.00|1.00|0.0000|
#> |x| 8| 2| 3| 2.028117223| 16|00:00:00|+0.50|0.56|0.2500|
#> |x| 9| 2| 3| 1.813206546| 18|00:00:00|+0.56|0.60|0.2222|
#> |x| 10| 2| 3| 1.637106795| 20|00:00:00|+0.60|0.64|0.2000|
#> |x| 11| 2| 3| 1.489877626| 22|00:00:00|+0.64|0.67|0.1818|
#> |x| 12| 2| 3| 1.607226214| 24|00:00:00|+0.39|0.44|0.3030|
#> |x| 13| 2| 3| 1.719058795| 26|00:00:00|+0.23|0.29|0.3846|
#> |x| 14| 2| 3| 1.602902468| 28|00:00:00|+0.27|0.33|0.3626|
#> |x| 15| 2| 3| 1.506645410| 30|00:00:00|+0.31|0.36|0.3429|
#> |x| 16| 2| 3| 1.422276696| 32|00:00:00|+0.20|0.25|0.4000|
#> |x| 17| 2| 3| 1.340430635| 34|00:00:00|+0.24|0.28|0.3824|
#> |x| 18| 2| 3| 1.271702194| 36|00:00:00|+0.15|0.20|0.4248|
#> |x| 19| 2| 3| 1.205092205| 38|00:00:00|+0.18|0.22|0.4094|
#> |x| 20| 2| 3| 1.149951901| 40|00:00:00|+0.21|0.25|0.3947|
#> |x| 21| 2| 3| 1.105985406| 42|00:00:00|+0.14|0.18|0.4286|
#> |x| 22| 2| 3| 1.059378233| 44|00:00:00|+0.17|0.21|0.4156|
#> |x| 23| 2| 3| 1.018417448| 46|00:00:00|+0.19|0.23|0.4032|
#> |x| 24| 2| 3| 0.9792428202| 48|00:00:00|+0.22|0.25|0.3913|
#> |x| 25| 2| 3| 1.381378481| 50|00:00:00|+0.16|0.19|0.4200|
#> |x| 26| 2| 3| 1.334540841| 52|00:00:00|+0.18|0.21|0.4092|
#> |x| 27| 2| 3| 1.290420952| 54|00:00:00|+0.20|0.23|0.3989|
#> |x| 28| 2| 3| 1.358871524| 56|00:00:00|+0.15|0.18|0.4233|
#> |x| 29| 2| 3| 1.316117711| 58|00:00:00|+0.17|0.20|0.4138|
#> |x| 30| 2| 3| 1.282241735| 60|00:00:00|+0.19|0.22|0.4046|
#> |x| 31| 2| 3| 1.335972765| 62|00:00:00|+0.21|0.23|0.3957|
#> |x| 32| 2| 3| 1.311449793| 64|00:00:00|+0.17|0.19|0.4173|
#> |x| 33| 2| 3| 1.392561366| 66|00:00:00|+0.13|0.16|0.4356|
#> |x| 34| 2| 3| 1.353914054| 68|00:00:00|+0.14|0.17|0.4278|
#> |x| 35| 2| 3| 1.316268540| 70|00:00:00|+0.16|0.18|0.4202|
#> |x| 36| 2| 3| 1.281549622| 72|00:00:00|+0.17|0.20|0.4127|
#> |x| 37| 2| 3| 1.332297343| 74|00:00:00|+0.14|0.16|0.4294|
#> |x| 38| 2| 3| 1.297820991| 76|00:00:00|+0.16|0.18|0.4225|
#> |x| 39| 2| 3| 1.267363142| 78|00:00:00|+0.17|0.19|0.4157|
#> |x| 40| 2| 3| 1.236738279| 80|00:00:00|+0.18|0.20|0.4090|
#> |x| 41| 2| 3| 1.207791606| 82|00:00:00|+0.20|0.21|0.4024|
#> |x| 42| 2| 3| 1.179182022| 84|00:00:00|+0.21|0.23|0.3961|
#> |x| 43| 2| 3| 1.151976396| 86|00:00:00|+0.22|0.24|0.3898|
#> |x| 44| 2| 3| 1.131526135| 88|00:00:00|+0.23|0.25|0.3837|
#> |x| 45| 2| 3| 1.117333811| 90|00:00:00|+0.24|0.26|0.3778|
#> |x| 46| 2| 3| 1.100296333| 92|00:00:00|+0.26|0.27|0.3720|
#> |x| 47| 2| 3| 1.141735069| 94|00:00:00|+0.27|0.28|0.3663|
#> |x| 48| 2| 3| 1.179765732| 96|00:00:00|+0.23|0.25|0.3830|
#> |x| 49| 2| 3| 1.160016379| 98|00:00:00|+0.24|0.26|0.3776|
#> |x| 50| 2| 3| 1.195139788| 100|00:00:00|+0.21|0.23|0.3927|
#> |x| 51| 2| 3| 1.171962330| 102|00:00:00|+0.23|0.24|0.3875|
#> |x| 52| 2| 3| 1.151006176| 104|00:00:00|+0.24|0.25|0.3824|
#> |x| 53| 2| 3| 1.130924126| 106|00:00:00|+0.25|0.26|0.3774|
#> |x| 54| 2| 3| 1.112310323| 108|00:00:00|+0.26|0.27|0.3725|
#> |x| 55| 2| 3| 1.094157199| 110|00:00:00|+0.26|0.28|0.3677|
#> |x| 56| 2| 3| 1.093339186| 112|00:00:00|+0.24|0.25|0.3818|
#> |x| 57| 2| 3| 1.076446177| 114|00:00:00|+0.25|0.26|0.3772|
#> |x| 58| 2| 3| 1.129874610| 116|00:00:00|+0.22|0.23|0.3902|
#> |x| 59| 2| 3| 1.112464268| 118|00:00:00|+0.23|0.24|0.3857|
#> |x| 60| 2| 3| 1.095213235| 120|00:00:00|+0.24|0.25|0.3814|
#> |x| 61| 2| 3| 1.128412592| 122|00:00:00|+0.21|0.23|0.3934|
#> |x| 62| 2| 3| 1.112073440| 124|00:00:00|+0.22|0.23|0.3892|
#> |x| 63| 2| 3| 1.099587043| 126|00:00:00|+0.20|0.21|0.4004|
#> |x| 64| 2| 3| 1.082668425| 128|00:00:00|+0.21|0.22|0.3963|
#> |x| 65| 2| 3| 1.110392658| 130|00:00:00|+0.19|0.20|0.4067|
#> |x| 66| 2| 3| 1.094658908| 132|00:00:00|+0.19|0.21|0.4028|
#> |x| 67| 2| 3| 1.080934558| 134|00:00:00|+0.20|0.21|0.3989|
#> |x| 68| 2| 3| 1.102144388| 136|00:00:00|+0.18|0.19|0.4087|
#> |x| 69| 2| 3| 1.087018761| 138|00:00:00|+0.19|0.20|0.4049|
#> |x| 70| 2| 3| 1.072328075| 140|00:00:00|+0.20|0.21|0.4012|
#> |x| 71| 2| 3| 1.101456529| 142|00:00:00|+0.18|0.19|0.4105|
#> |x| 72| 2| 3| 1.311554010| 144|00:00:00|+0.16|0.17|0.4190|
#> |x| 73| 2| 3| 1.294146365| 146|00:00:00|+0.17|0.18|0.4155|
#> |x| 74| 2| 3| 1.278135084| 148|00:00:00|+0.18|0.19|0.4121|
#> |x| 75| 2| 3| 1.315038849| 150|00:00:00|+0.18|0.19|0.4086|
#> |x| 76| 2| 3| 1.299486254| 152|00:00:00|+0.19|0.20|0.4053|
#> |x| 77| 2| 3| 1.293694049| 154|00:00:00|+0.20|0.21|0.4019|
#> |x| 78| 2| 3| 1.277131635| 156|00:00:00|+0.20|0.21|0.3986|
#> |x| 79| 2| 3| 1.299557525| 158|00:00:00|+0.19|0.20|0.4070|
#> |x| 80| 2| 3| 1.320645512| 160|00:00:00|+0.17|0.18|0.4149|
#> |x| 81| 2| 3| 1.306995772| 162|00:00:00|+0.16|0.17|0.4222|
#> |x| 82| 2| 3| 1.291682955| 164|00:00:00|+0.16|0.17|0.4192|
#> |x| 83| 2| 3| 1.311980899| 166|00:00:00|+0.15|0.16|0.4261|
#> |x| 84| 2| 3| 1.332779034| 168|00:00:00|+0.13|0.15|0.4326|
#> |x| 85| 2| 3| 1.329258574| 170|00:00:00|+0.14|0.15|0.4297|
#> |x| 86| 2| 3| 1.315279309| 172|00:00:00|+0.15|0.16|0.4268|
#> |x| 87| 2| 3| 1.302027620| 174|00:00:00|+0.15|0.16|0.4240|
#> |x| 88| 2| 3| 1.287823863| 176|00:00:00|+0.16|0.17|0.4211|
#> |x| 89| 2| 3| 1.276012392| 178|00:00:00|+0.16|0.17|0.4183|
#> |x| 90| 2| 3| 1.261927653| 180|00:00:00|+0.17|0.18|0.4155|
#> |x| 91| 2| 3| 1.248189192| 182|00:00:00|+0.17|0.18|0.4127|
#> |x| 92| 2| 3| 1.236557061| 184|00:00:00|+0.18|0.19|0.4099|
#> |x| 93| 2| 3| 1.254637605| 186|00:00:00|+0.17|0.18|0.4165|
#> |x| 94| 2| 3| 1.241301700| 188|00:00:00|+0.17|0.18|0.4139|
#> |x| 95| 2| 3| 1.230232450| 190|00:00:00|+0.18|0.19|0.4112|
#> |x| 96| 2| 3| 1.252351995| 192|00:00:00|+0.16|0.17|0.4175|
#> |x| 97| 2| 3| 1.341165168| 194|00:00:00|+0.15|0.16|0.4235|
#> |x| 98| 2| 3| 1.328377019| 196|00:00:00|+0.16|0.17|0.4210|
#> |x| 99| 2| 3| 1.345603337| 198|00:00:00|+0.15|0.16|0.4267|
#> |-| 100| 1| 3| 1.332873203| 200|00:00:00| NA| NA| NA|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> Best-so-far configuration: 3 mean value: 1.332873203
#> Description of the best-so-far configuration:
#> .ID. tmax temp .PARENT.
#> 3 3 95 35.5549 1
#>
#> # Best changed parameters:
#> # temp : 93.1484 -> 35.5549
#> # 2023-09-26 11:03:02 UTC: Final best configuration:
#> tmax temp
#> 3 95 35.5549
plotAblation(ablog)
# Execute ablation between two selected configurations, and selecting only a
# subset of parameters, directly reading the setup from the irace log file.
ablog <- ablation(logfile, src = 1, target = 10,
ab_params = c("temp"), ablationLogFile = NULL)
#> # Using 'train' instances:
#> 0.894501768540685
#> 0.921318169444232
#> 0.9127270466508
#> 0.925070250885822
#> 0.900772186574571
#> 0.885437572813138
#> 0.914622617613828
#> 0.900573881957391
#> 0.9228579443943
#> 0.895961758961795
#> 0.891047757730183
#> 0.876912345548979
#> 0.892886015887865
#> 0.884941229399852
#> 0.917668138646215
#> 0.910891987746372
#> 0.881553298710373
#> 0.901092171585161
#> 0.902031803627614
#> 0.914085655706059
#> 0.896423595951403
#> 0.942569317483396
#> 0.892524837658347
#> 0.872552502804592
#> 0.915453495888186
#> 0.896802546983848
#> 0.911830208760366
#> 0.91047463540408
#> 0.916623563667357
#> 0.889239778737424
#> 0.895994337525062
#> 0.91090782640865
#> 0.918750204407407
#> 0.872484855172359
#> 0.888630377770892
#> 0.916126434762543
#> 0.884102798618619
#> 0.896637686385077
#> 0.910947548603319
#> 0.905946020433283
#> 0.916241571247595
#> 0.901204726044066
#> 0.851441094230539
#> 0.913533327256422
#> 0.906476050074882
#> 0.889735901018944
#> 0.899574041486192
#> 0.879203191352201
#> 0.903736401459846
#> 0.908164659055373
#> 0.871044038125979
#> 0.908964407424915
#> 0.899268157781966
#> 0.884274191222221
#> 0.877919537771623
#> 0.905312266434585
#> 0.910806141774629
#> 0.889188178918031
#> 0.931552309589065
#> 0.911839172687047
#> 0.914829022961218
#> 0.869643119109939
#> 0.908672231247637
#> 0.893659790862889
#> 0.909710508057249
#> 0.902008315260975
#> 0.906100388329975
#> 0.902885437923853
#> 0.904158012025015
#> 0.912963908565568
#> 0.888556177926922
#> 0.869826356281433
#> 0.900227756754038
#> 0.870759613727054
#> 0.912927679092212
#> 0.892307806055868
#> 0.954991078324949
#> 0.91208289633115
#> 0.948684366545179
#> 0.932256699244544
#> 0.930208505175288
#> 0.851852668316431
#> 0.907473145128439
#> 0.900122344899903
#> 0.884782261416131
#> 0.884120367390293
#> 0.880192618333673
#> 0.943225985188736
#> 0.922909785871304
#> 0.880364522127397
#> 0.892029330457589
#> 0.840359683493198
#> 0.926762421256025
#> 0.877630876042559
#> 0.858535147438306
#> 0.937992215954751
#> 0.907232515219819
#> 0.891451173618053
#> 0.877057825268654
#> 0.893056773371777
#> # 2023-09-26 11:03:02 UTC: Starting ablation from 1 to 10
#> # Seed: 1234567
#> # Source configuration (row number is ID):
#> tmax temp
#> 1 95 93.1484
#> # Target configuration (row number is ID):
#> tmax temp
#> 10 1737 40.7362
#> # 2023-09-26 11:03:02 UTC: Executing source and target configurations on the given instances * nrep (100)...
#> # 2023-09-26 11:03:08 UTC: Final best configuration:
#> tmax temp
#> 2 1737 40.7362
plotAblation(ablog)
# }