Performs ablation between two configurations (from source to target).
Source:R/ablation.R
ablation.RdAblation is a method for analyzing the differences between two configurations.
Usage
ablation(
iraceResults,
src = 1L,
target = NULL,
ab_params = NULL,
type = c("full", "racing"),
nrep = 1L,
seed = 1234567L,
ablationLogFile = "log-ablation.Rdata",
instancesFile = "train",
...
)Arguments
- iraceResults
list()|character(1)
Object created by irace and typically saved in the log fileirace.Rdata. If a character string is given, then it is interpreted as the path to the log file from which theiraceResultsobject will be loaded.- src, target
(
integer(1)) Source and target configuration IDs. By default, the first configuration ever evaluated (ID 1) is used assrcand the best configuration found by irace is used as target.- ab_params
Specific parameter names to be used for the ablation. They must be in
parameters$names. By default, use all parameters.- type
Type of ablation to perform:
"full"will execute each configuration on alln_instancesto determine the best-performing one;"racing"will apply racing to find the best configurations.- nrep
(
integer(1)) Number of replications per instance used in"full"ablation. Whennrep > 1, each configuration will be executednreptimes on each instance with different random seeds.- seed
(
integer(1)) Integer value to use as seed for the random number generation.- ablationLogFile
(
character(1)) Log file to save the ablation log. IfNULL, the results are not saved to a file.- instancesFile
(
character(1)) Instances file used for ablation:'train','test'or a filename containing the list of instances.- ...
Further arguments to override scenario settings, e.g.,
debugLevel,parallel, etc.
Value
A list containing the following elements:
- configurations
Configurations tested in the ablation.
- instances
A matrix with the instances used in the experiments. First column has the instances IDs from
iraceResults$scenario$instances, second column the seed assigned to the instance.- experiments
A matrix with the results of the experiments (columns are configurations, rows are instances).
- scenario
Scenario object with the settings used for the experiments.
- trajectory
IDs of the best configurations at each step of the ablation.
- best
Best configuration found in the experiments.
References
C. Fawcett and H. H. Hoos. Analysing differences between algorithm configurations through ablation. Journal of Heuristics, 22(4):431–458, 2016.
Examples
# \donttest{
logfile <- system.file(package="irace", "exdata", "sann.rda")
# Execute ablation between the first and the best configuration found by irace.
ablog <- ablation(logfile, ablationLogFile = NULL)
#> # Using 'train' instances:
#> 0.905770965304296
#> 0.916261134951038
#> 0.900950296507216
#> 0.896050584302144
#> 0.90081393365621
#> 0.903273416747905
#> 0.90910947751087
#> 0.914998982846093
#> 0.8581206964259
#> 0.896133447525713
#> 0.890187747154988
#> 0.900224266505036
#> 0.905740992929333
#> 0.892614309811161
#> 0.884147764345707
#> 0.875439211895376
#> 0.900361154479465
#> 0.870940390848391
#> 0.876752586150611
#> 0.920149278316993
#> 0.899263605755574
#> 0.904534155527841
#> 0.90142985671259
#> 0.864684305372312
#> 0.866358385170101
#> 0.914594276449293
#> 0.88330042017361
#> 0.930833591877602
#> 0.89339536073163
#> 0.882900621404483
#> 0.86489389910577
#> 0.892817594935744
#> 0.877676322203827
#> 0.930627478291988
#> 0.953154257237073
#> 0.889402625939687
#> 0.900612875289483
#> 0.927447319355266
#> 0.886344588603625
#> 0.865608041580583
#> 0.887430702731119
#> 0.8917783930472
#> 0.889520804989122
#> 0.858515262553617
#> 0.873786339249632
#> 0.923984662476343
#> 0.939721846952254
#> 0.938971067011916
#> 0.885663985935876
#> 0.896168249903553
#> 0.889354994333634
#> 0.919709261960667
#> 0.896685045095138
#> 0.876701435403075
#> 0.877116004191802
#> 0.872431812267919
#> 0.922823850889682
#> 0.910348603017257
#> 0.869351038135099
#> 0.886468633474737
#> 0.920676878663717
#> 0.886513865512749
#> 0.902993843282458
#> 0.891940310168587
#> 0.893302528847964
#> 0.860624034010844
#> 0.909093272823887
#> 0.869566288781206
#> 0.864804296657299
#> 0.932851707058265
#> 0.893857487118153
#> 0.888291700043304
#> 0.894047573692867
#> 0.869488419557791
#> 0.874936346679489
#> 0.878073486125015
#> 0.913688765996476
#> 0.872972668090872
#> 0.910067584035446
#> 0.899297185399043
#> 0.941218299889351
#> 0.87623542727906
#> 0.901718207648815
#> 0.913089995660784
#> 0.895217103668734
#> 0.896549976639855
#> 0.872570747264888
#> 0.879654231681631
#> 0.914120605769352
#> 0.87589930527902
#> 0.891248629044158
#> 0.884513555169136
#> 0.892740582297866
#> 0.868154590203597
#> 0.912370876056576
#> 0.881647560929649
#> 0.907390542474778
#> 0.921558415339888
#> 0.94585700580415
#> 0.883700717593088
#> # 2024-11-21 13:44:33 UTC: Starting ablation from 1 to 83
#> # Seed: 1234567
#> # Source configuration (row number is ID):
#> tmax temp
#> 1 4418 96.2841
#> # Target configuration (row number is ID):
#> tmax temp
#> 83 1958 38.5869
#> # 2024-11-21 13:44:33 UTC: Executing source and target configurations on the given instances * nrep (100)...
#> # Generating configurations (row number is ID): tmax temp
#> tmax temp
#> 2 1958 96.2841
#> 3 4418 38.5869
#> # 2024-11-21 13:44:37 UTC: Ablation (full) of 2 configurations on 100 instances.
#> # Markers:
#> x No test is performed.
#> c Configurations are discarded only due to capping.
#> - The test is performed and some configurations are discarded.
#> = The test is performed but no configuration is discarded.
#> ! The test is performed and configurations could be discarded but elite configurations are preserved.
#> . All alive configurations are elite and nothing is discarded.
#>
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> | | Instance| Alive| Best| Mean best| Exp so far| W time| rho|KenW| Qvar|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> |x| 1| 2| 2| 0.1299124444| 2|00:00:00| NA| NA| NA|
#> |x| 2| 2| 2| 1.387237584| 4|00:00:00|-1.00|0.00|1.0000|
#> |x| 3| 2| 3| 0.4349154257| 6|00:00:00|-0.33|0.11|0.6667|
#> |x| 4| 2| 3| 0.3548609828| 8|00:00:00|+0.00|0.25|0.5000|
#> |x| 5| 2| 3| 1.086689331| 10|00:00:00|-0.20|0.04|0.6000|
#> |x| 6| 2| 2| 0.7990212319| 12|00:00:00|-0.20|0.00|0.6000|
#> |x| 7| 2| 2| 0.7599895474| 14|00:00:00|-0.14|0.02|0.5714|
#> |x| 8| 2| 2| 0.7631774175| 16|00:00:00|-0.14|0.00|0.5714|
#> |x| 9| 2| 2| 0.7585765493| 18|00:00:00|-0.11|0.01|0.5556|
#> |x| 10| 2| 2| 0.6854857525| 20|00:00:00|-0.07|0.04|0.5333|
#> |x| 11| 2| 2| 0.6527012298| 22|00:00:00|-0.02|0.07|0.5091|
#> |x| 12| 2| 2| 0.6529789075| 24|00:00:00|+0.03|0.11|0.4848|
#> |x| 13| 2| 2| 0.6209079038| 26|00:00:00|+0.08|0.15|0.4615|
#> |x| 14| 2| 2| 0.7885092057| 28|00:00:00|+0.12|0.18|0.4396|
#> |x| 15| 2| 2| 0.7789336577| 30|00:00:00|+0.16|0.22|0.4190|
#> |x| 16| 2| 2| 0.7880679259| 32|00:00:00|+0.20|0.25|0.4000|
#> |x| 17| 2| 2| 1.041561129| 34|00:00:00|+0.12|0.17|0.4412|
#> |x| 18| 2| 2| 1.605948691| 36|00:00:00|+0.06|0.11|0.4706|
#> |x| 19| 2| 2| 1.756562377| 38|00:00:00|+0.09|0.14|0.4561|
#> |x| 20| 2| 2| 1.676397593| 40|00:00:00|+0.12|0.16|0.4421|
#> |x| 21| 2| 2| 1.633002548| 42|00:00:00|+0.14|0.18|0.4286|
#> |x| 22| 2| 2| 1.567836557| 44|00:00:00|+0.17|0.21|0.4156|
#> |x| 23| 2| 2| 1.504734883| 46|00:00:00|+0.19|0.23|0.4032|
#> |x| 24| 2| 2| 1.445838767| 48|00:00:00|+0.22|0.25|0.3913|
#> |x| 25| 2| 2| 1.439032447| 50|00:00:00|+0.24|0.27|0.3800|
#> |x| 26| 2| 2| 1.545831168| 52|00:00:00|+0.26|0.29|0.3692|
#> |x| 27| 2| 2| 1.495711930| 54|00:00:00|+0.28|0.31|0.3590|
#> |x| 28| 2| 2| 1.497927440| 56|00:00:00|+0.30|0.33|0.3492|
#> |x| 29| 2| 2| 1.641823996| 58|00:00:00|+0.24|0.27|0.3793|
#> |x| 30| 2| 2| 1.596975397| 60|00:00:00|+0.26|0.28|0.3701|
#> |x| 31| 2| 2| 1.649018016| 62|00:00:00|+0.21|0.23|0.3957|
#> |x| 32| 2| 2| 1.621009700| 64|00:00:00|+0.23|0.25|0.3871|
#> |x| 33| 2| 2| 1.584676996| 66|00:00:00|+0.24|0.27|0.3788|
#> |x| 34| 2| 2| 1.624877833| 68|00:00:00|+0.26|0.28|0.3708|
#> |x| 35| 2| 2| 1.631571858| 70|00:00:00|+0.27|0.29|0.3630|
#> |x| 36| 2| 2| 1.692252908| 72|00:00:00|+0.29|0.31|0.3556|
#> |x| 37| 2| 2| 1.650251140| 74|00:00:00|+0.30|0.32|0.3483|
#> |x| 38| 2| 2| 1.626047880| 76|00:00:00|+0.26|0.28|0.3713|
#> |x| 39| 2| 2| 1.606928599| 78|00:00:00|+0.27|0.29|0.3644|
#> |x| 40| 2| 2| 1.574131185| 80|00:00:00|+0.28|0.30|0.3577|
#> |x| 41| 2| 2| 1.539541974| 82|00:00:00|+0.24|0.26|0.3780|
#> |x| 42| 2| 2| 1.579896157| 84|00:00:00|+0.21|0.23|0.3961|
#> |x| 43| 2| 2| 1.554741803| 86|00:00:00|+0.18|0.20|0.4120|
#> |x| 44| 2| 2| 1.533727036| 88|00:00:00|+0.19|0.21|0.4059|
#> |x| 45| 2| 2| 1.505837691| 90|00:00:00|+0.20|0.22|0.4000|
#> |x| 46| 2| 2| 1.478311132| 92|00:00:00|+0.21|0.23|0.3942|
#> |x| 47| 2| 2| 1.561861654| 94|00:00:00|+0.18|0.20|0.4089|
#> |x| 48| 2| 2| 1.599185297| 96|00:00:00|+0.16|0.17|0.4220|
#> |x| 49| 2| 2| 1.579670171| 98|00:00:00|+0.17|0.18|0.4167|
#> |x| 50| 2| 2| 1.562873253| 100|00:00:00|+0.18|0.19|0.4114|
#> |x| 51| 2| 2| 1.547940902| 102|00:00:00|+0.19|0.20|0.4063|
#> |x| 52| 2| 2| 1.528444458| 104|00:00:00|+0.20|0.21|0.4012|
#> |x| 53| 2| 2| 1.671028247| 106|00:00:00|+0.17|0.19|0.4136|
#> |x| 54| 2| 2| 1.641753426| 108|00:00:00|+0.18|0.20|0.4088|
#> |x| 55| 2| 2| 1.636822047| 110|00:00:00|+0.16|0.17|0.4202|
#> |x| 56| 2| 2| 1.627282524| 112|00:00:00|+0.17|0.18|0.4156|
#> |x| 57| 2| 2| 1.650975372| 114|00:00:00|+0.18|0.19|0.4110|
#> |x| 58| 2| 2| 1.634341405| 116|00:00:00|+0.19|0.20|0.4065|
#> |x| 59| 2| 2| 1.639817999| 118|00:00:00|+0.20|0.21|0.4021|
#> |x| 60| 2| 2| 1.613522168| 120|00:00:00|+0.20|0.22|0.3977|
#> |x| 61| 2| 2| 1.593087150| 122|00:00:00|+0.18|0.20|0.4087|
#> |x| 62| 2| 2| 1.570251233| 124|00:00:00|+0.19|0.20|0.4045|
#> |x| 63| 2| 2| 1.547005576| 126|00:00:00|+0.20|0.21|0.4004|
#> |x| 64| 2| 2| 1.524017399| 128|00:00:00|+0.21|0.22|0.3963|
#> |x| 65| 2| 2| 1.503488334| 130|00:00:00|+0.22|0.23|0.3923|
#> |x| 66| 2| 2| 1.490968766| 132|00:00:00|+0.22|0.24|0.3883|
#> |x| 67| 2| 2| 1.475633002| 134|00:00:00|+0.23|0.24|0.3844|
#> |x| 68| 2| 2| 1.499340621| 136|00:00:00|+0.24|0.25|0.3806|
#> |x| 69| 2| 2| 1.480451817| 138|00:00:00|+0.25|0.26|0.3768|
#> |x| 70| 2| 2| 1.518510431| 140|00:00:00|+0.22|0.24|0.3876|
#> |x| 71| 2| 2| 1.500742424| 142|00:00:00|+0.23|0.24|0.3839|
#> |x| 72| 2| 2| 1.513059207| 144|00:00:00|+0.21|0.22|0.3940|
#> |x| 73| 2| 2| 1.495732543| 146|00:00:00|+0.22|0.23|0.3904|
#> |x| 74| 2| 2| 1.507865583| 148|00:00:00|+0.23|0.24|0.3869|
#> |x| 75| 2| 2| 1.531761381| 150|00:00:00|+0.21|0.22|0.3964|
#> |x| 76| 2| 2| 1.660724750| 152|00:00:00|+0.19|0.20|0.4053|
#> |x| 77| 2| 2| 1.666857884| 154|00:00:00|+0.17|0.18|0.4135|
#> |x| 78| 2| 2| 1.684752626| 156|00:00:00|+0.16|0.17|0.4212|
#> |x| 79| 2| 2| 1.774203563| 158|00:00:00|+0.14|0.15|0.4284|
#> |x| 80| 2| 2| 1.804059903| 160|00:00:00|+0.15|0.16|0.4253|
#> |x| 81| 2| 2| 1.782628542| 162|00:00:00|+0.16|0.17|0.4222|
#> |x| 82| 2| 2| 1.763069193| 164|00:00:00|+0.16|0.17|0.4192|
#> |x| 83| 2| 2| 1.754172083| 166|00:00:00|+0.17|0.18|0.4161|
#> |x| 84| 2| 2| 1.742703883| 168|00:00:00|+0.17|0.18|0.4131|
#> |x| 85| 2| 2| 1.724901348| 170|00:00:00|+0.18|0.19|0.4101|
#> |x| 86| 2| 2| 1.712342085| 172|00:00:00|+0.17|0.18|0.4172|
#> |x| 87| 2| 2| 1.696111586| 174|00:00:00|+0.17|0.18|0.4143|
#> |x| 88| 2| 2| 1.686668356| 176|00:00:00|+0.18|0.19|0.4114|
#> |x| 89| 2| 2| 1.669373214| 178|00:00:00|+0.18|0.19|0.4086|
#> |x| 90| 2| 2| 1.656180580| 180|00:00:00|+0.19|0.20|0.4057|
#> |x| 91| 2| 2| 1.694513192| 182|00:00:00|+0.17|0.18|0.4127|
#> |x| 92| 2| 2| 1.697415661| 184|00:00:00|+0.18|0.19|0.4099|
#> |x| 93| 2| 2| 1.681976283| 186|00:00:00|+0.17|0.18|0.4165|
#> |x| 94| 2| 2| 1.665733072| 188|00:00:00|+0.17|0.18|0.4139|
#> |x| 95| 2| 2| 1.697758404| 190|00:00:00|+0.16|0.17|0.4202|
#> |x| 96| 2| 2| 1.692535274| 192|00:00:00|+0.16|0.17|0.4175|
#> |x| 97| 2| 2| 1.695935060| 194|00:00:00|+0.15|0.16|0.4235|
#> |x| 98| 2| 2| 1.681528852| 196|00:00:00|+0.16|0.17|0.4210|
#> |x| 99| 2| 2| 1.676874666| 198|00:00:00|+0.15|0.16|0.4267|
#> |-| 100| 1| 2| 1.668563079| 200|00:00:00| NA| NA| NA|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> Best-so-far configuration: 2 mean value: 1.668563079
#> Description of the best-so-far configuration:
#> .ID. tmax temp .PARENT.
#> 2 2 1958 96.2841 1
#>
#> # Best changed parameters:
#> # tmax : 4418 -> 1958
#> # 2024-11-21 13:44:42 UTC: Final best configuration:
#> tmax temp
#> 4 1958 38.5869
plotAblation(ablog)
# Execute ablation between two selected configurations, and selecting only a
# subset of parameters, directly reading the setup from the irace log file.
ablog <- ablation(logfile, src = 1, target = 10,
ab_params = c("temp"), ablationLogFile = NULL)
#> # Using 'train' instances:
#> 0.905770965304296
#> 0.916261134951038
#> 0.900950296507216
#> 0.896050584302144
#> 0.90081393365621
#> 0.903273416747905
#> 0.90910947751087
#> 0.914998982846093
#> 0.8581206964259
#> 0.896133447525713
#> 0.890187747154988
#> 0.900224266505036
#> 0.905740992929333
#> 0.892614309811161
#> 0.884147764345707
#> 0.875439211895376
#> 0.900361154479465
#> 0.870940390848391
#> 0.876752586150611
#> 0.920149278316993
#> 0.899263605755574
#> 0.904534155527841
#> 0.90142985671259
#> 0.864684305372312
#> 0.866358385170101
#> 0.914594276449293
#> 0.88330042017361
#> 0.930833591877602
#> 0.89339536073163
#> 0.882900621404483
#> 0.86489389910577
#> 0.892817594935744
#> 0.877676322203827
#> 0.930627478291988
#> 0.953154257237073
#> 0.889402625939687
#> 0.900612875289483
#> 0.927447319355266
#> 0.886344588603625
#> 0.865608041580583
#> 0.887430702731119
#> 0.8917783930472
#> 0.889520804989122
#> 0.858515262553617
#> 0.873786339249632
#> 0.923984662476343
#> 0.939721846952254
#> 0.938971067011916
#> 0.885663985935876
#> 0.896168249903553
#> 0.889354994333634
#> 0.919709261960667
#> 0.896685045095138
#> 0.876701435403075
#> 0.877116004191802
#> 0.872431812267919
#> 0.922823850889682
#> 0.910348603017257
#> 0.869351038135099
#> 0.886468633474737
#> 0.920676878663717
#> 0.886513865512749
#> 0.902993843282458
#> 0.891940310168587
#> 0.893302528847964
#> 0.860624034010844
#> 0.909093272823887
#> 0.869566288781206
#> 0.864804296657299
#> 0.932851707058265
#> 0.893857487118153
#> 0.888291700043304
#> 0.894047573692867
#> 0.869488419557791
#> 0.874936346679489
#> 0.878073486125015
#> 0.913688765996476
#> 0.872972668090872
#> 0.910067584035446
#> 0.899297185399043
#> 0.941218299889351
#> 0.87623542727906
#> 0.901718207648815
#> 0.913089995660784
#> 0.895217103668734
#> 0.896549976639855
#> 0.872570747264888
#> 0.879654231681631
#> 0.914120605769352
#> 0.87589930527902
#> 0.891248629044158
#> 0.884513555169136
#> 0.892740582297866
#> 0.868154590203597
#> 0.912370876056576
#> 0.881647560929649
#> 0.907390542474778
#> 0.921558415339888
#> 0.94585700580415
#> 0.883700717593088
#> # 2024-11-21 13:44:42 UTC: Starting ablation from 1 to 10
#> # Seed: 1234567
#> # Source configuration (row number is ID):
#> tmax temp
#> 1 4418 96.2841
#> # Target configuration (row number is ID):
#> tmax temp
#> 10 1606 15.0341
#> # 2024-11-21 13:44:42 UTC: Executing source and target configurations on the given instances * nrep (100)...
#> # 2024-11-21 13:44:46 UTC: Final best configuration:
#> tmax temp
#> 2 1606 15.0341
plotAblation(ablog)
# }