Performs ablation between two configurations (from source to target).
Source:R/ablation.R
ablation.RdAblation is a method for analyzing the differences between two configurations.
Usage
ablation(
iraceResults,
src = 1L,
target = NULL,
ab_params = NULL,
type = c("full", "racing"),
nrep = 1L,
seed = 1234567L,
ablationLogFile = "log-ablation.Rdata",
instancesFile = "train",
...
)Arguments
- iraceResults
(
list()|character(1))
Object created by irace and typically saved in the log fileirace.Rdata. If a character string is given, then it is interpreted as the path to the log file from which theiraceResultsobject will be loaded.- src, target
(
integer(1)) Source and target configuration IDs. By default, the first configuration ever evaluated (ID 1) is used assrcand the best configuration found by irace is used as target.- ab_params
Specific parameter names to be used for the ablation. They must be in
parameters$names. By default, use all parameters.- type
Type of ablation to perform:
"full"will execute each configuration on alln_instancesto determine the best-performing one;"racing"will apply racing to find the best configurations.- nrep
(
integer(1)) Number of replications per instance used in"full"ablation. Whennrep > 1, each configuration will be executednreptimes on each instance with different random seeds.- seed
(
integer(1)) Integer value to use as seed for the random number generation.- ablationLogFile
(
character(1)) Log file to save the ablation log. IfNULL, the results are not saved to a file.- instancesFile
(
character(1)) Instances file used for ablation:'train','test'or a filename containing the list of instances.- ...
Further arguments to override scenario settings, e.g.,
debugLevel,parallel, etc.
Value
A list containing the following elements:
- configurations
Configurations tested in the ablation.
- instances
A matrix with the instances used in the experiments. First column has the instances IDs from
iraceResults$scenario$instances, second column the seed assigned to the instance.- experiments
A matrix with the results of the experiments (columns are configurations, rows are instances).
- scenario
Scenario object with the settings used for the experiments.
- trajectory
IDs of the best configurations at each step of the ablation.
- best
Best configuration found in the experiments.
References
C. Fawcett and H. H. Hoos. Analysing differences between algorithm configurations through ablation. Journal of Heuristics, 22(4):431–458, 2016.
Examples
# \donttest{
logfile <- system.file(package="irace", "exdata", "sann.rda")
# Execute ablation between the first and the best configuration found by irace.
ablog <- ablation(logfile, ablationLogFile = NULL)
#> # Using 'train' instances:
#> 0.860339554929799
#> 0.876351447362025
#> 0.851547238879318
#> 0.917566078416809
#> 0.86940213766692
#> 0.885882063795202
#> 0.917368119602368
#> 0.883024727887818
#> 0.890294216500032
#> 0.895404843930028
#> 0.892579401664355
#> 0.906981916446107
#> 0.891674680707451
#> 0.865660299916002
#> 0.928345131539666
#> 0.897726629923715
#> 0.877897452414984
#> 0.902239678173651
#> 0.906944953717246
#> 0.87245818600224
#> 0.926312601695457
#> 0.896927094601937
#> 0.904934754996882
#> 0.92674141993335
#> 0.912193252463897
#> 0.893435861899156
#> 0.930690427450913
#> 0.922217221705376
#> 0.923589473036879
#> 0.89650026551228
#> 0.896645293829647
#> 0.908282873079767
#> 0.903208618722223
#> 0.903806858499259
#> 0.888341627876823
#> 0.909295370597863
#> 0.916136847125745
#> 0.929707668376692
#> 0.903574934275889
#> 0.928222747303035
#> 0.894365900214697
#> 0.89146318414938
#> 0.899106313921096
#> 0.927827737699432
#> 0.894980802020092
#> 0.884677402247161
#> 0.877391174321647
#> 0.905879197545356
#> 0.902517543448092
#> 0.934435100968938
#> 0.896488946031878
#> 0.889792016859481
#> 0.93205946775441
#> 0.904406318022476
#> 0.861951675921213
#> 0.887254366164604
#> 0.894569044108253
#> 0.916620582217757
#> 0.908233921051475
#> 0.911466373008508
#> 0.868747313654071
#> 0.876987196942168
#> 0.883716154459711
#> 0.905650639104779
#> 0.875875120955676
#> 0.898097349407182
#> 0.877113017486763
#> 0.899117077399166
#> 0.915767481943537
#> 0.864303203877886
#> 0.899435222259812
#> 0.908638680393453
#> 0.885949034833742
#> 0.864815677984075
#> 0.897128093965593
#> 0.897705133266575
#> 0.906189652087981
#> 0.887251074294492
#> 0.888048007833716
#> 0.867678607147691
#> 0.896301473917458
#> 0.906024594977991
#> 0.910204611154315
#> 0.897099736612278
#> 0.939058099560877
#> 0.914812206404204
#> 0.917608779289547
#> 0.884803266726778
#> 0.867047613966726
#> 0.913843012371863
#> 0.912971185210657
#> 0.900795296337931
#> 0.917676623002888
#> 0.872093954050175
#> 0.90683251371825
#> 0.900993153764746
#> 0.852097355243314
#> 0.883994277751485
#> 0.90890056586115
#> 0.8651423676284
#> # 2024-03-22 14:22:30 UTC: Starting ablation from 1 to 100
#> # Seed: 1234567
#> # Source configuration (row number is ID):
#> tmax temp
#> 1 3210 70.0396
#> # Target configuration (row number is ID):
#> tmax temp
#> 100 2284 50.4926
#> # 2024-03-22 14:22:30 UTC: Executing source and target configurations on the given instances * nrep (100)...
#> # Generating configurations (row number is ID): tmax temp
#> tmax temp
#> 2 2284 70.0396
#> 3 3210 50.4926
#> # 2024-03-22 14:22:34 UTC: Ablation (full) of 2 configurations on 100 instances.
#> # Markers:
#> x No test is performed.
#> c Configurations are discarded only due to capping.
#> - The test is performed and some configurations are discarded.
#> = The test is performed but no configuration is discarded.
#> ! The test is performed and configurations could be discarded but elite configurations are preserved.
#> . All alive configurations are elite and nothing is discarded.
#>
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> | | Instance| Alive| Best| Mean best| Exp so far| W time| rho|KenW| Qvar|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> |x| 1| 2| 3| 0.3019193419| 2|00:00:00| NA| NA| NA|
#> |x| 2| 2| 3| 0.2092730133| 4|00:00:00|+1.00|1.00|0.0000|
#> |x| 3| 2| 3| 1.753651778| 6|00:00:00|-0.33|0.11|0.6667|
#> |x| 4| 2| 3| 1.353411381| 8|00:00:00|+0.00|0.25|0.5000|
#> |x| 5| 2| 3| 1.119213320| 10|00:00:00|+0.20|0.36|0.4000|
#> |x| 6| 2| 3| 1.430983630| 12|00:00:00|+0.33|0.44|0.3333|
#> |x| 7| 2| 3| 1.662576113| 14|00:00:00|+0.43|0.51|0.2857|
#> |x| 8| 2| 3| 2.022866042| 16|00:00:00|+0.14|0.25|0.4286|
#> |x| 9| 2| 3| 1.803506094| 18|00:00:00|+0.22|0.31|0.3889|
#> |x| 10| 2| 3| 1.680327461| 20|00:00:00|+0.07|0.16|0.4667|
#> |x| 11| 2| 3| 1.879342999| 22|00:00:00|-0.02|0.07|0.5091|
#> |x| 12| 2| 3| 1.774527337| 24|00:00:00|-0.06|0.03|0.5303|
#> |x| 13| 2| 3| 2.138649418| 26|00:00:00|-0.08|0.01|0.5385|
#> |x| 14| 2| 3| 2.054464990| 28|00:00:00|-0.05|0.02|0.5275|
#> |x| 15| 2| 3| 2.187355861| 30|00:00:00|-0.07|0.00|0.5333|
#> |x| 16| 2| 2| 2.131579650| 32|00:00:00|-0.07|0.00|0.5333|
#> |x| 17| 2| 2| 2.020615109| 34|00:00:00|-0.06|0.00|0.5294|
#> |x| 18| 2| 2| 1.916789089| 36|00:00:00|-0.05|0.01|0.5229|
#> |x| 19| 2| 2| 1.825538970| 38|00:00:00|-0.03|0.02|0.5146|
#> |x| 20| 2| 2| 1.737711584| 40|00:00:00|-0.01|0.04|0.5053|
#> |x| 21| 2| 2| 1.668375808| 42|00:00:00|+0.01|0.06|0.4952|
#> |x| 22| 2| 2| 1.733600338| 44|00:00:00|-0.01|0.03|0.5065|
#> |x| 23| 2| 2| 1.671016983| 46|00:00:00|+0.00|0.05|0.4980|
#> |x| 24| 2| 2| 1.656971904| 48|00:00:00|-0.01|0.03|0.5072|
#> |x| 25| 2| 2| 1.593554851| 50|00:00:00|+0.00|0.04|0.5000|
#> |x| 26| 2| 2| 1.689227926| 52|00:00:00|-0.02|0.02|0.5077|
#> |x| 27| 2| 2| 1.636128163| 54|00:00:00|-0.03|0.01|0.5128|
#> |x| 28| 2| 2| 1.598179205| 56|00:00:00|-0.02|0.02|0.5079|
#> |x| 29| 2| 2| 1.684695210| 58|00:00:00|-0.02|0.01|0.5123|
#> |x| 30| 2| 2| 1.639338705| 60|00:00:00|-0.02|0.02|0.5080|
#> |x| 31| 2| 2| 1.693796060| 62|00:00:00|-0.01|0.03|0.5032|
#> |x| 32| 2| 2| 1.737939307| 64|00:00:00|-0.02|0.02|0.5081|
#> |x| 33| 2| 2| 1.698619965| 66|00:00:00|-0.01|0.02|0.5038|
#> |x| 34| 2| 2| 1.824909616| 68|00:00:00|-0.02|0.01|0.5080|
#> |x| 35| 2| 2| 1.893901600| 70|00:00:00|-0.02|0.01|0.5109|
#> |x| 36| 2| 2| 1.848453269| 72|00:00:00|-0.02|0.01|0.5079|
#> |x| 37| 2| 2| 1.799979388| 74|00:00:00|-0.01|0.02|0.5045|
#> |x| 38| 2| 2| 1.762256307| 76|00:00:00|-0.00|0.02|0.5007|
#> |x| 39| 2| 2| 1.723924967| 78|00:00:00|+0.01|0.03|0.4966|
#> |x| 40| 2| 2| 1.759018086| 80|00:00:00|-0.00|0.02|0.5013|
#> |x| 41| 2| 2| 1.748725157| 82|00:00:00|-0.01|0.01|0.5049|
#> |x| 42| 2| 2| 1.724149397| 84|00:00:00|-0.02|0.01|0.5075|
#> |x| 43| 2| 2| 1.688686829| 86|00:00:00|-0.01|0.01|0.5050|
#> |x| 44| 2| 2| 1.664085979| 88|00:00:00|-0.00|0.02|0.5021|
#> |x| 45| 2| 2| 1.702820140| 90|00:00:00|-0.01|0.01|0.5051|
#> |x| 46| 2| 2| 1.671741231| 92|00:00:00|-0.00|0.02|0.5024|
#> |x| 47| 2| 2| 1.640777307| 94|00:00:00|-0.01|0.01|0.5051|
#> |x| 48| 2| 2| 1.670456872| 96|00:00:00|-0.01|0.01|0.5071|
#> |x| 49| 2| 2| 1.641829060| 98|00:00:00|-0.01|0.01|0.5051|
#> |x| 50| 2| 2| 1.633936761| 100|00:00:00|-0.01|0.01|0.5029|
#> |x| 51| 2| 2| 1.607432776| 102|00:00:00|-0.00|0.02|0.5004|
#> |x| 52| 2| 2| 1.581709458| 104|00:00:00|+0.00|0.02|0.4977|
#> |x| 53| 2| 2| 1.605095842| 106|00:00:00|+0.01|0.03|0.4949|
#> |x| 54| 2| 2| 1.584130411| 108|00:00:00|+0.00|0.02|0.4983|
#> |x| 55| 2| 2| 1.559158342| 110|00:00:00|-0.00|0.02|0.5010|
#> |x| 56| 2| 2| 1.584376259| 112|00:00:00|+0.00|0.02|0.4987|
#> |x| 57| 2| 2| 1.569248784| 114|00:00:00|+0.01|0.02|0.4962|
#> |x| 58| 2| 2| 1.595024145| 116|00:00:00|+0.01|0.03|0.4936|
#> |x| 59| 2| 2| 1.574047621| 118|00:00:00|+0.01|0.02|0.4968|
#> |x| 60| 2| 2| 1.555217968| 120|00:00:00|+0.00|0.02|0.4994|
#> |x| 61| 2| 2| 1.541157929| 122|00:00:00|+0.01|0.02|0.4973|
#> |x| 62| 2| 2| 1.524217229| 124|00:00:00|+0.00|0.02|0.4997|
#> |x| 63| 2| 2| 1.569374265| 126|00:00:00|-0.00|0.01|0.5018|
#> |x| 64| 2| 2| 1.551487479| 128|00:00:00|+0.00|0.02|0.5000|
#> |x| 65| 2| 2| 1.573527707| 130|00:00:00|-0.00|0.01|0.5019|
#> |x| 66| 2| 2| 1.561126692| 132|00:00:00|-0.00|0.01|0.5002|
#> |x| 67| 2| 2| 1.547773207| 134|00:00:00|-0.00|0.01|0.5020|
#> |x| 68| 2| 2| 1.528449208| 136|00:00:00|-0.00|0.01|0.5004|
#> |x| 69| 2| 2| 1.552107237| 138|00:00:00|-0.00|0.01|0.5021|
#> |x| 70| 2| 2| 1.584472984| 140|00:00:00|-0.01|0.01|0.5035|
#> |x| 71| 2| 2| 1.564866899| 142|00:00:00|-0.00|0.01|0.5022|
#> |x| 72| 2| 2| 1.552768432| 144|00:00:00|-0.00|0.01|0.5008|
#> |x| 73| 2| 2| 1.533363666| 146|00:00:00|+0.00|0.02|0.4992|
#> |x| 74| 2| 2| 1.514904761| 148|00:00:00|+0.00|0.02|0.4976|
#> |x| 75| 2| 2| 1.521779437| 150|00:00:00|+0.00|0.01|0.4995|
#> |x| 76| 2| 2| 1.504161853| 152|00:00:00|+0.00|0.02|0.4979|
#> |x| 77| 2| 2| 1.486637638| 154|00:00:00|+0.00|0.01|0.4997|
#> |x| 78| 2| 2| 1.476008478| 156|00:00:00|+0.00|0.02|0.4982|
#> |x| 79| 2| 2| 1.502978198| 158|00:00:00|+0.00|0.01|0.4998|
#> |x| 80| 2| 2| 1.496751693| 160|00:00:00|-0.00|0.01|0.5013|
#> |x| 81| 2| 2| 1.481243830| 162|00:00:00|+0.00|0.01|0.5000|
#> |x| 82| 2| 2| 1.464258576| 164|00:00:00|+0.00|0.01|0.4986|
#> |x| 83| 2| 2| 1.447948310| 166|00:00:00|+0.01|0.02|0.4972|
#> |x| 84| 2| 2| 1.434119231| 168|00:00:00|+0.01|0.02|0.4957|
#> |x| 85| 2| 2| 1.450767359| 170|00:00:00|+0.01|0.02|0.4975|
#> |x| 86| 2| 2| 1.436137651| 172|00:00:00|+0.01|0.02|0.4960|
#> |x| 87| 2| 2| 1.426348902| 174|00:00:00|+0.01|0.02|0.4945|
#> |x| 88| 2| 2| 1.433535478| 176|00:00:00|+0.01|0.03|0.4929|
#> |x| 89| 2| 2| 1.417996652| 178|00:00:00|+0.02|0.03|0.4913|
#> |x| 90| 2| 2| 1.448872937| 180|00:00:00|+0.01|0.02|0.4934|
#> |x| 91| 2| 2| 1.434229236| 182|00:00:00|+0.02|0.03|0.4918|
#> |x| 92| 2| 2| 1.423509762| 184|00:00:00|+0.02|0.03|0.4902|
#> |x| 93| 2| 2| 1.414284319| 186|00:00:00|+0.02|0.03|0.4923|
#> |x| 94| 2| 2| 1.409966502| 188|00:00:00|+0.01|0.02|0.4942|
#> |x| 95| 2| 2| 1.403622022| 190|00:00:00|+0.01|0.02|0.4959|
#> |x| 96| 2| 2| 1.394239657| 192|00:00:00|+0.01|0.02|0.4974|
#> |x| 97| 2| 2| 1.423065519| 194|00:00:00|+0.00|0.01|0.4987|
#> |x| 98| 2| 2| 1.413465023| 196|00:00:00|+0.00|0.01|0.4999|
#> |x| 99| 2| 2| 1.403760985| 198|00:00:00|+0.00|0.01|0.4989|
#> |-| 100| 1| 2| 1.394391702| 200|00:00:00| NA| NA| NA|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> Best-so-far configuration: 2 mean value: 1.394391702
#> Description of the best-so-far configuration:
#> .ID. tmax temp .PARENT.
#> 2 2 2284 70.0396 1
#>
#> # Best changed parameters:
#> # tmax : 3210 -> 2284
#> # 2024-03-22 14:22:39 UTC: Final best configuration:
#> tmax temp
#> 4 2284 50.4926
plotAblation(ablog)
# Execute ablation between two selected configurations, and selecting only a
# subset of parameters, directly reading the setup from the irace log file.
ablog <- ablation(logfile, src = 1, target = 10,
ab_params = c("temp"), ablationLogFile = NULL)
#> # Using 'train' instances:
#> 0.860339554929799
#> 0.876351447362025
#> 0.851547238879318
#> 0.917566078416809
#> 0.86940213766692
#> 0.885882063795202
#> 0.917368119602368
#> 0.883024727887818
#> 0.890294216500032
#> 0.895404843930028
#> 0.892579401664355
#> 0.906981916446107
#> 0.891674680707451
#> 0.865660299916002
#> 0.928345131539666
#> 0.897726629923715
#> 0.877897452414984
#> 0.902239678173651
#> 0.906944953717246
#> 0.87245818600224
#> 0.926312601695457
#> 0.896927094601937
#> 0.904934754996882
#> 0.92674141993335
#> 0.912193252463897
#> 0.893435861899156
#> 0.930690427450913
#> 0.922217221705376
#> 0.923589473036879
#> 0.89650026551228
#> 0.896645293829647
#> 0.908282873079767
#> 0.903208618722223
#> 0.903806858499259
#> 0.888341627876823
#> 0.909295370597863
#> 0.916136847125745
#> 0.929707668376692
#> 0.903574934275889
#> 0.928222747303035
#> 0.894365900214697
#> 0.89146318414938
#> 0.899106313921096
#> 0.927827737699432
#> 0.894980802020092
#> 0.884677402247161
#> 0.877391174321647
#> 0.905879197545356
#> 0.902517543448092
#> 0.934435100968938
#> 0.896488946031878
#> 0.889792016859481
#> 0.93205946775441
#> 0.904406318022476
#> 0.861951675921213
#> 0.887254366164604
#> 0.894569044108253
#> 0.916620582217757
#> 0.908233921051475
#> 0.911466373008508
#> 0.868747313654071
#> 0.876987196942168
#> 0.883716154459711
#> 0.905650639104779
#> 0.875875120955676
#> 0.898097349407182
#> 0.877113017486763
#> 0.899117077399166
#> 0.915767481943537
#> 0.864303203877886
#> 0.899435222259812
#> 0.908638680393453
#> 0.885949034833742
#> 0.864815677984075
#> 0.897128093965593
#> 0.897705133266575
#> 0.906189652087981
#> 0.887251074294492
#> 0.888048007833716
#> 0.867678607147691
#> 0.896301473917458
#> 0.906024594977991
#> 0.910204611154315
#> 0.897099736612278
#> 0.939058099560877
#> 0.914812206404204
#> 0.917608779289547
#> 0.884803266726778
#> 0.867047613966726
#> 0.913843012371863
#> 0.912971185210657
#> 0.900795296337931
#> 0.917676623002888
#> 0.872093954050175
#> 0.90683251371825
#> 0.900993153764746
#> 0.852097355243314
#> 0.883994277751485
#> 0.90890056586115
#> 0.8651423676284
#> # 2024-03-22 14:22:39 UTC: Starting ablation from 1 to 10
#> # Seed: 1234567
#> # Source configuration (row number is ID):
#> tmax temp
#> 1 3210 70.0396
#> # Target configuration (row number is ID):
#> tmax temp
#> 10 3204 49.8704
#> # 2024-03-22 14:22:39 UTC: Executing source and target configurations on the given instances * nrep (100)...
#> # 2024-03-22 14:22:44 UTC: Final best configuration:
#> tmax temp
#> 2 3204 49.8704
plotAblation(ablog)
# }