Talks about how manual algorithm tuning is tedious and not always done and automated tuning can be useful

Gaussian distribution for numerical parameters, discrete distribution for categorical.

Each parameter being tuned has a statistical field which is sampled from to create candidates and is updated based on the best results.

"Soft-restart" does some stuff to try and avoid prematurely concluding an algorithm was optimized.

Tournament

• Create a list of candidates where each parameter is sampled from the models.
• Run some number of steps.
• Eliminate the worst performing configuration.
• Update statistical fields of parameters to prefer generating winners.
• Repeat until done.

Steps

Run the algorithm with a given configuration and collect scoring data.

Repeat until you have enough data you are confident in the benchmark.

General thesis

• Sample new configuration from statistical distribution
• Select best candidate configuration by racing
• Update statistical distribution towards the higher performing parameters