Configure Colosseum#

\(\texttt{Colosseum}\) allows configuring global directories for the hyperparameters optimization and the benchmarking procedures, the settings regarding multiprocessing, and some other functionalities. The config module provide the configuring functions.

Configuring directories

Benchmarking

As a running example, we assume that the goal is to run the tabular benchmark separately for some model-free and model-based agents. So we’ll create a main experiment folder called tabular and two sub-folders for the two types of agents.

main_experiments_folder = "tabular"

When we benchmark the model-free agents, we set the name of the experiment as model-free and communicate it to \(\texttt{Colosseum}\).

current_experiment_folder_name = "model_free"

# Set the experiment folder and the related hyperoptimization folder
config.set_experiments_folder(main_experiments_folder, current_experiment_folder_name)
config.set_hyperopt_folder(main_experiments_folder, current_experiment_folder_name)

# Show the folder structure for the benchmarking results and for the hyperparameters optimizations results
print("Model-free experiment folder: ", config.get_experiments_folder())
print("Model-free hyperoptimization folder", config.get_hyperopt_folder())

# Code for benchamrking the model-free agents goes here

Model-free experiment folder:  tabular/benchmarking/model_free/
Model-free hyperoptimization folder tabular/hyperopt/model_free/

When instead we benchmark the model-based agents, we set the name of the experiment as model-based and similarly communicate it to the package.

current_experiment_folder_name = "model_based"

# Set the experiment folder and the related hyperoptimization folder
config.set_experiments_folder(main_experiments_folder, current_experiment_folder_name)
config.set_hyperopt_folder(main_experiments_folder, current_experiment_folder_name)

# Show the folder structure for the benchmarking results and for the hyperparameters optimizations results
print("Model-based experiment folder: ", config.get_experiments_folder())
print("Model-based hyperoptimization folder", config.get_hyperopt_folder())

# Code for benchmarking the model-based agents goes here

Model-based experiment folder:  tabular/benchmarking/model_based/
Model-based hyperoptimization folder tabular/hyperopt/model_based/

Hardness analysis

The package includes cached values of the hardness measures for the benchmark environments and automatically caches the values for new environments locally by creating a copy of the cached folder from the package.

print("Default hardness measures cache folder: ", config.get_hardness_measures_cache_folder())

# If you prefer, you can set a different folder path
config.set_hardness_measures_cache_folder("my_cached_hardness_measures_folder")
print("Custom hardness measures cache folder: ", config.get_hardness_measures_cache_folder())

Default hardness measures cache folder:  cached_hardness_measures/
Custom hardness measures cache folder:  my_cached_hardness_measures_folder/

Verbosity

\(\texttt{Colosseum}\) can provide verbose logging for the agent/MDP interaction, computing the hardness measures, and some time-consuming visualizations. Note that verbosity is turned off by default.

# Enable verbosity
config.enable_verbose_logging()
# Disable verbosity
config.disable_verbose_logging()

Multiprocessing

\(\texttt{Colosseum}\) can leverage multiple cores for benchmarking agents and computing hardness measures. Note that multiprocessing is turned off by default.

When multiprocessing is enabled, \(\texttt{Colosseum}\) sets the number of available cores to the total number of cores available minus two.

config.enable_multiprocessing()
print("Number of cores available to the package: ", config.get_available_cores())

Number of cores available to the package:  30

However, it is possible to manually set the number of cores the package will use.

config.set_available_cores(5)
print("Number of cores available to the package: ", config.get_available_cores())

Number of cores available to the package:  5

Once multiprocessing has been enabled, it can be disabled.

# Disable multiprocessing
config.disable_multiprocessing()
print("Number of cores available to the package: ", config.get_available_cores())

Number of cores available to the package:  1