OmniSafe Experiment Grid#

ExperimentGrid([exp_name])

Tool for running many experiments given hyper-parameters ranges.

Experiment Grid#

Documentation

class omnisafe.common.experiment_grid.ExperimentGrid(exp_name='')[source]#

Tool for running many experiments given hyper-parameters ranges.

Parameters:

exp_name (str, optional) – Name of the experiment grid. Defaults to ‘ ‘.

Variables:

  • keys (list[str]) – The keys of the configurations for the experiments.

  • vals (list[Any]) – The values of the configurations for the experiments.

  • shs (list[str]) – The shorthands of the configurations for the experiments.

  • in_names (list[bool]) – Whether the shorthand is included in the name of the experiment.

  • div_line_width (int) – The width of the dividing line.

  • name (str) – Name of the experiment grid.

  • default_shorthand (bool) – Whether GridSearch provides default shorthands.

  • wait_defore_launch (int) – Tells the GridSearch how many seconds to pause for before launching experiments.

  • foce_datastamp (bool) – Whether to automatically insert a date and time stamp into the names of save directories.

  • log_dir (str) – The directory for saving the logs.

Initialize an instance of ExperimentGrid.

_default_shorthand(key)[source]#

Get the default shorthand.

Create a default shorthand for the key, built from the first three letters of each colon-separated part. But if the first three letters contains something which isn’t alphanumeric, shear that off.

Examples

>>> _default_shorthand('env_name:SafetyPointGoal1-v0')
'env'
Parameters:

key (str) – Name of parameter.

Returns:

Shorthand of parameter.

Return type:

str

_init_statistical_tools()[source]#

Initialize statistical tools.

Return type:

None

_variants(keys, vals)[source]#

Recursively builds list of valid variants.

Parameters:

  • (keys (keys) – list[str]): List of keys.

  • vals (list[Any]) – List of values.

Returns:

List of valid variants.

Return type:

list[dict[str, Any]]

add(key, vals, shorthand=None, in_name=False)[source]#

Add a parameter (key) to the grid config, with potential values (vals).

By default, if a shorthand isn’t given, one is automatically generated from the key using the first three letters of each colon-separated term.

Hint

This function is called in omnisafe/examples/benchmarks/run_experiment_grid.py.

Examples

>>> add('env_id', ['SafetyPointGoal1-v0', 'SafetyAntVelocity-v1'])
>>> add('algo', ['SAC', 'DDPG', 'TD3'])
>>> add('seed', [0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
Parameters:

  • key (str) – Name of parameter.

  • vals (list or object) – Possible values for parameter.

  • shorthand (str, optional) – Shorthand for parameter.

  • in_name (bool, optional) – Whether to include this parameter in the experiment name.

Return type:

None

analyze(parameter, values=None, compare_num=None, cost_limit=None, show_image=False)[source]#

Analyze the experiment results.

Note

values and compare_num cannot be set at the same time.

Parameters:

  • parameter (str) – Name of parameter to analyze.

  • values (list[Any] or None, optional) – Specific values of attribute, if it is specified, will only compare values in it. Defaults to None.

  • compare_num (int or None, optional) – Number of values to compare, if it is specified, will combine any potential combination to compare. Defaults to None.

  • cost_limit (float or None, optional) – Value for one line showed on graph to indicate cost. Defaults to None.

  • show_image (bool) – Whether to show graph image in GUI windows.

Return type:

None

check_variant_vaild(variant)[source]#

Check if the variant is valid.

Parameters:

variant (dict[str, Any]) – Experiment variant to be checked.

Return type:

None

evaluate(num_episodes=10, cost_criteria=1.0)[source]#

Agent Evaluation.

Parameters:

  • num_episodes (int, optional) – Number of episodes to evaluate. Defaults to 10.

  • cost_criteria (float, optional) – Cost criteria for evaluation. Defaults to 1.0.

Return type:

None

print()[source]#

Print a helpful report about the experiment grid.

This function prints a helpful report about the experiment grid, including the name of the experiment grid, the parameters being varied, and the possible values for each parameter.

In Command Line:

===================== ExperimentGrid [test] runs over parameters: =====================
env_name                                [env]
['SafetyPointGoal1-v0', 'SafetyAntVelocity-v1']
algo                                    [algo]
['SAC', 'DDPG', 'TD3']
seed                                    [seed]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
Return type:

None

render(num_episodes=10, render_mode='rgb_array', camera_name='track', width=256, height=256)[source]#

Evaluate and render some episodes.

Parameters:

  • num_episodes (int, optional) – Number of episodes to render. Defaults to 10.

  • render_mode (str, optional) – Render mode, can be ‘rgb_array’, ‘depth_array’ or ‘human’. Defaults to ‘rgb_array’.

  • camera_name (str, optional) – Camera name, specify the camera which you use to capture images. Defaults to ‘track’.

  • width (int, optional) – The width of the rendered image. Defaults to 256.

  • height (int, optional) – The height of the rendered image. Defaults to 256.

Return type:

None

run(thunk, num_pool=1, parent_dir=None, is_test=False, gpu_id=None)[source]#

Run each variant in the grid with function ‘thunk’.

Note: ‘thunk’ must be either a callable function, or a string. If it is a string, it must be the name of a parameter whose values are all callable functions.

Uses call_experiment to actually launch each experiment, and gives each variant a name using self.variant_name().

Maintenance note: the args for ExperimentGrid.run should track closely to the args for call_experiment. However, seed is omitted because we presume the user may add it as a parameter in the grid.

Parameters:

  • thunk (Callable) – Function to be called.

  • num_pool (int, optional) – Number of processes to run in parallel. Defaults to 1.

  • parent_dir (str or None, optional) – Parent directory to save the experiment results. Defaults to None.

  • is_test (bool, optional) – Whether to run the experiment in test mode. Defaults to False.

  • gpu_id (list of int or None, optional) – List of GPU IDs to use. Defaults to None.

Return type:

None

save_grid_config()[source]#

Save experiment grid configurations as json.

Return type:

None

save_results(exp_names, variants, results)[source]#

Save results to a file.

Parameters:

  • exp_names (list of str) – List of experiment names.

  • variants (list[dict[str, Any]]) – List of experiment variants.

  • results (list) – List of experiment results.

Return type:

None

save_same_exps_config(exps_log_dir, variant)[source]#

Save experiment grid configurations as json.

Parameters:

  • exps_log_dir (str) – Experiment log directory.

  • variant (dict[str, Any]) – Experiment variant.

Return type:

None

update_dict(total_dict, item_dict)[source]#

Updater of multi-level dictionary.

This function is used to update the total dictionary with the item dictionary.

Parameters:

  • total_dict (dict[str, Any]) – Total dictionary.

  • item_dict (dict[str, Any]) – Item dictionary.

Return type:

None

variant_name(variant)[source]#

Given a variant (dict of valid param/value pairs), make an exp_name.

A variant’s name is constructed as the grid name (if you’ve given it one), plus param names (or shorthands if available) and values separated by underscores.

..warning::

if seed is a parameter, it is not included in the name.

Examples

>>> variant_name({'env_id': 'SafetyPointGoal1-v0', 'algo': 'SAC', 'seed': 0})
'SafetyPointGoal1-v0_SAC_0'
Parameters:

variant (dict[str, Any]) – Variant dictionary.

Returns:

Experiment name.

Return type:

str

variants()[source]#

Makes a list of dict, where each dict is a valid config in the grid.

There is special handling for variant parameters whose names take the form 'full:param:name'.

The colons are taken to indicate that these parameters should have a nested dict structure. For example, if there are two params,

Key

Val

'base:param:a'

1

'base:param:b'

2

'base:param:c'

3

'special:d'

4

'special:e'

5

the variant dict will have the structure

{
    'base': {
        'param': {
            'a': 1,
            'b': 2,
            'c': 3
        }
    },
    'special': {
        'd': 4,
        'e': 5
    }
}
Returns:

List of valid and not duplicate variants.

Return type:

list[dict[str, Any]]