Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

1085 runhistory documentation #1175

Merged
merged 7 commits into from
Dec 2, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions docs/3_getting_started.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,7 @@ from smac import Scenario

scenario = Scenario(
configspace=cs,
name="experiment_name",
output_directory=Path("your_output_directory")
walltime_limit=120, # Limit to two minutes
n_trials=500, # Evaluated max 500 trials
Expand All @@ -83,9 +84,15 @@ scenario = Scenario(
)
```

!!! note
If no `name` is given, a hash of the experiment is used. Running the same experiment again at a later time will result in exactly the same hash. This is important, because the optimization will warmstart on the preexisting evaluations, if not otherwise specified in the [Facade][smac.facade.abstract_facade].


## Facade

!!! warn
By default Facades will try to warmstart on preexisting logs. This behavior can be specified using the `overwrite` parameter.

A [facade][smac.facade.abstract_facade] is the entry point to SMAC, which constructs a default optimization
pipeline for you. SMAC offers various facades, which satisfy many common use cases and are crucial to
achieving peak performance. The idea behind the facades is to provide a simple interface to all of SMAC's components,
Expand Down
107 changes: 107 additions & 0 deletions docs/advanced_usage/8_logging.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,113 @@ The table shows you the specific levels:
| 40 | ERROR |
| 50 | CRITICAL |

## Standard Logging Files

By default, SMAC generates several files to document the optimization process. These files are stored in the directory structure `./output_directory/name/seed`, where name is replaced by a hash if no name is explicitly provided. This behavior can be customized through the [Scenario][smac.scenario] configuration, as shown in the example below:
```python
Scenario(
configspace = some_configspace,
name = 'experiment_name',
output_directory = Path('some_directory'),
...
)
```
Notably, if an output already exists at `./some_directory/experiment_name/seed`, the behavior is determined by the overwrite parameter in the [facade's][smac/facade/abstract_facade] settings. This parameter specifies whether to continue the previous run (default) or start a new run.

The output is split into four different log files, and a copy of the utilized [Configuration Space of the ConfigSpace library](https://automl.github.io/ConfigSpace/latest/).

### intensifier.json
The [intensification][Intensification] is logged in `intensifier.json` and has the following structure:

```json
{
"incumbent_ids": [
65
],
"rejected_config_ids": [
1,
],
"incumbents_changed": 2,
"trajectory": [
{
"config_ids": [
1
],
"costs": [
0.45706284046173096
],
"trial": 1,
"walltime": 0.029736042022705078
},
#...
],
"state": {
"tracker": {},
"next_bracket": 0
}
}
```

### optimization.json
The optimization process is portrayed in `optimization.json` with the following structure

```json
{
"used_walltime": 184.87366724014282,
"used_target_function_walltime": 20.229533672332764,
"last_update": 1732703596.5609574,
"finished": false
}
```
### runhistory.json
The runhistory.json in split into four parts. `stats`, `data`, `configs`, and `config_origins`.
`stats` contains overall broad stats on the different evaluated configurations:
```json
"stats": {
"submitted": 73,
"finished": 73,
"running": 0
},
```

`data` contains a list of entries, one for each configuration.
```json
"data": [
[
1, # config_id
null, # instance or None
209652396, # seed or None
null, # budget or None
5.4345623938566385, # cost
6.699562072753906e-05, # time
6.299999999992423e-05, # cpu_time
1, # status
1733133181.2144582, # start_time
1733133181.21695, # end_time
{} # additional_info
],
...
]
```

`configs` is a human-readable dictionary of configurations, where the keys are the one-based `config_id`. It is important to note that in `runhistory.json`, the indexing is zero-based.
```json
"configs": {
"1": {
"x": -2.3312147893012
},
```

Lastly, `config_origins` specifies the source of a configuration, indicating whether it stems from the initial design or results from the maximization of an acquisition function.
```json
"config_origins": {
"1": "Initial Design: Sobol",
...
}
```

### scenario.json
The ´scenario.json´ file contains the overall state of the [Scenario][smac.scenario] logged to a json file.

## Custom File

Expand Down
Loading