Deploying to gh-pages from @ 3c319cd 🚀

_sources/source_tree_docs/index.md.txt

@@ -10,25 +10,24 @@ MLOS is a project to enable autotuning for systems.
 
 ## Contents
 
-<!-- TOC -->
-
-- [MLOS](#mlos)
-    - [Contents](#contents)
-    - [Overview](#overview)
-    - [Organization](#organization)
-    - [Contributing](#contributing)
-    - [Getting Started](#getting-started)
-        - [conda activation](#conda-activation)
-        - [Usage Examples](#usage-examples)
-            - [mlos-core](#mlos-core)
-            - [mlos-bench](#mlos-bench)
-            - [mlos-viz](#mlos-viz)
-    - [Installation](#installation)
-    - [See Also](#see-also)
-        - [Examples](#examples)
-        - [Publications](#publications)
-
-<!-- /TOC -->
+<!-- mdformat-toc start --slug=github --no-anchors --maxlevel=4 --minlevel=2 -->
+
+- [Contents](#contents)
+- [Overview](#overview)
+- [Organization](#organization)
+- [Contributing](#contributing)
+- [Getting Started](#getting-started)
+  - [`conda` activation](#conda-activation)
+  - [Usage Examples](#usage-examples)
+    - [`mlos-core`](#mlos-core)
+    - [`mlos-bench`](#mlos-bench)
+    - [`mlos-viz`](#mlos-viz)
+- [Installation](#installation)
+- [See Also](#see-also)
+  - [Examples](#examples)
+  - [Publications](#publications)
+
+<!-- mdformat-toc end -->
 
 ## Overview
 
@@ -43,9 +42,12 @@ To accomplish this, the general flow involves
 - Apply that configuration to the target system.
 - Repeat until either the exploration budget is consumed or the configurations' performance appear to have converged.
 
-<!-- markdownlint-disable-next-line MD033 -->
+<!-- markdownlint-disable MD033 -->
+
 <img src="../_static/llamatune-loop.png" style="width:700px" alt="optimization loop" />
 
+<!-- markdownlint-enable MD033 -->
+
 > Source: [LlamaTune: VLDB 2022](https://arxiv.org/abs/2203.05128)
 
 For a brief overview of some of the features and capabilities of MLOS, please see the following video:
@@ -60,7 +62,7 @@ To do this this repo provides three Python modules, which can be used independen
 
 - [`mlos-viz`](https://github.com/microsoft/MLOS/tree/main/././mlos_viz/) provides some simple APIs to help automate visualizing the results of benchmark experiments and their trials.
 
-    It provides a simple `plot(experiment_data)` API, where `experiment_data` is obtained from the `mlos_bench.storage` module.
+  It provides a simple `plot(experiment_data)` API, where `experiment_data` is obtained from the `mlos_bench.storage` module.
 
 - [`mlos-core`](https://github.com/microsoft/MLOS/tree/main/././mlos_core/) provides an abstraction around existing optimization frameworks (e.g., [FLAML](https://github.com/microsoft/FLAML), [SMAC](https://github.com/automl/SMAC3), etc.)
 
@@ -92,26 +94,26 @@ For instructions on setting up the development environment please try one of the
 
 1. Create the `mlos` Conda environment.
 
-     ```sh
-    conda env create -f conda-envs/mlos.yml
-    ```
+   ```sh
+   conda env create -f conda-envs/mlos.yml
+   ```
 
-    > See the [`conda-envs/`](https://github.com/microsoft/MLOS/tree/main/././conda-envs/) directory for additional conda environment files, including those used for Windows (e.g. [`mlos-windows.yml`](https://github.com/microsoft/MLOS/tree/main/././conda-envs/mlos-windows.yml)).
+   > See the [`conda-envs/`](https://github.com/microsoft/MLOS/tree/main/././conda-envs/) directory for additional conda environment files, including those used for Windows (e.g. [`mlos-windows.yml`](https://github.com/microsoft/MLOS/tree/main/././conda-envs/mlos-windows.yml)).
 
    or
 
-    ```sh
-    # This will also ensure the environment is update to date using "conda env update -f conda-envs/mlos.yml"
-    make conda-env
-    ```
+   ```sh
+   # This will also ensure the environment is update to date using "conda env update -f conda-envs/mlos.yml"
+   make conda-env
+   ```
 
-    > Note: the latter expects a *nix environment.
+   > Note: the latter expects a \*nix environment.
 
 1. Initialize the shell environment.
 
-    ```sh
-    conda activate mlos
-    ```
+   ```sh
+   conda activate mlos
+   ```
 
 ### Usage Examples
 
@@ -186,7 +188,7 @@ Details on using a local version from git are available in [CONTRIBUTING.md](htt
 
 - [sqlite-autotuning](https://github.com/Microsoft-CISL/sqlite-autotuning)
 
-    Working example of tuning `sqlite` with MLOS.
+  Working example of tuning `sqlite` with MLOS.
 
 These can be used as starting points for new autotuning projects outside of the main MLOS repository if you want to keep your tuning experiment configs separate from the MLOS codebase.
 

_sources/source_tree_docs/mlos_bench/index.md.txt

@@ -10,59 +10,57 @@ It's available for `pip install` via the pypi repository at [mlos-bench](https:/
 
 <!-- markdownlint-disable MD007 -->
 
-<!-- TOC -->
-
-- [mlos-bench](#mlos-bench)
-    - [Table of Contents](#table-of-contents)
-    - [Description](#description)
-    - [Features](#features)
-    - [Quickstart](#quickstart)
-        - [Install and activate the conda environment](#install-and-activate-the-conda-environment)
-        - [Make sure that you have Azure CLI tool installed and working](#make-sure-that-you-have-azure-cli-tool-installed-and-working)
-        - [Generate access tokens to interact with Azure resources](#generate-access-tokens-to-interact-with-azure-resources)
-        - [Create a JSON config with DB credentials Optional](#create-a-json-config-with-db-credentials-optional)
-        - [Create a top-level configuration file for your MLOS setup](#create-a-top-level-configuration-file-for-your-mlos-setup)
-        - [Create another config file for the parameters specific to your experiment](#create-another-config-file-for-the-parameters-specific-to-your-experiment)
-            - [Importance of the Experiment ID config](#importance-of-the-experiment-id-config)
-        - [Run the benchmark](#run-the-benchmark)
-    - [Optimization](#optimization)
-        - [Resuming interrupted experiments](#resuming-interrupted-experiments)
-    - [Analyzing Results](#analyzing-results)
-    - [Debugging](#debugging)
-    - [See Also](#see-also)
-
-<!-- /TOC -->
-<!-- /TOC -->
+<!-- mdformat-toc start --slug=github --no-anchors --maxlevel=4 --minlevel=2 -->
+
+- [Table of Contents](#table-of-contents)
+- [Description](#description)
+- [Features](#features)
+- [Quickstart](#quickstart)
+  - [Install and activate the conda environment](#install-and-activate-the-conda-environment)
+  - [Make sure that you have Azure CLI tool installed and working](#make-sure-that-you-have-azure-cli-tool-installed-and-working)
+  - [Generate access tokens to interact with Azure resources](#generate-access-tokens-to-interact-with-azure-resources)
+  - [Create a JSON config with DB credentials (Optional)](#create-a-json-config-with-db-credentials-optional)
+  - [Create a top-level configuration file for your MLOS setup](#create-a-top-level-configuration-file-for-your-mlos-setup)
+  - [Create another config file for the parameters specific to your experiment](#create-another-config-file-for-the-parameters-specific-to-your-experiment)
+    - [Importance of the Experiment ID config](#importance-of-the-experiment-id-config)
+  - [Run the benchmark](#run-the-benchmark)
+- [Optimization](#optimization)
+  - [Resuming interrupted experiments](#resuming-interrupted-experiments)
+- [Analyzing Results](#analyzing-results)
+- [Debugging](#debugging)
+- [See Also](#see-also)
+
+<!-- mdformat-toc end -->
 
 <!-- markdownlint-enable MD007 -->
 
 ## Description
 
 `mlos-bench` is an end-to-end benchmarking service that can be independently launched for experimentation but is also integrated with `mlos-core` as its optimizer for OS tuning.
- Given a user-provided VM configuration, `mlos-bench` provisions a configured environment and remotely executes benchmarks on the cloud.
- Experiment results (benchmark results & telemetry) are stored as input to the `mlos-core` optimization engine loop to evaluate proposed configuration parameters and produce new results.
+Given a user-provided VM configuration, `mlos-bench` provisions a configured environment and remotely executes benchmarks on the cloud.
+Experiment results (benchmark results & telemetry) are stored as input to the `mlos-core` optimization engine loop to evaluate proposed configuration parameters and produce new results.
 
 ## Features
 
 With a [JSON5](https://spec.json5.org) [config file](https://github.com/microsoft/MLOS/tree/main/mlos_bench/./mlos_bench/config/) and command line parameters as input, `mlos-bench` streamlines workload performance measurement by automating the following benchmarking steps:
 
 1. Set up & clean up benchmark and application configuration
-    - **Ease of use:** Mlos-bench abstracts away controls for managing VMs in Azure, e.g., setup, teardown, stop, deprovision, and reboot. Get visibility into VM status through Azure Portal, ensuring that a VM is provisioned & running before issuing commands to it.
-    - **Versatility:** Mlos-bench provides a common interface to control a collection of environments (application, OS, VM), regardless of where or which cloud they come from. This allows changes to easily propagate to all environment layers when a new set of kernel parameters are applied.
-    - **Efficiency:** In adapting an environment to new parameters, mlos-bench optimizes for low re-configuration costs during optimization. For example, considering that not all OS kernel parameter adjustments require a full reboot, as some can be changed during run-time.
-2. Run benchmarks in the provisioned environment & standardize results for the optimizer
-    - Through Azure File Share, access docker scripts to run benchmarks & store results as input for optimization. For example, execute Redis benchmark uploaded to the file share, running a benchmark docker container with specified parameters. The file share is mounted to VMs via remote execution, instead of ARM templates.
-    - **Configurable:** Specify a python script in the initial config to post-process & standardize benchmark results. An example post-processing script for Redis benchmarks is included.
-    - **Local & remote benchmark execution:** Benchmarks can be run both locally in Hyper-V and remotely on Azure. Local execution allows better accuracy, while Azure runs are required to estimate the benchmark noise and understand the VM behavior when using cloud storage.
-    - **Cloud agnostic:** Mlos-bench can remotely execute benchmarks on other clouds, outside of Azure - e.g., controls for EC2 instances and ability to provision environments on AWS with Terraform.
-    - **Persistence:** Storage integration is available to persist experiment parameters and track results for re-use, either for analysis during & after trials, or warm-starting future experiments.
+   - **Ease of use:** Mlos-bench abstracts away controls for managing VMs in Azure, e.g., setup, teardown, stop, deprovision, and reboot. Get visibility into VM status through Azure Portal, ensuring that a VM is provisioned & running before issuing commands to it.
+   - **Versatility:** Mlos-bench provides a common interface to control a collection of environments (application, OS, VM), regardless of where or which cloud they come from. This allows changes to easily propagate to all environment layers when a new set of kernel parameters are applied.
+   - **Efficiency:** In adapting an environment to new parameters, mlos-bench optimizes for low re-configuration costs during optimization. For example, considering that not all OS kernel parameter adjustments require a full reboot, as some can be changed during run-time.
+1. Run benchmarks in the provisioned environment & standardize results for the optimizer
+   - Through Azure File Share, access docker scripts to run benchmarks & store results as input for optimization. For example, execute Redis benchmark uploaded to the file share, running a benchmark docker container with specified parameters. The file share is mounted to VMs via remote execution, instead of ARM templates.
+   - **Configurable:** Specify a python script in the initial config to post-process & standardize benchmark results. An example post-processing script for Redis benchmarks is included.
+   - **Local & remote benchmark execution:** Benchmarks can be run both locally in Hyper-V and remotely on Azure. Local execution allows better accuracy, while Azure runs are required to estimate the benchmark noise and understand the VM behavior when using cloud storage.
+   - **Cloud agnostic:** Mlos-bench can remotely execute benchmarks on other clouds, outside of Azure - e.g., controls for EC2 instances and ability to provision environments on AWS with Terraform.
+   - **Persistence:** Storage integration is available to persist experiment parameters and track results for re-use, either for analysis during & after trials, or warm-starting future experiments.
 
 ## Quickstart
 
 To get started, we can adapt an example configuration to test out running `mlos-bench`.
 For these instructions, we will be using Azure for our resources.
 
-### 1. Install and activate the conda environment
+### Install and activate the conda environment
 
 From here onwards we assume we are in the project root directory.
 Ensure you have a conda environment (`mlos`) set up for executing `mlos_bench`.
@@ -75,7 +73,7 @@ conda activate mlos
 
 > Note: if you are running inside the devcontainer, this should be done automatically.
 
-### 2. Make sure that you have Azure CLI tool installed and working
+### Make sure that you have Azure CLI tool installed and working
 
 > Installation instructions for `az` (Azure CLI) [can be found here](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli).
 >
@@ -94,7 +92,7 @@ az config set storage.account=MyStorageAccount --local
 az account set --subscription "..."
 ```
 
-### 3. Generate access tokens to interact with Azure resources
+### Generate access tokens to interact with Azure resources
 
 A script at `./scripts/generate-azure-credentials-config` produces a JSON config snippet with necessary Azure credentials.
 
@@ -114,7 +112,7 @@ This data produced in the `global_config_azure.jsonc` file is in the format that
 
 > Note: On Linux, this script also requires `jq` to also be installed (comes preinstalled in the devcontainer).
 
-### 4. Create a JSON config with DB credentials (Optional)
+### Create a JSON config with DB credentials (Optional)
 
 If you plan to store the information about experiments and benchmarks in a (remote) database like PostgreSQL or MySQL, create a JSON/JSONC file with the DB hostname and password.
 See [`mysql.jsonc`](https://github.com/microsoft/MLOS/tree/main/mlos_bench/./mlos_bench/config/storage/mysql.jsonc) or [`postgresql.jsonc`](https://github.com/microsoft/MLOS/tree/main/mlos_bench/./mlos_bench/config/storage/postgresql.jsonc) configuration files for examples with a more complete list of DB parameters supported by underlying the [SqlAlchemy](https://www.sqlalchemy.org/library.html#reference) library.
@@ -136,7 +134,7 @@ See [`sqlite.jsonc`](https://github.com/microsoft/MLOS/tree/main/mlos_bench/./ml
 
 > Note: if no storage is specified, a basic sqlite config will be used by default.
 
-### 5. Create a top-level configuration file for your MLOS setup
+### Create a top-level configuration file for your MLOS setup
 
 We provide a few examples of such files in [`./mlos_bench/config/cli/`](https://github.com/microsoft/MLOS/tree/main/mlos_bench/./mlos_bench/config/cli).
 For example, [`azure-redis-opt.jsonc`](https://github.com/microsoft/MLOS/tree/main/mlos_bench/./mlos_bench/config/cli/azure-redis-opt.jsonc) is a configuration for optimizing Redis VM on Azure and saving the results in a local SQLite database.
@@ -159,7 +157,7 @@ For the purpose of this tutorial, we will assume that we reuse the existing [`az
 In a more realistic scenario, however, you might need to change and/or create new config files for some parts of your benchmarking environment.
 We'll give more details on that below.
 
-### 5. Create another config file for the parameters specific to your experiment
+### Create another config file for the parameters specific to your experiment
 
 Copy one of our examples, e.g., [`experiment_RedisBench.jsonc`](https://github.com/microsoft/MLOS/tree/main/mlos_bench/./mlos_bench/config/experiments/experiment_RedisBench.jsonc) and name it after your experiment, e.g. `experiment_MyBenchmark.jsonc`.
 
@@ -177,7 +175,7 @@ As such this value should be unique for each experiment.
 Be sure to change it whenever *"incompatible"* changes are made to the experiment configuration or scripts.
 Unfortunately, determining what constitutes and *"incompatible"* change for any given system is not always possible, so `mlos_bench` largely leaves this up to the user.
 
-### 6. Run the benchmark
+### Run the benchmark
 
 Now we can run our configuration with `mlos_bench`:
 
@@ -230,6 +228,7 @@ For example:
 
 ```python
 from mlos_bench.storage import from_config
+
 # Specify the experiment_id used for your experiment.
 experiment_id = "YourExperimentId"
 trial_id = 1

_sources/source_tree_docs/mlos_core/index.md.txt

@@ -15,16 +15,16 @@ These can be evaluated by [`mlos-bench`](https://github.com/microsoft/MLOS/tree/
 Since the tunable parameter search space is often extremely large, `mlos-core` automates the following steps to efficiently generate optimal task-specific kernel and application configurations.
 
 1. Reduce the search space by identifying a promising set of tunable parameters
-    - Map out the configuration search space: Automatically track and manage the discovery of new Linux kernel parameters and their default values across versions.
-    Filter out non-tunable parameters (e.g., not writable) and track which kernel parameters exist for a given kernel version.
-    - Leverage parameter knowledge for optimization: Information on ranges, sampling intervals, parameter correlations, workload type sensitivities for tunable parameters are tracked and currently manually curated.
-    In the future, this can be automatically maintained by scraping documentation pages on kernel parameters.
-    - Tailored to application: Consider prior knowledge of the parameter's impact & an application's workload profile (e.g. network heavy, disk heavy, CPU bound, multi-threaded, latency sensitive, throughput oriented, etc.) to identify likely impactful candidates of tunable parameters, specific to a particular application.
-2. Sampling to warm-start optimization in a high dimensional search space
-3. Produce optimal configurations through Bayesian optimization
-    - Support for various optimizer algorithms (default Bayesian optimizer, Flaml, SMAC, and random for baseline comparison), that handle multiple types of constraints.
-    This includes cost-aware optimization, that considers experiment costs given current tunable parameters.
-    - Integrated with `mlos-bench`, proposed configurations are logged and evaluated.
+   - Map out the configuration search space: Automatically track and manage the discovery of new Linux kernel parameters and their default values across versions.
+     Filter out non-tunable parameters (e.g., not writable) and track which kernel parameters exist for a given kernel version.
+   - Leverage parameter knowledge for optimization: Information on ranges, sampling intervals, parameter correlations, workload type sensitivities for tunable parameters are tracked and currently manually curated.
+     In the future, this can be automatically maintained by scraping documentation pages on kernel parameters.
+   - Tailored to application: Consider prior knowledge of the parameter's impact & an application's workload profile (e.g. network heavy, disk heavy, CPU bound, multi-threaded, latency sensitive, throughput oriented, etc.) to identify likely impactful candidates of tunable parameters, specific to a particular application.
+1. Sampling to warm-start optimization in a high dimensional search space
+1. Produce optimal configurations through Bayesian optimization
+   - Support for various optimizer algorithms (default Bayesian optimizer, Flaml, SMAC, and random for baseline comparison), that handle multiple types of constraints.
+     This includes cost-aware optimization, that considers experiment costs given current tunable parameters.
+   - Integrated with `mlos-bench`, proposed configurations are logged and evaluated.
 
 ## See Also