From b9a7c5bdb02ac093ca39708d10da46d03592206f Mon Sep 17 00:00:00 2001 From: Jaime Cernuda Date: Sat, 28 Oct 2023 19:36:06 -0500 Subject: [PATCH] Hermes Wiki update --- docs/03-Hermes/{Home.md => 01-Hermes.md} | 14 +++++++------- .../01-Getting-Started.md} | 2 +- .../01-Performance-Analysis} | 18 +++++++++--------- .../3.-Hermes-Configuration.md | 0 .../01-Basic-Programming} | 0 .../01-Buffer-Organizer.md} | 0 .../02-Buffer-Pool.md} | 0 .../03-Buffering-Target.md} | 0 .../04-Data-Placement-Strategies.md} | 0 .../06-Data-Staging.md} | 0 .../07-Hermes-Runtime.md} | 0 .../08-Metadata-Manager.md} | 0 .../09-Prefetcher.md} | 0 .../10-Storage-Hierarchy.md} | 0 .../11-Tags.md} | 0 .../12-Topology-Manager.md} | 0 .../01-Adapters.md} | 0 .../01-Docker-Containers.md} | 0 .../01-Developer's-Guide.md} | 0 .../01-Doxygen.md} | 0 docs/03-Hermes/{6.-FAQ.md => 11-FAQ/01-FAQ.md} | 0 21 files changed, 17 insertions(+), 17 deletions(-) rename docs/03-Hermes/{Home.md => 01-Hermes.md} (89%) rename docs/03-Hermes/{1.-Getting-Started.md => 02-Getting-Started/01-Getting-Started.md} (99%) rename docs/03-Hermes/{2.-Performance-Analysis.md => 03-Performance-Analysis/01-Performance-Analysis} (95%) rename docs/03-Hermes/{ => 04-Hermes-Configuration}/3.-Hermes-Configuration.md (100%) rename docs/03-Hermes/{4.-Basic-Programming.md => 05-Basic-programming/01-Basic-Programming} (100%) rename docs/03-Hermes/{Buffer-Organizer.md => 06-Hermes-components/01-Buffer-Organizer.md} (100%) rename docs/03-Hermes/{Buffer-Pool.md => 06-Hermes-components/02-Buffer-Pool.md} (100%) rename docs/03-Hermes/{Buffering-Target.md => 06-Hermes-components/03-Buffering-Target.md} (100%) rename docs/03-Hermes/{Data-Placement-Strategies.md => 06-Hermes-components/04-Data-Placement-Strategies.md} (100%) rename docs/03-Hermes/{Data-Staging.md => 06-Hermes-components/06-Data-Staging.md} (100%) rename docs/03-Hermes/{Hermes-Runtime.md => 06-Hermes-components/07-Hermes-Runtime.md} (100%) rename docs/03-Hermes/{Metadata-Manager.md => 06-Hermes-components/08-Metadata-Manager.md} (100%) rename docs/03-Hermes/{Prefetcher.md => 06-Hermes-components/09-Prefetcher.md} (100%) rename docs/03-Hermes/{Storage-Hierarchy.md => 06-Hermes-components/10-Storage-Hierarchy.md} (100%) rename docs/03-Hermes/{Tags.md => 06-Hermes-components/11-Tags.md} (100%) rename docs/03-Hermes/{Topology-Manager.md => 06-Hermes-components/12-Topology-Manager.md} (100%) rename docs/03-Hermes/{5. Adapters.md => 07-Adapters/01-Adapters.md} (100%) rename docs/03-Hermes/{Docker-Containers.md => 08-Docker/01-Docker-Containers.md} (100%) rename docs/03-Hermes/{Developer's-Guide.md => 09-Developers-Guide/01-Developer's-Guide.md} (100%) rename docs/03-Hermes/{7. Doxygen.md => 10-Doxygen/01-Doxygen.md} (100%) rename docs/03-Hermes/{6.-FAQ.md => 11-FAQ/01-FAQ.md} (100%) diff --git a/docs/03-Hermes/Home.md b/docs/03-Hermes/01-Hermes.md similarity index 89% rename from docs/03-Hermes/Home.md rename to docs/03-Hermes/01-Hermes.md index f8d408f4..d5b8524e 100644 --- a/docs/03-Hermes/Home.md +++ b/docs/03-Hermes/01-Hermes.md @@ -3,7 +3,7 @@ [[/images/Hermes_hierachy.jpg|Deep Distributed Storage Hierarchy (DDSH)]] Consider an HPC cluster equipped with a deep, distributed [storage -hierarchy](Storage-Hierarchy) (DDSH), the bottom layer of +hierarchy](06-Hermes-components/10-Storage-Hierarchy.md) (DDSH), the bottom layer of which is typically a parallel file system (PFS). DDSH was introduced to boost or to at least improve the I/O (POSIX, MPI-IO, HDF5, ...) performance of applications performing poorly otherwise. Unfortunately, @@ -44,24 +44,24 @@ with the following characteristics: where in DDSH a given data item is best/well/optimally-placed at that point in time. - To that end, the system consists of the following major components: - - [Strategies and algorithms](./Data-Placement-Strategies) that + - [Strategies and algorithms](06-Hermes-components/04-Data-Placement-Strategies.md) that implement policies and facilitate data placement decisions. Speculative data placement for read operations is also known as - [prefetching](./Prefetcher). + [prefetching](06-Hermes-components/09-Prefetcher.md). - These strategies work with (dynamic) sets of [buffering - target](Buffering-Target)s and are applicable more + target](06-Hermes-components/03-Buffering-Target.md)s and are applicable more broadly. - The physical buffering resources are managed in a distributed - [buffer pool](Buffer-Pool) (see also [Batching + [buffer pool](06-Hermes-components/02-Buffer-Pool.md) (see also [Batching System](Batching-System)). - - [Buffer Organizer](./Buffer-Organizer) + - [Buffer Organizer](06-Hermes-components/01-Buffer-Organizer.md) - [Profiler](./Profiler) - To separate concerns and for portability, system buffers are **not** directly exposed to applications. There is a set of intermediate [primitives](Primitives) targeted by [adapters](./Adapters) for different I/O libraries. A - generic [metadata manager](./Metadata-Manager) (MDM), + generic [metadata manager](06-Hermes-components/08-Metadata-Manager.md) (MDM), supports the bookkeeping needs of the various components. - The whole system is deployed in a server-less fashion. diff --git a/docs/03-Hermes/1.-Getting-Started.md b/docs/03-Hermes/02-Getting-Started/01-Getting-Started.md similarity index 99% rename from docs/03-Hermes/1.-Getting-Started.md rename to docs/03-Hermes/02-Getting-Started/01-Getting-Started.md index 1d5afbd6..a1cb7810 100644 --- a/docs/03-Hermes/1.-Getting-Started.md +++ b/docs/03-Hermes/02-Getting-Started/01-Getting-Started.md @@ -261,7 +261,7 @@ daemon, `LD_PRELOAD` a Hermes adapter and set some environment variables. We spawn a daemon on each node, then run our app with the appropriate environment variables, similar to the process described -[above](./1.-Getting-Started#hermes-services-running-in-separate-process-as-a-daemon). +[above](01-Getting-Started.md#hermes-services-running-in-separate-process-as-a-daemon). ```bash HERMES_CONF_PATH=/absolute/path/to/hermes.yaml diff --git a/docs/03-Hermes/2.-Performance-Analysis.md b/docs/03-Hermes/03-Performance-Analysis/01-Performance-Analysis similarity index 95% rename from docs/03-Hermes/2.-Performance-Analysis.md rename to docs/03-Hermes/03-Performance-Analysis/01-Performance-Analysis index 3efa2d23..4621a8f1 100644 --- a/docs/03-Hermes/2.-Performance-Analysis.md +++ b/docs/03-Hermes/03-Performance-Analysis/01-Performance-Analysis @@ -30,7 +30,7 @@ First we perform a strong scaling study. 2. Processes vary between 1 and 48 3. Transfer size of 1MB -| ![run workloads](images/performance/ssd-scale.svg) | +| ![run workloads](../images/performance/ssd-scale.svg) | |:--:| |SSD strong scaling| @@ -45,7 +45,7 @@ we want to measure the impact of garbage collection and OS caching. 2. Processes fixed at 4 3. Transfer size of 1MB -| ![run workloads](images/performance/ssd-dset.svg) | +| ![run workloads](../images/performance/ssd-dset.svg) | |:--:| |SSD dataset size scaling| @@ -62,7 +62,7 @@ First we perform a strong scaling study. 2. Processes vary between 1 and 48 3. Transfer size of 1MB -| ![run workloads](images/performance/nvme-scale.svg) | +| ![run workloads](../images/performance/nvme-scale.svg) | |:--:| |NVMe strong scaling| @@ -77,7 +77,7 @@ we want to measure the impact of garbage collection and OS caching. 2. Processes fixed at 4 3. Transfer size of 1MB -| ![run workloads](images/performance/nvme-dset.svg) | +| ![run workloads](../images/performance/nvme-dset.svg) | |:--:| |NVMe dataset size scaling| @@ -99,7 +99,7 @@ processes to be between 1 and 48. Each case performs a total of 100GB of I/O with transfer sizes of 1MB using the Hermes native Put/Get API. This evaluation was conducted only over a single node. -| ![multi-core scaling (NVMe)](images/performance/multicore-nvme-scale.svg) | +| ![multi-core scaling (NVMe)](../images/performance/multicore-nvme-scale.svg) | |:--:| |NVMe dataset size scaling| @@ -128,7 +128,7 @@ updated based on its current value. In addition, each workload contains a "Load" phase which perform insert-only workloads. Unlike an update, insert replaces the entire record. -| ![load workloads](images/performance/ycsb-load.svg) | +| ![load workloads](../images/performance/ycsb-load.svg) | |:--:| |Performance of KVS for the LOAD phase of YCSB| @@ -139,7 +139,7 @@ KVS adapter, since insert operations replace data. In our KVS, a record directly to a single Put operation in the Hermes KVS. This demonstrates that Hermes can perform comparably to well-established in-memory KVS. -| ![run workloads](images/performance/ycsb-run.svg) | +| ![run workloads](../images/performance/ycsb-run.svg) | |:--:| |Performance of KVS for the RUN phase of YCSB| @@ -158,7 +158,7 @@ In this evaluation, we run a multi-tiered experiment using Hermes native API. The workload sequentially PUTs 10GB per-node. We ran this experiment with 16 nodes and 16 processes per node. The overall dataset size is 160GB. -| ![run workloads](images/performance/tiering.svg) | +| ![run workloads](../images/performance/tiering.svg) | |:--:| |Performance of Hermes for varying Tiers| @@ -183,7 +183,7 @@ synthetic workload. Each rank produces a total of 1GB of data, there are 16 ranks per node, and a total of 4 nodes. The total dataset size produced is 160GB. We use a hierarchical setup with RAM, NVMe, and SATA SSD. -| ![run workloads](images/performance/dpe.svg) | +| ![run workloads](../images/performance/dpe.svg) | |:--:| |Performance of Hermes for varying DPEs| diff --git a/docs/03-Hermes/3.-Hermes-Configuration.md b/docs/03-Hermes/04-Hermes-Configuration/3.-Hermes-Configuration.md similarity index 100% rename from docs/03-Hermes/3.-Hermes-Configuration.md rename to docs/03-Hermes/04-Hermes-Configuration/3.-Hermes-Configuration.md diff --git a/docs/03-Hermes/4.-Basic-Programming.md b/docs/03-Hermes/05-Basic-programming/01-Basic-Programming similarity index 100% rename from docs/03-Hermes/4.-Basic-Programming.md rename to docs/03-Hermes/05-Basic-programming/01-Basic-Programming diff --git a/docs/03-Hermes/Buffer-Organizer.md b/docs/03-Hermes/06-Hermes-components/01-Buffer-Organizer.md similarity index 100% rename from docs/03-Hermes/Buffer-Organizer.md rename to docs/03-Hermes/06-Hermes-components/01-Buffer-Organizer.md diff --git a/docs/03-Hermes/Buffer-Pool.md b/docs/03-Hermes/06-Hermes-components/02-Buffer-Pool.md similarity index 100% rename from docs/03-Hermes/Buffer-Pool.md rename to docs/03-Hermes/06-Hermes-components/02-Buffer-Pool.md diff --git a/docs/03-Hermes/Buffering-Target.md b/docs/03-Hermes/06-Hermes-components/03-Buffering-Target.md similarity index 100% rename from docs/03-Hermes/Buffering-Target.md rename to docs/03-Hermes/06-Hermes-components/03-Buffering-Target.md diff --git a/docs/03-Hermes/Data-Placement-Strategies.md b/docs/03-Hermes/06-Hermes-components/04-Data-Placement-Strategies.md similarity index 100% rename from docs/03-Hermes/Data-Placement-Strategies.md rename to docs/03-Hermes/06-Hermes-components/04-Data-Placement-Strategies.md diff --git a/docs/03-Hermes/Data-Staging.md b/docs/03-Hermes/06-Hermes-components/06-Data-Staging.md similarity index 100% rename from docs/03-Hermes/Data-Staging.md rename to docs/03-Hermes/06-Hermes-components/06-Data-Staging.md diff --git a/docs/03-Hermes/Hermes-Runtime.md b/docs/03-Hermes/06-Hermes-components/07-Hermes-Runtime.md similarity index 100% rename from docs/03-Hermes/Hermes-Runtime.md rename to docs/03-Hermes/06-Hermes-components/07-Hermes-Runtime.md diff --git a/docs/03-Hermes/Metadata-Manager.md b/docs/03-Hermes/06-Hermes-components/08-Metadata-Manager.md similarity index 100% rename from docs/03-Hermes/Metadata-Manager.md rename to docs/03-Hermes/06-Hermes-components/08-Metadata-Manager.md diff --git a/docs/03-Hermes/Prefetcher.md b/docs/03-Hermes/06-Hermes-components/09-Prefetcher.md similarity index 100% rename from docs/03-Hermes/Prefetcher.md rename to docs/03-Hermes/06-Hermes-components/09-Prefetcher.md diff --git a/docs/03-Hermes/Storage-Hierarchy.md b/docs/03-Hermes/06-Hermes-components/10-Storage-Hierarchy.md similarity index 100% rename from docs/03-Hermes/Storage-Hierarchy.md rename to docs/03-Hermes/06-Hermes-components/10-Storage-Hierarchy.md diff --git a/docs/03-Hermes/Tags.md b/docs/03-Hermes/06-Hermes-components/11-Tags.md similarity index 100% rename from docs/03-Hermes/Tags.md rename to docs/03-Hermes/06-Hermes-components/11-Tags.md diff --git a/docs/03-Hermes/Topology-Manager.md b/docs/03-Hermes/06-Hermes-components/12-Topology-Manager.md similarity index 100% rename from docs/03-Hermes/Topology-Manager.md rename to docs/03-Hermes/06-Hermes-components/12-Topology-Manager.md diff --git a/docs/03-Hermes/5. Adapters.md b/docs/03-Hermes/07-Adapters/01-Adapters.md similarity index 100% rename from docs/03-Hermes/5. Adapters.md rename to docs/03-Hermes/07-Adapters/01-Adapters.md diff --git a/docs/03-Hermes/Docker-Containers.md b/docs/03-Hermes/08-Docker/01-Docker-Containers.md similarity index 100% rename from docs/03-Hermes/Docker-Containers.md rename to docs/03-Hermes/08-Docker/01-Docker-Containers.md diff --git a/docs/03-Hermes/Developer's-Guide.md b/docs/03-Hermes/09-Developers-Guide/01-Developer's-Guide.md similarity index 100% rename from docs/03-Hermes/Developer's-Guide.md rename to docs/03-Hermes/09-Developers-Guide/01-Developer's-Guide.md diff --git a/docs/03-Hermes/7. Doxygen.md b/docs/03-Hermes/10-Doxygen/01-Doxygen.md similarity index 100% rename from docs/03-Hermes/7. Doxygen.md rename to docs/03-Hermes/10-Doxygen/01-Doxygen.md diff --git a/docs/03-Hermes/6.-FAQ.md b/docs/03-Hermes/11-FAQ/01-FAQ.md similarity index 100% rename from docs/03-Hermes/6.-FAQ.md rename to docs/03-Hermes/11-FAQ/01-FAQ.md