V0.1.0

.gitignore

@@ -1,8 +1,13 @@
+machine
+tmp/
 # Generated by Cargo
 # will have compiled files and executables
 debug/
 target/
 
+# Files generated for benchmarking gnark memory
+frameworks/gnark/tmp
+
 # Remove Cargo.lock from gitignore if creating an executable, leave it for libraries
 # More information here https://doc.rust-lang.org/cargo/guide/cargo-toml-vs-cargo-lock.html
 Cargo.lock
@@ -55,7 +60,7 @@ __pycache__/
 *.so
 
 # Distribution / packaging
-bin/
+./bin/
 build/
 develop-eggs/
 dist/
@@ -106,6 +111,14 @@ pipenv/
 node_modules/
 
 # Gnark symlinks
-gnark/gnark
-gnark/template
+frameworks/gnark/gnark
+frameworks/gnark/groth16*
+frameworks/gnark/plonk*
+frameworks/gnark/template
 .idea/
+
+tags/
+
+# Vim
+*.*.swp
+*tags

.gitmodules

@@ -0,0 +1,6 @@
+[submodule "circom/rapidsnark"]
+	path = frameworks/circom/rapidsnark
+	url = https://github.com/iden3/rapidsnark.git
+[submodule "benchmarks"]
+	path = benchmarks
+	url = https://github.com/zkCollective/bench-data.git

ARCHITECTURE.md

@@ -0,0 +1,156 @@
+# Architecture
+
+![Alt text](./architecture.png?raw=true "Title")
+
+Our benchmarking framework is designed to support all types of different frameworks.
+We specify a generic set of interfaces, such that benchmarks can be invoked through 
+a configuration file `config.json`, which produces a standardized output 
+for a given benchmarking scenario.
+
+## Overview
+
+On a high level, zk-Harness takes as input a configuration file. 
+The “Config Reader” reads the standardized config and invokes the ZKP framework 
+as specified in the configuration file. You can find a description of the 
+configuration file below. Each integrated ZKP framework exposes a runner 
+that take as an input the standardized configuration parameters to execute the 
+corresponding benchmarks. The output of benchmarking a given ZKP framework 
+is a log file in csv format with standardized metrics. 
+The log file is read by the “Log Analyzer”, 
+which compiles the logs into pandas dataframes that are used by the front-end and displayed on the public website. You can find the standardized logging 
+format in the following.
+
+### Config.json
+
+The ``config.json`` file contains information about what benchmarks to execute when invoking the zk-Harness. The following describes how to run benchmarks given a specific config file. Further, we specify each of the keys in a common config file and their expected values.
+
+## Running benchmarks for a specific config file
+
+Given a config file, you can run `python3 -m src.reader --config path/to/config.json` to run the benchmarks defined by a given config file.
+
+## config.json key specification
+
+### `project`
+
+The name of the project being benchmarked.
+
+### `project_url`
+
+The URL(s) to the repository for the project.
+
+### `category`
+
+The category for which the zk-Harness should benchmark a ZKP-framework. 
+Currently, it supports only `circuit`.
+
+### `count`
+
+The number of runs over which a given computation specified by the config should be averaged.
+This option is taken under consideration only if the benchmarking is performed
+manually and not through a benchmarking framework such as criterion.
+
+### `payload`
+
+`payload` specifies the exact algorithms to benchmark. 
+  - `backend`
+    The backend algorithm(s) to use for proving the specified circuit(s).
+  - `curves`
+    The curve(s) for which the ZKP-framework should be benchmarked.
+  - `circuits`
+    The name of the circuit to benchmark. 
+    Equivalent circuits across frameworks should have the same naming scheme for 
+    ease of comparison.
+    If a new circuit is added, which does not yet exist in any framework, 
+    one should create a new input specification in the 
+    `input/circuit/<circuit_name>/input_<input_related_identifier>.json`.
+  - `algorithm`
+    The algorithm to execute. Note that not all algorithms exist for every framework.
+    Valid algorithms to execute in a given framework are currently:
+    - `compile`
+    - `setup`
+    - `witness`
+    - `prove`
+    - `verify`
+
+    If a given algorithm is not specified for the configured framework, 
+    the execution will fail.
+
+## Logs
+
+In the following sections, we describe the columns in the CSV file for each benchmark category.
+Note that for "Field Arithmetic" and "Elliptic Curve Group Operations", the logs
+are generated by the front-end and used only for displaying data in the UI.
+
+### Circuits
+
+The following information is recorded for each circuit benchmark:
+
+* framework: the name of the framework (e.g., gnark)
+* category: the category of the benchmark (i.e., circuit)
+* backend: the backend used (e.g., groth16)
+* curve: the curve used (e.g., bn256)
+* circuit: the circuit being run
+* input: file path of the input used 
+* operation: the step being measured -- compile, witness, setup, prove, verify 
+* nbConstraints: the number of constraints in the circuit
+* nbSecret: number of secret inputs
+* nbPublic: number of public inputs
+* ram: memory consumed in **bytes**
+* time: elapsed time in **milliseconds**
+* proofSize: the size of the proof in **bytes** -- empty value when `operation != prove`
+* count: number of times that we run the benchmark
+
+### Field Arithmetic
+
+The following information is recorded for each field arithmetic benchmark:
+
+* framework: the name of the framework (e.g., gnark)
+* category: the category of the benchmark (i.e., arithmetic)
+* curve: the curve of which field we use
+* field: the benchmarked field (base or scalar)
+* operation: the operation performed (add, sub, mul, inv, exp)
+* time: elapsed time in **nanoseconds**
+* count: number of times that we run the benchmark
+
+### Elliptic Curve Group Operations
+
+The following information is recorded for each elliptic curve group operation benchmark:
+
+* framework: the name of the framework (e.g., gnark)
+* category: the category of the benchmark (i.e., ec)
+* curve: the benchmarked curve
+* operation: the operation performed -- `g1-scalar-multiplication`, `g2-scalar-multiplication`, `g1-multi-scalar-multiplication`, `g2-multi-scalar-multiplication`, `pairing`
+* input: Provided input
+* time: elapsed time in **nanoseconds**
+* count: number of times that we run the benchmark
+
+## Adding a new framework to the zk-Harness
+
+To integrate a framework, one should follow the following steps:
+
+1. First, fork the `zk-harness` repository
+2. Create a `frameworks/<framework_name>` folder in the root folder of the repository.
+3. Create a custom benchmarking script that *(i)* reads from the standardized 
+input of the `config.json` and outputs *(ii)* the standardized logs.
+    * For example, benchmarking for `gnark` is done through a custom CLI, 
+    based on [cobra](https://github.com/spf13/cobra)
+    * Your script should be able to take a variety of arguments as specified in 
+    the `config.json`, such that benchmarks can be easily executed and extended. 
+    For example, a common command in the gnark integration would be 
+    `./gnark groth16 --circuit=sha256 --input=_input/circuit/sha256/input_3.json --curve=bn254`
+4. Modify the `src/reader/process_circuit.py` scripts to include your newly 
+created script as described in step 3, which is called if the `project` field of 
+the respective config contains the `<framework_name>` of your newly added ZKP framework.
+  * The `src/reader/process_circuit.py` processing python scripts are invoked by 
+  `__main__.py`.
+5. Create a documentation in the `frameworks/<framework_name>/README.md` 
+to outline how others can include new circuits / benchmarks for another circuit 
+in the framework. 
+6. Add config files for running the benchmarks in `input/config/<framework_name>` 
+and add make rules for the new framework in the Makefile.
+
+If you follow the specified interfaces for config and logs, 
+your framework specific benchmarking should seamlessly integrate into the zk-Harness frontend.
+
+Once finished, please create a Pull Request and assign it to one of the maintainers 
+for review.

INSTALL.md

@@ -0,0 +1,74 @@
+# Installation Instructions
+
+To run most of the benchmarks, you will need to have the following software installed on your system:
+
+1. Rust: Rust is a systems programming language that is used by some of the benchmarks. You can install Rust by following the instructions provided on the official website: https://www.rust-lang.org/tools/install.
+2. Go: Go is a programming language that is used by some of the benchmarks. You can install Go by following the instructions provided on the official website: https://golang.org/doc/install.
+3. Node.js: Node.js is a JavaScript runtime environment that is used by some of the benchmarks. You can install Node.js by following the instructions provided on the official website: https://nodejs.org.
+4. Cargo Criterion: Cargo Criterion is a benchmarking library for Rust. To install Cargo Criterion, open a terminal and run the following command:
+
+```bash
+cargo install cargo-criterion
+```
+
+This will install Cargo Criterion and its dependencies.
+
+
+Once you have installed the above software, you should be ready to run most of the benchmarks. 
+
+## Additional Dependencies
+
+For running benchmarks for the following projects, you should follow the specific instructions.
+
+### Circom
+
+Install Circom compiler following the instructions here: https://docs.circom.io/getting-started/installation/#installing-dependencies.
+You also need to install [jq](https://jqlang.github.io/jq/).
+
+To install `jq` you can use `brew`, `apt`, or `yum`.
+
+Furthermore, if you are going to run the full (or larger) benchmarks you will need
+a larger powers of TAU file. The default one is downloaded from the 
+[Hermez ceremony](https://github.com/iden3/snarkjs#7-prepare-phase-2)
+and it is `powersOfTau28_hez_final_16.ptau` saved into
+`frameworks/circom/phase1/powersOfTau28_final.ptau`.
+If you want to benchmark larger circuits we suggest downloading the TAU file
+of power of 24 which is sufficient for the current benchmarks.
+Note that this file needs 18 GB of disk space.
+
+```
+wget -O frameworks/circom/phase1/powersOfTau28_final.ptau  https://hermez.s3-eu-west-1.amazonaws.com/powersOfTau28_hez_final_24.ptau
+```
+
+* Using SNARKJS
+
+```
+npm install snarkjs
+```
+
+* Using rapidsnark
+
+Note that this would only work in Intel64
+
+```
+cd frameworks/circom
+git submodule init && git submodule update
+cd rapidsnark
+# Following the instructions in the README
+```
+
+### ffiasm
+
+Note that this would only work in Intel64
+
+* Linux (ubuntu)
+
+```
+apt install libgmp-dev nasm
+```
+
+* Mac
+
+```
+brew install gmp nasm
+```