Code

Add interface for NoSQL storage#214

Code

Draft

mcopik wants to merge 114 commits intomasterspcl/serverless-benchmarks:masterfrom feature/nosql_storagespcl/serverless-benchmarks:feature/nosql_storageCopy head branch name to clipboard

+3,327-400

Commits

Commits on Jul 26, 2024

• [system] Add basic interface for allocating NoSQL storage

  mcopik

  committed
  0363899

  Copy full SHA for 0363899

Commits on Jul 27, 2024

• [aws] Implement allocation of DynamoDB tables

  mcopik

  committed
  dd3384c

  Copy full SHA for dd3384c

• [system] Allocate NoSQL tables for the selected benchmark

  mcopik

  committed
  847534f

  Copy full SHA for 847534f

• [system] Remove debug printouts

  mcopik

  committed
  6a62633

  Copy full SHA for 6a62633

• [aws] Add first version of DynamoDB wrapper

  mcopik

  committed
  1c30372

  Copy full SHA for 1c30372

• [azure] First implementation of wrapper for CosmosDB

  mcopik

  committed
  ed22ea3

  Copy full SHA for ed22ea3

• [aws] Update AWS wrapper with query interface

  mcopik

  committed
  b21cd34

  Copy full SHA for b21cd34

• [gcp] Add wrapper for google cloud datastore

  mcopik

  committed
  e85c348

  Copy full SHA for e85c348

• [system] Add packages needed to support storage API

  mcopik

  committed
  f28b496

  Copy full SHA for f28b496

Commits on Jul 28, 2024

• [aws] Formatting AWS nosql wrapper

  mcopik

  committed
  a1cd286

  Copy full SHA for a1cd286

• [system] Split system packages into benchmark modules

  mcopik

  committed
  9b3a284

  Copy full SHA for 9b3a284

• [aws] Support sorting keys in DynamoDB

  mcopik

  committed
  3f3b440

  Copy full SHA for 3f3b440

• [system] Fix incorrect variable name

  mcopik

  committed
  8ca8e24

  Copy full SHA for 8ca8e24

• [system] Adapt cache implementation of NoSQL to support different layouts across platforms

  mcopik

  committed
  1fdf04f

  Copy full SHA for 1fdf04f

• [azure] Implement CosmosDB management

  mcopik

  committed
  2cd1bc4

  Copy full SHA for 2cd1bc4

• [azure] Add management of CosmosDB accoutns

  mcopik

  committed
  f90971f

  Copy full SHA for f90971f

• [azure] Install locally CosmosDB library to allocate containers

  mcopik

  committed
  020fcd6

  Copy full SHA for 020fcd6

• [azure] Add missing handler file

  mcopik

  committed
  6959cd6

  Copy full SHA for 6959cd6

• [azure] Allocate clients to databases and containers when they already exist

  mcopik

  committed
  f089017

  Copy full SHA for f089017

• [system] Improved logging

  mcopik

  committed
  d6e14de

  Copy full SHA for d6e14de

• [azure] Export NoSQL database information

  mcopik

  committed
  0c7906a

  Copy full SHA for 0c7906a

• [system] Initialize storage before deploying function

  mcopik

  committed
  8314f41

  Copy full SHA for 8314f41

• [azure] Help mypy recognize that variable is never None

  mcopik

  committed
  b975d78

  Copy full SHA for b975d78

• [azure] Linting nosql wrapper

  mcopik

  committed
  b9ae6ca

  Copy full SHA for b9ae6ca

• [azure] Export the proper database name as env

  mcopik

  committed
  53bb33e

  Copy full SHA for 53bb33e

Commits on Jul 29, 2024

• [azure] Export tables as envs

  mcopik

  committed
  8d211a2

  Copy full SHA for 8d211a2

• [azure] Fix error of allocating the same container many times

  mcopik

  committed
  43680a1

  Copy full SHA for 43680a1

• [azure] Avoid creating duplicated HTTP triggers

  mcopik

  committed
  cf6fc22

  Copy full SHA for cf6fc22

• [aws] Remove Decimals from DynamoDB results

  mcopik

  committed
  f49714c

  Copy full SHA for f49714c

• [aws] Append environment variables to send Table data

  mcopik

  committed
  36dd13f

  Copy full SHA for 36dd13f

• [aws] Linting

  mcopik

  committed
  4312ebc

  Copy full SHA for 4312ebc

• [azure] Prevent overwriting existing variables

  mcopik

  committed
  ec62b9c

  Copy full SHA for ec62b9c

• [azure Add sleep to avoid problems with stale values of envs

  mcopik

  committed
  ab62ec9

  Copy full SHA for ab62ec9

• [gcp] Add Google Cloud manage image with gcloud

  mcopik

  committed
  e36cf80

  Copy full SHA for e36cf80

• [system] [aws] Move table management to each implementation to adjust for different hierarchies

  mcopik

  committed
  69fe546

  Copy full SHA for 69fe546

• [azure] Remove mapping of tables

  mcopik

  committed
  50f8ac6

  Copy full SHA for 50f8ac6

• [system] Fix bug in cache processing - now storage is updated before code package

  mcopik

  committed
  b836597

  Copy full SHA for b836597

• [gcp] Add gcloud CLI handling

  mcopik

  committed
  853d083

  Copy full SHA for 853d083

• [system] Linting

  mcopik

  committed
  89560ca

  Copy full SHA for 89560ca

• [gcp] Extend Docker images

  mcopik

  committed
  e8288a0

  Copy full SHA for e8288a0

• [gcp] Update wrapper with database init

  mcopik

  committed
  4a944a0

  Copy full SHA for 4a944a0

• [gcp] Update docs with permissions

  mcopik

  committed
  e511885

  Copy full SHA for e511885

• [aws] Ensure we update envs

  mcopik

  committed
  ce9076b

  Copy full SHA for ce9076b

• [gcp] Add management of datastore instances

  mcopik

  committed
  8204715

  Copy full SHA for 8204715

• [gcp] Correctly mount read-only files in CLI

  mcopik

  committed
  8e66ca3

  Copy full SHA for 8e66ca3

• [gcp] Update function envs to support NoSQL storage

  mcopik

  committed
  994f107

  Copy full SHA for 994f107

• [benchmarks] Add initial config for 130.crud-api benchmark

  mcopik

  committed
  22333a5

  Copy full SHA for 22333a5

• [benchmarks] Update module configuration

  mcopik

  committed
  820c5b4

  Copy full SHA for 820c5b4

• [azure] [gcp] Fix nosql import

  mcopik

  committed
  85de381

  Copy full SHA for 85de381

• [azure] Move storage connection string from request to config

  mcopik

  committed
  7f23fe2

  Copy full SHA for 7f23fe2

• [system] Update benchmark input config

  mcopik

  committed
  2af6082

  Copy full SHA for 2af6082

• [gcp] Add missing dependency

  mcopik

  committed
  92c7ce6

  Copy full SHA for 92c7ce6

• [experiments] Reorder input processing and function generation

  mcopik

  committed
  ed00c14

  Copy full SHA for ed00c14

• [whisk] Skip nosql

  mcopik

  committed
  12b475d

  Copy full SHA for 12b475d

• [gcp] Fix mypy issues

  mcopik

  committed
  06ece22

  Copy full SHA for 06ece22

Commits on Jul 30, 2024

• [benchmarks] Add test implementation of the CRUD benchmark

  mcopik

  committed
  7118b58

  Copy full SHA for 7118b58

Commits on Jul 31, 2024

• [benchmarks] Work in progress on the CRUD benchmark

  mcopik

  committed
  6c78152

  Copy full SHA for 6c78152

• [storage] Adapt interface to multiple tiers of storage

  mcopik

  committed
  3d58e42

  Copy full SHA for 3d58e42

• [local] Adapt local to the new storage config

  mcopik

  committed
  045f077

  Copy full SHA for 045f077

• [system] First attempt to break the circular dependency between System and Benchmark

  Show description for 1c48a10
  mcopik

  committed
  1c48a10

  Copy full SHA for 1c48a10

• [local] Adapt new resource style

  mcopik

  committed
  b8b04f1

  Copy full SHA for b8b04f1

• [system] Expand storage configuration

  mcopik

  committed
  c9e2635

  Copy full SHA for c9e2635

• [system] remove dbg output

  mcopik

  committed
  e9c954b

  Copy full SHA for e9c954b

• [minio] Support data volume and explicit version

  mcopik

  committed
  f4bd76f

  Copy full SHA for f4bd76f

• [minio] Fix bug in setting volume path

  mcopik

  committed
  afa5d7e

  Copy full SHA for afa5d7e

• [scylla] Add ScyllaDB as a local NoSQL backend

  mcopik

  committed
  a0c88ad

  Copy full SHA for a0c88ad

• [scylla] Add stop for the container

  mcopik

  committed
  898328f

  Copy full SHA for 898328f

• [aws] Restructurize storage allocation

  mcopik

  committed
  18f4692

  Copy full SHA for 18f4692

• [benchmarks] Updated version of 130.crud-api

  mcopik

  committed
  67fa42b

  Copy full SHA for 67fa42b

• [system] Add initialization of NoSQL results

  mcopik

  committed
  da0e009

  Copy full SHA for da0e009

• [aws] Implement initialization of DynamoDB

  mcopik

  committed
  f953768

  Copy full SHA for f953768

• [system] Fix cache issue - we now prepare benchmark data before code package

  mcopik

  committed
  e5d72b8

  Copy full SHA for e5d72b8

• [benchmarks] Update input API

  mcopik

  committed
  dc60969

  Copy full SHA for dc60969

Commits on Aug 1, 2024

• [system] Additional logging

  mcopik

  committed
  ef54402

  Copy full SHA for ef54402

• [benchmarks] Update input API

  mcopik

  committed
  ff9314c

  Copy full SHA for ff9314c

• [benchmarks] Fix incorrect IDs and division by zero

  mcopik

  committed
  c6463e2

  Copy full SHA for c6463e2

• [bechmarks] Implement final sizes for 130.crud-api

  mcopik

  committed
  9916028

  Copy full SHA for 9916028

Commits on Aug 2, 2024

• [aws] Implement the update method for DynamoDB

  mcopik

  committed
  9dc5d03

  Copy full SHA for 9dc5d03

• [aws] Fix waiting condition when creating DynamoDB table

  mcopik

  committed
  078bdbc

  Copy full SHA for 078bdbc

• [gcp] Implement new resource class

  mcopik

  committed
  3d7d1dc

  Copy full SHA for 3d7d1dc

• [gcp] Implement upload function for Datastore

  mcopik

  committed
  51fafd4

  Copy full SHA for 51fafd4

• [gcp] Add wrapper for update function

  mcopik

  committed
  0843b00

  Copy full SHA for 0843b00

• [gcp] Fix incorrect update implementation

  mcopik

  committed
  1a7325f

  Copy full SHA for 1a7325f

• [gcp] Fix linting issues

  mcopik

  committed
  23ed66b

  Copy full SHA for 23ed66b

• [azure] Move resources to a different file

  mcopik

  committed
  d277923

  Copy full SHA for d277923

• [azure] Linting

  mcopik

  committed
  ff1a696

  Copy full SHA for ff1a696

• [azure] Implement the missing function for CosmosDB

  mcopik

  committed
  1077605

  Copy full SHA for 1077605

• [azure] Add the new abstraction of system resources

  mcopik

  committed
  6758863

  Copy full SHA for 6758863

• [azure] Replace insert with upsert

  mcopik

  committed
  b9dfc92

  Copy full SHA for b9dfc92

• [azure] Add NoSQL update

  mcopik

  committed
  383a62b

  Copy full SHA for 383a62b

Commits on Aug 3, 2024

• [azure] Fix allocation of database client

  mcopik

  committed
  ecc2816

  Copy full SHA for ecc2816

Commits on Aug 7, 2024

• [local] Add base methods for local storage config

  mcopik

  committed
  0e1e1fc

  Copy full SHA for 0e1e1fc

• [local] Make storage paths consistent

  mcopik

  committed
  945b0fd

  Copy full SHA for 945b0fd

• [local] Drop unused storage config in output

  mcopik

  committed
  be85773

  Copy full SHA for be85773

• [local] Ignore Docker volumes in git

  mcopik

  committed
  7bb8d87

  Copy full SHA for 7bb8d87

• [system] Allow for multiple input files with storage definition

  mcopik

  committed
  a5977b7

  Copy full SHA for a5977b7

• [local] Better error description

  mcopik

  committed
  c168c0f

  Copy full SHA for c168c0f

• [system] Remove debug printout

  mcopik

  committed
  2caa564

  Copy full SHA for 2caa564

• [local] Make storage implementation cacheable and more generic by offloading Minio-specific code

  mcopik

  committed
  2e6f75a

  Copy full SHA for 2e6f75a

• [local] Final standardization of volume names

  mcopik

  committed
  92d610b

  Copy full SHA for 92d610b

• [storage] Remove unnecessary storage classes

  mcopik

  committed
  3ac12e7

  Copy full SHA for 3ac12e7

• [system] Fix type hint

  mcopik

  committed
  a76c7d9

  Copy full SHA for a76c7d9

• [local] Support ScyllaDB serialization

  mcopik

  committed
  0a678b5

  Copy full SHA for 0a678b5

• [local] Linting

  mcopik

  committed
  cc9c103

  Copy full SHA for cc9c103

• [system] Support removing functions that are no longer available

  mcopik

  committed
  cf66c2c

  Copy full SHA for cf66c2c

• [aws] Linting

  mcopik

  committed
  8db684c

  Copy full SHA for 8db684c

• [storage] Implement allocation of ScyllaDB tables

  mcopik

  committed
  223f368

  Copy full SHA for 223f368

• [storage] Fix correct deployment name in output

  mcopik

  committed
  124cf79

  Copy full SHA for 124cf79

• [storage] Linting

  mcopik

  committed
  ecc5cbf

  Copy full SHA for ecc5cbf

• [local] Export NoSQL settings

  mcopik

  committed
  dae8ec1

  Copy full SHA for dae8ec1

• [storage] Add default implementation of envs for NoSQL

  mcopik

  committed
  1796ca8

  Copy full SHA for 1796ca8

• [local] Add ScyllaDB wrapper

  mcopik

  committed
  e504674

  Copy full SHA for e504674

• [local] Implement function update for local container

  mcopik

  committed
  0642eaa

  Copy full SHA for 0642eaa

• [storage] Linting

  mcopik

  committed
  b293f7a

  Copy full SHA for b293f7a