WIP: Docker integ test with async API #1003

normanj-bitquill · 2024-12-31T18:44:29Z

Description

Update the integration test docker stack to support the OpenSearch Async API and using Minio as an S3 storage engine. Also includes having everything configured on startup.

Related Issues

#992

Check List

Updated documentation (docs/ppl-lang/README.md)
Implemented unit tests
Implemented tests for combination with other commands
New added source code should include a copyright header
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Norman Jordan <[email protected]>

normanj-bitquill · 2024-12-31T18:49:05Z

This is not complete yet. Still need to:

Create a replacement for EMRServerlessClient that will use the local docker environment. This will include starting a docker container of Spark to run the query. Given that the idea is to start the container with spark-submit, it should be possible to easily change to the Spark EMR image instead.
Alter the Python integration tests to use the OpenSearch Async API instead
Remove the spark and spark-worker containers since Spark would be run on demand

YANG-DB · 2024-12-31T19:48:10Z

@normanj-bitquill I've also started working on some similar PR with Iceberg based docker-compose -
lets see if we can share knowledge ...

normanj-bitquill · 2025-01-07T23:19:29Z

Current Status

In my testing, I have added a second OpenSearch container. This resolves issues with the cluster and indices going into yellow state.

Changes in OpenSearch node:

Bind the docker socket
Replace the aws-java-sdk-emrserverless-*.jar file with on that uses a different EMRClient. The new EMRClient writes a file to a directory (/tmp/docker) with the arguments for running docker.
A script is run on the container that reads files from /tmp/docker and runs docker with the arguments.
Disable system indices and system indices permissions
Connect with second OpenSearch container

What works:

Initiate query with aysnc-api on OpenSearch container
Starts a new Spark docker container to call spark-submit using information from the start job request that the EMRClient received
New container connects to spark cluster to execute the job
Creates the request and result indices
Saves the result to the result index

What is missing:

Detecting that the job has finished (this is on the spark-submit container)
The query is for a table on an S3 data source, query fails with table not found

Can submit an async query and the result is written to the result index. Need to create the external table in Spark before submitting the query Signed-off-by: Norman Jordan <[email protected]>

normanj-bitquill · 2025-01-11T00:00:19Z

@YANG-DB I have partially working async API in the latest commit. These are my testing steps:

Start the cluster using docker compose up
Run spark-shell on the master spark container.
1. Create an external table with a location under s3a://integ-test/
2. Put some data in the table

Submit a query:

curl -u '...' -X POST -H 'Content-Type: application/json' -d '{"datasource": "mys3", "lang": 
"sql", "query": "SELECT * FROM mys3.default.foo"}

Retrieve the result (will need to wait until it is ready, maybe about 1 minute)

curl -u '...' -X POST -H 'Content-Type: application/json' -d '{}' 'http://localhost:9200/query_execution_result_mys3/_search?pretty'

The OpenSearch container will need to bind the docker socket /var/run/docker.sock. By default this isn't available on Mac, but can be enabled.

The OpenSearch container will start another container to process the async query. This is the place where we could slip in the EMR spark container (if there is any value from it).

normanj-bitquill · 2025-01-11T00:02:13Z

I have not tested retrieving results using the Async API. This is likely broken, since cannot check on the EMR job status. I also haven't tested a streaming query (also likely broken).

YANG-DB · 2025-01-11T05:13:44Z

docker/integ-test/docker-compose.yml

@@ -8,29 +23,35 @@ services:
      - "${UI_PORT:-4040}:4040"
      - "${SPARK_CONNECT_PORT}:15002"
    entrypoint: /opt/bitnami/scripts/spark/master-entrypoint.sh
+    user: root


@normanj-bitquill why is this mandatory ?

YANG-DB · 2025-01-11T05:16:42Z

docker/integ-test/opensearch/docker-command-runner.sh

can you plz explain what is the issue here we need to solve ?

missing license header

YANG-DB · 2025-01-11T05:17:13Z

...t/opensearch/emr-src/com/amazonaws/services/emrserverless/AWSEMRServerlessClientBuilder.java

@@ -0,0 +1,29 @@
+package com.amazonaws.services.emrserverless;


missing license header

YANG-DB · 2025-01-11T05:18:07Z

docker/integ-test/metastore/Dockerfile

missing license header

YANG-DB · 2025-01-11T05:18:31Z

docker/integ-test/opensearch/Dockerfile

missing license header

YANG-DB · 2025-01-11T05:18:44Z

docker/integ-test/opensearch/docker-command-runner.sh

missing license header

YANG-DB · 2025-01-11T05:18:55Z

...eg-test/opensearch/emr-src/org/opensearch/spark/emrserverless/DockerEMRServerlessClient.java

missing license header

YANG-DB · 2025-01-11T05:24:53Z

docker/integ-test/opensearch/opensearch-docker-it-entrypoint.sh

+
+su opensearch ./opensearch-docker-entrypoint.sh "$@"
+
+kill -TERM `cat /var/run/docker-command-runner.pid`


why is this called ?
can you plz explain the entire flow for the docker compose ?

YANG-DB · 2025-01-11T05:27:06Z

docker/integ-test/spark/spark-master-entrypoint.sh

@@ -0,0 +1,88 @@
+#!/bin/bash


this is nice - can you add some documentation of what is done here ?

WIP: Docker integ test with async API

d5052a6

Signed-off-by: Norman Jordan <[email protected]>

normanj-bitquill requested review from dai-chen, mengweieric, penghuo, seankao-az, anirudha, kaituo, YANG-DB, noCharger, LantaoJin and ykmr1224 as code owners December 31, 2024 18:44

normanj-bitquill mentioned this pull request Dec 31, 2024

(RFC) Integration Tests on Apache Spark and Spark EMR #992

Open

YANG-DB approved these changes Dec 31, 2024

View reviewed changes

YANG-DB mentioned this pull request Jan 4, 2025

Introduce basic sanity test for MV used by Observability Integrations #995

Open

5 tasks

YANG-DB added the testing test related feature label Jan 5, 2025

YANG-DB mentioned this pull request Jan 5, 2025

[FEATURE] Understandable logical documentation #1008

Open

Partially have Async API working

ca30131

Can submit an async query and the result is written to the result index. Need to create the external table in Spark before submitting the query Signed-off-by: Norman Jordan <[email protected]>

YANG-DB reviewed Jan 11, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: Docker integ test with async API #1003

WIP: Docker integ test with async API #1003

normanj-bitquill commented Dec 31, 2024

normanj-bitquill commented Dec 31, 2024

YANG-DB commented Dec 31, 2024

normanj-bitquill commented Jan 7, 2025

normanj-bitquill commented Jan 11, 2025

normanj-bitquill commented Jan 11, 2025

YANG-DB Jan 11, 2025

YANG-DB Jan 11, 2025

YANG-DB Jan 11, 2025

YANG-DB Jan 11, 2025

YANG-DB Jan 11, 2025

YANG-DB Jan 11, 2025

YANG-DB Jan 11, 2025

YANG-DB Jan 11, 2025

YANG-DB Jan 11, 2025

YANG-DB Jan 11, 2025

		@@ -0,0 +1,29 @@
		package com.amazonaws.services.emrserverless;


		su opensearch ./opensearch-docker-entrypoint.sh "$@"

		kill -TERM `cat /var/run/docker-command-runner.pid`

WIP: Docker integ test with async API #1003

Are you sure you want to change the base?

WIP: Docker integ test with async API #1003

Conversation

normanj-bitquill commented Dec 31, 2024

Description

Related Issues

Check List

normanj-bitquill commented Dec 31, 2024

YANG-DB commented Dec 31, 2024

normanj-bitquill commented Jan 7, 2025

Current Status

normanj-bitquill commented Jan 11, 2025

normanj-bitquill commented Jan 11, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment