Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add kinesis integration testing #75

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open

Add kinesis integration testing #75

wants to merge 5 commits into from
+607 −0

Conversation

qhool
Copy link
Contributor

@qhool qhool commented Jul 7, 2023

  • docker compose.yml creates a testing environment: * Use kinesalite and dynalite (github.com/mhart) to provide kinesis and dynamodb work-alikes * Create ca-certificate and server cert signed with this ca, so that kinesalite and dynalite can pose as the real deals * Insert the ca-cert into trusted certificats for the test container * Create dns aliases for standard aws endpoints
  • New consume_SUITE.erl and load_from_dir script enable testing against real AWS or using the docker compose env
  • Add Makefile to hold the required commands to start tests under docker

@qhool
Add kinesis integration testing
c0d7d9c 
  * docker compose.yml creates a testing environment:
    * Use kinesalite and dynalite (github.com/mhart) to provide
      kinesis and dynamodb work-alikes
    * Create ca-certificate and server cert signed with this ca,
      so that kinesalite and dynalite can pose as the real deals
    * Insert the ca-cert into trusted certificats for the test container
    * Create dns aliases for standard aws endpoints
  * New consume_SUITE.erl and load_from_dir script enable testing against
    real AWS or using the docker compose env
  * Add Makefile to hold the required commands to start tests under docker
raz-adroll
raz-adroll reviewed Jul 7, 2023
View reviewed changes
Copy link
Contributor

@raz-adroll raz-adroll left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm on a running it in my mac and I get:

error: failed switching to "work_user": unable to find user work_user: no matching entries in passwd file

I highly recommend documenting the setup in the README.

test/compose/ca/cacert.sh
openssl genrsa -des3 -passout pass:foobar -out ca/ca.key $KEY_SIZE || exit 1

openssl req -new -x509 -passin pass:foobar \
-subj "/C=US/ST=CA/O=ACME/CN=acme.fake/[email protected]" \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I got that reference
image

test/compose/dynamo/Dockerfile Outdated Show resolved Hide resolved
test/compose/test_runner/Dockerfile Outdated Show resolved Hide resolved
test/bin/dump_to_dir Outdated Show resolved Hide resolved
@qhool
Copy link
Contributor Author

qhool commented Jul 7, 2023

I'm on a running it in my mac and I get:

error: failed switching to "work_user": unable to find user work_user: no matching entries in passwd file

What command did you use exactly?

I highly recommend documenting the setup in the README.

I'll add something once I understand what's going on.. for me the setup was apt install docker-ce then make test; obviously that lacks .. something?

qhool added 3 commits July 7, 2023 12:22
@qhool
Remove dead code and appease hank
4a25c42
@qhool
Fix format
016a7d2
@qhool
Work around Docker desktop
5ba2891 
  Docker desktop uses a utility vm to host the containers.
  In addition to making things slower, this has other advantages:
  for instance, by running as root and mapping file ownership to root,
  it can break workflows that assume the docker user is unpriviledged.
@raz-adroll
Copy link
Contributor

Almost working, but I still got this error while I was running the last commit.

test-tests-1    | ===> Running Common Test suites...
test-tests-1    | %%% consume_SUITE:
test-tests-1    | =INFO REPORT==== 7-Jul-2023::20:59:02.262740 ===
test-tests-1    | erlmld_sup_simple_consume_test listening on 40593
test-tests-1    |
test-tests-1    | =INFO REPORT==== 7-Jul-2023::20:59:02.265542 ===
test-tests-1    | erlmld_runner_simple_consume_test launched "/home/work/copy/_build/test/lib/erlmld/priv/run_kinesis.sh" (pid 1136)
test-tests-1    |
test-tests-1    | =SUPERVISOR REPORT==== 7-Jul-2023::20:59:02.608564 ===
test-tests-1    |     supervisor: {local,erlmld_sup_simple_consume_test}
test-tests-1    |     errorContext: child_terminated
test-tests-1    |     reason: {child_exited,1136,{exit_status,256}}
test-tests-1    |     offender: [{pid,<0.2132.0>},
test-tests-1    |                {id,mld_runner},
test-tests-1    |                {mfargs,{erlmld_runner,start_link,
test-tests-1    |                                       [erlmld_runner_simple_consume_test,
test-tests-1    |                                        "/tmp/erlmld/erlmld.simple_consume_test.properties",
test-tests-1    |                                        kinesis]}},
test-tests-1    |                {restart_type,permanent},
test-tests-1    |                {significant,false},
test-tests-1    |                {shutdown,brutal_kill},
test-tests-1    |                {child_type,worker}]
test-tests-1    |
test-tests-1    | =INFO REPORT==== 7-Jul-2023::20:59:02.609100 ===
test-tests-1    | erlmld_runner_simple_consume_test launched "/home/work/copy/_build/test/lib/erlmld/priv/run_kinesis.sh" (pid 1157)
test-tests-1    |
test-tests-1    | =SUPERVISOR REPORT==== 7-Jul-2023::20:59:02.966878 ===
test-tests-1    |     supervisor: {local,erlmld_sup_simple_consume_test}
test-tests-1    |     errorContext: child_terminated
test-tests-1    |     reason: {child_exited,1157,{exit_status,256}}
test-tests-1    |     offender: [{pid,<0.2135.0>},
test-tests-1    |                {id,mld_runner},
test-tests-1    |                {mfargs,{erlmld_runner,start_link,
test-tests-1    |                                       [erlmld_runner_simple_consume_test,
test-tests-1    |                                        "/tmp/erlmld/erlmld.simple_consume_test.properties",
test-tests-1    |                                        kinesis]}},
test-tests-1    |                {restart_type,permanent},
test-tests-1    |                {significant,false},
test-tests-1    |                {shutdown,brutal_kill},
test-tests-1    |                {child_type,worker}]
test-tests-1    |
test-tests-1    | =INFO REPORT==== 7-Jul-2023::20:59:02.967439 ===
test-tests-1    | erlmld_runner_simple_consume_test launched "/home/work/copy/_build/test/lib/erlmld/priv/run_kinesis.sh" (pid 1178)
test-tests-1    |
test-tests-1    | =SUPERVISOR REPORT==== 7-Jul-2023::20:59:03.307353 ===
test-tests-1    |     supervisor: {local,erlmld_sup_simple_consume_test}
test-tests-1    |     errorContext: child_terminated
test-tests-1    |     reason: {child_exited,1178,{exit_status,256}}
test-tests-1    |     offender: [{pid,<0.2138.0>},
test-tests-1    |                {id,mld_runner},
test-tests-1    |                {mfargs,{erlmld_runner,start_link,
test-tests-1    |                                       [erlmld_runner_simple_consume_test,
test-tests-1    |                                        "/tmp/erlmld/erlmld.simple_consume_test.properties",
test-tests-1    |                                        kinesis]}},
test-tests-1    |                {restart_type,permanent},
test-tests-1    |                {significant,false},
test-tests-1    |                {shutdown,brutal_kill},
test-tests-1    |                {child_type,worker}]
test-tests-1    |
test-tests-1    | =INFO REPORT==== 7-Jul-2023::20:59:03.307840 ===
test-tests-1    | erlmld_runner_simple_consume_test launched "/home/work/copy/_build/test/lib/erlmld/priv/run_kinesis.sh" (pid 1199)
test-tests-1    |
test-tests-1    | =SUPERVISOR REPORT==== 7-Jul-2023::20:59:03.652304 ===
test-tests-1    |     supervisor: {local,erlmld_sup_simple_consume_test}
test-tests-1    |     errorContext: child_terminated
test-tests-1    |     reason: {child_exited,1199,{exit_status,256}}
test-tests-1    |     offender: [{pid,<0.2141.0>},
test-tests-1    |                {id,mld_runner},
test-tests-1    |                {mfargs,{erlmld_runner,start_link,
test-tests-1    |                                       [erlmld_runner_simple_consume_test,
test-tests-1    |                                        "/tmp/erlmld/erlmld.simple_consume_test.properties",
test-tests-1    |                                        kinesis]}},
test-tests-1    |                {restart_type,permanent},
test-tests-1    |                {significant,false},
test-tests-1    |                {shutdown,brutal_kill},
test-tests-1    |                {child_type,worker}]
test-tests-1    |
test-tests-1    | =INFO REPORT==== 7-Jul-2023::20:59:03.652876 ===
test-tests-1    | erlmld_runner_simple_consume_test launched "/home/work/copy/_build/test/lib/erlmld/priv/run_kinesis.sh" (pid 1220)
test-tests-1    |
test-tests-1    | =SUPERVISOR REPORT==== 7-Jul-2023::20:59:04.028469 ===
test-tests-1    |     supervisor: {local,erlmld_sup_simple_consume_test}
test-tests-1    |     errorContext: child_terminated
test-tests-1    |     reason: {child_exited,1220,{exit_status,256}}
test-tests-1    |     offender: [{pid,<0.2144.0>},
test-tests-1    |                {id,mld_runner},
test-tests-1    |                {mfargs,{erlmld_runner,start_link,
test-tests-1    |                                       [erlmld_runner_simple_consume_test,
test-tests-1    |                                        "/tmp/erlmld/erlmld.simple_consume_test.properties",
test-tests-1    |                                        kinesis]}},
test-tests-1    |                {restart_type,permanent},
test-tests-1    |                {significant,false},
test-tests-1    |                {shutdown,brutal_kill},
test-tests-1    |                {child_type,worker}]
test-tests-1    |
test-tests-1    | =INFO REPORT==== 7-Jul-2023::20:59:04.029812 ===
test-tests-1    | erlmld_runner_simple_consume_test launched "/home/work/copy/_build/test/lib/erlmld/priv/run_kinesis.sh" (pid 1241)
test-tests-1    |
test-tests-1    | =SUPERVISOR REPORT==== 7-Jul-2023::20:59:04.417206 ===
test-tests-1    |     supervisor: {local,erlmld_sup_simple_consume_test}
test-tests-1    |     errorContext: child_terminated
test-tests-1    |     reason: {child_exited,1241,{exit_status,256}}
test-tests-1    |     offender: [{pid,<0.2147.0>},
test-tests-1    |                {id,mld_runner},
test-tests-1    |                {mfargs,{erlmld_runner,start_link,
test-tests-1    |                                       [erlmld_runner_simple_consume_test,
test-tests-1    |                                        "/tmp/erlmld/erlmld.simple_consume_test.properties",
test-tests-1    |                                        kinesis]}},
test-tests-1    |                {restart_type,permanent},
test-tests-1    |                {significant,false},
test-tests-1    |                {shutdown,brutal_kill},
test-tests-1    |                {child_type,worker}]
test-tests-1    |
test-tests-1    | =INFO REPORT==== 7-Jul-2023::20:59:04.417784 ===
test-tests-1    | erlmld_runner_simple_consume_test launched "/home/work/copy/_build/test/lib/erlmld/priv/run_kinesis.sh" (pid 1262)
test-tests-1    |
test-tests-1    | =SUPERVISOR REPORT==== 7-Jul-2023::20:59:04.764537 ===
test-tests-1    |     supervisor: {local,erlmld_sup_simple_consume_test}
test-tests-1    |     errorContext: child_terminated
test-tests-1    |     reason: {child_exited,1262,{exit_status,256}}
test-tests-1    |     offender: [{pid,<0.2150.0>},
test-tests-1    |                {id,mld_runner},
test-tests-1    |                {mfargs,{erlmld_runner,start_link,
test-tests-1    |                                       [erlmld_runner_simple_consume_test,
test-tests-1    |                                        "/tmp/erlmld/erlmld.simple_consume_test.properties",
test-tests-1    |                                        kinesis]}},
test-tests-1    |                {restart_type,permanent},
test-tests-1    |                {significant,false},
test-tests-1    |                {shutdown,brutal_kill},
test-tests-1    |                {child_type,worker}]
test-tests-1    |
test-tests-1    | =INFO REPORT==== 7-Jul-2023::20:59:04.765446 ===
test-tests-1    | erlmld_runner_simple_consume_test launched "/home/work/copy/_build/test/lib/erlmld/priv/run_kinesis.sh" (pid 1283)
test-tests-1    |
test-tests-1    | =SUPERVISOR REPORT==== 7-Jul-2023::20:59:05.152821 ===
test-tests-1    |     supervisor: {local,erlmld_sup_simple_consume_test}
test-tests-1    |     errorContext: child_terminated
test-tests-1    |     reason: {child_exited,1283,{exit_status,256}}
test-tests-1    |     offender: [{pid,<0.2153.0>},
test-tests-1    |                {id,mld_runner},
test-tests-1    |                {mfargs,{erlmld_runner,start_link,
test-tests-1    |                                       [erlmld_runner_simple_consume_test,
test-tests-1    |                                        "/tmp/erlmld/erlmld.simple_consume_test.properties",
test-tests-1    |                                        kinesis]}},
test-tests-1    |                {restart_type,permanent},
test-tests-1    |                {significant,false},
test-tests-1    |                {shutdown,brutal_kill},
test-tests-1    |                {child_type,worker}]
test-tests-1    |
test-tests-1    | =INFO REPORT==== 7-Jul-2023::20:59:05.153767 ===
test-tests-1    | erlmld_runner_simple_consume_test launched "/home/work/copy/_build/test/lib/erlmld/priv/run_kinesis.sh" (pid 1304)
test-tests-1    |
test-tests-1    | =SUPERVISOR REPORT==== 7-Jul-2023::20:59:05.495754 ===
test-tests-1    |     supervisor: {local,erlmld_sup_simple_consume_test}
test-tests-1    |     errorContext: child_terminated
test-tests-1    |     reason: {child_exited,1304,{exit_status,256}}
test-tests-1    |     offender: [{pid,<0.2156.0>},
test-tests-1    |                {id,mld_runner},
test-tests-1    |                {mfargs,{erlmld_runner,start_link,
test-tests-1    |                                       [erlmld_runner_simple_consume_test,
test-tests-1    |                                        "/tmp/erlmld/erlmld.simple_consume_test.properties",
test-tests-1    |                                        kinesis]}},
test-tests-1    |                {restart_type,permanent},
test-tests-1    |                {significant,false},
test-tests-1    |                {shutdown,brutal_kill},
test-tests-1    |                {child_type,worker}]
test-tests-1    |
test-tests-1    | =INFO REPORT==== 7-Jul-2023::20:59:05.496381 ===
test-tests-1    | erlmld_runner_simple_consume_test launched "/home/work/copy/_build/test/lib/erlmld/priv/run_kinesis.sh" (pid 1325)
test-tests-1    |
test-tests-1    | =SUPERVISOR REPORT==== 7-Jul-2023::20:59:05.838265 ===
test-tests-1    |     supervisor: {local,erlmld_sup_simple_consume_test}
test-tests-1    |     errorContext: child_terminated
test-tests-1    |     reason: {child_exited,1325,{exit_status,256}}
test-tests-1    |     offender: [{pid,<0.2159.0>},
test-tests-1    |                {id,mld_runner},
test-tests-1    |                {mfargs,{erlmld_runner,start_link,
test-tests-1    |                                       [erlmld_runner_simple_consume_test,
test-tests-1    |                                        "/tmp/erlmld/erlmld.simple_consume_test.properties",
test-tests-1    |                                        kinesis]}},
test-tests-1    |                {restart_type,permanent},
test-tests-1    |                {significant,false},
test-tests-1    |                {shutdown,brutal_kill},
test-tests-1    |                {child_type,worker}]
test-tests-1    |
test-tests-1    | =INFO REPORT==== 7-Jul-2023::20:59:05.838710 ===
test-tests-1    | erlmld_runner_simple_consume_test launched "/home/work/copy/_build/test/lib/erlmld/priv/run_kinesis.sh" (pid 1346)
test-tests-1    |
test-tests-1    | =SUPERVISOR REPORT==== 7-Jul-2023::20:59:06.211647 ===
test-tests-1    |     supervisor: {local,erlmld_sup_simple_consume_test}
test-tests-1    |     errorContext: child_terminated
test-tests-1    |     reason: {child_exited,1346,{exit_status,256}}
test-tests-1    |     offender: [{pid,<0.2162.0>},
test-tests-1    |                {id,mld_runner},
test-tests-1    |                {mfargs,{erlmld_runner,start_link,
test-tests-1    |                                       [erlmld_runner_simple_consume_test,
test-tests-1    |                                        "/tmp/erlmld/erlmld.simple_consume_test.properties",
test-tests-1    |                                        kinesis]}},
test-tests-1    |                {restart_type,permanent},
test-tests-1    |                {significant,false},
test-tests-1    |                {shutdown,brutal_kill},
test-tests-1    |                {child_type,worker}]
test-tests-1    |
test-tests-1    | =SUPERVISOR REPORT==== 7-Jul-2023::20:59:06.211692 ===
test-tests-1    |     supervisor: {local,erlmld_sup_simple_consume_test}
test-tests-1    |     errorContext: shutdown
test-tests-1    |     reason: reached_max_restart_intensity
test-tests-1    |     offender: [{pid,<0.2162.0>},
test-tests-1    |                {id,mld_runner},
test-tests-1    |                {mfargs,{erlmld_runner,start_link,
test-tests-1    |                                       [erlmld_runner_simple_consume_test,
test-tests-1    |                                        "/tmp/erlmld/erlmld.simple_consume_test.properties",
test-tests-1    |                                        kinesis]}},
test-tests-1    |                {restart_type,permanent},
test-tests-1    |                {significant,false},
test-tests-1    |                {shutdown,brutal_kill},
test-tests-1    |                {child_type,worker}]
test-tests-1    |
test-tests-1    |
test-tests-1    | %%% consume_SUITE ==> simple_consume_test: FAILED
test-tests-1    | %%% consume_SUITE ==> {'EXIT',shutdown}
test-tests-1    |
test-tests-1    | EXPERIMENTAL: Writing retry specification at /home/work/copy/_build/test/logs/retry.spec
test-tests-1    |               call rebar3 ct with '--retry' to re-run failing cases.
test-tests-1    | Failed 1 tests. Passed 0 tests.
test-tests-1    | Results written to "/home/work/copy/_build/test/logs/index.html".
test-tests-1    | ===> Failures occurred running tests: 1

pbrudnick
pbrudnick reviewed Jul 10, 2023
View reviewed changes
test/compose.yml
depends_on:
ca:
condition: service_healthy
domainname: kinesis.us-east-1.amazaonaws.com
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo here

Suggested change
domainname: kinesis.us-east-1.amazaonaws.com
domainname: kinesis.us-east-1.amazonaws.com

@tak30
Copy link
Contributor

tak30 commented Jul 11, 2023

Please do not merge this PR before it has been tested in mac arm and mac x86. As soon as we get it working on x86 I'll test it in arm. Thaaanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@raz-adroll raz-adroll raz-adroll left review comments

@pbrudnick pbrudnick pbrudnick left review comments

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@qhool @raz-adroll @tak30 @pbrudnick