Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding HA health-check auto-heal, Zero Downtime Deployments #136

Merged
merged 2 commits into from
Nov 5, 2024
Merged

Adding HA health-check auto-heal, Zero Downtime Deployments #136

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
42564e2
Adding high availability Setup, Proxy, health-checks & auto-healing
mhmdksh Nov 5, 2024
cc8562d
Adding Zero Downtime deployments
mhmdksh Nov 5, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions .env.template
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,2 +1,3 @@
MY_URL="https://mydomain.com"
IP_WHITELIST="0.0.0.0/0"
MY_URL="https://mydomain.com" # Backend URL
IP_WHITELIST="0.0.0.0/0" # Backend Whitelisted IP Addresses
DOCKER_IMAGE="ghcr.io/generalmagicio/qacc-be:staging" # For Production: ghcr.io/generalmagicio/qacc-be:main
55 changes: 52 additions & 3 deletions .github/workflows/main-deploy-only.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,57 @@ jobs:
port: ${{ secrets.SSH_PORT }}
script: |
cd QAcc-BE
git reset --hard HEAD~1
git checkout main
git pull
docker compose -f docker-compose-production.yml pull
docker compose -f docker-compose-production.yml up --force-recreate -d
git pull origin main
docker image prune -a --force
docker compose pull

## Update each backend service one by one
## First Deployment
docker compose up -d --no-deps --scale qacc-be-graph-ql1=0 qacc-be-graph-ql2=1 qacc-be-graph-ql3=1
docker compose up -d --no-deps --scale qacc-be-job1=0 qacc-be-job2=1 qacc-be-job3=1
docker compose up -d
# Check the health of qacc-be-graph-ql1
if [ "$(docker inspect --format='{{json .State.Status}}' qacc-be-graph-ql1)" != "\"running\"" ]; then
echo "qacc-be-graph-ql1 is not running, stopping deployment"
exit 1
fi
# Check the health of qacc-be-job1
if [ "$(docker inspect --format='{{json .State.Status}}' qacc-be-job1)" != "\"running\"" ]; then
echo "qacc-be-job1 is not running, stopping deployment"
exit 1
fi

## Second Deployment
docker compose up -d --no-deps --scale qacc-be-graph-ql1=1 qacc-be-graph-ql2=0 qacc-be-graph-ql3=1
docker compose up -d --no-deps --scale qacc-be-job1=1 qacc-be-job2=0 qacc-be-job3=1
docker compose up -d
# Check the health of qacc-be-graph-ql2
if [ "$(docker inspect --format='{{json .State.Status}}' qacc-be-graph-ql2)" != "\"running\"" ]; then
echo "qacc-be-graph-ql2 is not running, stopping deployment"
exit 1
fi

# Check the health of qacc-be-job2
if [ "$(docker inspect --format='{{json .State.Status}}' qacc-be-job2)" != "\"running\"" ]; then
echo "qacc-be-job2 is not running, stopping deployment"
exit 1
fi


## Third Deployment
docker compose up -d --no-deps --scale qacc-be-graph-ql1=1 qacc-be-graph-ql2=1 qacc-be-graph-ql3=0
docker compose up -d --no-deps --scale qacc-be-job1=1 qacc-be-job2=1 qacc-be-job3=0
docker compose up -d
# Check the health of qacc-be-graph-ql3
if [ "$(docker inspect --format='{{json .State.Status}}' qacc-be-graph-ql3)" != "\"running\"" ]; then
echo "qacc-be-graph-ql3 is not running, stopping deployment"
exit 1
fi

# Check the health of qacc-be-job3
if [ "$(docker inspect --format='{{json .State.Status}}' qacc-be-job3)" != "\"running\"" ]; then
echo "qacc-be-job3 is not running, stopping deployment"
exit 1
fi
55 changes: 52 additions & 3 deletions .github/workflows/main-pipeline.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -158,8 +158,57 @@ jobs:
port: ${{ secrets.SSH_PORT }}
script: |
cd QAcc-BE
git reset --hard HEAD~1
git checkout main
git pull
docker compose -f docker-compose-production.yml pull
docker compose -f docker-compose-production.yml up -d
git pull origin main
docker image prune -a --force
docker compose pull

## Update each backend service one by one
## First Deployment
docker compose up -d --no-deps --scale qacc-be-graph-ql1=0 qacc-be-graph-ql2=1 qacc-be-graph-ql3=1
docker compose up -d --no-deps --scale qacc-be-job1=0 qacc-be-job2=1 qacc-be-job3=1
docker compose up -d
# Check the health of qacc-be-graph-ql1
if [ "$(docker inspect --format='{{json .State.Status}}' qacc-be-graph-ql1)" != "\"running\"" ]; then
echo "qacc-be-graph-ql1 is not running, stopping deployment"
exit 1
fi
# Check the health of qacc-be-job1
if [ "$(docker inspect --format='{{json .State.Status}}' qacc-be-job1)" != "\"running\"" ]; then
echo "qacc-be-job1 is not running, stopping deployment"
exit 1
fi

## Second Deployment
docker compose up -d --no-deps --scale qacc-be-graph-ql1=1 qacc-be-graph-ql2=0 qacc-be-graph-ql3=1
docker compose up -d --no-deps --scale qacc-be-job1=1 qacc-be-job2=0 qacc-be-job3=1
docker compose up -d
# Check the health of qacc-be-graph-ql2
if [ "$(docker inspect --format='{{json .State.Status}}' qacc-be-graph-ql2)" != "\"running\"" ]; then
echo "qacc-be-graph-ql2 is not running, stopping deployment"
exit 1
fi

# Check the health of qacc-be-job2
if [ "$(docker inspect --format='{{json .State.Status}}' qacc-be-job2)" != "\"running\"" ]; then
echo "qacc-be-job2 is not running, stopping deployment"
exit 1
fi


## Third Deployment
docker compose up -d --no-deps --scale qacc-be-graph-ql1=1 qacc-be-graph-ql2=1 qacc-be-graph-ql3=0
docker compose up -d --no-deps --scale qacc-be-job1=1 qacc-be-job2=1 qacc-be-job3=0
docker compose up -d
# Check the health of qacc-be-graph-ql3
if [ "$(docker inspect --format='{{json .State.Status}}' qacc-be-graph-ql3)" != "\"running\"" ]; then
echo "qacc-be-graph-ql3 is not running, stopping deployment"
exit 1
fi

# Check the health of qacc-be-job3
if [ "$(docker inspect --format='{{json .State.Status}}' qacc-be-job3)" != "\"running\"" ]; then
echo "qacc-be-job3 is not running, stopping deployment"
exit 1
fi
55 changes: 52 additions & 3 deletions .github/workflows/main-publish-deploy.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,8 +63,57 @@ jobs:
port: ${{ secrets.SSH_PORT }}
script: |
cd QAcc-BE
git reset --hard HEAD~1
git checkout main
git pull
docker compose -f docker-compose-production.yml pull
docker compose -f docker-compose-production.yml up --force-recreate -d
git pull origin main
docker image prune -a --force
docker compose pull

## Update each backend service one by one
## First Deployment
docker compose up -d --no-deps --scale qacc-be-graph-ql1=0 qacc-be-graph-ql2=1 qacc-be-graph-ql3=1
docker compose up -d --no-deps --scale qacc-be-job1=0 qacc-be-job2=1 qacc-be-job3=1
docker compose up -d
# Check the health of qacc-be-graph-ql1
if [ "$(docker inspect --format='{{json .State.Status}}' qacc-be-graph-ql1)" != "\"running\"" ]; then
echo "qacc-be-graph-ql1 is not running, stopping deployment"
exit 1
fi
# Check the health of qacc-be-job1
if [ "$(docker inspect --format='{{json .State.Status}}' qacc-be-job1)" != "\"running\"" ]; then
echo "qacc-be-job1 is not running, stopping deployment"
exit 1
fi

## Second Deployment
docker compose up -d --no-deps --scale qacc-be-graph-ql1=1 qacc-be-graph-ql2=0 qacc-be-graph-ql3=1
docker compose up -d --no-deps --scale qacc-be-job1=1 qacc-be-job2=0 qacc-be-job3=1
docker compose up -d
# Check the health of qacc-be-graph-ql2
if [ "$(docker inspect --format='{{json .State.Status}}' qacc-be-graph-ql2)" != "\"running\"" ]; then
echo "qacc-be-graph-ql2 is not running, stopping deployment"
exit 1
fi

# Check the health of qacc-be-job2
if [ "$(docker inspect --format='{{json .State.Status}}' qacc-be-job2)" != "\"running\"" ]; then
echo "qacc-be-job2 is not running, stopping deployment"
exit 1
fi


## Third Deployment
docker compose up -d --no-deps --scale qacc-be-graph-ql1=1 qacc-be-graph-ql2=1 qacc-be-graph-ql3=0
docker compose up -d --no-deps --scale qacc-be-job1=1 qacc-be-job2=1 qacc-be-job3=0
docker compose up -d
# Check the health of qacc-be-graph-ql3
if [ "$(docker inspect --format='{{json .State.Status}}' qacc-be-graph-ql3)" != "\"running\"" ]; then
echo "qacc-be-graph-ql3 is not running, stopping deployment"
exit 1
fi

# Check the health of qacc-be-job3
if [ "$(docker inspect --format='{{json .State.Status}}' qacc-be-job3)" != "\"running\"" ]; then
echo "qacc-be-job3 is not running, stopping deployment"
exit 1
fi
55 changes: 52 additions & 3 deletions .github/workflows/staging-deploy-only.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,57 @@ jobs:
port: ${{ secrets.SSH_PORT }}
script: |
cd QAcc-BE
git reset --hard HEAD~1
git checkout staging
git pull
docker compose -f docker-compose-staging.yml pull
docker compose -f docker-compose-staging.yml up --force-recreate -d
git pull origin staging
docker image prune -a --force
docker compose pull

## Update each backend service one by one
## First Deployment
docker compose up -d --no-deps --scale qacc-be-graph-ql1=0 qacc-be-graph-ql2=1 qacc-be-graph-ql3=1
docker compose up -d --no-deps --scale qacc-be-job1=0 qacc-be-job2=1 qacc-be-job3=1
docker compose up -d
# Check the health of qacc-be-graph-ql1
if [ "$(docker inspect --format='{{json .State.Status}}' qacc-be-graph-ql1)" != "\"running\"" ]; then
echo "qacc-be-graph-ql1 is not running, stopping deployment"
exit 1
fi
# Check the health of qacc-be-job1
if [ "$(docker inspect --format='{{json .State.Status}}' qacc-be-job1)" != "\"running\"" ]; then
echo "qacc-be-job1 is not running, stopping deployment"
exit 1
fi

## Second Deployment
docker compose up -d --no-deps --scale qacc-be-graph-ql1=1 qacc-be-graph-ql2=0 qacc-be-graph-ql3=1
docker compose up -d --no-deps --scale qacc-be-job1=1 qacc-be-job2=0 qacc-be-job3=1
docker compose up -d
# Check the health of qacc-be-graph-ql2
if [ "$(docker inspect --format='{{json .State.Status}}' qacc-be-graph-ql2)" != "\"running\"" ]; then
echo "qacc-be-graph-ql2 is not running, stopping deployment"
exit 1
fi

# Check the health of qacc-be-job2
if [ "$(docker inspect --format='{{json .State.Status}}' qacc-be-job2)" != "\"running\"" ]; then
echo "qacc-be-job2 is not running, stopping deployment"
exit 1
fi


## Third Deployment
docker compose up -d --no-deps --scale qacc-be-graph-ql1=1 qacc-be-graph-ql2=1 qacc-be-graph-ql3=0
docker compose up -d --no-deps --scale qacc-be-job1=1 qacc-be-job2=1 qacc-be-job3=0
docker compose up -d
# Check the health of qacc-be-graph-ql3
if [ "$(docker inspect --format='{{json .State.Status}}' qacc-be-graph-ql3)" != "\"running\"" ]; then
echo "qacc-be-graph-ql3 is not running, stopping deployment"
exit 1
fi

# Check the health of qacc-be-job3
if [ "$(docker inspect --format='{{json .State.Status}}' qacc-be-job3)" != "\"running\"" ]; then
echo "qacc-be-job3 is not running, stopping deployment"
exit 1
fi
4 changes: 2 additions & 2 deletions .github/workflows/staging-funding-pot-execution.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,5 +22,5 @@ jobs:
port: ${{ secrets.SSH_PORT }}
script: |
cd QAcc-BE
docker compose -f docker-compose-staging.yml exec qacc-be npm run execute:inverter:production
docker compose -f docker-compose-staging.yml logs qacc-be
docker compose exec qacc-be-graph-ql1 npm run execute:inverter:production
docker compose logs qacc-be-graph-ql1
55 changes: 52 additions & 3 deletions .github/workflows/staging-pipeline.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -158,8 +158,57 @@ jobs:
port: ${{ secrets.SSH_PORT }}
script: |
cd QAcc-BE
git reset --hard HEAD~1
git checkout staging
git pull
docker compose -f docker-compose-staging.yml pull
docker compose -f docker-compose-staging.yml up -d
git pull origin staging
docker image prune -a --force
docker compose pull

## Update each backend service one by one
## First Deployment
docker compose up -d --no-deps --scale qacc-be-graph-ql1=0 qacc-be-graph-ql2=1 qacc-be-graph-ql3=1
docker compose up -d --no-deps --scale qacc-be-job1=0 qacc-be-job2=1 qacc-be-job3=1
docker compose up -d
# Check the health of qacc-be-graph-ql1
if [ "$(docker inspect --format='{{json .State.Status}}' qacc-be-graph-ql1)" != "\"running\"" ]; then
echo "qacc-be-graph-ql1 is not running, stopping deployment"
exit 1
fi
# Check the health of qacc-be-job1
if [ "$(docker inspect --format='{{json .State.Status}}' qacc-be-job1)" != "\"running\"" ]; then
echo "qacc-be-job1 is not running, stopping deployment"
exit 1
fi

## Second Deployment
docker compose up -d --no-deps --scale qacc-be-graph-ql1=1 qacc-be-graph-ql2=0 qacc-be-graph-ql3=1
docker compose up -d --no-deps --scale qacc-be-job1=1 qacc-be-job2=0 qacc-be-job3=1
docker compose up -d
# Check the health of qacc-be-graph-ql2
if [ "$(docker inspect --format='{{json .State.Status}}' qacc-be-graph-ql2)" != "\"running\"" ]; then
echo "qacc-be-graph-ql2 is not running, stopping deployment"
exit 1
fi

# Check the health of qacc-be-job2
if [ "$(docker inspect --format='{{json .State.Status}}' qacc-be-job2)" != "\"running\"" ]; then
echo "qacc-be-job2 is not running, stopping deployment"
exit 1
fi


## Third Deployment
docker compose up -d --no-deps --scale qacc-be-graph-ql1=1 qacc-be-graph-ql2=1 qacc-be-graph-ql3=0
docker compose up -d --no-deps --scale qacc-be-job1=1 qacc-be-job2=1 qacc-be-job3=0
docker compose up -d
# Check the health of qacc-be-graph-ql3
if [ "$(docker inspect --format='{{json .State.Status}}' qacc-be-graph-ql3)" != "\"running\"" ]; then
echo "qacc-be-graph-ql3 is not running, stopping deployment"
exit 1
fi

# Check the health of qacc-be-job3
if [ "$(docker inspect --format='{{json .State.Status}}' qacc-be-job3)" != "\"running\"" ]; then
echo "qacc-be-job3 is not running, stopping deployment"
exit 1
fi
16 changes: 15 additions & 1 deletion Caddyfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,17 @@
{$MY_URL} {
reverse_proxy qacc-be:4000
route {
@allowed {
path /*
remote_ip {$IP_WHITELIST}
}
reverse_proxy @allowed {
to qacc-be-graph-ql1:4000 qacc-be-graph-ql2:4000 qacc-be-graph-ql3:4000
lb_policy round_robin
health_uri /health
health_interval 5s
health_timeout 2s
health_status 200
}
respond 403
}
}
2 changes: 1 addition & 1 deletion Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ COPY tsconfig.json .


RUN apk add --update alpine-sdk
RUN apk add git python3
RUN apk add git python3 curl
RUN apk add --no-cache chromium --repository=http://dl-cdn.alpinelinux.org/alpine/v3.18/main
RUN npm ci
RUN npm i -g ts-node
Expand Down
2 changes: 1 addition & 1 deletion docker-compose-local.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,4 +101,4 @@ volumes:
networks:
qacc:
name: qacc-be
external: true
external: true
Loading
Loading