Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update the Docker setup #114

Closed
wants to merge 121 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
121 commits
Select commit Hold shift + click to select a range
adbe612
Update docker config
hopsoft Jan 7, 2024
f73d888
Update Docker setup to use Ruby 3.2.2 and clean some things up
hopsoft Jan 7, 2024
31554fe
Update GH Action config
hopsoft Jan 7, 2024
56198ed
Attempt to make GH Actions happy
hopsoft Jan 7, 2024
561d3c9
Yak shaving
hopsoft Jan 7, 2024
f1a20fc
More yak shaving
hopsoft Jan 7, 2024
f9466f8
Attempt to take more control over GH Action environment
hopsoft Jan 7, 2024
50319f8
Fix references to APP_NAME var
hopsoft Jan 7, 2024
9532db5
Pull APP_NAME from env
hopsoft Jan 7, 2024
a144b1e
Breakup Dockerfile into a multi-stage build
hopsoft Jan 7, 2024
da5b9c3
Update to only install chromium for playwright
hopsoft Jan 7, 2024
392fbb0
Shaving the yak
hopsoft Jan 7, 2024
fa00998
Fighting with GH Actions and Playwright
hopsoft Jan 7, 2024
8e9527d
Getting closer... I think
hopsoft Jan 7, 2024
74c7ae0
Maybe?
hopsoft Jan 7, 2024
74b7d30
Too many undos
hopsoft Jan 7, 2024
8f439bf
Iterating...
hopsoft Jan 7, 2024
bbb18d4
Another shot
hopsoft Jan 7, 2024
d4a0c73
...
hopsoft Jan 7, 2024
55553e8
Another attempt
hopsoft Jan 7, 2024
401a1c0
Get some visibility
hopsoft Jan 7, 2024
99543ec
...
hopsoft Jan 7, 2024
da4a537
Pathing...
hopsoft Jan 7, 2024
82c3619
Peek
hopsoft Jan 7, 2024
10f2c04
Update pathing
hopsoft Jan 7, 2024
04cf403
closer? ...
hopsoft Jan 7, 2024
eb959f8
Yakety Yak
hopsoft Jan 7, 2024
4c5af61
fingers crossed...
hopsoft Jan 7, 2024
6b51ddd
yak attack
hopsoft Jan 7, 2024
9376e04
yak shack
hopsoft Jan 7, 2024
32c52f8
more yaks
hopsoft Jan 7, 2024
190a95a
Still blocked by playwright
hopsoft Jan 7, 2024
6653c84
Give up
hopsoft Jan 8, 2024
8d8dee7
yaaaks
hopsoft Jan 8, 2024
135d5ef
Jesus...
hopsoft Jan 8, 2024
6358dc4
Skip browser install
hopsoft Jan 8, 2024
8052c22
Install playwright deps myself
hopsoft Jan 8, 2024
9606ef7
...
hopsoft Jan 8, 2024
724a979
experiment
hopsoft Jan 8, 2024
ce076dd
commit
hopsoft Jan 8, 2024
07ed185
only install chromium
hopsoft Jan 8, 2024
71a9750
force run
hopsoft Jan 8, 2024
a33a226
Set explicit timeout mins
hopsoft Jan 8, 2024
88c3b46
Attempt to install playwright deps again
hopsoft Jan 8, 2024
ef48902
Again
hopsoft Jan 8, 2024
d2047cb
Fail fast
hopsoft Jan 8, 2024
58df1ee
Cache playwright browsers
hopsoft Jan 8, 2024
62ea828
2nd attempt
hopsoft Jan 8, 2024
71ff7c9
Another attempt at caching playwright
hopsoft Jan 8, 2024
215b597
Remove all cache
hopsoft Jan 8, 2024
a8f93c0
Experimenting with drone ci
hopsoft Jan 14, 2024
43c1705
Attemp to get drone to build an image
hopsoft Jan 14, 2024
5eff490
Attempt to get drone to build the Dockerfile
hopsoft Jan 14, 2024
2121d5a
Use docker in docker to build
hopsoft Jan 14, 2024
511469a
tinkering
hopsoft Jan 14, 2024
4b2ac8b
Another experiment
hopsoft Jan 14, 2024
5a70f49
Use the docker plugin
hopsoft Jan 14, 2024
e0fc2a5
Fix indentation
hopsoft Jan 14, 2024
9f54a0a
Attempt to fix format
hopsoft Jan 14, 2024
6e039ad
Experimenting with tags
hopsoft Jan 14, 2024
ebcd13e
Attempt without explicit tag
hopsoft Jan 14, 2024
b57a885
Still messing with tags
hopsoft Jan 14, 2024
5a4577c
Try auto tagging
hopsoft Jan 15, 2024
8fb4b3c
Attempt to cache layers
hopsoft Jan 15, 2024
3c85b6f
yak shaving tags
hopsoft Jan 15, 2024
27bd154
Trying quotes for the tag
hopsoft Jan 15, 2024
6f36b43
Attempt a full tag name
hopsoft Jan 15, 2024
e7da4f0
Setup a repo value
hopsoft Jan 15, 2024
8a1ae3d
Still at it
hopsoft Jan 15, 2024
275254a
shaving the yak
hopsoft Jan 15, 2024
6e0bc00
Setup a docker registry
hopsoft Jan 15, 2024
c34ebd6
Specify the registry
hopsoft Jan 15, 2024
9b2d3c2
Attempt to get drone to honor registry name
hopsoft Jan 15, 2024
edd910c
digging deeper into registry
hopsoft Jan 15, 2024
8b3a961
Refinements
hopsoft Jan 15, 2024
9cc140d
Update env example
hopsoft Jan 15, 2024
fc3d339
fighting with drone
hopsoft Jan 15, 2024
1b3e5ba
maybe maybe ?
hopsoft Jan 15, 2024
782414c
Update registry config
hopsoft Jan 15, 2024
d4de8d5
Attempt with protocol
hopsoft Jan 15, 2024
e8ec173
experiment
hopsoft Jan 15, 2024
37f6007
Getting closer
hopsoft Jan 15, 2024
d58e1c0
remove protocol from repo
hopsoft Jan 15, 2024
409081f
allow for http
hopsoft Jan 15, 2024
1d9ce39
testing
hopsoft Jan 15, 2024
98c8b11
omit pull
hopsoft Jan 15, 2024
83617bb
more testing
hopsoft Jan 15, 2024
ca76cac
revert repo name
hopsoft Jan 15, 2024
5425194
Revert repo and repository names
hopsoft Jan 15, 2024
24c389b
auto tag
hopsoft Jan 15, 2024
1526338
Revert auto tag
hopsoft Jan 15, 2024
3a03d7f
Attempt to add host
hopsoft Jan 15, 2024
2bdd4a6
host tweak
hopsoft Jan 15, 2024
863999a
Attempt to share network
hopsoft Jan 15, 2024
9d5d9cd
Attempt to use the built image
hopsoft Jan 15, 2024
98e30dd
Attempt to get docker layer caching working
hopsoft Jan 15, 2024
9b03ca3
try with the protocol
hopsoft Jan 15, 2024
e8deea9
Attempt to use http to pull image
hopsoft Jan 15, 2024
43fb0d5
See if we can get away without pulling
hopsoft Jan 15, 2024
dfa5a86
2nd attempt
hopsoft Jan 15, 2024
3da0bc1
Revert
hopsoft Jan 15, 2024
f43273e
Setup a network test
hopsoft Jan 16, 2024
a41d5c2
Experiment
hopsoft Jan 16, 2024
0a52f1f
Fix cache image
hopsoft Jan 16, 2024
b963f4d
Remove top level volume
hopsoft Jan 16, 2024
f332d03
dind test
hopsoft Jan 17, 2024
16f7bfc
remove privileged mode
hopsoft Jan 17, 2024
2ad32f4
yaks
hopsoft Jan 17, 2024
31be47f
Experiment with cache
hopsoft Jan 17, 2024
35de82d
Attempt without pull
hopsoft Jan 17, 2024
3bd6f53
use correct tag
hopsoft Jan 17, 2024
dcfa2e4
Use full image name
hopsoft Jan 17, 2024
70b1e22
rebuild cache and peek
hopsoft Jan 17, 2024
7946154
Deeper peek
hopsoft Jan 17, 2024
36c0dd5
full peek
hopsoft Jan 17, 2024
35327b4
experiment with mounts
hopsoft Jan 17, 2024
f4bc13c
getting closer
hopsoft Jan 17, 2024
4d6673a
Isolate prepare and setup basic docker registry test
hopsoft Jan 17, 2024
62f8e94
trying with basic docker image
hopsoft Jan 17, 2024
58bca22
tweaking paths
hopsoft Jan 17, 2024
b34f3e3
dind
hopsoft Jan 17, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
83 changes: 83 additions & 0 deletions .drone.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
---
kind: pipeline
type: docker
name: default

steps:
- name: restore-cache
image: meltwater/drone-cache:v1
settings:
backend: filesystem
restore: true
archive_format: gzip
filesystem_cache_root: /cache
mount:
- /mnt/external
- /var/lib/docker
volumes:
- name: cache
path: /cache
- name: external
path: /mnt/external

- name: build-image
image: plugins/docker
settings:
context: .
registry: registry:5000
repo: registry:5000/hopsoft/turbo_boost-commands
tags: ci
#tags: ci-${DRONE_COMMIT_SHA}
compress: true
force_tag: true
insecure: true
storage_path: /var/lib/docker
use_cache: true


- name: rebuild-cache
image: meltwater/drone-cache:v1
settings:
backend: filesystem
rebuild: true
archive_format: gzip
filesystem_cache_root: /cache
mount:
- /mnt/external
- /var/lib/docker
volumes:
- name: cache
path: /cache
- name: external
path: /mnt/external

- name: peek
image: alpine
commands:
- apk update
- apk add tree
- tree /cache
volumes:
- name: cache
path: /cache
- name: external
path: /mnt/external

- name: prepare
image: docker:24-dind
settings:
commands:
- docker run hello-world
volumes:
- name: external
path: /mnt/external

#- name: check-standardrb
#image: hopsoft/turbo_boost-commands:ci-${DRONE_COMMIT_SHA}
#pull: never
#commands:
#- shopt -s globstar
#- bundle exec standardrb --format progress
#volumes:
#- name: docker
#path: /var/lib/docker
23 changes: 23 additions & 0 deletions .drone/.env.example
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# private
DRONE_GITHUB_CLIENT_ID="SEE: https://docs.drone.io/server/reference/drone-github-client-id/"
DRONE_GITHUB_CLIENT_SECRET="SEE: https://docs.drone.io/server/reference/drone-github-client-secret/"
DRONE_RPC_SECRET="SEE: https://docs.drone.io/server/reference/drone-rpc-secret/"
DRONE_UI_PASSWORD="SEE: https://docs.drone.io/runner/ssh/configuration/reference/drone-ui-password/"
DRONE_UI_USERNAME="SEE: https://docs.drone.io/runner/ssh/configuration/reference/drone-ui-username/"
DRONE_USER_FILTER="SEE: https://docs.drone.io/server/reference/drone-user-filter/"

# public
DRONE_DATABASE_DATASOURCE=/data/database.sqlite
DRONE_DATABASE_DRIVER=sqlite3
DRONE_LOGS_COLOR="true"
DRONE_LOGS_DEBUG="true"
DRONE_LOGS_PRETTY="true"
DRONE_LOGS_TEXT="true"
DRONE_RPC_HOST="server"
DRONE_RPC_PROTO="http"
DRONE_RUNNER_CAPACITY=1
DRONE_RUNNER_NAME="runner"
DRONE_RUNNER_NETWORKS="docker_primary"
DRONE_RUNNER_VOLUMES="/home/docker/cache:/cache"
DRONE_SERVER_HOST="server"
DRONE_SERVER_PROTO="http"
3 changes: 3 additions & 0 deletions .drone/Caddyfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
https://cicd.hopsoft.io {
reverse_proxy http://server
}
53 changes: 53 additions & 0 deletions .drone/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
# Drone CI (self hosted)

## Initial Setup and Configuration

1. [Create a GitHub OAuth application](https://docs.drone.io/server/provider/github/)
1. Configure your DNS by setting `A/AAAA` record(s) to point to your server
1. Optionally configure `SSL/TLS` to use [`Full (Strict)` mode](https://developers.cloudflare.com/ssl/origin-configuration/ssl-modes/full-strict/) if using a DNS service like Cloudflare
1. `ssh` to your server

```sh
ssh root@SERVER
```

1. Open ports `80` and `443`

```sh
ufw allow 80/tcp
ufw allow 443/tcp
ufw allow 443/udp
```

1. Create a non-root `sudo` user _(`docker` for example)_ to run Docker

> [!NOTE] This step isn't necessary if your server already has a Docker user

```sh
adduser docker
```

1. Switch to the new user and verify docker works

```sh
su - docker
docker info
```

1. Copy the `.env`, `Caddyfile`, and `compose.yml` config files to the server _(location of your choice, `/home/docker/.config/drone` for example)_

```sh
mkdir /home/docker/cache # host volume shared with the drone runner to cache docker builds
touch /home/docker/.env /home/docker/Caddyfile /home/docker/compose.yml /home/docker/registry.yml
chmod 600 /home/docker/.env
chown -R docker:docker /home/docker
# copy contents to each file
```

1. Update the `.env` file with your specific values
1. Start the Docker configuration

```sh
# as root
su docker -c "docker compose -f /home/docker/compose.yml up -d"
```
73 changes: 73 additions & 0 deletions .drone/compose.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
networks:
primary:

volumes:
caddy_config:
caddy_data:
docker_registry_data:
drone_server_data:

services:
registry:
image: registry:2
container_name: registry
networks:
- primary
restart: unless-stopped
volumes:
- docker_registry_data:/var/lib/registry
- /home/docker/registry.yml:/etc/docker/registry/config.yml

server:
image: drone/drone:2
container_name: server
depends_on:
registry:
condition: service_started
env_file: .env
networks:
- primary
restart: unless-stopped
healthcheck:
test: wget --tries=1 -S -O /dev/null http://localhost:80 || exit 1
interval: 30s
timeout: 3s
retries: 5
start_period: 10s
volumes:
- drone_server_data:/data
#- /home/docker/daemon.json:/etc/docker/daemon.json

runner:
image: drone/drone-runner-docker:1
container_name: runner
depends_on:
registry:
condition: service_started
server:
condition: service_healthy
env_file: .env
networks:
- primary
restart: unless-stopped
volumes:
- /var/run/docker.sock:/var/run/docker.sock
#- /home/docker/daemon.json:/etc/docker/daemon.json

proxy:
image: caddy:2
container_name: proxy
depends_on:
server:
condition: service_healthy
ports:
- "80:80"
- "443:443"
- "443:443/udp"
networks:
- primary
restart: unless-stopped
volumes:
- /home/docker/Caddyfile:/etc/caddy/Caddyfile
- caddy_config:/config
- caddy_data:/data
3 changes: 3 additions & 0 deletions .drone/daemon.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{
"insecure-registries" : [ "registry:5000" ]
}
19 changes: 19 additions & 0 deletions .drone/registry.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
version: 0.1
http:
addr: 0.0.0.0:5000
secret: 35x03e9d593142529264643dc668d02b
log:
accesslog:
disabled: false
level: debug
storage:
filesystem:
rootdirectory: /var/lib/registry
delete:
enabled: true
validation:
manifests:
urls:
allow:
- ^http://runner
- ^http://server
4 changes: 2 additions & 2 deletions .github/workflows/prettier.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,12 +17,12 @@ jobs:
- name: Setup Node
uses: actions/setup-node@v3
with:
version: '16.x'
version: '20.x'
cache: 'yarn'

- name: Yarn install
run: yarn install --frozen-lockfile

- name: Run Prettier
run: yarn run prettier --write package.json prettier.config.js bin/build.mjs app/javascript/**/**/**/**/*.js test/dummy/app/javascript/**/**/**/**/*.js test/dummy/app/assets/stylesheets/**/**/**/**/*.css test/dummy/app/views/**/**/**/**/*.css
run: shopt -s globstar && yarn run prettier --write package.json prettier.config.js bin/build.mjs app/javascript/**/*.js test/dummy/app/assets/stylesheets/**/*.css test/dummy/app/views/**/*.css

4 changes: 2 additions & 2 deletions .github/workflows/standardrb.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
ruby-version: ['2.7']
ruby-version: ['3.0']

steps:
- uses: actions/checkout@v3
Expand All @@ -25,4 +25,4 @@ jobs:
bundler-cache: true

- name: Run StandardRB
run: bundle exec standardrb --format progress
run: shopt -s globstar && bundle exec standardrb --format progress
53 changes: 30 additions & 23 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
@@ -1,37 +1,44 @@
name: Tests

on:
pull_request:
branches:
- '*'
push:
branches:
- main
pull_request:
branches:
- '*'

jobs:
ruby_test:
name: Ruby Test Action
test:
runs-on: ubuntu-latest
timeout-minutes: 15
strategy:
matrix:
ruby-version: ['2.7', '3.0', '3.1', '3.2', '3.3']

steps:
- uses: actions/checkout@v3
- name: Set up Ruby ${{ matrix.ruby-version }}
uses: ruby/setup-ruby@v1
with:
ruby-version: ${{ matrix.ruby-version }}
bundler-cache: true

- name: Setup Node
uses: actions/setup-node@v3
with:
node-version: '16.x'
cache: 'yarn'

- name: Yarn install
run: yarn install --frozen-lockfile

- name: Run ruby tests
run: bundle exec rake test
- uses: actions/checkout@v2
- uses: actions/setup-node@v3
with:
node-version: 18

- name: Set up Ruby ${{ matrix.ruby-version }}
uses: ruby/setup-ruby@v1
with:
ruby-version: ${{ matrix.ruby-version }}

- name: Yarn install
run: yarn install --frozen-lockfile --network-concurrency=3

- name: Get Playwright Version
id: playwright-version
run: echo "version=$(yarn playwright --version)" >> $GITHUB_OUTPUT

- name: Install Playwright Browsers
run: yarn playwright install

- name: Install Playwright Dependencies
run: yarn playwright install-deps

- name: Run ruby tests
run: bundle exec rake MINITEST_FAIL_FAST=true
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
*.gem
*.key
.containers.yml

/.bundle/
/doc/
/log/*.log
Expand All @@ -13,5 +14,7 @@
/test/dummy/storage/
/test/dummy/tmp/
/tmp/

Gemfile.lock
test/dummy/app/assets/builds/
test/dummy/app/javascript/@turbo-boost
9 changes: 6 additions & 3 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM ruby:3.0.3-slim-bullseye
FROM ruby:3.2.2-slim-bullseye

RUN apt-get -y update && \
apt-get -y --no-install-recommends install \
Expand All @@ -14,9 +14,12 @@ npm install -g npm@latest yarn

RUN apt-get clean
RUN gem update --system
RUN bundle config set --local clean 'true'

RUN mkdir -p /mnt/external/node_modules /mnt/external/yarn/.cache /mnt/external/gems /mnt/external/database
RUN mkdir -p \
/mnt/external/node_modules \
/mnt/external/yarn/.cache \
/mnt/external/gems \
/mnt/external/database

COPY . /app
WORKDIR /app
Expand Down
Loading