diff --git a/.config/dictionaries/project.dic b/.config/dictionaries/project.dic index b0f2f0d0729..171774a4278 100644 --- a/.config/dictionaries/project.dic +++ b/.config/dictionaries/project.dic @@ -21,6 +21,7 @@ autorecalculates autoresizing backendpython bech +bento bimap bindgen bkioshn @@ -51,6 +52,7 @@ codepoints commitlog coti coverallsapp +cpu CQLSH cryptoxide Cunego @@ -106,6 +108,7 @@ icudtl ideascale idents ilap +inotify Instantitation integ Intellij @@ -113,14 +116,17 @@ interps inversed iohk iphoneos +iscsi jdbc jetbrains +jetstack Joaquín jorm jormungandr Jörmungandr junitreport junitxml +justfile Keyhash keychains keyserver @@ -171,6 +177,7 @@ netifas netkey nextest Nodetool +noninteractive OCSP Oleksandr onboarded @@ -190,6 +197,7 @@ postcss Pozhylenkov Precache Precertificate +preconfiguration preprod projectcatalyst Prokhorenko @@ -272,6 +280,7 @@ Unstaked UTXO Utxos varint +vcpu Vespr vite vitss @@ -297,4 +306,4 @@ xctest xctestrun xcworkspace xvfb -yoroi +yoroi \ No newline at end of file diff --git a/utilities/local-cluster/Readme.md b/utilities/local-cluster/Readme.md index 43032c69757..a2b7b46fd55 100644 --- a/utilities/local-cluster/Readme.md +++ b/utilities/local-cluster/Readme.md @@ -1,15 +1,45 @@ # Local Cluster +* [Local Cluster](#local-cluster) + * [Requirements](#requirements) + * [Cluster Architecture](#cluster-architecture) + * [Default Services](#default-services) + * [Scylla DB](#scylla-db) + * [Deploying the Cluster](#deploying-the-cluster) + * [Setup hosts on Windows](#setup-hosts-on-windows) + * [Startup](#startup) + * [Linux/Windows](#linuxwindows) + * [macOS](#macos) + * [Getting Basic Cluster details](#getting-basic-cluster-details) + * [Suspending the Cluster](#suspending-the-cluster) + * [Resuming a suspended the Cluster](#resuming-a-suspended-the-cluster) + * [Stopping the Cluster](#stopping-the-cluster) + * [Catalyst Voices Services](#catalyst-voices-services) + * [Deploying Catalyst Voices Frontend and Backend Services](#deploying-catalyst-voices-frontend-and-backend-services) + * [Deploying Catalyst Voices Documentation Service](#deploying-catalyst-voices-documentation-service) + * [Debugging the cluster](#debugging-the-cluster) + * [SSH into a running VM](#ssh-into-a-running-vm) + * [Local UI to access ScyllaDB](#local-ui-to-access-scylladb) + * [Troubleshooting](#troubleshooting) + * [Issues with Vagrant configurations](#issues-with-vagrant-configurations) + * [Issues with running command with just](#issues-with-running-command-with-just) + * [Vagrant cannot forward the specified ports on a VM](#vagrant-cannot-forward-the-specified-ports-on-a-vm) + * [Check if Port 6443 is in Use](#check-if-port-6443-is-in-use) + * [List all running Vagrant VMs to see if any are currently active](#list-all-running-vagrant-vms-to-see-if-any-are-currently-active) + * [Halt or Destroy the Existing VM](#halt-or-destroy-the-existing-vm) + * [Kill the VM Process Manually (If Necessary)](#kill-the-vm-process-manually-if-necessary) + * [Confirm the Process is Terminated](#confirm-the-process-is-terminated) + +## Requirements + Integration Tests and local testing will require a running local cluster. To eliminate variability and simplify local deployment, we have standardized the local cluster around: -* [VirtualBox](https://www.virtualbox.org/) +* [VirtualBox](https://www.virtualbox.org/) for Linux or [Parallels](https://www.parallels.com/) for macOS * [Vagrant](https://developer.hashicorp.com/vagrant/install?product_intent=vagrant) * [kubectl](https://kubernetes.io/docs/tasks/tools/) * [helm](https://helm.sh/docs/intro/install/) - -These tools allow us to define VMs that are consistent and provide a uniform Kubernetes environment -for local testing. +* [just](https://github.com/casey/just) ## Cluster Architecture @@ -56,14 +86,14 @@ For testing purposes, the ScyllaDB is accessible on the Cluster IP Address: `192 ### Setup hosts on Windows -On Windows, you need to set up the hosts before starting the cluster -From the Windows terminal to open the hosts file: +On Windows you need to setup the hosts before starting the cluster +From Windows terminal open the hosts file: ```sh -notepad %SystemRoot%\System32\drivers\etc\hosts -``` +notepad %SystemRoot%\System32\drivers\etc\hosts +``` -and copy the hosts from `./shared/extra.hosts` into the Windows host file +and copy the hosts from `./shared/extra.hosts` into the Windows host file ### Startup @@ -176,13 +206,13 @@ vagrant ssh agent99 Found (and tested) description how to connect using only open-source via DBeaver: -1. Download DBeaver (Community Edition) -2. Download Cassandra JDBC jar files: +1. Download dbeaver (Community Edition) [dbeaver](https://dbeaver.io/download) +2. Download cassandra jdbc jar files: [cassandra-jdbc-drive](http://www.dbschema.com/cassandra-jdbc-driver.html) (Downloading and Testing the Driver Binaries section have links to binary and source) -3. extract Cassandra JDBC zip -4. run DBeaver -5. go to Database > Driver Manager -6. click New +3. Extract Cassandra JDBC zip +4. Run DBeaver +5. Go to `Database` > `Driver Manager` +6. Click `New` 7. Fill in details as follows: * Driver Name: `Cassandra` (or whatever you want it to say) * Driver Type: `Generic` @@ -192,7 +222,115 @@ Found (and tested) description how to connect using only open-source via DBeaver * Embedded: `no` * Category: * Description: `Cassandra` (or whatever you want it to say) -8. click Add File and add all the jars in the Cassandra JDBC zip file. -9. click Find Class to make sure the Class Name is found okay -10. click OK -11. Create New Connection, selecting the database driver you just added +8. Click Add File and add all the jars in the Cassandra JDBC zip file. +9. Click Find Class to make sure the Class Name is found okay. +10. Click `OK`. +11. Create `New Connection`, selecting the database driver you just added. + +## Troubleshooting + +### Issues with Vagrant configurations + +If you encounter any weird issues with the Vagrant, you can try the following: + +```sh +rm -rf .vagrant +``` + +and then restart the cluster. + +### Issues with running command with just + +If you encounter any issues with running the commands with `just`, +you can try running the commands directly. + +Instead of: + +```sh +just stop-cluster +``` + +run: + +```sh +vagrant destroy -f +``` + +For more see [justfile](./justfile) + +### Vagrant cannot forward the specified ports on a VM + +#### Check if Port 6443 is in Use + +```sh +sudo lsof -i :6443 +``` + +You can see the output like this: + +```sh +COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME +parallels-syst 42886 username 19u IPv4 0x9af1245915ca3e76 0t0 TCP *:sun-sr-https (LISTEN) +``` + +The process parallels-syst, with PID 42886, is listening on port 6443. +This means that a parallels virtual machine (can be VirtualBox) is currently running and using port 6443 on your host machine. + +Since you’re using the VirtualBox/Parallels provider with Vagrant +it’s likely that an existing VM is still running and occupying port 6443. +This is preventing your new Vagrant VM from forwarding port 6443, resulting in the error + +#### List all running Vagrant VMs to see if any are currently active + +```sh +vagrant global-status +``` + +This command will list all Vagrant environments on your system. +Look for any VMs that are in the running state. + +```sh +id name provider state directory +-------------------------------------------------------------------------------------- +abcd123 control parallels running /path/to/your/project +abcd456 other parallels running /another/project +``` + +#### Halt or Destroy the Existing VM + +If you find that a VM is running that you no longer need, you can halt or destroy it. + +**To halt the VM:** + +```sh +vagrant halt +``` + +**To destroy the VM:** + +```sh +vagrant destroy +``` + +Replace ID with the ID of the VM from the vagrant global-status output. +For example: `vagrant halt abcd123` + +#### Kill the VM Process Manually (If Necessary) + +If the VM is not managed by Vagrant, or if it did not shut down properly, +you may need to kill the VM process manually. + +```sh +kill 42886 +``` + +Replace 42886 with the PID from your lsof output. + +#### Confirm the Process is Terminated + +```sh +sudo lsof -i :6443 +``` + +This should return no output, indicating that no process is listening on port 6443. +Retry bringing up the Vagrant VM diff --git a/utilities/local-cluster/Vagrantfile b/utilities/local-cluster/Vagrantfile index 68c633b9041..71c413a2d0b 100644 --- a/utilities/local-cluster/Vagrantfile +++ b/utilities/local-cluster/Vagrantfile @@ -1,13 +1,14 @@ # -*- mode: ruby -*- # vi: set ft=ruby : -# cspell: words vcpu, noninteractive, iscsi, preconfiguration, inotify, jetstack + +# These variables are kept for reference in scripts, but we're not using them for network configuration control_ip = "192.168.58.10" # Determine the maximum number of agents, and set their IP addresses agents = { "agent86" => "192.168.58.86", "agent99" => "192.168.58.99" } -# This is sized so that a machine with 16 threads and 16GB will allocate at most +# This is sized so that a machine with 16 threads and 16GB will allocate at most # ~3/4 of its resources to the cluster. control_vcpu = "4" # Number of vCPUs in the VM control_memory = "5192" # 5G of Memory (An extra gig because its the control-plane node) @@ -42,7 +43,7 @@ helm_install_script = <<-SHELL curl https://baltocdn.com/helm/signing.asc | gpg --dearmor | sudo tee /usr/share/keyrings/helm.gpg > /dev/null echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/helm.gpg] https://baltocdn.com/helm/stable/debian/ all main" | tee /etc/apt/sources.list.d/helm-stable-debian.list apt-get update - apt-get install helm + apt-get install helm SHELL control_plane_script = <<-SHELL @@ -101,7 +102,7 @@ longhorn_install_script = <<-SHELL helm repo update helm install longhorn longhorn/longhorn --namespace longhorn-system --create-namespace --version 1.6.2 kubectl apply -f /vagrant_shared/k3s/longhorn/storage-classes.yaml - kubectl apply -f /vagrant_shared/k3s/longhorn/ingress.yaml + kubectl apply -f /vagrant_shared/k3s/longhorn/ingress.yaml SHELL local_path_provisioner_script = <<-SHELL @@ -123,7 +124,7 @@ monitoring_install_script = <<-SHELL export KUBECONFIG=/etc/rancher/k3s/k3s.yaml helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm repo add grafana https://grafana.github.io/helm-charts - helm repo update + helm repo update helm install prometheus-stack --version 58.6.0 -f /vagrant_shared/k3s/grafana-prometheus/prometheus-values.yaml prometheus-community/kube-prometheus-stack --create-namespace --namespace monitoring --wait kubectl apply -f /vagrant_shared/k3s/grafana-prometheus/alert-manager-ingress.yaml kubectl apply -f /vagrant_shared/k3s/grafana-prometheus/prometheus-ingress.yaml @@ -153,27 +154,33 @@ update_sysctl_limits = <<-SHELL SHELL Vagrant.configure("2") do |config| - config.vm.box = "generic/debian12" + if is_darwin_arm64 + config.vm.box = "bento/ubuntu-20.04-arm64" + else + config.vm.box = "bento/debian-12" + end + + config.vm.network "forwarded_port", guest: 6443, host: 6443 config.vm.define "control", primary: true do |control| control.vm.network "private_network", ip: control_ip control.vm.hostname = "control" - if !is_darwin_arm64 - # x86 anything should work with this + if is_darwin_arm64 + control.vm.synced_folder "./shared", "/vagrant_shared", type: "rsync" + control.vm.provider "parallels" do |p| + p.cpus = control_vcpu + p.memory = control_memory + end + else + control.vm.network "private_network", ip: control_ip control.vm.synced_folder "./shared", "/vagrant_shared" control.vm.provider "virtualbox" do |vb| - vb.memory = control_memory vb.cpus = control_vcpu - end - else - # Specific config just for Arm Macs. - control.vm.synced_folder "./shared", "/vagrant_shared", type: "smb" - control.vm.provider "qemu" do |qe| - qe.memory = control_memory - qe.smp = control_vcpu + vb.memory = control_memory end end + control.vm.provision "shell", inline: update_sysctl_limits control.vm.provision "shell", inline: static_ips control.vm.provision "shell", inline: debian_packages @@ -190,21 +197,22 @@ Vagrant.configure("2") do |config| agents.each do |agent_name, agent_ip| config.vm.define agent_name do |agent| - agent.vm.network "private_network", ip: agent_ip agent.vm.hostname = agent_name - if !is_darwin_arm64 + + if is_darwin_arm64 + agent.vm.synced_folder "./shared", "/vagrant_shared", type: "rsync" + agent.vm.provider "parallels" do |p| + p.cpus = agent_vcpu + p.memory = agent_memory + end + else agent.vm.synced_folder "./shared", "/vagrant_shared" agent.vm.provider "virtualbox" do |vb| - vb.memory = agent_memory vb.cpus = agent_vcpu - end - else - agent.vm.synced_folder "./shared", "/vagrant_shared", type: "smb" - agent.vm.provider "qemu" do |qe| - qe.memory = agent_memory - qe.smp = agent_vcpu + vb.memory = agent_memory end end + agent.vm.provision "shell", inline: update_sysctl_limits agent.vm.provision "shell", inline: static_ips agent.vm.provision "shell", inline: debian_packages diff --git a/utilities/local-cluster/justfile b/utilities/local-cluster/justfile index 0a202096dec..24f65c0a019 100644 --- a/utilities/local-cluster/justfile +++ b/utilities/local-cluster/justfile @@ -24,10 +24,9 @@ _check-cluster-prereqs: start-cluster: _check-cluster-prereqs vagrant up -# Start the development cluster - only on a aarch64 macos machine start-cluster-aarch64-macos: _check-cluster-prereqs - VAGRANT_DISABLE_STRICT_DEPENDENCY_ENFORCEMENT=1 vagrant plugin install vagrant-qemu - vagrant up --provider qemu + VAGRANT_LOG=debug vagrant plugin install vagrant-parallels + vagrant up control --provider=parallels # Pause the development cluster - does not destroy it suspend-cluster: @@ -55,8 +54,8 @@ ssh-into-agent86: # SSH into the agent99 node ssh-into-agent99: - vagrant ssh agent99 + vagrant ssh agent99 get-all-logs: - mkdir -p logs + mkdir -p logs kail --log-level=error --since 6h > cluster.logs