Expand formatting (#74)

* Expand formatter config

* Exclude vendor directory

* Apply formatting

* Exclude vendor dir

* Update to use pre-commit

* Update format

.editorconfig

@@ -1,11 +1,22 @@
 root = true
 
 [*]
+charset = utf-8
 end_of_line = lf
 insert_final_newline = true
+trim_trailing_whitespace = true
+max_line_length = 140
 
-[*.{js,py}]
-charset = utf-8
+[*.py]
+indent_style = space
+indent_size = 4
+
+[*.{js,html,css,json,yml,yaml}]
+indent_style = space
+indent_size = 2
 
 [Makefile]
 indent_style = tab
+
+[*.md]
+trim_trailing_whitespace = false

.github/FUNDING.yml

@@ -1,6 +1,5 @@
 # These are supported funding model platforms
 buy_me_a_coffee: griesel
-
 # github: # Replace with up to 4 GitHub Sponsors-enabled usernames e.g., [user1, user2]
 # patreon: # Replace with a single Patreon username
 # open_collective: # Replace with a single Open Collective username

.github/workflows/lint-and-format.yml

@@ -5,20 +5,31 @@ on:
   pull_request:
 
 jobs:
-  lint:
+  pre-commit:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v3
+
       - name: Set up Python
         uses: actions/setup-python@v4
         with:
           python-version: 3.12
+
+      - name: Set up Node
+        uses: actions/setup-node@v3
+        with:
+          node-version: "18"
+
       - name: Install dependencies
         run: |
-          pip install black isort flake8
-      - name: Run Black
-        run: black --check .
-      - name: Run isort
-        run: isort --check-only .
-      - name: Run Flake8
-        run: flake8 .
+          python -m pip install --upgrade pip
+          pip install pre-commit
+
+      - name: Cache pre-commit hooks
+        uses: actions/cache@v3
+        with:
+          path: ~/.cache/pre-commit
+          key: pre-commit-${{ hashFiles('.pre-commit-config.yaml') }}
+
+      - name: Run pre-commit
+        run: pre-commit run --all-files

.gitignore

@@ -31,4 +31,3 @@ output
 /node_modules
 /docs/.vitepress/cache
 *.bak
-

.pre-commit-config.yaml

@@ -1,13 +1,50 @@
 repos:
+  # Python
   - repo: https://github.com/psf/black
     rev: 23.9.1
     hooks:
       - id: black
+        language_version: python3
+
   - repo: https://github.com/pycqa/isort
     rev: 5.12.0
     hooks:
       - id: isort
+        args: ["--profile", "black"]
+
   - repo: https://github.com/pycqa/flake8
     rev: 6.1.0
     hooks:
       - id: flake8
+
+  # HTML, CSS, JS, TS, MD, ...
+  - repo: https://github.com/pre-commit/mirrors-prettier
+    rev: v3.0.3
+    hooks:
+      - id: prettier
+        types_or: [javascript, jsx, ts, tsx, css, html, json, yaml, markdown]
+        exclude: ^services/web/src/static/vendor/
+        additional_dependencies:
+          - [email protected]
+          - "@prettier/[email protected]"
+
+  # Generic
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v4.4.0
+    hooks:
+      - id: trailing-whitespace
+        stages: [pre-commit]
+        exclude: ^services/web/src/static/vendor/
+      - id: end-of-file-fixer
+        stages: [pre-commit]
+        exclude: ^services/web/src/static/vendor/
+      - id: check-yaml
+        stages: [pre-commit]
+        exclude: ^services/web/src/static/vendor/
+      - id: check-json
+        stages: [pre-commit]
+        exclude: ^services/web/src/static/vendor/
+      - id: mixed-line-ending
+        stages: [pre-commit]
+        exclude: ^services/web/src/static/vendor/
+        args: [--fix=lf]

CODE_OF_CONDUCT.md

@@ -17,23 +17,23 @@ diverse, inclusive, and healthy community.
 Examples of behavior that contributes to a positive environment for our
 community include:
 
-* Demonstrating empathy and kindness toward other people
-* Being respectful of differing opinions, viewpoints, and experiences
-* Giving and gracefully accepting constructive feedback
-* Accepting responsibility and apologizing to those affected by our mistakes,
+- Demonstrating empathy and kindness toward other people
+- Being respectful of differing opinions, viewpoints, and experiences
+- Giving and gracefully accepting constructive feedback
+- Accepting responsibility and apologizing to those affected by our mistakes,
   and learning from the experience
-* Focusing on what is best not just for us as individuals, but for the
+- Focusing on what is best not just for us as individuals, but for the
   overall community
 
 Examples of unacceptable behavior include:
 
-* The use of sexualized language or imagery, and sexual attention or
+- The use of sexualized language or imagery, and sexual attention or
   advances of any kind
-* Trolling, insulting or derogatory comments, and personal or political attacks
-* Public or private harassment
-* Publishing others' private information, such as a physical or email
+- Trolling, insulting or derogatory comments, and personal or political attacks
+- Public or private harassment
+- Publishing others' private information, such as a physical or email
   address, without their explicit permission
-* Other conduct which could reasonably be considered inappropriate in a
+- Other conduct which could reasonably be considered inappropriate in a
   professional setting
 
 ## Enforcement Responsibilities
@@ -106,7 +106,7 @@ Violating these terms may lead to a permanent ban.
 ### 4. Permanent Ban
 
 **Community Impact**: Demonstrating a pattern of violation of community
-standards, including sustained inappropriate behavior,  harassment of an
+standards, including sustained inappropriate behavior, harassment of an
 individual, or aggression toward or disparagement of classes of individuals.
 
 **Consequence**: A permanent ban from any sort of public interaction within

docker/docker-compose.base.yml

@@ -4,19 +4,19 @@ x-healthcheck: &default-healthcheck
 
 # Base service configuration
 x-ollama: &ollama-base
-    hostname: ollama
-    image: ollama/ollama:latest
-    pull_policy: always
-    tty: true
-    restart: unless-stopped
-    ports:
-      - 21240:11434
-    environment:
-      - OLLAMA_KEEP_ALIVE=24h
-    volumes:
-      - ollama-data:/root/.ollama
-    networks:
-      - default
+  hostname: ollama
+  image: ollama/ollama:latest
+  pull_policy: always
+  tty: true
+  restart: unless-stopped
+  ports:
+    - 21240:11434
+  environment:
+    - OLLAMA_KEEP_ALIVE=24h
+  volumes:
+    - ollama-data:/root/.ollama
+  networks:
+    - default
 
 services:
   # NVIDIA GPU configuration
@@ -72,7 +72,7 @@ services:
       test: ["CMD", "curl", "-f", "http://localhost:9200/_cat/health"]
     deploy:
       resources:
-         limits:
+        limits:
           memory: 8g
 
 networks:

docker/docker-compose.dev.yml

@@ -5,8 +5,8 @@ x-healthcheck: &default-healthcheck
 x-shared-deploy: &shared-deploy
   resources:
     limits:
-      memory: '2g'
-      
+      memory: "2g"
+
 services:
   elasticvue:
     image: cars10/elasticvue

docker/docker-compose.prod.yml

@@ -5,7 +5,7 @@ x-healthcheck: &default-healthcheck
 x-shared-deploy: &shared-deploy
   resources:
     limits:
-      memory: '2g'
+      memory: "2g"
 
 services:
   api:
@@ -40,7 +40,7 @@ services:
 
   nginx-proxy:
     # ref: https://nginxproxymanager.com/guide/#quick-setup
-    
+
     image: jc21/nginx-proxy-manager:latest
     ports:
       - 80:80

docs/.vitepress/theme/index.ts

@@ -1,17 +1,17 @@
 // https://vitepress.dev/guide/custom-theme
-import { h } from 'vue'
-import type { Theme } from 'vitepress'
-import DefaultTheme from 'vitepress/theme'
-import './style.css'
+import { h } from "vue";
+import type { Theme } from "vitepress";
+import DefaultTheme from "vitepress/theme";
+import "./style.css";
 
 export default {
   extends: DefaultTheme,
   Layout: () => {
     return h(DefaultTheme.Layout, null, {
       // https://vitepress.dev/guide/extending-default-theme#layout-slots
-    })
+    });
   },
   enhanceApp({ app, router, siteData }) {
     // ...
-  }
-} satisfies Theme
+  },
+} satisfies Theme;

docs/demo.md

@@ -3,14 +3,15 @@
 Experience Chipper live right here! Please note that the demo's functionality is subject to limitations. It utilizes the **Hugging Face serverless inference API**, which operates under strict quotas and rate limits. Each user is allowed up to **8 prompts per day**. The embedded documents are the chipper test data.
 
 **Maybe you ask Chipper now:**
+
 ```plain
 Tell me a story about Chipper, the brilliant golden retriever.
 ```
 
 <div class="demo">
  <iframe
    title="Chipper AI Demo"
-   width="100%" 
+   width="100%"
    height="900"
    src="https://demo.chipper.tilmangriesel.com/">
  </iframe>
@@ -23,21 +24,24 @@ This demo **is limited by the free resources** available from Hugging Face and m
 Additionally, the serverless inference API may sometimes queue your request, leading to delays. During this period, you’ll see the "**Chipper is thinking...**" message. Keep in mind that the performance of Hugging Face inference in this demo may vary and does **not represent the experience of a self-hosted setup**.
 
 ### Experimentation 🧪
+
 Since Chipper uses embeddings, you can ask him about his adventures based on the [embedded stories](https://github.com/TilmanGriesel/chipper/tree/main/tools/embed/testdata/md/internal).
 
 ### Models Used
+
 - **Inference**: [meta-llama/Meta-Llama-3-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct)
 - **Embedding**: [sentence-transformers/all-mpnet-base-v2](https://huggingface.co/sentence-transformers/all-mpnet-base-v2)
 
 ### Demo Server Specs
+
 The demo operates on a **Scaleway Stardust 1** instance, utilizing the latest Chipper stack along with Elasticsearch from Docker Hub, powered by the **Hugging Face** provider.
 
-| **Specification**   | **Details**            |
-|----------------------|------------------------|
-| **CPU**             | AMD EPYC 7282 (2.8 GHz) |
-| **CPU Architecture** | amd64                 |
+| **Specification**    | **Details**                   |
+| -------------------- | ----------------------------- |
+| **CPU**              | AMD EPYC 7282 (2.8 GHz)       |
+| **CPU Architecture** | amd64                         |
 | **Sizing**           | 1 vCPU, 1 GiB RAM, 8 GiB Swap |
-| **Storage**          | 25 GB                 |
+| **Storage**          | 25 GB                         |
 
 [More about the stardust instance](https://www.scaleway.com/en/docs/compute/instances/reference-content/instances-datasheet/#stardust1-instances)
 

docs/get-started.md

@@ -5,7 +5,7 @@ Chipper started as a set of Python scripts designed to experiment with local RAG
 
 <p align="center">
 <div style="position: relative; padding-top: 56.25%">
-  <iframe 
+  <iframe
     style="position: absolute; top: 0; left: 0; width: 100%; height: 100%; border-radius: 24px;"  src="https://www.youtube.com/embed/WFAGn9BjN9M" title="Chipper AI -  AI interface for tinkerers (Ollama, Haystack RAG, Python)" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>
 </div>
 </p>
@@ -129,7 +129,7 @@ You can set the index using the `--es-index <name>` parameter, specify the embed
 
 ## Step 5: Next Steps and Exploration 🔍
 
-First off, if you’ve made it this far, let me unravel the mystery behind why Chipper is called Chipper the Golden Retriever. For starters, I adore golden retrievers! But there’s more to it: they love to *chip* wood, just like we need to split and chip the data we want to embed. And as for *retriever*, - well ...
+First off, if you’ve made it this far, let me unravel the mystery behind why Chipper is called Chipper the Golden Retriever. For starters, I adore golden retrievers! But there’s more to it: they love to _chip_ wood, just like we need to split and chip the data we want to embed. And as for _retriever_, - well ...
 
 Jokes aside, this project offers plenty more tools to explore. You can transcribe audio files into text and embed it, scrape websites (only your own or with proper consent), or dive into the frontend, here you can write `/help` to see some options or backend to customize and change Chipper to suit your needs.
 

docs/public/assets/favicon/site.webmanifest

@@ -18,4 +18,4 @@
   "theme_color": "#ffffff",
   "background_color": "#ffffff",
   "display": "standalone"
-}
+}

docs/public/assets/fonts/LICENSE_JustAnotherHand-Regular.txt

@@ -57,4 +57,4 @@ Limitation of Liability
 In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages.
 
 Accepting Warranty or Additional Liability
-While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability.
+While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability.