Merge branch 'main' into feature-support-kenlm

.github/workflows/pylint.yml

@@ -0,0 +1,25 @@
+name: Pylint
+
+on: [push]
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.8", "3.9", "3.10"]
+    steps:
+    - uses: actions/checkout@v3
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: actions/setup-python@v3
+      with:
+        python-version: ${{ matrix.python-version }}
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        curl -sSL https://install.python-poetry.org | python -
+        poetry install # takes no arguments
+        pip install pylint
+    - name: Analysing the code with pylint
+      run: |
+        pylint $(git ls-files '*.py')

.github/workflows/python-publish.yml

@@ -0,0 +1,42 @@
+# This workflow will upload a Python Package using Twine when a release is created
+# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python#publishing-to-package-registries
+
+# This workflow uses actions that are not certified by GitHub.
+# They are provided by a third-party and are governed by
+# separate terms of service, privacy policy, and support
+# documentation.
+
+name: Upload Python Package
+
+on:
+  release:
+    types: [published]
+
+permissions:
+  contents: read
+
+jobs:
+  deploy:
+
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/checkout@v3
+    - name: Set up Python
+      uses: actions/setup-python@v3
+      with:
+        python-version: '3.x'
+    - name: Install dependencies
+      run: |
+        sudo apt install curl
+        python -m pip install --upgrade pip
+        curl -sSL https://install.python-poetry.org | python -
+        poetry install # no adtional args
+        pip install build
+    - name: Build package
+      run: poetry build
+    - name: Publish package
+      uses: pypa/gh-action-pypi-publish@27b31702a0e7fc50959f5ad993c78deac1bdfc29
+      with:
+        user: __token__
+        password: ${{ secrets.PYPI_API_TOKEN }}

README.md

@@ -2,7 +2,8 @@
 Compute surprisal from language models!
 
 `surprisal` supports most Causal Language Models (`GPT2`- and `GPTneo`-like models) from Huggingface or local checkpoint, 
-as well as `GPT3` models from OpenAI using their API!
+as well as `GPT3` models from OpenAI using their API! We also support `KenLM` N-gram based language models using the
+KenLM Python interface.
 
 Masked Language Models (`BERT`-like models) are in the pipeline and will be supported at a future time. 
 
@@ -12,6 +13,10 @@ The snippet below computes per-token surprisals for a list of sentences
 ```python
 from surprisal import AutoHuggingFaceModel
 
+from surprisal import KenLMModel
+k = KenLMModel(model_path='./literature.arpa')
+
+
 sentences = [
     "The cat is on the mat",
     "The cat is on the hat",
@@ -26,6 +31,9 @@ m.to('cuda') # optionally move your model to GPU!
 
 for result in m.surprise(sentences):
     print(result)
+
+for result in k.surprise(sentences):
+    print(result)
 ```
 and produces output of this sort:
 ```
@@ -105,8 +113,27 @@ python -m surprisal -m distilgpt2 "I went to the space station today."
 
 
 ## Installing
-`pip install surprisal`
+Because `surprisal` is used by people from different communities for different
+purposes, by default, core dependencies related to language modeling are marked
+optional. Depending on your use case, install `surprisal` with the appropriate
+extras.
+
+- For Huggingface transformers support:
+`pip install surprisal[transformers]`
+- For KenLM support:
+`pip install surprisal[kenlm]`
+- For OpenAI support:
+`pip install surprisal[openai]`
+
+### To install all extras:
+```bash
+pip install surprisal[transformers,openai,kenlm]
+```
 
+### Install using `poetry`
+```bash
+poetry add surprisal -E transformers -E openai -E kenlm
+```
 
 ## Acknowledgments