Configuring Typesense in GCE

[flauc]

Typesense is a great open-source alternative to elasticsearch and algolia.

It provides a docker image that can easily be self hosted on a VM anywhere. You can do so in GCE by following the steps below.

1. Run docker on your local machine and pull the typesense image by running docker pull typesense/typesense
2. Make sure you have gcloud cli installed and authenticated then set your project gcloud config set [your-project]
3. Now tag the typesense image with the path to your desired google artifact registry location and push the image (instructions [here](docker tag SOURCE-IMAGE LOCATION-docker.pkg.dev/PROJECT-ID/REPOSITORY/IMAGE:TAG)).
4. Now create a google container instance (even the smallest instance can host typesense)
   Configure the docker container in the following way:
   
   • Provide the following arguments
   • --enable-cors
   • --api-key={key} - key: some random value
   • --data-dir=/data
   • --ssl-certificate=/cert/fullchain.pem
   • --ssl-certificate-key=/cert/privkey.pem
   • Create the following environment variables
   • TYPESENSE_API_PORT=443
   • Add two volume mounts
   • Volume type: Directory, Mounth path: /data, Host path: /home/data, Mode: read/write
   • Volume type: Directory, Mounth path: /cert, Host path: /home/letsencrypt/archive/search.scoutz-internal-20.com/, Mode: read/write
5. Enable https traffic on the container.
6. Once created you can ssh in to the VM and run docker ps -a to verify that the container has been created and docker logs [container-id] to confirm it's running properly.

Great work at this point you have a running typesense container.

Configure SSL Certificate

It's important to configure an SSL certificate in order to support https communication which is a requirement for any web application.
The easiest way to do so is through letsencrypt and a certbot authenticating through google dns.

1. Create a service account in GCE with the "DNS Admin" role
2. Copy that service account json file in to the VM under the name "service-account.json"
3. Now run the following command:

docker run --rm -v /home/letsencrypt:/etc/letsencrypt:rw -v ${PWD}/service-account.json:/service-account.json certbot/dns-google certonly --dns-google --dns-google-credentials /service-account.json --dns-google-propagation-seconds 90 --agree-tos -m [email protected] --non-interactive -d search.scoutz-internal-20.com

4. Give everyone permission to read the certificates sudo chmod u=rwx,g=rwx,o=rwx /home/letsencrypt/ -R
5. Now you'll need to manually restart the instance for changes to take effect.

Note: In order for this to work the domain needs to be registered within google Cloud DNS of the same project.

Accessing the system from a client

The API key you created is an admin key with elevated permissions you shouldn't use it from any client application directly. Instead you should create a search only API key for the website or app and scope it to only public collections.

curl 'http://${your-ip}/keys' \
    -X POST \
    -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \
    -H 'Content-Type: application/json' \
    -d '{"description":"Search-only key.","actions": ["documents:search"], "collections": ["*"]}'

You can read more about API keys here.

https://extensions.dev/extensions/typesense/firestore-typesense-search

Indexing collections with Firestore

The easiest way to add new indexes to typesense if you're using firebase is by using the firebase typesense extension.

https://extensions.dev/extensions/typesense/firestore-typesense-search

The extension tracks changes in a give collection and automatically in real time syncs them with a collection in typesense. You can install the extension multiple times to track multiple collections. However, you need to create the collection before installing the extension. You can do so through the API:

POST https://search.scoutz-internal-20.com/collections
x-typesense-api-key: 9tMUyVICflkU24cIx1vn
Content-Type: application/json

{
	"name": "teams",
	"fields": [
		{"name": "name", "type": "string"},
		{"name": "city", "type": "string"}
	]
}

Note: You don't need to include the id as a field.

If you need to back-fill the collection in typesense you can do so by creating a document typesense_sync/backfill with the data {triggered: true}