Kser is a bundle of python library whose purpose is to serialize tasks to be executed on Kafka consumers. It supports farious extensions:
You can choose to use the C bindings using confluent-kafka:
$ pip install kser[confluent]
Note
You need to install manually librdkafka, see confluent-kafka-python documentation
You can choose the pure python library kafka-python:
$ pip install kser[pykafka]
There is also a light version using HTTP ideal to produce messages (not recommended for consumption)
$ pip install kser[http]
There is a dummy version for tests, no additional import required, just use / patch with the dummy module.
It is possible to encrypt messages in the Kafka queue using libsodium.
$ pip install kser[crypto]
Note
You need to install manually libsodium, see libsodium documentation
You can export metrics using the prometheus format:
$ pip install kser[prometheus]
A few set of environment variables allow to manage the exporter:
| Environment variable | Default value |
|---|---|
| KSER_METRICS_ENABLED | no |
| KSER_METRICS_ADDRESS | 0.0.0.0 |
| KSER_METRICS_PORT | 8888 |
The exporter has only 2 metrics defined by default, it's just a sample. A good way to implement your own is to override the triggers methods (prefixed with '_') like the following example:
from kser import KSER_METRICS_ENABLED
from prometheus_client import Counter
from kser.entry import Entrypoint
MY_METRIC = Counter('kser_my_metric', 'a usefull metric')
class MyEntrypoint(Entrypoint):
def _run(self):
if KSER_METRICS_ENABLED == "yes":
MY_METRIC.inc()
return self.run()See also:
- prometheus_client documentation
- Documentation of the python Prometheus client
You can use opentracing with Jaeger to follow operation and task:
$ pip install kser[opentracing]
Configuration can be set using environment variable:
| Environment variable | Default value |
|---|---|
| JAEGER_HOST | localhost |
See also:
- Opentracing support with Jaeger
- Kser documentation
- Flask extension: Flask integration.
- Python 3.x
- Project: http://kser.readthedocs.io/
- Libsodium: https://download.libsodium.org/doc/
- confluent-kafka-python: http://docs.confluent.io/current/clients/confluent-kafka-python
- kafka-python: http://kafka-python.readthedocs.io/en/master/
- PyPI: https://pypi.python.org/pypi/kser
- Project issues: https://github.com/cdumay/kser/issues
Licensed under BSD 3-Clause License or https://opensource.org/licenses/BSD-3-Clause.