REST API Service

Simple prototype on sqlite database that has two endpoints:

• Save - This endpoint is responsible to store the log data.
• Query - This endpoint is responsible to query the stored logs data.

Installation

$ git clone $this_repo_somehow$
$ cd app/
$ pip install -r requirements.txt --user
$ python app.py

Only after first run, to initialize sqlite database, open anohter shell, go to same folder as above, and call for db.create_all() via Python shell:

$ cd app/
$ python
>>> from app import db
>>> db.create_all()

It will create a logserver.sqlite database in the same folder, which would be used to persist data.

Endpoints supported

• /save - This endpoint is responsible to store the log data. Expects:

{
    'hostname': varchar 100
    'filename': varchar 200
    'filecontent': BASE64 BLOB, no length
}

• /query/<hostname> - This endpoint is responsible to query the stored logs data. Query API will return filtered query result in JSON format. It currently supports:

Example request:

http://localhost:5000/query/almond.local

Example response:

    [
    {
        "filecontent": "dHJhdGF0YQo=",
        "filename": "/tmp/app101/blahtxt.txt",
        "hostname": "almond.local"
    }
]

To get the file content, use any BASE64 decoder in your downstream client.

Roadmap

If you have a user story, please help us prioritize backlog. So far:

• Database initialization script
• Pluggable backend API for different endpoints to different database types
  • For files metadata - relational DB
  • For file content - columnar DB plug-in (hbase)
• Pluggable Authentication
  • LDAP (i.e. flask-simpleldap)
  • OAuth2 (i.e. flask-oauthlib)
• Verbs Auto-documentation (i.e. using Swagger UI)

Known Bugs

Please submit your bugs

• (minor) JSON serialization needed, TypeError: <__main__.Log object at 0x1036e3850> is not JSON serializable

Log Forwarder

Simple prototype, relies on filesystem watchdog that has a configuration file and allows to specify:

• API server
• Log file path
• Log file pattern

Installation

Configure the filepath , fileextension and url

$ cd forwarder
$ pip install -r requirements.txt --user
$ vi conf.py
$ python forwarder.py

Depending on Operating System, there could be multiple ways to daemonize forwarder.py:

• Ubuntu - https://stackoverflow.com/questions/4705564/python-script-as-linux-service-daemon
• OS X - https://stackoverflow.com/questions/9522324/running-python-in-background-on-os-x

Roadmap:

If you have a user story, please help us prioritize backlog. So far:

• File metadata (timestamps, POSIX, etc.)
• Check logserver availability and failover
• Maintain upload work tasks in cache, if log server unavailable

Known Bugs:

Please submit your bugs