Skip to content

Latest commit

 

History

History
231 lines (173 loc) · 8.39 KB

README.md

File metadata and controls

231 lines (173 loc) · 8.39 KB

hawk.collector

Go Report Card

Powerful module that can handle most errors around the web.

Build and run

Install RabbitMQ manually and build Hawk.collector

make build
./bin/hawk.collector

Build for specific os

make build-linux
make build-windows
make build-darwin

or build all

make build-all

Import RabbitMQ definitions

rabbitmqadmin import config/rabbit.definitions.json

Run in Docker

make docker

Data flows

Request from errors catcher

The following structure represents data that go through the HTTP request (POST request to '/' with Content-Type: application/json)

name type description
token string (base64) JWT in base64 format
payload raw JSON Nested valid JSON
catcherType string Type of the catcher (errors/golang, errors/php)

JSON example

{
  "token": "...",
  "catcherType": "errors/golang",
  "payload": {
    "title": "Test exception",
    "timestamp": 1545203808
  }
}

Request to upload sourcemap

The following structure represents data got through the HTTP request (POST request to '/release' with Content-Type: multipart/form-data)

Form values

name type description
release string Release name
file multipart (optional) Content of the binary file
commits commitMessage (optional) Suspected commits

commitMessage has the following format:

name type description
hash string Commit hash
title string Commit description
author string Commit author
date string Commit date

Authentication is made via bearer token.

cURL sending example

curl --request POST \
 -F 'release=1.0.1'\
 -F 'commits=[{"hash":"557940a440352d9d86ad5610f2e366aafb2729e4","title":"Add some stuff","author":"[email protected]","date":"Wed May 6 13:37:00 2021 +0300"}]'\
 -F "repository=https://github.com/codex-team/hawk.api.nodejs"\
 -F file=@"main.min.js.map"\
 -H "Authorization: Bearer TOKEN" https://test.stage-k1.hawk.so/release

Response message

HTTP response from the collector. It is provided as JSON with HTTP status code.

name type description
code int internal error code
error bool if the error was occurred
message string result details

For now there are two possible HTTP status codes: 200 (OK) and 400 (Bad request).

Examples

{"error": true, "message": "Token is empty", "code": 200}

{"error": true, "message": "Invalid JSON format", "code": 400}

No body will be returned for the valid response (200).

Websocket transport

Errors can be sent via websockets (for example with the help of wscat util).

~# wscat -c wss://test.stage-k1.hawk.so/ws                                                                                                                                                                                                              ✔  11720  20:53:23
connected (press CTRL+C to quit)
> f
< {"code":400,"error":true,"message":"Invalid JSON format"}
> {"token": "...","catcherType": "errors/golang","payload": {"title": "Test exception","timestamp": 1545203808}}
< {"code":200,"error":false,"message":"OK"}

Message broker

For now we support RabbitMQ as a general AMQP broker. We declare a durable exchange with errors name. The valid payload JSON from Request structure goes directly to the exchange with the route specified by catcherType value.

Environment variables

Basic configuration is taken from .env file.

variable example value description
BROKER_URL amqp://guest:guest@localhost:5672/ Connection URI to RabbitMQ
EXCHANGE errors Basic exchange for errors
RELEASE_EXCHANGE release Basic exchange for releases
RETRY_NUMBER 10 Try to establish connection with broker for N times
RETRY_INTERVAL 4 Wait N seconds before retry to establish connection with broker
JWT_SECRET qwerty JWT token secret key
MAX_REQUEST_BODY_SIZE 20000000 Maximum available HTTP body size for any request (in bytes)
MAX_ERROR_CATCHER_MESSAGE_SIZE 25000 Maximum available HTTP body size for error request (in bytes)
MAX_SOURCEMAP_CATCHER_MESSAGE_SIZE 250000 Maximum available HTTP body size for sourcemap request (in bytes)
LISTEN localhost:3000 Listen host and port
REDIS_URL localhost:6379 Redis address
REDIS_PASSWORD password Redis password
REDIS_DISABLED_PROJECT_SET DisabledProjectsSet Name of set that contains disabled projects IDs
REDIS_BLACKLIST_IP_SET BlacklistIPsSet Name of set that contains IPs blacklist
REDIS_ALL_IPS_MAP AllIPsMap Name of map with all IPs and their request counters
REDIS_CURRENT_PERIOD_MAP CurrentPeriodMap Name of map that contains IPs and their request counters for current period
BLOCKED_PROJECTS_UPDATE_PERIOD 5s Time interval to update blocked projects list
BLACKLIST_UPDATE_PERIOD 15s Time interval to update blacklist
BLACKLIST_THRESHOLD 10000 Amount of requests, which, when achieved, forces IP to get blocked
NOTIFY_URL https://notify.bot.ifmo.su/u/ABCD1234 Address to send alerts in case of too many requests
TOKEN_UPDATE_PERIOD 10s Time interval to update token cache
PROJECTS_LIMITS_UPDATE_PERIOD 3600 Time interval to update projects limits cache (in seconds)

Rate Limiting

Rate limiting is implemented using Redis to track and enforce request limits per project. The system supports configurable limits at the project, workspace and plan level.

Configuration

Rate limits can be configured at multiple levels and applied in the following order (highest to lowest):

  1. Project level - Individual project-specific limits
  2. Workspace level - Limits that apply to all projects in a workspace
  3. Plan level - Default limits from the workspace's tariff plan

Implementation

Rate limits are tracked in rate_limit Redis set with the following pattern:

// Key: "project_id" -> value: "timestamp:count"
// example: "6762b5db032b200023854b2c" -> "1737483572:5"

Each project's rate limit data contains:

  • Timestamp of the current window
  • Request count in the current window

Rate Limit Parameters

Two main parameters control the rate limiting:

  • EventsLimit - Maximum number of events allowed in the period
  • EventsPeriod - Time window in seconds for the limit (in seconds)

Configuration

Rate limits are fetched from MongoDB. You can find them in the rateLimitSettings field of the plans,workspaces,projects collections.

rateLimitSettings is object with two fields:

  • N - Maximum number of events allowed in the period (int64)
  • T - Time window in seconds for the limit (in seconds) (int64)
{
  "rateLimitSettings": {
    "N": {
      "$numberLong": "15"
    },
    "T": {
      "$numberLong": "100"
    }
  }
}

Rate limits are automatically enforced for all incoming error and release events. No additional configuration is needed at the client level.

When a rate limit is exceeded, clients will receive a response like:

{
  "code": 402,
  "error": true,
  "message": "Rate limit exceeded"
}