-
Notifications
You must be signed in to change notification settings - Fork 1
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
0 parents
commit 22ff869
Showing
17 changed files
with
4,525 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,12 @@ | ||
root = true | ||
|
||
[*] | ||
indent_style = space | ||
indent_size = 2 | ||
end_of_line = lf | ||
charset = utf-8 | ||
trim_trailing_whitespace = true | ||
insert_final_newline = true | ||
|
||
[*.md] | ||
trim_trailing_whitespace = false |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,39 @@ | ||
{ | ||
"extends": [ | ||
"eslint-config-airbnb-base" | ||
], | ||
"env": { | ||
"node": true, | ||
"mocha": true | ||
}, | ||
"rules": { | ||
"arrow-body-style": 0, | ||
"consistent-this": [ | ||
"error", | ||
"thiz" | ||
], | ||
"comma-dangle": 0, | ||
"global-require": 0, | ||
"no-unused-vars": [ | ||
"error", | ||
{ | ||
"vars": "all", | ||
"args": "none" | ||
} | ||
], | ||
"prefer-template": 0, | ||
"prefer-const": [ | ||
2, | ||
{ | ||
"destructuring": "any", | ||
"ignoreReadBeforeAssign": true | ||
} | ||
], | ||
"space-before-function-paren": 0, | ||
"prefer-object-spread": 0, | ||
"strict": 0 | ||
}, | ||
"plugins": [ | ||
"json" | ||
] | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,13 @@ | ||
node_modules/ | ||
npm-debug.log | ||
|
||
coverage/ | ||
|
||
.vscode | ||
.project | ||
.settings | ||
|
||
*.log | ||
*~ | ||
.idea/ | ||
.nyc_output |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
.* | ||
coverage | ||
test | ||
docker-compose.yml |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
# 0.1.0 / 2023-08-11 | ||
|
||
- Change Elasticsearch client to Opensearch client. | ||
- fork from https://github.com/vanthome/winston-elasticsearch/tree/fb870f958d90dcfc9fd75e2eb52260d16a4dc6f5 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,22 @@ | ||
The MIT License (MIT) | ||
|
||
Copyright (c) 2015 - 2018 Thomas Hoppe. | ||
Copyright (c) 2013 Jacques-Olivier D. Bernier. | ||
|
||
Permission is hereby granted, free of charge, to any person obtaining a copy | ||
of this software and associated documentation files (the "Software"), to deal | ||
in the Software without restriction, including without limitation the rights | ||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell | ||
copies of the Software, and to permit persons to whom the Software is | ||
furnished to do so, subject to the following conditions: | ||
|
||
The above copyright notice and this permission notice shall be included in | ||
all copies or substantial portions of the Software. | ||
|
||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR | ||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, | ||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE | ||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER | ||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, | ||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN | ||
THE SOFTWARE. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,207 @@ | ||
# winston-opensearch | ||
|
||
transport for the [winston](https://github.com/winstonjs/winston) logging toolkit. | ||
|
||
Most of code is from the [winston-elasticsearch](https://github.com/vanthome/winston-elasticsearch) version 0.17.3 | ||
|
||
## Features | ||
|
||
- [logstash](https://www.elastic.co/products/logstash) compatible message structure. | ||
- Thus consumable with [kibana](https://www.elastic.co/products/kibana). | ||
- Date pattern based index names. | ||
- Custom transformer function to transform logged data into a different message structure. | ||
- Buffering of messages in case of unavailability of ES. The limit is the memory as all unwritten messages are kept in memory. | ||
|
||
### Compatibility | ||
|
||
I tested **Winston 3.8.2**, **Opensearch 2.7** for AWS usage. | ||
|
||
## Installation | ||
|
||
```sh | ||
npm install --save winston winston-opensearch | ||
``` | ||
|
||
## Usage | ||
|
||
```js | ||
const winston = require('winston'); | ||
const { OpensearchTransport } = require('winston-opensearch'); | ||
|
||
const esTransportOpts = { | ||
level: 'info', | ||
}; | ||
const esTransport = new OpensearchTransport(esTransportOpts); | ||
const logger = winston.createLogger({ | ||
transports: [esTransport], | ||
}); | ||
// Compulsory error handling | ||
logger.on('error', (error) => { | ||
console.error('Error in logger caught', error); | ||
}); | ||
esTransport.on('error', (error) => { | ||
console.error('Error in logger caught', error); | ||
}); | ||
``` | ||
|
||
The [winston API for logging](https://github.com/winstonjs/winston#streams-objectmode-and-info-objects) | ||
can be used with one restriction: Only one JS object can only be logged and indexed as such. | ||
If multiple objects are provided as arguments, the contents are stringified. | ||
|
||
## Options | ||
|
||
- `level` [`info`] Messages logged with a severity greater or equal to the given one are logged to ES; others are discarded. | ||
- `index` [none | when `dataStream` is `true`, `logs-app-default`] The index to be used. This option is mutually exclusive with `indexPrefix`. | ||
- `indexPrefix` [`logs`] The prefix to use to generate the index name according to the pattern `<indexPrefix>-<indexSuffixPattern>`. Can be string or function, returning the string to use. | ||
- `indexSuffixPattern` [`YYYY.MM.DD`] a Day.js compatible date/ time pattern. | ||
- `transformer` [see below] A transformer function to transform logged data into a different message structure. | ||
- `useTransformer` [`true`] If set to `true`, the given `transformer` will be used (or the default). Set to `false` if you want to apply custom transformers during Winston's `createLogger`. | ||
- `ensureIndexTemplate` [`true`] If set to `true`, the given `indexTemplate` is checked/ uploaded to ES when the module is sending the first log message to make sure the log messages are mapped in a sensible manner. | ||
- `indexTemplate` [see file `index-template-mapping.json`] the mapping template to be ensured as parsed JSON. | ||
`ensureIndexTemplate` is `true` and `indexTemplate` is `undefined` | ||
- `flushInterval` [`2000`] Time span between bulk writes in ms. | ||
- `retryLimit` [`400`] Number of retries to connect to ES before giving up. | ||
- `healthCheckTimeout` [`30s`] Timeout for one health check (health checks will be retried forever). | ||
- `healthCheckWaitForStatus` [`yellow`] Status to wait for when check upon health. See [its API docs](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-health.html) for supported options. | ||
- `healthCheckWaitForNodes` [`>=1`] Nodes to wait for when check upon health. See [its API docs](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-health.html) for supported options. | ||
- `client` An [opensearch client](https://www.npmjs.com/package/@elastic/elasticsearch) instance. If given, the `clientOpts` are ignored. | ||
- `clientOpts` An object passed to the ES client. See [its docs](https://www.elastic.co/guide/en/elasticsearch/client/javascript-api/current/client-configuration.html) for supported options. | ||
- `waitForActiveShards` [`1`] Sets the number of shard copies that must be active before proceeding with the bulk operation. | ||
- `pipeline` [none] Sets the pipeline id to pre-process incoming documents with. See [the bulk API docs](https://www.elastic.co/guide/en/elasticsearch/client/javascript-api/current/api-reference.html#api-bulk). | ||
- `buffering` [`true`] Boolean flag to enable or disable messages buffering. The `bufferLimit` option is ignored if set to `false`. | ||
- `bufferLimit` [`null`] Limit for the number of log messages in the buffer. | ||
- `source` [none] the source of the log message. This can be useful for microservices to understand from which service a log message origins. | ||
- `internalLogger` [`console.error`] A logger of last resort to log internal errors. | ||
|
||
### Logging of OpenSearch Client | ||
|
||
The default client and options will log through `console`. | ||
|
||
### Interdependencies of Options | ||
|
||
When changing the `indexPrefix` and/or the `transformer`, | ||
make sure to provide a matching `indexTemplate`. | ||
|
||
## Transformer | ||
|
||
The transformer function allows mutation of log data as provided | ||
by winston into a shape more appropriate for indexing in Opensearch. | ||
|
||
The default transformer generates a `@timestamp` and rolls any `meta` | ||
objects into an object called `fields`. | ||
|
||
Params: | ||
|
||
- `logdata` An object with the data to log. Properties are: | ||
- `timestamp` [`new Date().toISOString()`] The timestamp of the log entry | ||
- `level` The log level of the entry | ||
- `message` The message for the log entry | ||
- `meta` The meta data for the log entry | ||
|
||
Returns: Object with the following properties | ||
|
||
- `@timestamp` The timestamp of the log entry | ||
- `severity` The log level of the entry | ||
- `message` The message for the log entry | ||
- `fields` The meta data for the log entry | ||
|
||
The default transformer function's transformation is shown below. | ||
|
||
Input A: | ||
|
||
```js | ||
{ | ||
"message": "Some message", | ||
"level": "info", | ||
"meta": { | ||
"method": "GET", | ||
"url": "/sitemap.xml", | ||
... | ||
} | ||
} | ||
``` | ||
|
||
Output A: | ||
|
||
```js | ||
{ | ||
"@timestamp": "2019-09-30T05:09:08.282Z", | ||
"message": "Some message", | ||
"severity": "info", | ||
"fields": { | ||
"method": "GET", | ||
"url": "/sitemap.xml", | ||
... | ||
} | ||
} | ||
``` | ||
|
||
The default transformer can be imported and extended | ||
|
||
### Example | ||
|
||
```js | ||
const { OpensearchTransformer } = require('winston-opensearch'); | ||
const esTransportOpts = { | ||
transformer: (logData) => { | ||
const transformed = OpensearchTransformer(logData); | ||
transformed.fields.customField = 'customValue'; | ||
return transformed; | ||
}, | ||
}; | ||
const esTransport = new OpensearchTransformer(esTransportOpts); | ||
``` | ||
|
||
Note that in current logstash versions, the only "standard fields" are | ||
`@timestamp` and `@version`, anything else is just free. | ||
|
||
A custom transformer function can be provided in the options initiation. | ||
|
||
## Events | ||
|
||
- `error`: in case of any error. | ||
|
||
## Example | ||
|
||
An example assuming default settings. | ||
|
||
### Log Action | ||
|
||
```js | ||
logger.info('Some message', {}); | ||
``` | ||
|
||
Only JSON objects are logged from the `meta` field. Any non-object is ignored. | ||
|
||
### Generated Message | ||
|
||
The log message generated by this module has the following structure: | ||
|
||
```js | ||
{ | ||
"@timestamp": "2019-09-30T05:09:08.282Z", | ||
"message": "Some log message", | ||
"severity": "info", | ||
"fields": { | ||
"method": "GET", | ||
"url": "/sitemap.xml", | ||
"headers": { | ||
"host": "www.example.com", | ||
"user-agent": "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)", | ||
"accept": "*/*", | ||
"accept-encoding": "gzip,deflate", | ||
"from": "googlebot(at)googlebot.com", | ||
"if-modified-since": "Tue, 30 Sep 2019 11:34:56 GMT", | ||
"x-forwarded-for": "66.249.78.19" | ||
} | ||
} | ||
} | ||
``` | ||
|
||
### Target Index | ||
|
||
This message would be POSTed to the following endpoint: | ||
|
||
http://localhost:9200/logs-2019.09.30/log/ | ||
|
||
So the default mapping uses an index pattern `logs-*`. |
Oops, something went wrong.