Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: elasticio/csv-component-v3
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: 3.1.3
Choose a base ref
...
head repository: elasticio/csv-component-v3
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: master
Choose a head ref
  • 9 commits
  • 14 files changed
  • 6 contributors

Commits on May 6, 2022

  1. Fix memory leak on Emit Batch (#12)

    Fix memory leak on `Emit Batch`
    if0s authored May 6, 2022
    Copy the full SHA
    e187c12 View commit details

Commits on Aug 26, 2022

  1. user-agent (#13)

    EmptyInfinity authored Aug 26, 2022
    Copy the full SHA
    8701988 View commit details

Commits on Dec 7, 2022

  1. Copy the full SHA
    ead995b View commit details
  2. Merge pull request #14 from elasticio/sailor_2.7.1

    Update Sailor version to 2.7.1
    Olha Virolainen authored Dec 7, 2022
    Copy the full SHA
    89dd70c View commit details

Commits on May 2, 2023

  1. Improves (#16)

    if0s authored May 2, 2023
    Copy the full SHA
    7a4d92a View commit details

Commits on May 21, 2024

  1. New Read URL trigger

    A3a3e1 authored May 21, 2024
    Copy the full SHA
    ad3c290 View commit details

Commits on Jan 13, 2025

  1. Update README.md

    erroneous0ne authored Jan 13, 2025
    Copy the full SHA
    65508b2 View commit details
  2. Update README.md

    erroneous0ne authored Jan 13, 2025
    Copy the full SHA
    9a21e5d View commit details
  3. Merge pull request #19 from elasticio/erroneous0ne-patch-1

    Update README.md
    erroneous0ne authored Jan 13, 2025
    Copy the full SHA
    26de77a View commit details
Showing with 5,153 additions and 2,560 deletions.
  1. +0 −10 .circleci/build_slug.sh
  2. +5 −3 .circleci/config.yml
  3. +24 −0 CHANGELOG.md
  4. +125 −11 README.md
  5. +153 −5 component.json
  6. +24 −26 lib/actions/read.js
  7. +35 −14 lib/actions/write.js
  8. +118 −0 lib/triggers/read.js
  9. +10 −0 lib/util.js
  10. +4,468 −2,380 package-lock.json
  11. +7 −7 package.json
  12. +0 −95 spec/read.spec.js
  13. +182 −0 spec/triggers/read.spec.js
  14. +2 −9 spec/write.spec.js
10 changes: 0 additions & 10 deletions .circleci/build_slug.sh

This file was deleted.

8 changes: 5 additions & 3 deletions .circleci/config.yml
Original file line number Diff line number Diff line change
@@ -72,14 +72,14 @@ commands:
jobs:
test:
docker:
- image: circleci/node:14-stretch
- image: circleci/node:16-stretch
steps:
- checkout
- node/install:
node-version: << pipeline.parameters.node-version >>
- run:
name: Audit Dependencies
command: npm audit --audit-level=high
command: npm audit --production --audit-level=high
- node/install-packages:
cache-path: ./node_modules
override-ci-command: npm install
@@ -88,7 +88,7 @@ jobs:
command: npm test
build:
docker:
- image: circleci/node:14-stretch
- image: circleci/node:16-stretch
user: root
steps:
- checkout
@@ -118,6 +118,8 @@ workflows:
jobs:
- build:
name: "Build and publish docker image"
context:
- componentspusher
filters:
branches:
ignore: /.*/
24 changes: 24 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,27 @@
## 3.3.0 (May 21, 2024)
* Added new trigger - `Read CSV file from URL`

## 3.2.0 (May 02, 2023)
* Added new config fields to `Read CSV attachment` action:
* `Skip empty lines`
* `Comment char`
* Added new config fields to `Create CSV From Message Stream` and `Create CSV From JSON Array` actions:
* `New line delimiter`
* `Escape formulae`

## 3.1.6 (December 16, 2022)

* Update Sailor version to 2.7.1

## 3.1.5 (August 26, 2022)

* Update Sailor version to 2.6.29
* Get rid of vulnerabilities in dependencies
* Update component-commons-library version to 3.0.2

## 3.1.4 (May 06, 2022)
* Fix memory leak on `Emit Batch` behavior for `Read CSV attachment` action

## 3.1.3 (April 22, 2022)
* Fix a bug when emit strategy 'Emit Batch' did not process correctly

136 changes: 125 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
@@ -11,6 +11,8 @@
* [Read CSV attachment](#read-CSV-attachment)
* [Create CSV From Message Stream](#create-CSV-from-message-stream)
* [Create CSV From JSON Array](#create-CSV-from-JSON-array)
* [Triggers](#triggers)
* [Read CSV file from URL](#read-csv-file-from-url)
* [Limitations](#limitations)

## Description
@@ -48,16 +50,48 @@ To configure this action the following fields can be used:

#### Config Fields

* `Emit Behavior` - this selector configures output behavior of the component. If the option is `Fetch All` - the component emits an array of messages; `Emit Individually` - the component emits a message per row; `Emit Batch` - component will produce a series of message where each message has an array of max length equal to the `Batch Size`;
* `Emit Behavior` (dropdown, required) - this selector configures output behavior of the component.
* `Fetch All` - the component emits an array of messages;
* `Emit Individually` - the component emits a message per row;
* `Emit Batch` - component will produce a series of message where each message has an array of max length equal to the `Batch Size`;
* `Skip empty lines` (checkbox, optional) - by default, empty lines are parsed if checked they will be skipped
* `Comment char` (string, optional) - if specified, skips lines starting with this string

#### Input Metadata

* `URL` - We will fetch this URL and parse it as CSV file
* `Contains headers` - if true, the first row of parsed data will be interpreted as field names, false by default.
* `Delimiter` - The delimiting character. Leave blank to auto-detect from a list of most common delimiters.
* `Convert Data types` - numeric, date and boolean data will be converted to their type instead of remaining strings, false by default.
* `Delimiter` - The delimiting character. Leave blank to auto-detect from a list of most common delimiters or provide your own
<details><summary>Example</summary>
if you use "$" as Delimiter, this CSV:

```
a$b$c$d
```
can be parsed into this JSON
``` json
{
"column0": "a",
"column1": "b",
"column2": "c",
"column3": "d"
}
```
</details>
* `Convert Data types` - numeric data and boolean data will be converted to their type instead of remaining strings, false by default.
If `Emit Behavior` equals to `Emit Batch` - new field appears: `Batch Size` - max length of array for each message
#### Output Metadata
- For `Fetch page` and `Emit Batch`: An object with key ***result*** that has an array as its value
- For `Emit Individually`: Each object fill the entire message
#### Limitations
* If you use `Fetch All` then component needs to store whole file and object in memory that cause big memory usage
* In `Emit Batch` use wisely `Batch Size`, bigger number cause bigger memory usage
* Possible exception: `[ERR_STREAM_PREMATURE_CLOSE]` could be thrown when flow stopped before finish emiting all data in file, as stream stopped
### Create CSV From Message Stream
This action will combine multiple incoming events into a CSV file until there is a gap
@@ -66,9 +100,29 @@ and attached to the outgoing message.
#### Config Fields
* `Upload CSV as file to attachments` - If checked store the generated CSV data as an attachment. If unchecked, place the CSV as a string in the outbound message.
* `Separator` - A single char used to delimit the CSV file. Default to `,`
* `Column Order` - A string delimited with the separator indicating which columns & in what order the columns should appear in the resulting file. If omitted, the column order in the resulting file will not be deterministic. Columns names will be trimmed (removed spaces in beginning and end of column name, for example: 'col 1,col 2 ,col 3, col 4' => ['col 1', 'col 2', 'col 3', 'col 4'])
* `Upload CSV as file to attachments` (checkbox, optional) - If checked store the generated CSV data as an attachment. If unchecked, place the CSV as a string in the outbound message.
* `Separator` (string, optional) - A single char used to delimit the CSV file. Default to "`,`" but you can set any
<details><summary>Example</summary>
if you use "$" as Delimiter, this CSV:
```
a$b$c$d
```
can be parsed into this JSON
``` json
{
"column0": "a",
"column1": "b",
"column2": "c",
"column3": "d"
}
```
</details>
* `Column Order` (string, optional) - A string delimited with the separator indicating which columns & in what order the columns should appear in the resulting file. If omitted, the column order in the resulting file will not be deterministic. Columns names will be trimmed (removed spaces in beginning and end of column name, for example: 'col 1,col 2 ,col 3, col 4' => ['col 1', 'col 2', 'col 3', 'col 4'])
* `New line delimiter` (string, optional, defaults to `\r\n`) - The character used to determine newline sequence.
* `Escape formulae` (checkbox, optional) - If checked, field values that begin with `=`, `+`, `-`, `@`, `\t`, or `\r`, will be prepended with a ` ` ` to defend against injection attacks, because Excel and LibreOffice will automatically parse such cells as formulae
#### Input Metadata
@@ -94,9 +148,24 @@ This action will convert an incoming array into a CSV file
#### Config Fields
* `Upload CSV as file to attachments` - If checked store the generated CSV data as an attachment. If unchecked, place the CSV as a string in the outbound message.
* `Separator` - A single char used to delimit the CSV file. Default to `,`
* `Column Order` - A string delimited with the separator indicating which columns & in what order the columns should appear in the resulting file. If omitted, the column order in the resulting file will not be deterministic.
* `Upload CSV as file to attachments` (checkbox, optional) - If checked store the generated CSV data as an attachment. If unchecked, place the CSV as a string in the outbound message.
* `Separator` (string, optional) - A single char used to delimit the CSV file. Default to "`,`" but you can set any
<details><summary>Example </summary>
default:
```
a,b,c,d
```
using "`;`" as separator:
```
a;b;c;d
```
</details>
* `Column Order` (string, optional) - A string delimited with the separator indicating which columns & in what order the columns should appear in the resulting file. If omitted, the column order in the resulting file will not be deterministic. Columns names will be trimmed (removed spaces in beginning and end of column name, for example: 'col 1,col 2 ,col 3, col 4' => ['col 1', 'col 2', 'col 3', 'col 4'])
* `New line delimiter` (string, optional, defaults to `\r\n`) - The character used to determine newline sequence.
* `Escape formulae` (checkbox, optional) - If checked, field values that begin with `=`, `+`, `-`, `@`, `\t`, or `\r`, will be prepended with a ` ` ` to defend against injection attacks, because Excel and LibreOffice will automatically parse such cells as formulae
#### Input Metadata
@@ -116,12 +185,57 @@ This action will convert an incoming array into a CSV file
* `attachmentExpiryTime` - When the attachment is set to expire
* `contentType` - Always set to `text/csv`
## Triggers
### Read CSV file from URL
This trigger read the CSV file from the URL provided in the configuration fields and output the result as a JSON object.
The trigger works pretty much the same as the [Read CSV attachment action](#read-CSV-attachment). The difference is that all the settings are to be provided in the configuration fields, not in the body message. As the triggers do not have input messages.
#### Config Fields
* `Emit Behavior` (dropdown, required) - this selector configures output behavior of the component.
* `Fetch All` - the component emits an array of messages;
* `Emit Individually` - the component emits a message per row;
* `Emit Batch` - component will produce a series of message where each message has an array of max length equal to the `Batch Size`;
* `Skip empty lines` (checkbox, optional) - by default, empty lines are parsed if checked they will be skipped
* `Comment char` (string, optional) - if specified, skips lines starting with this string
#### Input Metadata
* `URL` (string, required) - URL of the CSV file to parse
* `Contains headers` (boolean, optional) - If true, the first row of parsed data will be interpreted as field names, false by default.
* `Delimiter` (string, optional) - The delimiting character. Leave blank to auto-detect from a list of most common delimiters or provide your own
<details><summary>Example</summary>
if you use "$" as Delimiter, this CSV:
```
a$b$c$d
```
can be parsed into this JSON
``` json
{
"column0": "a",
"column1": "b",
"column2": "c",
"column3": "d"
}
```
</details>
* `Convert Data types` (boolean, optional) - Numeric data and boolean data will be converted to their type instead of remaining strings, false by default.
#### Output Metadata
- For `Fetch page` and `Emit Batch`: An object with key ***result*** that has an array as its value
- For `Emit Individually`: Each object fill the entire message
## Limitations
### General
* You may get `Component run out of memory and terminated.` error during run-time, that means that component needs more memory, please add
`EIO_REQUIRED_RAM_MB` environment variable with an appropriate value (e.g. value `1024` means that 1024 MB will be allocated) for the component in this case.
* Maximal possible size for an attachment is 10 MB.
* The maximum possible size for an attachment is only limited by the available RAM memory of the component. In case of running into the limit the platform will give an error message, and if problem persists after a number of restarts the platform will suspend the complete integration flow.
* Attachments mechanism does not work with [Local Agent Installation](https://docs.elastic.io/getting-started/local-agent.html)
* Inbound message in `Message Stream` and each element of `JSON Array` should be a plain Object, if value not a primitive type it will be set as `[object Object]`
* Inbound message in `Message Stream` and each element of `JSON Array` should be a plain Object, if value not a primitive type it will be set as `[object Object]`
Loading