Skip to content

Commit

Permalink
Merge branch 'opensource-merge-1.5' into main_1.5
Browse files Browse the repository at this point in the history 
Link: https://code.alibaba-inc.com/sls/ilogtail/codereview/12768482
* merge opensource 1.5

* Squashed commit of the following:

commit ac09462
Author: yyuuttaaoo <[email protected]>
Date:   Fri May 26 19:24:01 2023 +0800

    hotfix version 1.5.2

commit 52d70fd
Author: Tom Yu <[email protected]>
Date:   Fri May 26 17:04:23 2023 +0800

    close #872, fix container file fd not close if multiple configs monitor the same dir (#873)

commit 7fd7263
Author: linrunqi08 <[email protected]>
Date:   Thu May 18 09:58:40 2023 +0800

    fix #851 (#852)

    * fix createAliyunLogOperationWrapper

    * fix fetch token interval

    * fix lint

    * fix create wrapper

    * fix UpdateTokenFunction

    * fix comment

    * fix comment

commit 44a9dcd
Author: linrunqi08 <[email protected]>
Date:   Thu May 11 16:23:23 2023 +0800

    add tags to profile log (#840)

    * add tags to shennong log

    * fix compile

    * fix compile

    * fix comment

    * fix comment

commit a42a021
Author: linrunqi08 <[email protected]>
Date:   Thu May 11 10:10:49 2023 +0800

    fix env config cache (#844)

commit 03dfa0b
Author: Evan <[email protected]>
Date:   Mon May 8 14:20:26 2023 +0800

    fix system disk total val (#835)

    * fix system disk total val

    ---------

    Co-authored-by: liujiapeng.ljp <[email protected]>

commit 126497d
Author: Tom Yu <[email protected]>
Date:   Thu May 4 10:44:04 2023 +0800

    Fix: stale env config (#825)

    * fix: do not read env config from exited containers

commit b0e659c
Author: Tao Yu <[email protected]>
Date:   Wed May 3 00:12:30 2023 +0800

    fix windows 64-bit cgo interface

commit baf0f66
Author: yyuuttaaoo <[email protected]>
Date:   Tue May 2 01:00:15 2023 +0800

    add 1.5.0 artifacts

commit 164c96b
Author: Tom Yu <[email protected]>
Date:   Sun Apr 30 23:58:10 2023 +0800

    bump version to 1.5.0 (#822)

commit 3c3cc4f
Author: liukezhi2 <[email protected]>
Date:   Sun Apr 30 23:15:13 2023 +0800

    feat: support dynamic tag (#810)

    * feat: support dynamic tag

    * read dynamic tag from env ALIYUN_LOG_FILE_TAGS

commit ce4eaeb
Author: dragonYang200 <[email protected]>
Date:   Sat Apr 29 16:00:23 2023 +0800

    windows compile scripts (#745)

    * windows compile script

    ---------

    Co-authored-by: Tao Yu <[email protected]>

commit 251b108
Author: Tom Yu <[email protected]>
Date:   Thu Apr 27 15:10:39 2023 +0800

    fix: processor filter key filter log even if key matches (#815)

commit c24dcce
Author: Evan <[email protected]>
Date:   Wed Apr 26 19:58:21 2023 +0800

    ping add local mode support (#807)

commit 503caf4
Author: henryzhx8 <[email protected]>
Date:   Wed Apr 26 11:10:15 2023 +0800

    add switch policy for sls data server endpoint (#707)

    * add switch policy for sls data server endpoint

commit 80e5eaa
Author: henryzhx8 <[email protected]>
Date:   Wed Apr 26 11:09:10 2023 +0800

    add support for proxy through env (#790)

    * add support for proxy through env

commit e392e2e
Author: Tom Yu <[email protected]>
Date:   Wed Apr 26 11:07:47 2023 +0800

    feat: print response time in sender (#814)

commit 1c11345
Author: Tom Yu <[email protected]>
Date:   Mon Apr 24 14:13:52 2023 +0800

    close #798 (#799)

    Fix index when creating logstore in query mode

commit 8276977
Author: haorui <[email protected]>
Date:   Mon Apr 24 14:12:24 2023 +0800

    [FIX] remove upperbound of autosplit (#797)

commit 2794cbd
Author: Liu Haoyang <[email protected]>
Date:   Tue Apr 18 14:52:43 2023 +0800

    feat: move Log Body to LogContent (#800)

    * refactor Log Body too Contents

    * Fix test build error

    * update docs

commit bcf8a41
Author: 郭刚平 <[email protected]>
Date:   Thu Apr 13 06:47:06 2023 +0800

    doc: fix example error of ext_groupinfo_filter (#785)

commit a9ac34d
Author: 郭刚平 <[email protected]>
Date:   Wed Apr 12 10:50:45 2023 +0800

    fix: docs typo error (#783)

commit 130aba6
Author: Zhu Shunjia <[email protected]>
Date:   Tue Apr 11 15:38:57 2023 +0800

    feat: otlp-log-v2 (#778)

    * otlp decoder implements decodev2 for logs

    * implement otlp log converter

commit ba50697
Author: Evan <[email protected]>
Date:   Mon Apr 10 17:16:08 2023 +0800

    Feature support goprofile pull (#764)

    * feature-support-goprofile-pull

    * support go profile pull

    * feature-support-goprofile-pull

    * feature-support-goprofile-pull

    * feature-support-goprofile-pull

    * feature-support-goprofile-pull

    * polish codes

    ---------

    Co-authored-by: liujiapeng.ljp <[email protected]>

commit df051e9
Author: PjJinchen <[email protected]>
Date:   Mon Apr 10 10:50:34 2023 +0800

    Feature add support of quote in processor_split_key_value plugin (#766)

    * feat: add support of quote in processor_split_key_value plugin

    * feat: add support of quote in processor_split_key_value plugin add tests

    * feat: add support of quote in processor_split_key_value add docs

commit cde3e53
Author: louis <[email protected]>
Date:   Fri Apr 7 19:39:39 2023 +0800

    Update processor-regex.md (#772)

    add KeepSourceIfParseError in processor_regex doc

commit a961b26
Author: 徐可甲 <[email protected]>
Date:   Thu Apr 6 12:12:59 2023 +0800

    Update developer.md (#767)

    * Update developer.md

commit aae107c
Author: PjJinchen <[email protected]>
Date:   Thu Apr 6 12:12:09 2023 +0800

    [DOC] Add daemon tutorial (#769)

    * doc: add linux daemon

commit 6340731
Author: PjJinchen <[email protected]>
Date:   Wed Apr 5 23:42:56 2023 +0800

    Processor regex replace (#757)

    * add processor_fields_with_condition to support process actions after switch-case conditions … (#139)

    * add processor_filter_compose_regex to support add fields after regex filter

commit 5518b55
Author: 郭刚平 <[email protected]>
Date:   Mon Apr 3 20:04:12 2023 +0800

    feat: check docker version to determine whether using docker buildkit (#749)

    * feat: check docker version to determine whether using docker buildkit

    * feat: use docker buildkit only if docker_version >=19.03 && SSH_AUTH_SOCK env set

    * fix: set dockerfile syntax to enable --mount=type=ssh support on v19.03

commit d7359a7
Author: PjJinchen <[email protected]>
Date:   Mon Apr 3 16:29:22 2023 +0800

    Jinchen #558 (#758)

    * upgrade: optimization processor_split_key_value plugin

commit c732ffd
Author: Takuka0311 <[email protected]>
Date:   Mon Apr 3 16:25:51 2023 +0800

    Fix ConfigServer's code (#743)

    * Standardize ConfigServer's code format.

    * Add mConfigServerAvailable

commit 61c4975
Author: Tom Yu <[email protected]>
Date:   Mon Apr 3 14:13:53 2023 +0800

    fix: missed import_plugins.sh (#763)

commit b1f07b4
Author: Zhu Shunjia <[email protected]>
Date:   Mon Apr 3 13:29:19 2023 +0800

    feat: move helper to pkg/helper and implement decodeV2 of prometheus (#742)

    * prometheus decoder supports DecodeV2

commit 16e18f3
Author: Tom Yu <[email protected]>
Date:   Mon Apr 3 13:28:43 2023 +0800

    fix: gozstd compatiblity in some environment (#761)

    * fix gozstd in some environment

commit d759e4a
Author: shalousun <[email protected]>
Date:   Fri Mar 31 15:09:09 2023 +0800

    feat: Support elasticsearch dynamic index expression formatting (#717)

    * feat: Support elasticsearch dynamic index expression formatting

    * update: Add a document to introduce how to format the es index

commit db1d5a9
Author: Bingchang Chen <[email protected]>
Date:   Fri Mar 31 13:58:13 2023 +0800

    feat: add loki flusher (#685)

    * feat: add loki flusher

    ---------

    Co-authored-by: henryzhx8 <[email protected]>
    Co-authored-by: Tom Yu <[email protected]>

commit 4fa2d6f
Author: shalousun <[email protected]>
Date:   Fri Mar 31 13:56:59 2023 +0800

    Fix: tag fields rename (#746)

    * fix #744

    * update: open enable_env_ref_in_config

    * update: add addTagIfRequired  function

    * update: optimised flusher-kafka_v2.md

    ---------

    Co-authored-by: Tom Yu <[email protected]>

commit f36d92f
Author: henryzhx8 <[email protected]>
Date:   Fri Mar 31 12:35:32 2023 +0800

    temporarily remove nginx e2e case (#753)

    * improve doc

    * temporarily remove nginx e2e case

commit 132c875
Author: PjJinchen <[email protected]>
Date:   Fri Mar 31 07:25:41 2023 +0800

    Add metrics system doc (#750)

    * add metric-system doc

commit 0a6a34d
Author: Tom Yu <[email protected]>
Date:   Thu Mar 30 23:16:28 2023 +0800

    fix: unittest (#751)

    * fix: unittest

    * update CMakeLists

commit 79c9783
Author: henryzhx8 <[email protected]>
Date:   Tue Mar 28 17:14:00 2023 +0800

    fix broken link in SUMMARY.md (#748)

commit d7ecee0
Author: 郭刚平 <[email protected]>
Date:   Mon Mar 27 14:21:21 2023 +0800

    feat: add Extension type plugin && http flusher support Authenticator、Filter (#648)

     feat: add Extension type plugin && http flusher support Authenticator

commit 246c91a
Author: dragonYang200 <[email protected]>
Date:   Fri Mar 24 14:50:51 2023 +0800

    change github.com/mindprince/gonvml to github.com/iLogtail/gonvml (#722)

    * compatible to windows compile

    * change github.com/mindprince/gonvml to github.com/iLogtail/gonvml

    * license

    * merge form master

    * refine license

    * refine license

    * refine license

    * refine license

    * refine license

commit 6869339
Author: Evan <[email protected]>
Date:   Thu Mar 23 11:37:36 2023 +0800

    add fork license markdown (#741)

    Co-authored-by: liujiapeng.ljp <[email protected]>

commit 5fc3807
Author: Evan <[email protected]>
Date:   Wed Mar 22 20:04:42 2023 +0800

    refine license (#739)

    Co-authored-by: liujiapeng.ljp <[email protected]>

commit bd6de9b
Author: Tom Yu <[email protected]>
Date:   Wed Mar 22 15:48:17 2023 +0800

    bump version to 1.4.0 (#732)

    * bump version to 1.4.0

    * polish license (#736)

    Co-authored-by: Evan <[email protected]>

commit 3a99420
Author: Weipeng Hong <[email protected]>
Date:   Wed Mar 22 15:36:36 2023 +0800

    fix: fix a typo in .gitignore file. (#735)

commit 28bfd10
Author: Evan <[email protected]>
Date:   Wed Mar 22 14:00:47 2023 +0800

    polish license (#736)

    Co-authored-by: liujiapeng.ljp <[email protected]>

commit 5f36f58
Author: zeroerror2023 <[email protected]>
Date:   Wed Mar 22 09:17:20 2023 +0800

    Update overview.md (#733)

    修正错别字

commit 59f30fc
Author: Evan <[email protected]>
Date:   Tue Mar 21 19:37:51 2023 +0800

    Add async counter for unittest (#711)

    * add async counter helper

    * extend-http-dumper-ut-time

commit dd505f1
Author: Evan <[email protected]>
Date:   Tue Mar 21 08:10:30 2023 +0800

    add test (#723)

    Co-authored-by: liujiapeng.ljp <[email protected]>

commit 1ddd4c0
Author: Zhu Shunjia <[email protected]>
Date:   Mon Mar 20 16:20:40 2023 +0800

    fix nil map panic (#721)

commit 044ba92
Author: Ao Qiao <[email protected]>
Date:   Mon Mar 20 15:16:01 2023 +0800

    Feat: Support elasticsearch flusher (#681)

    * Add flusher_elasticsearch.go

    * Add e2e test

    * Add licences

commit b32a22d
Author: Takuka0311 <[email protected]>
Date:   Mon Mar 20 15:12:51 2023 +0800

    fix some bugs of configserver (#703)

    * fix some bugs

    * change int to Json::Value::ArrayIndex

commit 1a77d57
Author: henryzhx8 <[email protected]>
Date:   Mon Mar 20 15:09:50 2023 +0800

    add stability level doc (#705)

    * add stability level doc

    * add stability info for exsiting plugins

commit e0c5ec4
Author: 郭刚平 <[email protected]>
Date:   Mon Mar 20 14:42:32 2023 +0800

    feat: retry on request timeout (#712)

commit 8bc931f
Author: 千乘(Qian Sheng) <[email protected]>
Date:   Mon Mar 20 11:36:55 2023 +0800

    change property namespace to service.namespace (#718)

commit f391911
Author: linrunqi08 <[email protected]>
Date:   Fri Mar 17 11:15:39 2023 +0800

    add ecs meta to status metric (#709)

    * add ecs meta to status metric

    * fix comment

    * add ecs region id

    * try fix test

    * try fix test

    * reset test code

commit 1c861ec
Author: Evan <[email protected]>
Date:   Wed Mar 15 16:26:44 2023 +0800

    Add ecs meta (#692)

    * add ecs meta

    * add ecs meta

    * add ecs meta

    * add ecs meta

    * add ecs meta

    * add ecs meta

    * add ecs meta

    * add ecs meta

    * polish codes

    * add ecs meta

    * add ecs meta

    * add ecs meta

    * add ecs meta

    * polish codes

    * add ecs meta

    * fix lint

    * polish codes

    * polish codes

    * polish codes

    * polish codes

    * polish codes

    * polish codes

    * polish codes

    * polish codes

    * polish codes

    * polish codes

    * polish codes

    * polish codes

    * polish codes

    * polish codes

    * polish codes

    * polish codes

    * add examples

    * add examples

    * add examples

    * add examples

    ---------

    Co-authored-by: liujiapeng.ljp <[email protected]>

commit 1a14bc4
Author: linrunqi08 <[email protected]>
Date:   Tue Mar 14 16:16:04 2023 +0800

    add switch for collect containers (#699)

    * add_switch_for_collect_containers

commit bbb54ba
Author: Liu Haoyang <[email protected]>
Date:   Wed Mar 8 15:58:26 2023 +0800

    feat: suppprt log event model (#647)

    * feat: suppprt log event model

commit 9f6b8b6
Author: dragonYang200 <[email protected]>
Date:   Wed Mar 8 15:45:44 2023 +0800

    compatible to windows compile (#694)

commit 7477801
Author: Evan <[email protected]>
Date:   Tue Mar 7 11:56:35 2023 +0800

    unify-profile-cpu-unit (#678)

    * unify-profile-cpu-unit

    * fix jfr unit

    * fix cpu and wall

    ---------

    Co-authored-by: liujiapeng.ljp <[email protected]>
    Co-authored-by: linrunqi08 <[email protected]>

commit 7a46a38
Author: linrunqi08 <[email protected]>
Date:   Tue Mar 7 11:50:35 2023 +0800

    add more env field (#688)

    * add more env field to support sls logstore options

    * add hot ttl to product api

commit f2b7f4b
Author: 徐可甲 <[email protected]>
Date:   Tue Mar 7 11:33:14 2023 +0800

    Update README.md

commit 74d423e
Author: Tom Yu <[email protected]>
Date:   Sun Mar 5 23:04:51 2023 +0800

    Add logs for container matching in stdout plugin (#689)

    * Add logs for container matching in stdout plugin

    Set user agent for sls client

commit 7a1a8b7
Author: Evan <[email protected]>
Date:   Tue Feb 28 17:15:17 2023 +0800

    enhance prometheus support staleness (#684)

    * enhance prometheus

    * enhance prometheus

    * enhance prometheus

    * enhance prometheus

    ---------

    Co-authored-by: liujiapeng.ljp <[email protected]>

commit d0d2be2
Author: 郭刚平 <[email protected]>
Date:   Sat Feb 25 00:58:34 2023 +0800

    fix: ensure generating plugins-imports outside docker container (#682)

commit 745d822
Author: Evan <[email protected]>
Date:   Wed Feb 22 19:53:04 2023 +0800

    Add pyroscope input new v1 (#673)

    * add pyroscope pprof profile support

    * polish codes

    * polish codes

    * polish codes

    * polish codes

    * polish codes

    * polish codes

    * add pyroscope pprof profile support

    * pyroscope tire support

    * pyroscope tire support

    * add pyroscope jfr (#604)

    * add jfr

    * supply go lables

    * supply go lables

    * polish codes

    * fix comment (#609)

    * polish codes

    * polish codes

    * polish codes

    * polish codes

    * polish codes

    * fix jfr data format (#610)

    * polish codes

    * polish codes

    * add profile input v2

    * fix static check

    * fix static check

    * fix test

    * polish codes

    * add profile input v2

    * add profile input v2

    * add profile input v2

    * add profile input v2

    * polish codes

    * polish codes

    * polish codes

    * polish codes

    * polish codes

    * polish codes

    * polish codes

    * polish codes

    * polish codes

    * polish codes

    * polish codes

    * fix unit

    * fix copy error

    * polish codes

    * polish codes

    * polish codes

    * polish codes

    * polish codes

    * polish codes

    * polish codes

    * polish codes

    * polish codes

    * polish codes

    * polish codes

    * polish codes

    * polish codes

    * polish codes

    * remove v2

    * polish codes

    * polish codes

    * polish codes

    * polish codes

    * polish codes

    * polish codes

    * polish codes

    * polish Pyroscope demo

    * polish Pyroscope demo

    * add changelog

    ---------

    Co-authored-by: liujiapeng.ljp <[email protected]>
    Co-authored-by: linrunqi08 <[email protected]>
    Co-authored-by: linrunqi08 <[email protected]>

commit cd6ec5b
Author: 郭刚平 <[email protected]>
Date:   Tue Feb 21 22:41:14 2023 +0800

    feat: support private plugin integration (#638)

    * feat: support private plugin integration

    * fix: optimize code

    * fix: ignore git config -l error

    * fix: add import_plugins step for some Makefile command

    * fix: ignore git config -l error

    * fix: remove useless file

    * fix: modify Dockerfile according BUILD_KIT_ENABLE

    * fix: licenses

    * fix: unit tests

    * fix: e2e tests fix

    * doc: add changelog

    * doc: adjust plugin dev docs

    * doc: add plugins.yml config docs

    * fix: typo

    * fix: unit tests

commit d18b0a2
Author: 徐可甲 <[email protected]>
Date:   Mon Feb 20 20:12:04 2023 +0800

    2022 open source report (#672)

commit 4294216
Author: 千乘(Qian Sheng) <[email protected]>
Date:   Mon Feb 20 16:38:29 2023 +0800

    Fix NPE Exception (#671)

commit 5302d63
Author: Takuka0311 <[email protected]>
Date:   Fri Feb 17 19:57:38 2023 +0800

    doc: configserver (#658)

    * Add doc on how to configure ConfigServer at Agent

commit 5bbf887
Author: Ao Qiao <[email protected]>
Date:   Fri Feb 17 19:33:34 2023 +0800

    Fix docker user privilege problem (#664)

commit e7374e9
Author: Tom Yu <[email protected]>
Date:   Fri Feb 17 09:14:32 2023 +0800

    fix #661, fix update time to make container removable (#662)

    * fix #661, fix update time to make container removable

commit ba1166f
Author: Zhu Shunjia <[email protected]>
Date:   Wed Feb 15 18:02:56 2023 +0800

    feat: flusher_oltp plugin supports logs(v1) and metrics/traces(v2) (#646)

    * move otlp constant from helper to pkg to avoid import cycle

    * pass ut and link

    * flusher_otlp support seperate configs for logs/metrics/traces

    * update documents

    * feat: disable some output

    * fix ut

    * Revert "fix ut"

    This reverts commit 77f0330.

    * Revert "feat: disable some output"

    This reverts commit f409a76.

    * change: specific configturaions for logs/traces/metrics

    * fix typos in docs and logs

    * fix comments

    * make lint

    * rename ProtocolOtlpAllV1 to ProtocolOtlpV1

    * remove unused constant

    * fix: skip leveKey

commit 1c3f93d
Author: Evan <[email protected]>
Date:   Wed Feb 15 17:17:27 2023 +0800

    Fix metadata aggregator (#654)

    * fix-metadata-aggregator

    * fix-metadata-aggregator

    * fix-metadata-aggregator

    * fix-metadata-aggregator

    * fix-metadata-aggregator

    * polish codes

    ---------

    Co-authored-by: liujiapeng.ljp <[email protected]>

commit b78911d
Author: Takuka0311 <[email protected]>
Date:   Wed Feb 15 09:18:27 2023 +0800

    feat: config server (#613)

    * Modified from pr#506

    * Update AppConfig.cpp

    ---------

    Co-authored-by: henryzhx8 <[email protected]>
    Co-authored-by: Tao Yu <[email protected]>

commit 39237b8
Author: 郭刚平 <[email protected]>
Date:   Tue Feb 14 13:34:54 2023 +0800

    feat: http flusher support variable config in Header (#643)

    * feat: http flusher support variable config in Header

    * fix: lint

    * doc: http flusher header support variables

    * doc: add docs

commit a7ff0a6
Author: 郭刚平 <[email protected]>
Date:   Tue Feb 14 11:21:07 2023 +0800

    feat: http flusher support convert Metric Event to influxdb protocol (#641)

    * feat: http flusher support convert Metric Event to influxdb protocol

    * feat: support config behavior on unexpected data

commit e8e92e0
Author: kl7sn <[email protected]>
Date:   Mon Feb 13 23:46:52 2023 +0800

    feat: add clickhouse flusher (#554) (#580)

    * feat: add clickhouse flusher (#554)

    * test: complete the e2e test #554

    * docs: docs update #554

    * feat: automatically create buffer tables and support parameter configuration

    Signed-off-by: kl7sn <[email protected]>
    Co-authored-by: Tao Yu <[email protected]>

commit 3b687e3
Author: linrunqi08 <[email protected]>
Date:   Mon Feb 13 22:25:02 2023 +0800

    fix processor_desensitize (#640)

    * fix Chinese desensitize

commit 66daff1
Author: Tom Yu <[email protected]>
Date:   Mon Feb 13 16:49:29 2023 +0800

    Grok processor report match errors by default (#645)

    * close #644, grok report match errors on default

    meanwhile
    change pure plugin LogtailSysConfDir to current dir
    fix plugin version in logs by using ldflag to set go plugin version

commit 2b7b060
Author: Zhu Shunjia <[email protected]>
Date:   Wed Feb 8 23:33:45 2023 +0800

    feat: otlp grpc/http input service for logs, metrics, and traces (v2) (#611)

    * feat: add oltp grpc input

    * feat: add cn docs for service_oltp_grpc

    * opentelemetry plugin supports http protocal

commit a201ee3
Author: 郭刚平 <[email protected]>
Date:   Wed Feb 8 11:33:48 2023 +0800

    fix: influxdb should allow empty string value (#631)

commit 700be9e
Author: 徐可甲 <[email protected]>
Date:   Wed Feb 8 11:10:25 2023 +0800

    update contributing (#635)

    * update contributing

    ---------

    Co-authored-by: Tao Yu <[email protected]>

commit 2695aa8
Author: 徐可甲 <[email protected]>
Date:   Tue Feb 7 10:53:30 2023 +0800

    chatgroup update (#633)

commit 29b72d0
Author: 徐可甲 <[email protected]>
Date:   Sun Feb 5 21:56:30 2023 +0800

    符号错误修正 (#629)

commit d29707c
Author: 徐可甲 <[email protected]>
Date:   Sun Feb 5 10:41:32 2023 +0800

    Update developer-guide.md (#628)

commit 7d0d606
Author: 徐可甲 <[email protected]>
Date:   Fri Feb 3 21:39:12 2023 +0800

    add use-cases (#627)

commit b7fda19
Author: 徐可甲 <[email protected]>
Date:   Fri Feb 3 21:20:56 2023 +0800

    Update ilogtail.md (#626)

commit 420b3ff
Author: 郭刚平 <[email protected]>
Date:   Fri Feb 3 17:30:28 2023 +0800

    refactor: move pipeline models to pkg module (#624)

    * refactor: move pipeline models to pkg module

commit 73c84d6
Author: henryzhx8 <[email protected]>
Date:   Fri Feb 3 16:25:23 2023 +0800

    fix blocking problem caused by alwaysonline config update (#622)

    * resume config before stopping it

commit 89456fa
Author: shalousun <[email protected]>
Date:   Thu Feb 2 22:12:53 2023 +0800

    feat: flusher pulsar (#603)

    * feat: flusher pulsar

commit 77b65fb
Author: urnotsally <[email protected]>
Date:   Thu Feb 2 21:20:07 2023 +0800

    feat: http flusher add Convert.Separator config field (#616)

    * feat: http flusher add Convert.Separator config field

    * fix: add docs for http flusher convert separator

    * feat: add docs for raw protocol

commit 6cb1ff3
Author: henryzhx8 <[email protected]>
Date:   Fri Jan 20 10:54:10 2023 +0800

    improve logs on the lifecycle of log reader and checkpoint (#595)

    * improve logs on the lifecycle of log reader and checkpoint

commit 92180b0
Author: urnotsally <[email protected]>
Date:   Tue Jan 17 15:46:58 2023 +0800

    feat: add metadata group aggregator supporting group pipelineGroupEvents by metadata keys & enable http flusher and http server input supporting raw protocol  (#599)

    * fix: KeyValuesImpl fix Contains func

    * fix: observePipeCollector fix ToArray func

    * feat: http server input support metric data model and raw format

    * feat: add metadata group aggregator

    * feat: http flusher support raw protocol

    * feat: stdout flusher support the events of byteArray type

    * feat: add docs for flusher/http-server, aggregator/metadata-group, input/http-service

    * fix: handle errors in writeByteArray for stdout flusher

    * fix: add DecoderV2 to interface Decoder

commit 96614fe
Author: Evan <[email protected]>
Date:   Mon Jan 16 17:56:19 2023 +0800

    fix prometheus stream fetch mode block (#607)

    Co-authored-by: liujiapeng.ljp <[email protected]>

commit 5270316
Author: linrunqi08 <[email protected]>
Date:   Mon Jan 16 17:30:06 2023 +0800

    add container info (#567)

    add container info

commit 59079d6
Author: henryzhx8 <[email protected]>
Date:   Mon Jan 16 15:05:40 2023 +0800

    change the order of dumping checkpoint to avoid checkpoint overwritten caused by inode multiplexing (#596)

    * change the order of dumping checkpoint

commit 459fd7a
Author: shalousun <[email protected]>
Date:   Sat Jan 14 07:19:39 2023 +0800

    feat: flusher kafka v2 support TLS and Kerberos (#602)

    * feat: flusher kafka support tls and Kerberos

commit c90d573
Author: henryzhx8 <[email protected]>
Date:   Tue Jan 10 17:12:56 2023 +0800

    fix time parsing problem (#552)

    * fix time parsing problem

commit 59667ee
Author: 郭刚平 <[email protected]>
Date:   Fri Dec 30 11:43:16 2022 +0800

    fix: metric labels compare error (#594)

commit a01057b
Author: Takuka0311 <[email protected]>
Date:   Fri Dec 30 09:00:51 2022 +0800

    Update processor-desensitize.md (#593)

    fix some wrong descriptions

commit 2639133
Author: 郭刚平 <[email protected]>
Date:   Fri Dec 30 08:00:31 2022 +0800

    fix: decode error when input influxdb data contains multi fields (#583)

    * fix: decode error when input influxdb data contains multi fields

commit bb03db1
Author: chenxujun <[email protected]>
Date:   Wed Dec 28 10:44:36 2022 +0800

    Update performance-compare-with-filebeat.md (#590)

commit b939cae
Author: chenxujun <[email protected]>
Date:   Wed Dec 28 10:43:38 2022 +0800

    Update build.md (#587)

commit 8f3f4f4
Author: chenxujun <[email protected]>
Date:   Wed Dec 28 10:43:28 2022 +0800

    Update system-config.md (#588)

commit 79d5094
Author: chenxujun <[email protected]>
Date:   Wed Dec 28 10:43:00 2022 +0800

    Update metric-meta-host.md (#591)

commit 38a46cb
Author: Liu Haoyang <[email protected]>
Date:   Thu Dec 15 20:49:09 2022 +0800

    Define span model and byteArray model (#571)

    * Define the metrics data model and modify the plugin manager to support the new model

    * Fix build & lint error

    * metric_mock,processor_default,flusher_stdout support metrics models

    * update vscode ignore rule

    * update vscode ignore rule

    * Optimize the metric print of stdout flusher

    * Make SlsInput and PipelineInput compatible

    * fix InputMock

    * reset winmeta Opcodes

    * reset LocalCollector

    * reset codes

    * make both input run

    * reser

    * Fix metric_docker_file rdb mysql plugin run error

    * reset skywalking bp.go

    * reset skywalking pb.go

    * rename sls function name

    * reset

    * reset

    * reset

    * reset self_metric_imp_test

    * reset

    * reset

    * reset aggregators

    * reset aggregators

    * reset aggregators

    * refactor plugin runner

    * fix input mock

    * refactor: plugin runner

    * add pkg/constraints IntUintFloat

    * Fix restart pipeline

    * Fix flusher stop

    * Fix always run config tests

    * Add some ut

    * Add ut for plugin runner

    * Add math_helper_test

    * Add go version log

    * Update makefile

    * for test

    * reset constraints.IntUintFloat

    * refactor: rename plugin1 to xxxV1 % refactor pluginrunner

    * fix: Correctly handle the case comparison of version

    * refactor: Add Initialized check for pluginRunner

    * fix: support FlushOutStore overwrite

    * refactor: Optimize unsend data merge timing

    * define span and byteArray data model

    * resolve conflict

    * rename

    * rename

    * fix build error

    * add some notes

    * Fix gofmt lint

    * rename keyValuesEmpty to keyValuesNoop

commit 4587fd6
Author: 郭刚平 <[email protected]>
Date:   Wed Dec 14 14:01:34 2022 +0800

    fix: fix metric name field name split error when contains ":" (#578)

    * fix: fix metric name field name split error when contains ":"

    * fix: fix unit tests

commit ae2f89f
Author: Tom Yu <[email protected]>
Date:   Wed Dec 14 02:07:18 2022 +0800

    fix 3 bugs in v1.3.0 (#574)

    * fix ptr bug (#566)

    * update processor_desensitize (#568)

    * fix arm version compatibility (#569)

    * fix buffer_file_path error (#573)

    * update version to 1.3.1

    Co-authored-by: henryzhx8 <[email protected]>
    Co-authored-by: Takuka0311 <[email protected]>
    Co-authored-by: linrunqi08 <[email protected]>

commit 318be0d
Author: 郭刚平 <[email protected]>
Date:   Mon Dec 12 20:02:56 2022 +0800

    feat: do not replace influxdb acceptable chars when FieldsExtend enabled (#565)

    * feat: do not replace influxdb acceptable chars when FieldsExtend enabled

    * fix: add unit tests

    * fix: gofmt

commit ac73089
Author: 郭刚平 <[email protected]>
Date:   Fri Dec 9 18:54:01 2022 +0800

    feat: add http flusher supporting influxdb protocol & add group aggregator supproting group logs by keys (#521)

    * feat: add influxdb flusher & group aggregator

    * feat: http flusher support concurrency setting

commit 2900354
Author: henryzhx8 <[email protected]>
Date:   Thu Dec 8 15:48:47 2022 +0800

    fix windows compatability (#556)

    * fix windows compatability

commit f91ff3b
Author: shalousun <[email protected]>
Date:   Thu Dec 8 14:44:57 2022 +0800

    fix: Incorrect dnsPolicy setting in ilogtail-daemonset.yaml (#551)

    * fix: For pods using hostNetwork, their dnsPolicy should be set to "ClusterFirstWithHostNet"

    * update: The use case of kafka flusher changed to kafka flusher v2

    * update: Brokers describe error

    * update: Remove wrong comma in example template

commit 9d2242a
Author: Liu Haoyang <[email protected]>
Date:   Tue Dec 6 16:15:13 2022 +0800

    feat: Define the metrics data model and modify the plugin manager to support procress the metrics data (#519)

    * Define the metrics data model and modify the plugin manager to support the new model

    * metric_mock,processor_default,flusher_stdout support metrics models

    * Optimize the metric print of stdout flusher

commit 1a1a528
Author: 徐可甲 <[email protected]>
Date:   Sun Dec 4 13:54:38 2022 +0800

    Update ilogtail.md

* fix plugins

* Merge branch 'opensource-merge-1.5' into main_1.5
  • Loading branch information
yt348264 committed May 26, 2023
1 parent 4550bd5 commit 88c167a
Show file tree
Hide file tree
Showing 26 changed files with 169 additions and 93 deletions.
2 changes: 0 additions & 2 deletions .gitignore
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,6 @@
*.test
plugin_logger.xml
license_coverage.txt
go.work
go.work.sum

# C++
CMakeFiles
Expand Down
2 changes: 1 addition & 1 deletion CHANGELOG.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,6 @@ your changes, such as:

## Example

- [public] [both] [updated] add a new feature
- [public] [both] [fix] do not read env config from exited containers

## [Unreleased]
2 changes: 1 addition & 1 deletion Makefile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
# limitations under the License.

.DEFAULT_GOAL := all
VERSION ?= 1.5.1
VERSION ?= 1.5.2
DOCKER_PUSH ?= false
DOCKER_REPOSITORY ?= aliyun/ilogtail
BUILD_REPOSITORY ?= aliyun/ilogtail_build
Expand Down
2 changes: 1 addition & 1 deletion core/CMakeLists.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ option(ENABLE_STATIC_LINK_CRT "Build Logtail by linking CRT statically" OFF)

# Name/Version information.
if (NOT DEFINED LOGTAIL_VERSION)
set(LOGTAIL_VERSION "1.5.1")
set(LOGTAIL_VERSION "1.5.2")
endif ()
message(STATUS "Version: ${LOGTAIL_VERSION}")

Expand Down
2 changes: 2 additions & 0 deletions core/controller/EventDispatcherBase.cpp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1547,11 +1547,13 @@ bool EventDispatcherBase::ReadDSPacket(int eventFd) {
Config* config = ConfigManager::GetInstance()->FindDSConfigByCategory(logGroup.category());
MetricSender::SendMetric(logGroup); // Sender::Send() will erase log group
if (config != NULL) {
std::vector<sls_logs::LogTag> empty;
LogFileProfiler::GetInstance()->AddProfilingData(config->mConfigName,
config->mRegion,
config->mProjectName,
config->mCategory,
"",
empty,
msgStr.size(),
0,
logGroup.logs_size(),
Expand Down
25 changes: 16 additions & 9 deletions core/event_handler/EventHandler.cpp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -238,8 +238,14 @@ void CreateModifyHandler::Handle(const Event& event) {
}
if ((event.IsCreate() || event.IsMoveTo()) && isDir) {
mCreateHandlerPtr->Handle(event);
} else if (event.IsCreate() || event.IsModify() || event.IsMoveFrom() || event.IsMoveTo() || event.IsDeleted()
|| event.IsContainerStopped()) {
} else if (event.IsContainerStopped() && isDir) {
for (auto& pair : mModifyHandlerPtrMap) {
LOG_DEBUG(sLogger,
("Handle container stopped event, config", pair.first)("Source", event.GetSource())(
"Object", event.GetObject())("Dev", event.GetDev())("Inode", event.GetInode()));
pair.second->Handle(event);
}
} else if (event.IsCreate() || event.IsModify() || event.IsMoveFrom() || event.IsMoveTo() || event.IsDeleted()) {
if (!event.GetConfigName().empty()) {
Config* pConfig = ConfigManager::GetInstance()->FindConfigByName(event.GetConfigName());
if (pConfig != NULL) {
Expand Down Expand Up @@ -735,6 +741,7 @@ void ModifyHandler::Handle(const Event& event) {
reader->GetProjectName(),
reader->GetCategory(),
reader->GetConvertedPath(),
reader->GetExtraTags(),
reader->GetDevInode().dev,
reader->GetDevInode().inode,
reader->GetFileSize(),
Expand Down Expand Up @@ -952,10 +959,10 @@ void ModifyHandler::DeleteTimeoutReader(int32_t timeoutInterval) {
if (interval > timeoutInterval) {
LOG_INFO(sLogger,
("remove the corresponding reader from the log reader queue",
"current file has not been updated for a long time")(
"project", (*iter)->GetProjectName())("logstore", (*iter)->GetCategory())(
"config", mConfigName)("log reader queue name", (*iter)->GetLogPath())(
"log reader queue size", readerArray.size() - 1)(
"current file has not been updated for a long time")("project", (*iter)->GetProjectName())(
"logstore", (*iter)->GetCategory())("config", mConfigName)(
"log reader queue name", (*iter)->GetLogPath())("log reader queue size",
readerArray.size() - 1)(
"file device", (*iter)->GetDevInode().dev)("file inode", (*iter)->GetDevInode().inode)(
"file size", (*iter)->GetFileSize())("last file position", (*iter)->GetLastFilePos()));
mDevInodeReaderMap.erase((*iter)->GetDevInode());
Expand Down Expand Up @@ -989,9 +996,9 @@ void ModifyHandler::DeleteRollbackReader() {
LOG_INFO(
sLogger,
("remove the corresponding reader from the reader rotator pool",
"current file has not been updated for a long time")(
"project", readerIter->second->GetProjectName())("logstore", readerIter->second->GetCategory())(
"config", mConfigName)("file name", readerIter->second->GetRealLogPath())(
"current file has not been updated for a long time")("project", readerIter->second->GetProjectName())(
"logstore", readerIter->second->GetCategory())("config", mConfigName)(
"file name", readerIter->second->GetRealLogPath())(
"file device", readerIter->second->GetDevInode().dev)("file inode",
readerIter->second->GetDevInode().inode)(
"file size", readerIter->second->GetFileSize())("last file position",
Expand Down
2 changes: 2 additions & 0 deletions core/inner/streamlog/StreamLogManager.cpp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -58,11 +58,13 @@ const bool StreamLogMsgProcessor::SendData() {
LOG_DEBUG(sLogger, ("message", "fail to invoke sender function"));
return false;
} else {
std::vector<sls_logs::LogTag> tags;
LogFileProfiler::GetInstance()->AddProfilingData(mBufferConfig.mConfigName,
mBufferConfig.mRegion,
mBufferConfig.mProjectName,
mBufferConfig.mCategory,
"syslog",
tags,
mBufferPbSize,
0L,
0L,
Expand Down
1 change: 1 addition & 0 deletions core/processor/LogProcess.cpp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -592,6 +592,7 @@ void* LogProcess::ProcessLoop(int32_t threadNo) {
projectName,
category,
logPath,
logFileReader->GetExtraTags(),
readBytes,
skipBytes,
splitLines,
Expand Down
91 changes: 68 additions & 23 deletions core/profiler/LogFileProfiler.cpp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,6 @@
#include "common/RuntimeUtil.h"
#include "common/TimeUtil.h"
#include "common/ErrorUtil.h"
#include "log_pb/sls_logs.pb.h"
#include "logger/Logger.h"
#include "sender/Sender.h"
#include "config_manager/ConfigManager.h"
Expand Down Expand Up @@ -135,6 +134,15 @@ bool LogFileProfiler::GetProfileData(LogGroup& logGroup, LogStoreStatistic* stat
contentPtr->set_key("read_avg_delay");
contentPtr->set_value(ToString(statistic->mReadCount == 0 ? 0 : statistic->mReadDelaySum / statistic->mReadCount));

if (!statistic->mTags.empty()) {
const std::vector<sls_logs::LogTag>& extraTags = statistic->mTags;
for (size_t i = 0; i < extraTags.size(); ++i) {
contentPtr = logPtr->add_contents();
contentPtr->set_key(extraTags[i].key());
contentPtr->set_value(extraTags[i].value());
}
}

if (!statistic->mErrorLine.empty()) {
contentPtr = logPtr->add_contents();
contentPtr->set_key("error_line");
Expand Down Expand Up @@ -249,6 +257,7 @@ void LogFileProfiler::AddProfilingData(const std::string& configName,
const std::string& projectName,
const std::string& category,
const std::string& filename,
const std::vector<sls_logs::LogTag>& tags,
uint64_t readBytes,
uint64_t skipBytes,
uint64_t splitLines,
Expand All @@ -258,13 +267,14 @@ void LogFileProfiler::AddProfilingData(const std::string& configName,
uint64_t historyFailures,
uint64_t sendFailures,
const std::string& errorLine) {
if (filename.size() > (size_t)0) {
if (!filename.empty()) {
// logstore statistics
AddProfilingData(configName,
region,
projectName,
category,
"",
tags,
readBytes,
skipBytes,
splitLines,
Expand All @@ -291,32 +301,54 @@ void LogFileProfiler::AddProfilingData(const std::string& configName,
if ((iter->second)->mErrorLine.empty())
(iter->second)->mErrorLine = errorLine;
(iter->second)->mLastUpdateTime = time(NULL);
} else
statisticsMap.insert(std::pair<string, LogStoreStatistic*>(key,
new LogStoreStatistic(configName,
projectName,
category,
filename,
readBytes,
skipBytes,
splitLines,
parseFailures,
regexMatchFailures,
parseTimeFailures,
historyFailures,
sendFailures,
errorLine)));
} else {
LogStoreStatistic* statistic = NULL;
if (filename.empty()) {
std::vector<sls_logs::LogTag> empty;
statistic = new LogStoreStatistic(configName,
projectName,
category,
filename,
empty,
readBytes,
skipBytes,
splitLines,
parseFailures,
regexMatchFailures,
parseTimeFailures,
historyFailures,
sendFailures,
errorLine);
} else {
statistic = new LogStoreStatistic(configName,
projectName,
category,
filename,
tags,
readBytes,
skipBytes,
splitLines,
parseFailures,
regexMatchFailures,
parseTimeFailures,
historyFailures,
sendFailures,
errorLine);
}
statisticsMap.insert(std::pair<string, LogStoreStatistic*>(key, statistic));
}
}

void LogFileProfiler::AddProfilingSkipBytes(const std::string& configName,
const std::string& region,
const std::string& projectName,
const std::string& category,
const std::string& filename,
const std::vector<sls_logs::LogTag>& tags,
uint64_t skipBytes) {
if (filename.size() > (size_t)0) {
if (!filename.empty()) {
// logstore statistics
AddProfilingSkipBytes(configName, region, projectName, category, "", skipBytes);
AddProfilingSkipBytes(configName, region, projectName, category, "", tags, skipBytes);
}
string key = projectName + "_" + category + "_" + filename;
std::lock_guard<std::mutex> lock(mStatisticLock);
Expand All @@ -326,7 +358,13 @@ void LogFileProfiler::AddProfilingSkipBytes(const std::string& configName,
(iter->second)->mSkipBytes += skipBytes;
(iter->second)->mLastUpdateTime = time(NULL);
} else {
LogStoreStatistic* statistic = new LogStoreStatistic(configName, projectName, category, filename);
LogStoreStatistic* statistic = NULL;
if (filename.empty()) {
std::vector<sls_logs::LogTag> empty;
statistic = new LogStoreStatistic(configName, projectName, category, filename, empty);
} else {
statistic = new LogStoreStatistic(configName, projectName, category, filename, tags);
}
statistic->mSkipBytes += skipBytes;
statisticsMap.insert(std::pair<string, LogStoreStatistic*>(key, statistic));
}
Expand All @@ -337,15 +375,16 @@ void LogFileProfiler::AddProfilingReadBytes(const std::string& configName,
const std::string& projectName,
const std::string& category,
const std::string& filename,
const std::vector<sls_logs::LogTag>& tags,
uint64_t dev,
uint64_t inode,
uint64_t fileSize,
uint64_t readOffset,
int32_t lastReadTime) {
if (filename.size() > (size_t)0) {
if (!filename.empty()) {
// logstore statistics
AddProfilingReadBytes(
configName, region, projectName, category, "", dev, inode, fileSize, readOffset, lastReadTime);
configName, region, projectName, category, "", tags, dev, inode, fileSize, readOffset, lastReadTime);
}
string key = projectName + "_" + category + "_" + filename;
std::lock_guard<std::mutex> lock(mStatisticLock);
Expand All @@ -354,7 +393,13 @@ void LogFileProfiler::AddProfilingReadBytes(const std::string& configName,
if (iter != statisticsMap.end()) {
(iter->second)->UpdateReadInfo(dev, inode, fileSize, readOffset, lastReadTime);
} else {
LogStoreStatistic* statistic = new LogStoreStatistic(configName, projectName, category, filename);
LogStoreStatistic* statistic = NULL;
if (filename.empty()) {
std::vector<sls_logs::LogTag> empty;
statistic = new LogStoreStatistic(configName, projectName, category, filename, empty);
} else {
statistic = new LogStoreStatistic(configName, projectName, category, filename, tags);
}
statistic->UpdateReadInfo(dev, inode, fileSize, readOffset, lastReadTime);
statisticsMap.insert(std::pair<string, LogStoreStatistic*>(key, statistic));
}
Expand Down
7 changes: 7 additions & 0 deletions core/profiler/LogFileProfiler.h
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
#include <map>
#include <json/json.h>
#include "profile_sender/ProfileSender.h"
#include "log_pb/sls_logs.pb.h"

namespace sls_logs {
class LogGroup;
Expand All @@ -45,6 +46,7 @@ class LogFileProfiler {
const std::string& projectName,
const std::string& category,
const std::string& filename,
const std::vector<sls_logs::LogTag>& tags,
uint64_t readBytes,
uint64_t skipBytes,
uint64_t splitLines,
Expand All @@ -59,13 +61,15 @@ class LogFileProfiler {
const std::string& projectName,
const std::string& category,
const std::string& filename,
const std::vector<sls_logs::LogTag>& tags,
uint64_t skipBytes);

void AddProfilingReadBytes(const std::string& configName,
const std::string& region,
const std::string& projectName,
const std::string& category,
const std::string& filename,
const std::vector<sls_logs::LogTag>& tags,
uint64_t dev,
uint64_t inode,
uint64_t fileSize,
Expand Down Expand Up @@ -94,6 +98,7 @@ class LogFileProfiler {
const std::string& projectName,
const std::string& category,
const std::string& filename,
const std::vector<sls_logs::LogTag>& tags,
uint64_t readBytes = 0,
uint64_t skipBytes = 0,
uint64_t splitLines = 0,
Expand All @@ -107,6 +112,7 @@ class LogFileProfiler {
mProjectName(projectName),
mCategory(category),
mFilename(filename),
mTags(tags),
mReadBytes(readBytes),
mSkipBytes(skipBytes),
mSplitLines(splitLines),
Expand Down Expand Up @@ -160,6 +166,7 @@ class LogFileProfiler {
std::string mProjectName;
std::string mCategory;
std::string mFilename;
std::vector<sls_logs::LogTag> mTags;
// how many bytes processed
uint64_t mReadBytes;
// how many bytes skiped
Expand Down
2 changes: 1 addition & 1 deletion docker/Dockerfile_build
View file
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ WORKDIR /src
COPY . .

ARG HOST_OS=Linux
ARG VERSION=1.5.1
ARG VERSION=1.5.2

USER root

Expand Down
2 changes: 1 addition & 1 deletion docker/Dockerfile_development_part
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
FROM sls-opensource-registry.cn-shanghai.cr.aliyuncs.com/ilogtail-community-edition/ilogtail-build-linux:latest

ARG HOST_OS=Linux
ARG VERSION=1.5.1
ARG VERSION=1.5.2

USER root
WORKDIR /ilogtail
Expand Down
Loading

0 comments on commit 88c167a

Please sign in to comment.