ilogtail日志采集插件被停止 #1132

jingliming · 2023-09-19T04:05:50Z

jingliming
Sep 19, 2023

环境信息

ilogtail版本：1.3.1
集群版本： kubernetes v1.20.11-aliyun.1
集群容器运行时： docker://19.3.15
os image: Alibaba Cloud Linux (Aliyun Linux) 2.1903 LTS (Hunting Beagle)
kernel version: 4.19.91-26.al7.x86_64

采集配置

enable: true
inputs:
  - Type: service_docker_stdout
    Stderr: true
    Stdout: true
    IncludeEnv:
      BW_LOGS_CATALINA: stdout
    ExternalEnvTag:
      BW_LOGS_CATALINA_TAGS: BW_LOGS_CATALINA_TAGS
processors:
  - Type: processor_add_fields
    Fields:
      logCollector: ilogtail
    IgnoreIfExist: false
  - Type: processor_regex
    SourceKey: BW_LOGS_CATALINA_TAGS
    Regex: topic=(\S+)\,tenantId=(\S+)\,clusterCode=(\S+)\,envType=(\S+)\,envCode=(\S+)\,appCode=(\S+)\,projectCode=(\S+)
    Keys:
      - topic
      - tenantId
      - clusterCode
      - envType
      - envCode
      - appCode
      - projectCode
flushers:
  - Type: flusher_stdout
    OnlyStdout: true
  - Type: flusher_kafka_v2
    Brokers:
      - xx.xx.xx.xx:8080
      - xx.xx.xx.xx:8080
      - xx.xx.xx.xx:8080
    Topic: bizworks-logging-trace

现象

查看ilogtail plugin日志发现，提示warn之后，采集停止

相关日志打印

日志1

2023-09-18 03:38:58 [INF] [plugin_export.go:250] [setGCPercentForSlowStart] set startup GC percent from 30 to 20
2023-09-18 03:38:58 [INF] [plugin_export.go:234] [func1] init plugin base, version:0.1.0
2023-09-18 03:38:58 [INF] [global_config.go:55] [func1] load global config:{
	"HostIP" : "10.121.0.193",
	"Hostname" : "iZ2tn01rf2ktvox15q6y7tZ",
	"LogtailSysConfDir" : "/usr/local/ilogtail/./"
}

2023-09-18 03:38:58 [INF] [plugin_manager.go:96] [Init] init plugin, local env tags:[_node_name_ cn-shanghai-shyc-d01.10.121.0.193 _node_ip_ 10.121.0.193]
2023-09-18 03:38:58 [INF] [checkpoint_manager.go:96] [Init] init checkpoint:<nil>
2023-09-18 03:38:58 [INF] [logstore_config.go:799] [loadBuiltinConfig] load built-in config statistics, config name: shennong_log_profile, logstore: logtail_plugin_profile
2023-09-18 03:38:58 [INF] [logstore_config.go:799] [loadBuiltinConfig] load built-in config alarm, config name: logtail_alarm, logstore: logtail_alarm
2023-09-18 03:38:58 [INF] [logstore_config.go:788] [LoadLogstoreConfig] load config:config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml	logstore:
2023-09-18 03:38:58 [ERR] [logger.go:363] [1] AlarmType:CATCH_STANDARD_OUTPUT_ALARM	err:read |0: file already closed
2023-09-18 03:38:58 [ERR] [logger.go:363] [1] AlarmType:CATCH_STANDARD_OUTPUT_ALARM	err:read |0: file already closed
2023-09-18 03:38:58 [INF] [container_discover_controller.go:167] [Init] input:param	docker discover:true	cri discover:false	static discover:false
2023-09-18 03:38:58 [INF] [container_discover_controller.go:193] [Init] init docker center, fetch all seconds:5m0s
2023-09-18 03:38:58 [INF] [container_discover_controller.go:203] [Init] init docker center, fecth all success timeout:1h40m0s
2023-09-18 03:38:58 [INF] [container_discover_controller.go:213] [Init] init docker center, client request timeout:30s
2023-09-18 03:38:58 [INF] [container_discover_controller.go:224] [Init] init docker center, max fetchOne count per second:200
2023-09-18 03:38:58 [INF] [container_discover_controller.go:250] [Init] final:param	docker discover:true	cri discover:false	static discover:false
2023-09-18 03:38:58 [INF] [context.go:209] [Background] docker event listener:start
2023-09-18 03:38:58 [INF] [plugin_export.go:154] [Resume] Resume:start
2023-09-18 03:38:58 [INF] [logstore_config.go:146] [Start] [shennong_log_profile,logtail_plugin_profile]	config start:begin
2023-09-18 03:38:58 [INF] [logstore_config.go:184] [Start] [shennong_log_profile,logtail_plugin_profile]	config start:success
2023-09-18 03:38:58 [INF] [logstore_config.go:146] [Start] [logtail_alarm,logtail_alarm]	config start:begin
2023-09-18 03:38:58 [INF] [logstore_config.go:184] [Start] [logtail_alarm,logtail_alarm]	config start:success
2023-09-18 03:38:58 [INF] [logstore_config.go:146] [Start] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]	config start:begin
2023-09-18 03:38:58 [INF] [metric_wrapper.go:42] [Run] [shennong_log_profile,logtail_plugin_profile]	start run metric :&{0xc0000e2400}
2023-09-18 03:38:58 [INF] [service_wrapper.go:38] [Run] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]	start run service:&{map[] map[] map[BW_LOGS_CATALINA:stdout] map[] map[] map[] map[] map[] map[BW_LOGS_CATALINA_TAGS:BW_LOGS_CATALINA_TAGS] map[] 3000 1000 60  3000 10240 524288 60 131072 true true true    map[] map[] map[] map[] 0xc00066cec0 0xc0006c6310 0xc0005d70e0 0xc00000f6f8 0xc00000f710 map[] map[] <nil> {{} 0 0} 0xc0000e2b00 <nil> map[] map[] 0}
2023-09-18 03:38:58 [INF] [logstore_config.go:184] [Start] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]	config start:success
2023-09-18 03:38:58 [INF] [checkpoint_manager.go:115] [Resume] checkpoint:Resume
2023-09-18 03:38:58 [INF] [plugin_export.go:162] [Resume] Resume:success
2023-09-18 03:38:58 [INF] [metric_wrapper.go:42] [Run] [logtail_alarm,logtail_alarm]	start run metric :&{0xc0000e2600}
2023-09-18 03:38:58 [INF] [input_docker_stdout.go:278] [FlushAll] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]	update match list, firstStart: true, new: 1, delete: 0
2023-09-18 03:38:58 [WRN] [input_docker_stdout.go:94] [NewDockerFileSyner] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]	AlarmType:DOCKER_STDOUT_START_ALARM	log file too big, path:/logtail_host/var/lib/docker/containers/cb738f30292b74f562337b7814dc320d5b8b620f4f2b25f5e2eca9dce266e766/cb738f30292b74f562337b7814dc320d5b8b620f4f2b25f5e2eca9dce266e766-json.log	offset:65194534	logstore:	config:config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml
2023-09-18 03:38:58 [INF] [input_docker_stdout.go:116] [NewDockerFileSyner] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]	new stdout reader id:cb738f30292b	name:/k8s_tobacco-mp-uc-debe2-group-1_tobacco-mp-uc-debe2-group-1-v5-d8f6747b8-5n9kl_0c9ee14e7b_531d9fbf-cd40-4b76-af29-bd5b46a5cc3b_0	created:2023-09-12T08:06:15.051751611Z	status:running	checkpoint_logpath:/logtail_host/var/lib/docker/containers/cb738f30292b74f562337b7814dc320d5b8b620f4f2b25f5e2eca9dce266e766/cb738f30292b74f562337b7814dc320d5b8b620f4f2b25f5e2eca9dce266e766-json.log	in_docker:true
2023-09-18 03:38:58 [INF] [log_file_reader.go:111] [NewLogFileReader] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]	read file:/logtail_host/var/lib/docker/containers/cb738f30292b74f562337b7814dc320d5b8b620f4f2b25f5e2eca9dce266e766/cb738f30292b74f562337b7814dc320d5b8b620f4f2b25f5e2eca9dce266e766-json.log	first read:true
2023-09-18 03:38:58 [WRN] [docker_stdout_processor.go:202] [ParseContainerLogLine] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]	AlarmType:PARSE_DOCKER_LINE_ALARM	parse cri docker line error:invalid CRI log, timestamp not found	line:":"stdout","time":"2023-09-18T03:38:58.371220063Z"}
	logstore:	config:config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml
2023-09-18 03:39:48 [INF] [log_file_reader.go:156] [CheckFileChange] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]	file dev inode changed, read to end and force read from beginning, file:/logtail_host/var/lib/docker/containers/cb738f30292b74f562337b7814dc320d5b8b620f4f2b25f5e2eca9dce266e766/cb738f30292b74f562337b7814dc320d5b8b620f4f2b25f5e2eca9dce266e766-json.log	old:9437395-64784-99060080	new:9437396-64784-1314564	offset:100000055
2023-09-18 03:39:58 [INF] [input_docker_stdout.go:337] [ClearUselessCheckpoint] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]	delete checkpoint, id:84243d0bb124b744758fdabeb9dce5ca1b971b095c77045c0124e2d6f05c241e
2023-09-18 03:41:46 [INF] [log_file_reader.go:156] [CheckFileChange] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]	file dev inode changed, read to end and force read from beginning, file:/logtail_host/var/lib/docker/containers/cb738f30292b74f562337b7814dc320d5b8b620f4f2b25f5e2eca9dce266e766/cb738f30292b74f562337b7814dc320d5b8b620f4f2b25f5e2eca9dce266e766-json.log	old:9437396-64784-99244512	new:9437385-64784-428356	offset:99783687
2023-09-18 03:43:58 [INF] [plugin_export.go:256] [func1] resume GC percent from 20 to 30
2023-09-18 03:43:58 [INF] [container_discover_controller.go:263] [func1] container fetch all:start
2023-09-18 03:43:58 [INF] [container_discover_controller.go:261] [func1] container fetch all:end
2023-09-18 03:44:01 [INF] [log_file_reader.go:156] [CheckFileChange] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]	file dev inode changed, read to end and force read from beginning, file:/logtail_host/var/lib/docker/containers/cb738f30292b74f562337b7814dc320d5b8b620f4f2b25f5e2eca9dce266e766/cb738f30292b74f562337b7814dc320d5b8b620f4f2b25f5e2eca9dce266e766-json.log	old:9437385-64784-12037897	new:9437387-64784-91023112	offset:100000008
2023-09-18 03:45:10 [INF] [log_file_reader.go:156] [CheckFileChange] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]	file dev inode changed, read to end and force read from beginning, file:/logtail_host/var/lib/docker/containers/cb738f30292b74f562337b7814dc320d5b8b620f4f2b25f5e2eca9dce266e766/cb738f30292b74f562337b7814dc320d5b8b620f4f2b25f5e2eca9dce266e766-json.log	old:9437387-64784-91023112	new:9437388-64784-59609470	offset:100000032
2023-09-18 03:46:12 [INF] [log_file_reader.go:156] [CheckFileChange] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]	file dev inode changed, read to end and force read from beginning, file:/logtail_host/var/lib/docker/containers/cb738f30292b74f562337b7814dc320d5b8b620f4f2b25f5e2eca9dce266e766/cb738f30292b74f562337b7814dc320d5b8b620f4f2b25f5e2eca9dce266e766-json.log	old:9437388-64784-59609470	new:9437389-64784-11343869	offset:100000007
2023-09-18 03:46:29 [INF] [plugin_export.go:137] [HoldOn] Hold on:start	flag:1
2023-09-18 03:46:29 [INF] [plugin_manager.go:125] [func1] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]	Stop config in goroutine:begin
2023-09-18 03:46:29 [INF] [logstore_config.go:221] [Stop] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]	config stop:begin	exit:true
2023-09-18 03:46:29 [INF] [logstore_config.go:229] [Stop] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]	metric stop:done
2023-09-18 03:46:29 [INF] [input_docker_stdout.go:359] [Start] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]	docker stdout main runtime stop:begin
2023-09-18 03:46:29 [INF] [input_docker_stdout.go:363] [Start] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]	docker stdout main runtime stop:success
2023-09-18 03:46:29 [INF] [service_wrapper.go:45] [func1] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]	service done:the container stdout input plugin for iLogtail, which supports docker and containerd.
2023-09-18 03:46:29 [INF] [logstore_config.go:234] [Stop] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]	service stop:done
2023-09-18 03:46:29 [INF] [logstore_config.go:238] [Stop] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]	processor stop:done
2023-09-18 03:46:29 [INF] [logstore_config.go:243] [Stop] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]	aggregator stop:done
2023-09-18 03:46:29 [INF] [logstore_config.go:218] [Stop] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]	flusher goroutine stop:done
2023-09-18 03:46:29 [INF] [logstore_config.go:264] [Stop] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]	flusher stop:done
2023-09-18 03:46:29 [INF] [logstore_config.go:268] [Stop] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]	config stop:success
2023-09-18 03:46:29 [INF] [plugin_manager.go:130] [func1] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]	Stop config in goroutine:end
2023-09-18 03:46:29 [INF] [logstore_config.go:221] [Stop] [shennong_log_profile,logtail_plugin_profile]	config stop:begin	exit:true
2023-09-18 03:46:29 [INF] [metric_wrapper.go:64] [Stop] [shennong_log_profile,logtail_plugin_profile]	stop metric success:&{0xc0000e2400}
2023-09-18 03:46:29 [INF] [logstore_config.go:229] [Stop] [shennong_log_profile,logtail_plugin_profile]	metric stop:done
2023-09-18 03:46:29 [INF] [logstore_config.go:234] [Stop] [shennong_log_profile,logtail_plugin_profile]	service stop:done
2023-09-18 03:46:29 [INF] [logstore_config.go:238] [Stop] [shennong_log_profile,logtail_plugin_profile]	processor stop:done
2023-09-18 03:46:29 [INF] [logstore_config.go:243] [Stop] [shennong_log_profile,logtail_plugin_profile]	aggregator stop:done
2023-09-18 03:46:29 [INF] [logstore_config.go:218] [Stop] [shennong_log_profile,logtail_plugin_profile]	flusher goroutine stop:done
2023-09-18 03:46:29 [INF] [logstore_config.go:264] [Stop] [shennong_log_profile,logtail_plugin_profile]	flusher stop:done
2023-09-18 03:46:29 [INF] [logstore_config.go:268] [Stop] [shennong_log_profile,logtail_plugin_profile]	config stop:success
2023-09-18 03:46:29 [INF] [logstore_config.go:221] [Stop] [logtail_alarm,logtail_alarm]	config stop:begin	exit:true
2023-09-18 03:46:29 [INF] [metric_wrapper.go:64] [Stop] [logtail_alarm,logtail_alarm]	stop metric success:&{0xc0000e2600}
2023-09-18 03:46:29 [INF] [logstore_config.go:229] [Stop] [logtail_alarm,logtail_alarm]	metric stop:done
2023-09-18 03:46:29 [INF] [logstore_config.go:234] [Stop] [logtail_alarm,logtail_alarm]	service stop:done
2023-09-18 03:46:29 [INF] [logstore_config.go:238] [Stop] [logtail_alarm,logtail_alarm]	processor stop:done
2023-09-18 03:46:29 [INF] [logstore_config.go:243] [Stop] [logtail_alarm,logtail_alarm]	aggregator stop:done
2023-09-18 03:46:29 [INF] [logstore_config.go:218] [Stop] [logtail_alarm,logtail_alarm]	flusher goroutine stop:done
2023-09-18 03:46:29 [INF] [logstore_config.go:264] [Stop] [logtail_alarm,logtail_alarm]	flusher stop:done
2023-09-18 03:46:29 [INF] [logstore_config.go:268] [Stop] [logtail_alarm,logtail_alarm]	config stop:success
2023-09-18 03:46:29 [INF] [checkpoint_manager.go:106] [HoldOn] checkpoint:HoldOn
2023-09-18 03:46:29 [INF] [checkpoint_manager.go:126] [run] checkpoint:HoldOn success
2023-09-18 03:46:29 [INF] [plugin_export.go:145] [HoldOn] Hold on:success
2023-09-18 03:46:29 [INF] [plugin_export.go:147] [HoldOn] logger:close and recover

日志2

[root@iZ2tn01rf41bczqliw0preZ ilogtail]# cat logtail_plugin.LOG
2023-09-18 03:39:54 [INF] [plugin_export.go:250] [setGCPercentForSlowStart] set startup GC percent from 30 to 20
2023-09-18 03:39:54 [INF] [plugin_export.go:234] [func1] init plugin base, version:0.1.0
2023-09-18 03:39:54 [INF] [global_config.go:55] [func1] load global config:{
	"HostIP" : "10.121.0.186",
	"Hostname" : "iZ2tn01rf41bczqliw0preZ",
	"LogtailSysConfDir" : "/usr/local/ilogtail/./"
}

2023-09-18 03:39:54 [INF] [plugin_manager.go:96] [Init] init plugin, local env tags:[_node_name_ cn-shanghai-shyc-d01.10.121.0.186 _node_ip_ 10.121.0.186]
2023-09-18 03:39:54 [INF] [checkpoint_manager.go:96] [Init] init checkpoint:<nil>
2023-09-18 03:39:54 [INF] [logstore_config.go:799] [loadBuiltinConfig] load built-in config statistics, config name: shennong_log_profile, logstore: logtail_plugin_profile
2023-09-18 03:39:54 [INF] [logstore_config.go:799] [loadBuiltinConfig] load built-in config alarm, config name: logtail_alarm, logstore: logtail_alarm
2023-09-18 03:39:54 [INF] [logstore_config.go:788] [LoadLogstoreConfig] load config:config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml	logstore:
2023-09-18 03:39:54 [ERR] [logger.go:363] [1] AlarmType:CATCH_STANDARD_OUTPUT_ALARM	err:read |0: file already closed
2023-09-18 03:39:54 [ERR] [logger.go:363] [1] AlarmType:CATCH_STANDARD_OUTPUT_ALARM	err:read |0: file already closed
2023-09-18 03:39:54 [INF] [container_discover_controller.go:167] [Init] input:param	docker discover:true	cri discover:false	static discover:false
2023-09-18 03:39:54 [INF] [container_discover_controller.go:193] [Init] init docker center, fetch all seconds:5m0s
2023-09-18 03:39:54 [INF] [container_discover_controller.go:203] [Init] init docker center, fecth all success timeout:1h40m0s
2023-09-18 03:39:54 [INF] [container_discover_controller.go:213] [Init] init docker center, client request timeout:30s
2023-09-18 03:39:54 [INF] [container_discover_controller.go:224] [Init] init docker center, max fetchOne count per second:200
2023-09-18 03:39:54 [INF] [container_discover_controller.go:250] [Init] final:param	docker discover:true	cri discover:false	static discover:false
2023-09-18 03:39:54 [INF] [context.go:209] [Background] docker event listener:start
2023-09-18 03:39:54 [INF] [plugin_export.go:154] [Resume] Resume:start
2023-09-18 03:39:54 [INF] [logstore_config.go:146] [Start] [shennong_log_profile,logtail_plugin_profile]	config start:begin
2023-09-18 03:39:54 [INF] [logstore_config.go:184] [Start] [shennong_log_profile,logtail_plugin_profile]	config start:success
2023-09-18 03:39:54 [INF] [logstore_config.go:146] [Start] [logtail_alarm,logtail_alarm]	config start:begin
2023-09-18 03:39:54 [INF] [logstore_config.go:184] [Start] [logtail_alarm,logtail_alarm]	config start:success
2023-09-18 03:39:54 [INF] [logstore_config.go:146] [Start] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]	config start:begin
2023-09-18 03:39:54 [INF] [metric_wrapper.go:42] [Run] [shennong_log_profile,logtail_plugin_profile]	start run metric :&{0xc0004ac940}
2023-09-18 03:39:54 [INF] [service_wrapper.go:38] [Run] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]	start run service:&{map[] map[] map[BW_LOGS_CATALINA:stdout] map[] map[] map[] map[] map[] map[BW_LOGS_CATALINA_TAGS:BW_LOGS_CATALINA_TAGS] map[] 3000 1000 60  3000 10240 524288 60 131072 true true true    map[] map[] map[] map[] 0xc0002d4f00 0xc000336bd0 0xc00076ad20 0xc00012f1d0 0xc00012f1e8 map[] map[] <nil> {{} 0 0} 0xc0004acc40 <nil> map[] map[] 0}
2023-09-18 03:39:54 [INF] [logstore_config.go:184] [Start] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]	config start:success
2023-09-18 03:39:54 [INF] [metric_wrapper.go:42] [Run] [logtail_alarm,logtail_alarm]	start run metric :&{0xc0004acac0}
2023-09-18 03:39:54 [INF] [checkpoint_manager.go:115] [Resume] checkpoint:Resume
2023-09-18 03:39:54 [INF] [plugin_export.go:162] [Resume] Resume:success
2023-09-18 03:39:54 [INF] [input_docker_stdout.go:278] [FlushAll] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]	update match list, firstStart: true, new: 2, delete: 0
2023-09-18 03:39:54 [WRN] [input_docker_stdout.go:94] [NewDockerFileSyner] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]	AlarmType:DOCKER_STDOUT_START_ALARM	log file too big, path:/logtail_host/var/lib/docker/containers/dc090bf0f84d39b820da2c2e2aedaa4717c7451b27f9069dbddc1a794be56f2c/dc090bf0f84d39b820da2c2e2aedaa4717c7451b27f9069dbddc1a794be56f2c-json.log	offset:82616529	logstore:	config:config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml
2023-09-18 03:39:54 [INF] [input_docker_stdout.go:116] [NewDockerFileSyner] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]	new stdout reader id:dc090bf0f84d	name:/k8s_dsc-f29d6-group-1_dsc-f29d6-group-1-v2-55bf885686-25sc8_0c9ee14e7b_26567636-cc46-4820-93dd-b755f0b41f06_0	created:2023-09-11T09:26:18.444204158Z	status:running	checkpoint_logpath:/logtail_host/var/lib/docker/containers/dc090bf0f84d39b820da2c2e2aedaa4717c7451b27f9069dbddc1a794be56f2c/dc090bf0f84d39b820da2c2e2aedaa4717c7451b27f9069dbddc1a794be56f2c-json.log	in_docker:true
2023-09-18 03:39:54 [INF] [log_file_reader.go:111] [NewLogFileReader] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]	read file:/logtail_host/var/lib/docker/containers/dc090bf0f84d39b820da2c2e2aedaa4717c7451b27f9069dbddc1a794be56f2c/dc090bf0f84d39b820da2c2e2aedaa4717c7451b27f9069dbddc1a794be56f2c-json.log	first read:true
2023-09-18 03:39:54 [WRN] [input_docker_stdout.go:94] [NewDockerFileSyner] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]	AlarmType:DOCKER_STDOUT_START_ALARM	log file too big, path:/logtail_host/var/lib/docker/containers/8cdd2dc58ef1e65af1b988d109abfc910689e19cb8d67b6fb4d9700641585a73/8cdd2dc58ef1e65af1b988d109abfc910689e19cb8d67b6fb4d9700641585a73-json.log	offset:2040692	logstore:	config:config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml
2023-09-18 03:39:54 [INF] [input_docker_stdout.go:116] [NewDockerFileSyner] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]	new stdout reader id:8cdd2dc58ef1	name:/k8s_tobacco-mp-uc-debe2-group-1_tobacco-mp-uc-debe2-group-1-v5-d8f6747b8-4g6qx_0c9ee14e7b_e4786ed6-6e0c-48bb-8cf4-53c9443c76e0_0	created:2023-09-12T08:06:46.914419656Z	status:running	checkpoint_logpath:/logtail_host/var/lib/docker/containers/8cdd2dc58ef1e65af1b988d109abfc910689e19cb8d67b6fb4d9700641585a73/8cdd2dc58ef1e65af1b988d109abfc910689e19cb8d67b6fb4d9700641585a73-json.log	in_docker:true
2023-09-18 03:39:54 [INF] [log_file_reader.go:111] [NewLogFileReader] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]	read file:/logtail_host/var/lib/docker/containers/8cdd2dc58ef1e65af1b988d109abfc910689e19cb8d67b6fb4d9700641585a73/8cdd2dc58ef1e65af1b988d109abfc910689e19cb8d67b6fb4d9700641585a73-json.log	first read:true
2023-09-18 03:39:54 [WRN] [docker_stdout_processor.go:202] [ParseContainerLogLine] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]	AlarmType:PARSE_DOCKER_LINE_ALARM	parse cri docker line error:invalid CRI log, stream type not found	line:id::defaultBizId,appsvc::http://www.101259008001.com/szzt-ywzt/MODULE_CENTER/DataSecurity/application-service.EncryptionService] END\n","stream":"stdout","time":"2023-09-18T03:38:09.109138711Z"}
	logstore:	config:config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml
2023-09-18 03:40:54 [INF] [input_docker_stdout.go:337] [ClearUselessCheckpoint] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]	delete checkpoint, id:4848f5c2308049478b103ca16fd7621a15adeddc8f82a2e875ac74810729cba6
2023-09-18 03:41:51 [INF] [log_file_reader.go:156] [CheckFileChange] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]	file dev inode changed, read to end and force read from beginning, file:/logtail_host/var/lib/docker/containers/8cdd2dc58ef1e65af1b988d109abfc910689e19cb8d67b6fb4d9700641585a73/8cdd2dc58ef1e65af1b988d109abfc910689e19cb8d67b6fb4d9700641585a73-json.log	old:4202407-64784-99731043	new:4202409-64784-890647	offset:100000187
2023-09-18 03:44:09 [INF] [log_file_reader.go:156] [CheckFileChange] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]	file dev inode changed, read to end and force read from beginning, file:/logtail_host/var/lib/docker/containers/8cdd2dc58ef1e65af1b988d109abfc910689e19cb8d67b6fb4d9700641585a73/8cdd2dc58ef1e65af1b988d109abfc910689e19cb8d67b6fb4d9700641585a73-json.log	old:4202409-64784-7435930	new:4202413-64784-85012158	offset:100000028
2023-09-18 03:44:54 [INF] [plugin_export.go:256] [func1] resume GC percent from 20 to 30
2023-09-18 03:44:54 [INF] [container_discover_controller.go:263] [func1] container fetch all:start
2023-09-18 03:44:54 [INF] [container_discover_controller.go:261] [func1] container fetch all:end
2023-09-18 03:45:24 [INF] [log_file_reader.go:156] [CheckFileChange] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]	file dev inode changed, read to end and force read from beginning, file:/logtail_host/var/lib/docker/containers/8cdd2dc58ef1e65af1b988d109abfc910689e19cb8d67b6fb4d9700641585a73/8cdd2dc58ef1e65af1b988d109abfc910689e19cb8d67b6fb4d9700641585a73-json.log	old:4202413-64784-85012158	new:4202415-64784-60375610	offset:100000021
2023-09-18 03:46:22 [INF] [log_file_reader.go:156] [CheckFileChange] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]	file dev inode changed, read to end and force read from beginning, file:/logtail_host/var/lib/docker/containers/8cdd2dc58ef1e65af1b988d109abfc910689e19cb8d67b6fb4d9700641585a73/8cdd2dc58ef1e65af1b988d109abfc910689e19cb8d67b6fb4d9700641585a73-json.log	old:4202415-64784-60375610	new:4202416-64784-6505990	offset:100000200
2023-09-18 03:46:25 [INF] [plugin_export.go:137] [HoldOn] Hold on:start	flag:1
2023-09-18 03:46:25 [INF] [plugin_manager.go:125] [func1] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]	Stop config in goroutine:begin
2023-09-18 03:46:25 [INF] [logstore_config.go:221] [Stop] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]	config stop:begin	exit:true
2023-09-18 03:46:25 [INF] [logstore_config.go:229] [Stop] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]	metric stop:done
2023-09-18 03:46:25 [INF] [input_docker_stdout.go:359] [Start] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]	docker stdout main runtime stop:begin
2023-09-18 03:46:25 [INF] [log_file_reader.go:352] [ReadAndProcess] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]	receive stop signal when read data, path:/logtail_host/var/lib/docker/containers/8cdd2dc58ef1e65af1b988d109abfc910689e19cb8d67b6fb4d9700641585a73/8cdd2dc58ef1e65af1b988d109abfc910689e19cb8d67b6fb4d9700641585a73-json.log
2023-09-18 03:46:25 [INF] [input_docker_stdout.go:363] [Start] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]	docker stdout main runtime stop:success
2023-09-18 03:46:25 [INF] [service_wrapper.go:45] [func1] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]	service done:the container stdout input plugin for iLogtail, which supports docker and containerd.
2023-09-18 03:46:25 [INF] [logstore_config.go:234] [Stop] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]	service stop:done
2023-09-18 03:46:25 [INF] [logstore_config.go:238] [Stop] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]	processor stop:done
2023-09-18 03:46:26 [INF] [logstore_config.go:243] [Stop] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]	aggregator stop:done
2023-09-18 03:46:26 [INF] [logstore_config.go:218] [Stop] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]	flusher goroutine stop:done
2023-09-18 03:46:26 [INF] [logstore_config.go:264] [Stop] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]	flusher stop:done
2023-09-18 03:46:26 [INF] [logstore_config.go:268] [Stop] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]	config stop:success
2023-09-18 03:46:26 [INF] [plugin_manager.go:130] [func1] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]	Stop config in goroutine:end
2023-09-18 03:46:26 [INF] [logstore_config.go:221] [Stop] [shennong_log_profile,logtail_plugin_profile]	config stop:begin	exit:true
2023-09-18 03:46:26 [INF] [metric_wrapper.go:64] [Stop] [shennong_log_profile,logtail_plugin_profile]	stop metric success:&{0xc0004ac940}
2023-09-18 03:46:26 [INF] [logstore_config.go:229] [Stop] [shennong_log_profile,logtail_plugin_profile]	metric stop:done
2023-09-18 03:46:26 [INF] [logstore_config.go:234] [Stop] [shennong_log_profile,logtail_plugin_profile]	service stop:done
2023-09-18 03:46:26 [INF] [logstore_config.go:238] [Stop] [shennong_log_profile,logtail_plugin_profile]	processor stop:done
2023-09-18 03:46:26 [INF] [logstore_config.go:243] [Stop] [shennong_log_profile,logtail_plugin_profile]	aggregator stop:done
2023-09-18 03:46:26 [INF] [logstore_config.go:218] [Stop] [shennong_log_profile,logtail_plugin_profile]	flusher goroutine stop:done
2023-09-18 03:46:26 [INF] [logstore_config.go:264] [Stop] [shennong_log_profile,logtail_plugin_profile]	flusher stop:done
2023-09-18 03:46:26 [INF] [logstore_config.go:268] [Stop] [shennong_log_profile,logtail_plugin_profile]	config stop:success
2023-09-18 03:46:26 [INF] [logstore_config.go:221] [Stop] [logtail_alarm,logtail_alarm]	config stop:begin	exit:true
2023-09-18 03:46:26 [INF] [metric_wrapper.go:64] [Stop] [logtail_alarm,logtail_alarm]	stop metric success:&{0xc0004acac0}
2023-09-18 03:46:26 [INF] [logstore_config.go:229] [Stop] [logtail_alarm,logtail_alarm]	metric stop:done
2023-09-18 03:46:26 [INF] [logstore_config.go:234] [Stop] [logtail_alarm,logtail_alarm]	service stop:done
2023-09-18 03:46:26 [INF] [logstore_config.go:238] [Stop] [logtail_alarm,logtail_alarm]	processor stop:done
2023-09-18 03:46:26 [INF] [logstore_config.go:243] [Stop] [logtail_alarm,logtail_alarm]	aggregator stop:done
2023-09-18 03:46:26 [INF] [logstore_config.go:218] [Stop] [logtail_alarm,logtail_alarm]	flusher goroutine stop:done
2023-09-18 03:46:26 [INF] [logstore_config.go:264] [Stop] [logtail_alarm,logtail_alarm]	flusher stop:done
2023-09-18 03:46:26 [INF] [logstore_config.go:268] [Stop] [logtail_alarm,logtail_alarm]	config stop:success
2023-09-18 03:46:26 [INF] [checkpoint_manager.go:106] [HoldOn] checkpoint:HoldOn
2023-09-18 03:46:26 [INF] [checkpoint_manager.go:126] [run] checkpoint:HoldOn success
2023-09-18 03:46:26 [INF] [plugin_export.go:145] [HoldOn] Hold on:success
2023-09-18 03:46:26 [INF] [plugin_export.go:147] [HoldOn] logger:close and recover

linrunqi08 · 2023-09-19T11:51:37Z

linrunqi08
Sep 19, 2023
Collaborator

@jingliming ilogtail.LOG也贴一下吧

0 replies

jingliming · 2023-09-27T02:33:29Z

jingliming
Sep 27, 2023
Author

[2023-09-18 04:46:04.486634] [info] [000010] /src/core/app_config/AppConfigBase.cpp:151 AppConfigBase AppConfigBase:success
[2023-09-18 04:46:04.486651] [info] [000010] /src/core/app_config/AppConfigBase.cpp:314 load env tags from env key:node_name|node_ip
[2023-09-18 04:46:04.486669] [info] [000010] /src/core/app_config/AppConfigBase.cpp:322 load env, key:node_name value:cn-shanghai-shyc-d01.10.121.0.186
[2023-09-18 04:46:04.486671] [info] [000010] /src/core/app_config/AppConfigBase.cpp:322 load env, key:node_ip value:10.121.0.186
[2023-09-18 04:46:04.486677] [info] [000010] /src/core/app_config/AppConfigBase.cpp:379 purage container mode:true
[2023-09-18 04:46:04.486690] [info] [000010] /src/core/logtail.cpp:156 change working dir:/usr/local/ilogtail/1.3.1 fail, reason:No such file or directory
[2023-09-18 04:46:04.486704] [info] [000010] /src/core/logtail.cpp:158 change working dir:/usr/local/ilogtail/ result:0
[2023-09-18 04:46:04.486773] [info] [000010] /src/core/app_config/AppConfigBase.cpp:1120 set logtail sys conf dir:/usr/local/ilogtail/./ user local config path:/usr/local/ilogtail/./user_local_config.json user local config dir path:/usr/local/ilogtail/./user_config.d/ user local yaml config dir path:/usr/local/ilogtail/./user_yaml_config.d/
[2023-09-18 04:46:04.486805] [info] [000010] /src/core/app_config/AppConfigBase.cpp:283 load logtail config file, path:ilogtail_config.json
[2023-09-18 04:46:04.486808] [info] [000010] /src/core/app_config/AppConfigBase.cpp:284 load logtail config file, detail:{}

[2023-09-18 04:46:04.486905] [info] [000010] /src/core/app_config/AppConfigBase.cpp:595 logtail checkpoint path:/usr/local/ilogtail/checkpoint/logtail_check_point
[2023-09-18 04:46:04.486917] [info] [000010] /src/core/common/JsonUtil.cpp:138 load config from env:docker_file_cache_path value:checkpoint/docker_path_config.json
[2023-09-18 04:46:04.486921] [info] [000010] /src/core/common/JsonUtil.cpp:171 load parameter from env:docker_file_cache_path value:checkpoint/docker_path_config.json
[2023-09-18 04:46:04.486940] [info] [000010] /src/core/app_config/AppConfigBase.cpp:340 set cpu_usage_limit from env, value:1
[2023-09-18 04:46:04.486950] [info] [000010] /src/core/app_config/AppConfigBase.cpp:340 set mem_usage_limit from env, value:512
[2023-09-18 04:46:04.486970] [info] [000010] /src/core/app_config/AppConfigBase.cpp:1046 default cache size, polling total max stat:250000 per dir max stat:25000 per config max stat:25000 cache config max size:250000 modify max:25000watch dir count max:25000 max open files limit:32765 max reader open files limit:25000
[2023-09-18 04:46:04.486974] [info] [000010] /src/core/app_config/AppConfigBase.cpp:1050 send byte per second limit:26214400 batch send interval:3 batch send size:262144
[2023-09-18 04:46:04.487031] [info] [000010] /src/core/common/TimeUtil.cpp:232 get system boot time from /proc/uptime:1691148067
[2023-09-18 04:46:04.487085] [info] [000010] /src/core/config_manager/ConfigManagerBase.cpp:2138 invalid aliuid conf dir:/usr/local/ilogtail/./users error:No such file or directory
[2023-09-18 04:46:04.487122] [info] [000010] /src/core/config_manager/ConfigManagerBase.cpp:2145 recreate aliuid conf dir success:/usr/local/ilogtail/./users
[2023-09-18 04:46:04.497830] [info] [000010] /src/core/config_manager/ConfigManagerBase.cpp:2398 user yaml config file loaded:/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml
[2023-09-18 04:46:04.497848] [info] [000010] /src/core/config_manager/ConfigManagerBase.cpp:2407 user yaml config removed or added, last:0 now:1
[2023-09-18 04:46:04.497866] [info] [000010] /src/core/config_manager/ConfigManagerBase.cpp:2013 load user defined id from env:default
[2023-09-18 04:46:04.498169] [info] [000010] /src/core/config_manager/ConfigYamlToJson.cpp:245 Trans yaml to json:success config_name:/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml is_file_mode:false input_plugin_type:service_docker_stdout has_accelerate_processor:false accelerate_processor_plugin_type: log_split_processor: logtype:plugin user_local_json_config:{
"metrics" :
{
"config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml" :
{
"enable" : true,
"log_type" : "plugin",
"plugin" :
{
"flushers" :
[
{
"detail" :
{
"OnlyStdout" : true
},
"type" : "flusher_stdout"
},
{
"detail" :
{
"Brokers" :
[
"10.120.57.85:8080",
"10.120.176.238:8080",
"10.120.177.49:8080"
],
"Topic" : "bizworks-logging-trace"
},
"type" : "flusher_kafka_v2"
}
],
"inputs" :
[
{
"detail" :
{
"ExternalEnvTag" :
{
"BW_LOGS_CATALINA_TAGS" : "BW_LOGS_CATALINA_TAGS"
},
"IncludeEnv" :
{
"BW_LOGS_CATALINA" : "stdout"
},
"Stderr" : true,
"Stdout" : true
},
"type" : "service_docker_stdout"
}
],
"processors" :
[
{
"detail" :
{
"Fields" :
{
"logCollector" : "ilogtail"
},
"IgnoreIfExist" : false
},
"type" : "processor_add_fields"
},
{
"detail" :
{
"Keys" :
[
"topic",
"tenantId",
"clusterCode",
"envType",
"envCode",
"appCode",
"projectCode"
],
"Regex" : "topic=(\S+)\,tenantId=(\S+)\,clusterCode=(\S+)\,envType=(\S+)\,envCode=(\S+)\,appCode=(\S+)\,projectCode=(\S+)",
"SourceKey" : "BW_LOGS_CATALINA_TAGS"
},
"type" : "processor_regex"
}
]
}
}
}
}

[2023-09-18 04:46:04.498428] [info] [000010] /src/core/config_manager/ConfigManagerBase.cpp:1070 load user_yaml_config.d config:true file:/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml now config count:1
[2023-09-18 04:46:04.498601] [info] [000010] /src/core/monitor/Monitor.cpp:537 machine cpu cores:32
[2023-09-18 04:46:04.498819] [info] [000010] /src/core/sls_control/SLSControl.cpp:65 user agent:ilogtail/1.3.1 (Linux; 4.19.91; x86_64) ip/10.121.0.186 env/K8S-Daemonset
[2023-09-18 04:46:04.498832] [info] [000010] /src/core/sender/Sender.cpp:453 Set sender queue param depend value:10
[2023-09-18 04:46:04.498999] [info] [000018] /src/core/sender/Sender.cpp:1264 SendThread:start
[2023-09-18 04:46:04.499058] [info] [000010] /src/core/plugin/LogtailPlugin.cpp:307 load plugin base, config count:1 docker env config:false dl file:/usr/local/ilogtail/libPluginAdapter.so
[2023-09-18 04:46:04.499142] [info] [000010] /src/core/plugin/LogtailPlugin.cpp:328 check plugin adapter version success, version:300
[2023-09-18 04:46:04.553365] [info] [000010] /src/core/plugin/LogtailPlugin.cpp:439 init plugin base:success
[2023-09-18 04:46:04.695680] [info] [000010] /src/core/plugin/LogtailPlugin.cpp:116 logtail plugin Resume:start
[2023-09-18 04:46:04.696101] [info] [000010] /src/core/plugin/LogtailPlugin.cpp:118 logtail plugin Resume:success
[2023-09-18 04:46:04.696160] [info] [000010] /src/core/common/DynamicLibHelper.cpp:104 load glibc dynamic library:begin
[2023-09-18 04:46:04.696208] [info] [000010] /src/core/common/DynamicLibHelper.cpp:118 load glibc dynamic library:success
[2023-09-18 04:46:04.696282] [info] [000010] /src/core/checkpoint/CheckPointManager.cpp:154 load checkpoint, version:200 file check point:0 dir check point:0
[2023-09-18 04:46:04.696338] [info] [000010] /src/core/profiler/LogIntegrity.cpp:507 no integrity file to load:/usr/local/ilogtail/logtail_integrity_snapshot.json
[2023-09-18 04:46:04.696360] [info] [000010] /src/core/profiler/LogLineCount.cpp:263 no line count file to load:/usr/local/ilogtail/logtail_line_count_snapshot.json
[2023-09-18 04:46:04.696538] [info] [000010] /src/core/logtail.cpp:282 Logtail started, appInfo:{
"UUID" : "4641ABBC-55DE-11EE-97E6-00163E01096C",
"build_date" : "20221213",
"compiler" : "GCC 4.8.5",
"git_hash" : "1efeed384abafa6b1cae46c5f85aee4558f984e3",
"hostname" : "iZ2tn01rf41bczqliw0preZ",
"instance_id" : "4641A5EA-55DE-11EE-90E5-00163E01096C_10.121.0.186_1695012364",
"ip" : "10.121.0.186",
"logtail_version" : "1.3.1 Community Edition",
"os" : "Linux; 4.19.91-26.al7.x86_64; #1 SMP Tue May 24 12:11:55 CST 2022; x86_64",
"update_time" : "2023-09-18 04:46:04"
}

[2023-09-18 04:46:04.696686] [info] [000010] /src/core/controller/EventDispatcherBase.cpp:588 start add existed check point events, size:0
[2023-09-18 04:46:04.696697] [info] [000010] /src/core/controller/EventDispatcherBase.cpp:602 add existed checkpoint events, size:0 cache size:0 event size:0 delete size:0
[2023-09-18 04:46:04.696854] [info] [000044] /src/core/polling/PollingModify.cpp:224 PollingModify::Polling:start
[2023-09-18 04:46:04.697018] [info] [000047] /src/core/processor/LogProcess.cpp:205 local timezone offset second:0
[2023-09-18 05:01:14.597655] [info] [000010] /src/core/controller/EventDispatcherBase.cpp:1274 Start dump checkpoint, hold on LogInput:1695013274
[2023-09-18 05:01:14.603778] [info] [000010] /src/core/controller/EventDispatcherBase.cpp:1285 Finish dump checkpoint, LogInput resumed:1695013274
[2023-09-18 05:16:21.382064] [info] [000010] /src/core/controller/EventDispatcherBase.cpp:1274 Start dump checkpoint, hold on LogInput:1695014181
[2023-09-18 05:16:21.398052] [info] [000010] /src/core/controller/EventDispatcherBase.cpp:1285 Finish dump checkpoint, LogInput resumed:1695014181
[2023-09-18 05:31:25.848659] [info] [000010] /src/core/controller/EventDispatcherBase.cpp:1274 Start dump checkpoint, hold on LogInput:1695015085
[2023-09-18 05:31:25.864476] [info] [000010] /src/core/controller/EventDispatcherBase.cpp:1285 Finish dump checkpoint, LogInput resumed:1695015085
[2023-09-18 05:46:38.820474] [info] [000010] /src/core/controller/EventDispatcherBase.cpp:1274 Start dump checkpoint, hold on LogInput:1695015998
[2023-09-18 05:46:38.827952] [info] [000010] /src/core/controller/EventDispatcherBase.cpp:1285 Finish dump checkpoint, LogInput resumed:1695015998
[2023-09-18 06:01:51.300127] [info] [000010] /src/core/controller/EventDispatcherBase.cpp:1274 Start dump checkpoint, hold on LogInput:1695016911
[2023-09-18 06:01:51.317647] [info] [000010] /src/core/controller/EventDispatcherBase.cpp:1285 Finish dump checkpoint, LogInput resumed:1695016911
[2023-09-18 06:17:00.305316] [info] [000010] /src/core/controller/EventDispatcherBase.cpp:1274 Start dump checkpoint, hold on LogInput:1695017820
[2023-09-18 06:17:00.318430] [info] [000010] /src/core/controller/EventDispatcherBase.cpp:1285 Finish dump checkpoint, LogInput resumed:1695017820
[2023-09-18 06:32:09.114649] [info] [000010] /src/core/controller/EventDispatcherBase.cpp:1274 Start dump checkpoint, hold on LogInput:1695018729
[2023-09-18 06:32:09.123048] [info] [000010] /src/core/controller/EventDispatcherBase.cpp:1285 Finish dump checkpoint, LogInput resumed:1695018729
[2023-09-18 06:47:29.014398] [info] [000010] /src/core/controller/EventDispatcherBase.cpp:1274 Start dump checkpoint, hold on LogInput:1695019649
[2023-09-18 06:47:29.019906] [info] [000010] /src/core/controller/EventDispatcherBase.cpp:1285 Finish dump checkpoint, LogInput resumed:1695019649
[2023-09-18 06:55:05.124668] [error] [000015] /src/core/monitor/Monitor.cpp:179 Resource used by program exceeds upper limit:prepare restart Logtail cpu_usage:1.04016 mem_rss:130
[2023-09-18 06:55:05.201723] [info] [000010] /src/core/controller/EventDispatcherBase.cpp:1200 LogInput:hold on
[2023-09-18 06:55:05.219054] [info] [000010] /src/core/controller/EventDispatcherBase.cpp:1203 dump checkpoint to local:start
[2023-09-18 06:55:05.219080] [info] [000010] /src/core/controller/EventDispatcherBase.cpp:1205 dump checkpoint to local:30%
[2023-09-18 06:55:05.219323] [info] [000010] /src/core/controller/EventDispatcherBase.cpp:1209 dump checkpoint to local:success
[2023-09-18 06:55:05.219337] [info] [000010] /src/core/controller/EventDispatcherBase.cpp:1213 dump line count data to local file:start
[2023-09-18 06:55:05.219341] [info] [000010] /src/core/controller/EventDispatcherBase.cpp:1216 dump integrity data to local file:start
[2023-09-18 06:55:05.219344] [info] [000010] /src/core/controller/EventDispatcherBase.cpp:1220 flush log process buffer:start
[2023-09-18 06:55:05.219348] [info] [000010] /src/core/plugin/LogtailPlugin.cpp:100 logtail plugin HoldOn:start
[2023-09-18 06:55:05.731192] [info] [000010] /src/core/plugin/LogtailPlugin.cpp:104 logtail plugin HoldOn:success cost:512
[2023-09-18 06:55:05.731222] [info] [000010] /src/core/observer/network/NetworkObserver.cpp:54 hold on:observer
[2023-09-18 06:55:05.732319] [info] [000010] /src/core/controller/EventDispatcherBase.cpp:1242 flush log process buffer:success
[2023-09-18 06:55:05.732336] [info] [000010] /src/core/controller/EventDispatcherBase.cpp:1247 flush out sender data:start
[2023-09-18 06:55:05.832487] [info] [000010] /src/core/controller/EventDispatcherBase.cpp:1251 flush out sender data:success

0 replies

jingliming · 2023-09-27T02:38:21Z

jingliming
Sep 27, 2023
Author

0 replies

jingliming · 2023-09-27T02:38:57Z

jingliming
Sep 27, 2023
Author

apiVersion: apps/v1
kind: DaemonSet
metadata:
annotations:
deprecated.daemonset.template.generation: "1"
meta.helm.sh/release-name: ilogtail
meta.helm.sh/release-namespace: bizworks-system
creationTimestamp: "2023-08-18T07:50:57Z"
generation: 1
labels:
app.kubernetes.io/managed-by: Helm
k8s-app: logtail-ds
managedFields:

apiVersion: apps/v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:deprecated.daemonset.template.generation: {}
f:meta.helm.sh/release-name: {}
f:meta.helm.sh/release-namespace: {}
f:labels:
.: {}
f:app.kubernetes.io/managed-by: {}
f:k8s-app: {}
f:spec:
f:revisionHistoryLimit: {}
f:selector: {}
f:template:
f:metadata:
f:labels:
.: {}
f:k8s-app: {}
f:spec:
f:containers:
k:{"name":"logtail"}:
.: {}
f:env:
.: {}
k:{"name":"node_ip"}:
.: {}
f:name: {}
f:valueFrom:
.: {}
f:fieldRef:
.: {}
f:apiVersion: {}
f:fieldPath: {}
k:{"name":"node_name"}:
.: {}
f:name: {}
f:valueFrom:
.: {}
f:fieldRef:
.: {}
f:apiVersion: {}
f:fieldPath: {}
k:{"name":"ALIYUN_LOG_ENV_TAGS"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"cpu_usage_limit"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"default_access_key"}:
.: {}
f:name: {}
f:valueFrom:
.: {}
f:secretKeyRef:
.: {}
f:key: {}
f:name: {}
f:optional: {}
k:{"name":"default_access_key_id"}:
.: {}
f:name: {}
f:valueFrom:
.: {}
f:secretKeyRef:
.: {}
f:key: {}
f:name: {}
f:optional: {}
k:{"name":"mem_usage_limit"}:
.: {}
f:name: {}
f:value: {}
f:image: {}
f:imagePullPolicy: {}
f:name: {}
f:resources:
.: {}
f:limits:
.: {}
f:cpu: {}
f:memory: {}
f:requests:
.: {}
f:cpu: {}
f:memory: {}
f:terminationMessagePath: {}
f:terminationMessagePolicy: {}
f:volumeMounts:
.: {}
k:{"mountPath":"/logtail_host"}:
.: {}
f:mountPath: {}
f:mountPropagation: {}
f:name: {}
f:readOnly: {}
k:{"mountPath":"/usr/local/ilogtail/checkpoint"}:
.: {}
f:mountPath: {}
f:name: {}
k:{"mountPath":"/usr/local/ilogtail/user_yaml_config.d"}:
.: {}
f:mountPath: {}
f:name: {}
f:readOnly: {}
k:{"mountPath":"/var/run"}:
.: {}
f:mountPath: {}
f:name: {}
f:dnsPolicy: {}
f:hostNetwork: {}
f:imagePullSecrets:
.: {}
k:{"name":"ilogtail-imagessecret"}:
.: {}
f:name: {}
f:restartPolicy: {}
f:schedulerName: {}
f:securityContext: {}
f:terminationGracePeriodSeconds: {}
f:tolerations: {}
f:volumes:
.: {}
k:{"name":"checkpoint"}:
.: {}
f:hostPath:
.: {}
f:path: {}
f:type: {}
f:name: {}
k:{"name":"root"}:
.: {}
f:hostPath:
.: {}
f:path: {}
f:type: {}
f:name: {}
k:{"name":"run"}:
.: {}
f:hostPath:
.: {}
f:path: {}
f:type: {}
f:name: {}
k:{"name":"user-config"}:
.: {}
f:configMap:
.: {}
f:defaultMode: {}
f:name: {}
f:name: {}
f:updateStrategy:
f:rollingUpdate:
.: {}
f:maxUnavailable: {}
f:type: {}
manager: Go-http-client
operation: Update
time: "2023-08-18T07:50:57Z"
apiVersion: apps/v1
fieldsType: FieldsV1
fieldsV1:
f:status:
f:currentNumberScheduled: {}
f:desiredNumberScheduled: {}
f:numberAvailable: {}
f:numberReady: {}
f:observedGeneration: {}
f:updatedNumberScheduled: {}
manager: kube-controller-manager
operation: Update
time: "2023-09-18T04:49:08Z"
name: ilogtail-ds
namespace: bizworks-system
resourceVersion: "37802761"
uid: 815db977-5799-405d-9c09-2723095c223b
spec:
revisionHistoryLimit: 10
selector:
matchLabels:
k8s-app: logtail-ds
template:
metadata:
creationTimestamp: null
labels:
k8s-app: logtail-ds
spec:
containers:
- env:
  - name: ALIYUN_LOG_ENV_TAGS
    value: node_name|node_ip
  - name: node_name
    valueFrom:
    fieldRef:
    apiVersion: v1
    fieldPath: spec.nodeName
  - name: node_ip
    valueFrom:
    fieldRef:
    apiVersion: v1
    fieldPath: status.hostIP
  - name: cpu_usage_limit
    value: "1"
  - name: mem_usage_limit
    value: "512"
  - name: default_access_key_id
    valueFrom:
    secretKeyRef:
    key: access_key_id
    name: ilogtail-secret
    optional: true
  - name: default_access_key
    valueFrom:
    secretKeyRef:
    key: access_key
    name: ilogtail-secret
    optional: true
    image: cr.registry.res.zgs.yun/ubcp/ilogtail:1.3.1
    imagePullPolicy: IfNotPresent
    name: logtail
    resources:
    limits:
    cpu: "2"
    memory: 2Gi
    requests:
    cpu: 125m
    memory: 256Mi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
  - mountPath: /var/run
    name: run
  - mountPath: /logtail_host
    mountPropagation: HostToContainer
    name: root
    readOnly: true
  - mountPath: /usr/local/ilogtail/checkpoint
    name: checkpoint
  - mountPath: /usr/local/ilogtail/user_yaml_config.d
    name: user-config
    readOnly: true
    dnsPolicy: ClusterFirst
    hostNetwork: true
    imagePullSecrets:
- name: ilogtail-imagessecret
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  terminationGracePeriodSeconds: 30
  tolerations:
- operator: Exists
  volumes:
- hostPath:
  path: /var/run
  type: Directory
  name: run
- hostPath:
  path: /
  type: Directory
  name: root
- hostPath:
  path: /etc/ilogtail-ilogtail-ds/checkpoint
  type: DirectoryOrCreate
  name: checkpoint
- configMap:
  defaultMode: 420
  name: ilogtail-user-cm
  name: user-config
  updateStrategy:
  rollingUpdate:
  maxUnavailable: 1
  type: RollingUpdate
  status:
  currentNumberScheduled: 23
  desiredNumberScheduled: 23
  numberAvailable: 23
  numberMisscheduled: 0
  numberReady: 23
  observedGeneration: 1
  updatedNumberScheduled: 23

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ilogtail日志采集插件被停止 #1132

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 4 comments

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

ilogtail日志采集插件被停止 #1132

jingliming Sep 19, 2023

环境信息

采集配置

现象

相关日志打印

日志1

日志2

Replies: 4 comments

linrunqi08 Sep 19, 2023 Collaborator

jingliming Sep 27, 2023 Author

jingliming Sep 27, 2023 Author

jingliming Sep 27, 2023 Author

jingliming
Sep 19, 2023

linrunqi08
Sep 19, 2023
Collaborator

jingliming
Sep 27, 2023
Author

jingliming
Sep 27, 2023
Author

jingliming
Sep 27, 2023
Author