Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Loki start failed. memberlist-kv: invalid service state: Stopping, expected: Running #15788

Open
wh-song opened this issue Jan 16, 2025 · 2 comments
Open

Loki start failed. memberlist-kv: invalid service state: Stopping, expected: Running #15788

wh-song opened this issue Jan 16, 2025 · 2 comments

Comments

@wh-song
Copy link

wh-song commented Jan 16, 2025 •
edited
Loading

Loki: 3.3.0;
Install Loki locally;

Loki config:

auth_enabled: true

server:
  http_listen_address: 0.0.0.0
  grpc_listen_address: 0.0.0.0
  http_listen_port: 3100
  grpc_listen_port: 9095
  log_level: debug

common:
  path_prefix: /loki
  compactor_address: http://manager:3100
  replication_factor: 1
  storage:
    s3:
      endpoint: manager:18989
      insecure: true
      access_key_id: loki
      secret_access_key: 11111111
      bucketnames: loki
      s3forcepathstyle: true
  ring:
    kvstore:
      store: memberlist
#    instance_interface_names:
#    - eth0

storage_config:
  tsdb_shipper:
    active_index_directory: /loki1/tsdb-shipper-active
    cache_location: /loki1/tsdb-shipper-cache
    cache_ttl: 24h


memberlist:
  join_members: ["worker", "master", "manager"]
  dead_node_reclaim_time: 30s
  gossip_to_dead_nodes_time: 15s
  left_ingesters_timeout: 30s
  bind_port: 7946
  gossip_interval: 2s

ingester:
  lifecycler:
    join_after: 10s
    observe_period: 5s
    ring:
      replication_factor: 1
      kvstore:
        store: memberlist
    final_sleep: 0s
  chunk_idle_period: 1m
  wal:
    enabled: true
    dir: /loki/wal
  max_chunk_age: 1m
  chunk_retain_period: 30s
  chunk_encoding: snappy
  chunk_target_size: 1.572864e+06
  chunk_block_size: 262144
  flush_op_timeout: 10s

schema_config:
  configs:
  - from: 2024-03-29
    store: tsdb
    object_store: s3
    schema: v13
    index:
      prefix: index_
      period: 24h


limits_config:
  max_cache_freshness_per_query: '10m'
  reject_old_samples: true
  reject_old_samples_max_age: 30m
  ingestion_rate_mb: 10
  ingestion_burst_size_mb: 20
  # parallelize queries in 15min intervals
  split_queries_by_interval: 15m
  volume_enabled: true

table_manager:
  retention_deletes_enabled: true
  retention_period: 168h

query_range:
  # make queries more cache-able by aligning them with their step intervals
  align_queries_with_step: true
  max_retries: 5
  parallelise_shardable_queries: true
  cache_results: true

frontend:
  log_queries_longer_than: 5s
  compress_responses: true
  max_outstanding_per_tenant: 2048

query_scheduler:
  max_outstanding_requests_per_tenant: 32768

querier:
  max_concurrent: 16

compactor:
  working_directory: /tmp/compactor

error message:
`level=info ts=2025-01-16T08:50:09.088743575Z caller=main.go:126 msg="Starting Loki" version="(version=3.3.0, branch=k227, revision=19bbc448)"
level=info ts=2025-01-16T08:50:09.088789827Z caller=main.go:127 msg="Loading configuration file" filename=loki2.yaml
level=debug ts=2025-01-16T08:50:09.088899117Z caller=modules.go:1721 msg="initializing ingester query tags interceptors"
level=info ts=2025-01-16T08:50:09.089830388Z caller=server.go:351 msg="server listening on addresses" http=[::]:3100 grpc=[::]:9095
level=info ts=2025-01-16T08:50:09.092063272Z caller=memberlist_client.go:439 msg="Using memberlist cluster label and node name" cluster_label= node=manager-364cd10a
ts=2025-01-16T08:50:09.092164312Z caller=memberlist_logger.go:74 level=debug msg="configured Transport is not a NodeAwareTransport and some features may not work as desired"
level=debug ts=2025-01-16T08:50:09.092394724Z caller=netutil.go:85 msg="looking for addresses" inf="[eth0 en0 lo]" inet6enabled=false
level=debug ts=2025-01-16T08:50:09.0924887Z caller=netutil.go:102 msg="detected highest quality address" ipAddr=10.80.169.141 inf=eth0
level=debug ts=2025-01-16T08:50:09.092940416Z caller=modules.go:1011 msg="initializing query frontend tripperware"
level=info ts=2025-01-16T08:50:09.094472043Z caller=table_manager.go:136 index-store=tsdb-2024-03-29 msg="uploading tables"
level=info ts=2025-01-16T08:50:09.094514015Z caller=table_manager.go:299 index-store=tsdb-2024-03-29 msg="query readiness setup completed" duration=3.638µs distinct_users_len=0 distinct_users=
level=info ts=2025-01-16T08:50:09.094543994Z caller=shipper.go:160 index-store=tsdb-2024-03-29 msg="starting index shipper in RW mode"
level=info ts=2025-01-16T08:50:09.127776608Z caller=head_manager.go:308 index-store=tsdb-2024-03-29 component=tsdb-head-manager msg="loaded wals by period" groups=0
level=debug ts=2025-01-16T08:50:09.127808409Z caller=manager.go:263 index-store=tsdb-2024-03-29 component=tsdb-manager msg="building WALs" n=0 ts=2025-01-16T16:50:09.127805111+08:00
level=debug ts=2025-01-16T08:50:09.127826537Z caller=manager.go:287 index-store=tsdb-2024-03-29 component=tsdb-manager msg="recovering tenant heads"
level=info ts=2025-01-16T08:50:09.127843881Z caller=manager.go:86 index-store=tsdb-2024-03-29 component=tsdb-manager msg="loaded leftover local indices" err=null successful=true buckets=0 indices=0 failures=0
level=info ts=2025-01-16T08:50:09.127863748Z caller=head_manager.go:308 index-store=tsdb-2024-03-29 component=tsdb-head-manager msg="loaded wals by period" groups=1
level=debug ts=2025-01-16T08:50:09.127870474Z caller=manager.go:263 index-store=tsdb-2024-03-29 component=tsdb-manager msg="building WALs" n=1 ts=2025-01-16T16:50:09.127869131+08:00
level=debug ts=2025-01-16T08:50:09.127876074Z caller=manager.go:287 index-store=tsdb-2024-03-29 component=tsdb-manager msg="recovering tenant heads"
level=debug ts=2025-01-16T08:50:09.129297149Z caller=netutil.go:85 msg="looking for addresses" inf="[eth0 en0 lo]" inet6enabled=false
level=debug ts=2025-01-16T08:50:09.129403622Z caller=netutil.go:102 msg="detected highest quality address" ipAddr=10.80.169.141 inf=eth0
level=debug ts=2025-01-16T08:50:09.130916974Z caller=netutil.go:85 component=ingester msg="looking for addresses" inf="[eth0 en0 lo]" inet6enabled=false
level=debug ts=2025-01-16T08:50:09.130988474Z caller=netutil.go:102 component=ingester msg="detected highest quality address" ipAddr=10.80.169.141 inf=eth0
level=debug ts=2025-01-16T08:50:09.131559041Z caller=modules.go:676 msg=" pattern ingester tee service disabled"
level=debug ts=2025-01-16T08:50:09.132377688Z caller=modules.go:1087 msg="initializing query frontend" config="{Handler:{LogQueriesLongerThan:5s LogQueryRequestHeaders: MaxBodySize:10485760 QueryStatsEnabled:false} FrontendV1:{MaxOutstandingPerTenant:2048 QuerierForgetDelay:0s} FrontendV2:{SchedulerAddress: DNSLookupPeriod:10s WorkerConcurrency:5 GRPCClientConfig:{MaxRecvMsgSize:104857600 MaxSendMsgSize:104857600 GRPCCompression: RateLimit:0 RateLimitBurst:0 BackoffOnRatelimits:false BackoffConfig:{MinBackoff:100ms MaxBackoff:10s MaxRetries:10} InitialStreamWindowSize:65535 InitialConnectionWindowSize:65535 TLSEnabled:false TLS:{CertPath: KeyPath: CAPath: ServerName: InsecureSkipVerify:false CipherSuites: MinVersion: Reader:} ConnectTimeout:5s ConnectBackoffBaseDelay:1s ConnectBackoffMaxDelay:5s Middleware:[] StreamMiddleware:[] CustomCompressors:[]} GracefulShutdownTimeout:5m0s InfNames:[eth0 en0 lo] Addr: Port:0 Encoding:json} CompressResponses:true DownstreamURL: TailProxyURL: TLS:{CertPath: KeyPath: CAPath: ServerName: InsecureSkipVerify:false CipherSuites: MinVersion: Reader:}}"
level=debug ts=2025-01-16T08:50:09.132489888Z caller=modules.go:1116 msg="using query frontend" version=v2
level=info ts=2025-01-16T08:50:09.133244676Z caller=mapper.go:47 msg="cleaning up mapped rules directory" path=/loki/rules-temp
level=debug ts=2025-01-16T08:50:09.133957521Z caller=worker_service.go:32 msg="determining if querier is running as standalone target" runningStandalone=false queryFrontendEnabled=false queryScheduleEnabled=false readEnabled=false allEnabled=true
level=debug ts=2025-01-16T08:50:09.134021425Z caller=worker_service.go:32 msg="determining if querier is running as standalone target" runningStandalone=false queryFrontendEnabled=false queryScheduleEnabled=false readEnabled=false allEnabled=true
level=info ts=2025-01-16T08:50:09.134072512Z caller=worker.go:130 component=querier msg="Starting querier worker using query-scheduler and scheduler ring for addresses"
level=debug ts=2025-01-16T08:50:09.134713515Z caller=netutil.go:85 component=distributor msg="looking for addresses" inf="[eth0 en0 lo]" inet6enabled=false
level=debug ts=2025-01-16T08:50:09.134781238Z caller=netutil.go:102 component=distributor msg="detected highest quality address" ipAddr=10.80.169.141 inf=eth0
level=debug ts=2025-01-16T08:50:09.137514779Z caller=module_service.go:72 msg="module waiting for initialization" module=ruler waiting_for=ring
level=debug ts=2025-01-16T08:50:09.137548495Z caller=module_service.go:72 msg="module waiting for initialization" module=memberlist-kv waiting_for=server
level=debug ts=2025-01-16T08:50:09.137539457Z caller=module_service.go:72 msg="module waiting for initialization" module=query-scheduler-ring waiting_for=memberlist-kv
level=debug ts=2025-01-16T08:50:09.137562921Z caller=module_service.go:72 msg="module waiting for initialization" module=ring waiting_for=memberlist-kv
level=debug ts=2025-01-16T08:50:09.137575216Z caller=module_service.go:72 msg="module waiting for initialization" module=query-frontend-tripperware waiting_for=server
level=debug ts=2025-01-16T08:50:09.137566115Z caller=module_service.go:72 msg="module waiting for initialization" module=distributor waiting_for=ring
level=debug ts=2025-01-16T08:50:09.1375931Z caller=module_service.go:72 msg="module waiting for initialization" module=ingester waiting_for=analytics
level=debug ts=2025-01-16T08:50:09.137592564Z caller=module_service.go:72 msg="module waiting for initialization" module=query-scheduler waiting_for=memberlist-kv
level=info ts=2025-01-16T08:50:09.137624992Z caller=module_service.go:82 msg=starting module=server
level=debug ts=2025-01-16T08:50:09.138003346Z caller=module_service.go:72 msg="module waiting for initialization" module=compactor waiting_for=analytics
level=debug ts=2025-01-16T08:50:09.138033757Z caller=module_service.go:72 msg="module waiting for initialization" module=analytics waiting_for=memberlist-kv
level=debug ts=2025-01-16T08:50:09.138038857Z caller=module_service.go:72 msg="module waiting for initialization" module=querier waiting_for=memberlist-kv
level=debug ts=2025-01-16T08:50:09.137610227Z caller=module_service.go:72 msg="module waiting for initialization" module=ingester-querier waiting_for=memberlist-kv
level=info ts=2025-01-16T08:50:09.138043479Z caller=module_service.go:82 msg=starting module=cache-generation-loader
level=debug ts=2025-01-16T08:50:09.138049525Z caller=module_service.go:72 msg="module waiting for initialization" module=store waiting_for=server
level=debug ts=2025-01-16T08:50:09.138061046Z caller=module_service.go:72 msg="module waiting for initialization" module=store waiting_for=memberlist-kv
level=info ts=2025-01-16T08:50:09.138063164Z caller=module_service.go:82 msg=starting module=query-frontend-tripperware
level=info ts=2025-01-16T08:50:09.138067549Z caller=module_service.go:82 msg=starting module=memberlist-kv
level=debug ts=2025-01-16T08:50:09.137630591Z caller=module_service.go:72 msg="module waiting for initialization" module=query-frontend waiting_for=query-frontend-tripperware
level=debug ts=2025-01-16T08:50:09.13812589Z caller=module_service.go:72 msg="module waiting for initialization" module=query-frontend waiting_for=query-scheduler
level=error ts=2025-01-16T08:50:09.138172598Z caller=loki.go:558 msg="module failed" module=memberlist-kv error="starting module memberlist-kv: invalid service state: Stopping, expected: Running"
level=debug ts=2025-01-16T08:50:09.138199934Z caller=module_service.go:133 msg="module waiting for" module=query-frontend-tripperware waiting_for=query-frontend
level=error ts=2025-01-16T08:50:09.138215772Z caller=loki.go:558 msg="module failed" module=analytics error="failed to start analytics, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: starting module memberlist-kv: invalid service state: Stopping, expected: Running"
level=error ts=2025-01-16T08:50:09.138226012Z caller=loki.go:558 msg="module failed" module=store error="failed to start store, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: starting module memberlist-kv: invalid service state: Stopping, expected: Running"
level=debug ts=2025-01-16T08:50:09.138228938Z caller=module_service.go:133 msg="module waiting for" module=server waiting_for=analytics
level=debug ts=2025-01-16T08:50:09.13822386Z caller=module_service.go:110 msg=stopping module=query-frontend-tripperware
level=error ts=2025-01-16T08:50:09.138231998Z caller=loki.go:558 msg="module failed" module=ingester-querier error="failed to start ingester-querier, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: starting module memberlist-kv: invalid service state: Stopping, expected: Running"
level=debug ts=2025-01-16T08:50:09.138235905Z caller=module_service.go:133 msg="module waiting for" module=server waiting_for=ingester-querier
level=debug ts=2025-01-16T08:50:09.138241695Z caller=module_service.go:133 msg="module waiting for" module=server waiting_for=querier
level=info ts=2025-01-16T08:50:09.138240753Z caller=module_service.go:120 msg="module stopped" module=query-frontend-tripperware
level=debug ts=2025-01-16T08:50:09.138241467Z caller=module_service.go:133 msg="module waiting for" module=cache-generation-loader waiting_for=querier
level=debug ts=2025-01-16T08:50:09.13824643Z caller=module_service.go:133 msg="module waiting for" module=server waiting_for=ruler
level=debug ts=2025-01-16T08:50:09.138256202Z caller=module_service.go:133 msg="module waiting for" module=cache-generation-loader waiting_for=query-frontend
level=debug ts=2025-01-16T08:50:09.13826348Z caller=module_service.go:110 msg=stopping module=cache-generation-loader
level=error ts=2025-01-16T08:50:09.138317502Z caller=loki.go:558 msg="module failed" module=querier error="failed to start querier, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: starting module memberlist-kv: invalid service state: Stopping, expected: Running"
level=error ts=2025-01-16T08:50:09.138348655Z caller=loki.go:558 msg="module failed" module=ingester error="failed to start ingester, because it depends on module analytics, which has failed: context canceled"
level=error ts=2025-01-16T08:50:09.138358096Z caller=loki.go:558 msg="module failed" module=query-scheduler error="failed to start query-scheduler, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: starting module memberlist-kv: invalid service state: Stopping, expected: Running"
level=error ts=2025-01-16T08:50:09.138369759Z caller=loki.go:558 msg="module failed" module=ring error="failed to start ring, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: starting module memberlist-kv: invalid service state: Stopping, expected: Running"
level=error ts=2025-01-16T08:50:09.138377372Z caller=loki.go:558 msg="module failed" module=query-scheduler-ring error="failed to start query-scheduler-ring, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: starting module memberlist-kv: invalid service state: Stopping, expected: Running"
level=error ts=2025-01-16T08:50:09.138383692Z caller=loki.go:558 msg="module failed" module=compactor error="failed to start compactor, because it depends on module analytics, which has failed: context canceled"
level=error ts=2025-01-16T08:50:09.138388722Z caller=loki.go:558 msg="module failed" module=query-frontend error="failed to start query-frontend, because it depends on module query-scheduler, which has failed: context canceled"
level=error ts=2025-01-16T08:50:09.138393447Z caller=loki.go:558 msg="module failed" module=distributor error="failed to start distributor, because it depends on module ring, which has failed: context canceled"
level=error ts=2025-01-16T08:50:09.138398549Z caller=loki.go:558 msg="module failed" module=ruler error="failed to start ruler, because it depends on module ring, which has failed: context canceled"
level=debug ts=2025-01-16T08:50:09.138513153Z caller=module_service.go:133 msg="module waiting for" module=server waiting_for=compactor
level=debug ts=2025-01-16T08:50:09.138520409Z caller=module_service.go:133 msg="module waiting for" module=server waiting_for=ingester
level=debug ts=2025-01-16T08:50:09.138525476Z caller=module_service.go:133 msg="module waiting for" module=server waiting_for=memberlist-kv
level=debug ts=2025-01-16T08:50:09.13853039Z caller=module_service.go:133 msg="module waiting for" module=server waiting_for=query-scheduler
level=debug ts=2025-01-16T08:50:09.138535326Z caller=module_service.go:133 msg="module waiting for" module=server waiting_for=query-frontend
level=debug ts=2025-01-16T08:50:09.138540288Z caller=module_service.go:133 msg="module waiting for" module=server waiting_for=query-frontend-tripperware
level=debug ts=2025-01-16T08:50:09.138544361Z caller=module_service.go:133 msg="module waiting for" module=server waiting_for=ring
level=debug ts=2025-01-16T08:50:09.138549378Z caller=module_service.go:133 msg="module waiting for" module=server waiting_for=store
level=debug ts=2025-01-16T08:50:09.138553966Z caller=module_service.go:133 msg="module waiting for" module=server waiting_for=distributor
level=debug ts=2025-01-16T08:50:09.138558414Z caller=module_service.go:133 msg="module waiting for" module=server waiting_for=query-scheduler-ring
level=debug ts=2025-01-16T08:50:09.138563284Z caller=module_service.go:110 msg=stopping module=server
level=info ts=2025-01-16T08:50:09.138867702Z caller=module_service.go:120 msg="module stopped" module=cache-generation-loader
level=info ts=2025-01-16T08:50:09.13899292Z caller=modules.go:1917 msg="server stopped"
level=info ts=2025-01-16T08:50:09.139001479Z caller=module_service.go:120 msg="module stopped" module=server
level=info ts=2025-01-16T08:50:09.139011085Z caller=loki.go:542 msg="Loki stopped" running_time=64.6406ms
failed services
github.com/grafana/loki/v3/pkg/loki.(*Loki).Run
/src/loki/pkg/loki/loki.go:600
main.main
/src/loki/cmd/loki/main.go:129
runtime.main
/usr/local/go/src/runtime/proc.go:272
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1700
level=error ts=2025-01-16T08:50:09.139455112Z caller=log.go:216 msg="error running loki" err="failed services\ngithub.com/grafana/loki/v3/pkg/loki.(*Loki).Run\n\t/src/loki/pkg/loki/loki.go:600\nmain.main\n\t/src/loki/cmd/loki/main.go:129\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:272\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700"

`
loki.log
@wh-song
Copy link
Author

wh-song commented Jan 16, 2025

add:
Identical configuration parameters (excluding the hostname) boot successfully in one 3-node environment; in another 3-node environment, they fail to boot. The distinction is merely that the successful environment has a network card name of ens192, while the failed environment has a network card name of eth0.

@wh-song
Copy link
Author

wh-song commented Jan 16, 2025

Compared to issue6370, the error messages are not entirely the same.
https://github.com/grafana/loki/issues/6370

Additionally, I would like to ask if Loki's startup logs support configuring the trace level? Can the trace level be specified for error logging through the log_level: parameter?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@wh-song