Consistent timeouts on Trace Hierarchy view

[TylerBurnett]

For a bit of prior context, we've upgraded from 2023.3 to 2024.x, with the intent of utilizing the new trace capabilities in Seq

This is where it is occurring for a more visual representation of this issue.

We process a lot of logs, I've been quoted upwards of 80gb a day are incoming.

We've spent quite a bit of time digging from our side from which we've checked

• The index for @Traceid is present
• The index for @SpanId is present

What we've noticed from debugging is that the query used for the graph doesn't seem to be time-gated by the timeframe selected by the user?

select coalesce(@Resource.service.name, Application) as serviceName, 
        @StructuredMessage as structuredMessage,
        @Id as id, 
        @Level as level, 
        @TraceId as traceId, 
        @SpanId as spanId, 
        @ParentId as parentId, 
        TotalMilliseconds(@Elapsed) as elapsedMs,
        ToIsoString(@Start) as start, 
        ToIsoString(@Timestamp) as timestamp,
        @SpanKind as kind,
        @EventType as eventType
        , has(Environment) as has_col0, Environment as col0, has(AssemblyVersion) as has_col1, AssemblyVersion as col1, has(Application) as has_col2, Application as col2
        from stream 
        where Has(@Start) and Has(@SpanId) and @TraceId = '6abc394890094358ead442e7ce32b833'
        limit 1000

By adding a timeframe of 1hr to the above query we're able to bring it to under 4 seconds. We also tried changing the 'Default to search last' query time to 15 minutes and trying to load the view again, this didn't seem to make a difference.

Do you guys have any idea how we can fix this issue?

EDIT: We believe that some of this might be due to indexing, It has been about 24 hours since we upgraded.

[nblumhardt]

Hi Tyler, thanks for your message. The Data > Storage screen will show whether your indexes are up-to-date or not; it sounds like you might have a lot of historical data from pre-2024.3 that will need to be indexed before trace id searches will work.

Let me know how you go, and if things don't seem to be moving forward please drop us a line via [email protected] with some more information about the system.

Greetings from another BNE dev! 😎

Nick

[TylerBurnett]

Thanks, Nick, the index finished building and traces are now showing and are snappy when viewed.

See you at DDD!