Indexing properties

[Sune1337]

Background:
We're using Seq for our internal systems (about 10-15) to log useful informational messages, warnings and errors to Seq on one server. In total this is somewhere around 50Gb for the past 3 years, so i consider this fairly selective logging; ie we don't "log everything".

Current indexing:
We always log a property called "ApplicationName" and have created signals to display logs for each application one might be interested in to see logs for. Searching for logs by using these signals is very fast. I can see the logs for any application i choose for any period of time in less than a second. This is great.

Some applications:
Also log some context-property. For example there might be a property called "OrderNumber" that is included in all all messages from an application that also logs the property "ApplicationName = OrderProcessor".

Problem:
If i want to see all messages for "ApplicationName=OrderProcessor" where "OrderNumber=123456" which happened to be processed 2 years ago, the search will probably take somewhere around 5-10 minutes.

Question:
Is there a way to index properties without creating a signal? It is not realistic to create a signal for a specific "OrderNumber" i am currently interested in; generating that index will far exceed the time to just search the logs without creating the index.

Notes:
The way i see indexing work in Seq, it's pretty much low-cardinality (even single-cardinality?) indexing only by use of signals. Am i right in this assumption? If so, are there any plans to support high-cardinality indexing of some selected property?

[nblumhardt]

Hi @Sune1337 - thanks for getting in touch.

Your analysis is correct. We're planning to add high cardinality indexing support in Seq 2022.2 (midyear), which should speed your scenario up.

Hope this helps!
Nick

[Sune1337]

That sounds awesome! Can't wait to check that out.