Implement an indexer query manager

[asteriscos]

Description

We need to implement a common service that will handle all queries made to the indexer. This service will be used in all plugins that retrieve information from the indexer, such as stateless dashboards, stateful views, and even the home overview summary.

Non-functional requirements

• This service must avoid duplicate requests as possible, this can lead to a cache mechanism.
• It must use as much as possible native services provided by OpenSearch

Functional requirements

• It must be able to handle different index-patterns.
• It must be exposed as a decoupled service so different plugins can implement it.
• It must be able to handle predefined filters.
• It must be able to read preset filters in the URL state.
• It must be able to work with indexes generated by transform jobs.

Use cases

• Make custom queries for custom kpis
• In the same view, load information from different index patterns
• Implemented in a custom search bar, Integrated with dashboards by reference.
• If filters change the URL state must reflect it. This will probably be limited to a single index pattern.
• The user must be able to refresh a query result even if the filter values didn't change.
• The default index pattern selected may change if the user has configured custom index patterns (Eg: with a custom index pattern dropdown)

[Machi3mfl]

Designing Indexer Query Manager Service (Second iteration)

abstract class Criteria {
  protected page = 0;
  protected size = 10;
  protected sortFields: string[] = [];

  setPage(page: number): this {
    this.page = page;
    return this;
  }

  setSize(size: number): this {
    this.size = size;
    return this;
  }

  setSortFields(fields: string[]): this {
    this.sortFields = fields;
    return this;
  }

  abstract build(): object;
}

class IndexerCriteria extends Criteria {
  private filters: any[] = [];
  private mustMatch: any[] = [];
  private shouldMatch: any[] = [];

  addFilter(field: string, value: any): this {
    this.filters.push({ term: { [field]: value } });
    return this;
  }

  addMustMatch(field: string, value: any): this {
    this.mustMatch.push({ match: { [field]: value } });
    return this;
  }

  addShouldMatch(field: string, value: any): this {
    this.shouldMatch.push({ match: { [field]: value } });
    return this;
  }

  addRangeFilter(field: string, options: { gte?: any, lte?: any }): this {
    this.filters.push({
      range: {
        [field]: options
      }
    });
    return this;
  }

  build(): object {
    const query: any = {
      bool: {}
    };

    if (this.filters.length > 0) query.bool.filter = this.filters;
    if (this.mustMatch.length > 0) query.bool.must = this.mustMatch;
    if (this.shouldMatch.length > 0) query.bool.should = this.shouldMatch;

    return {
      from: this.page * this.size,
      size: this.size,
      sort: this.sortFields,
      query
    };
  }
}

// OpenSearch Implementation
class Indexer'DataSource implements IDataSource {
  private client: any;

  constructor(client: any) {
    this.client = client;
  }

  async fetch(pattern: string, criteria: OpenSearchCriteria): Promise<IndexData> {
    const response = await this.client.search({
      index: pattern,
      body: criteria.build()
    });

    return {
      hits: response.hits.hits,
      total: response.hits.total.value
    };
  }
}

// Abstract Factory
abstract class DataSourceFactory {
  abstract createDataSource(): IDataSource;
  abstract createCriteria(): Criteria;
}

class OpenSearchFactory extends DataSourceFactory {
  private client: any;

  constructor(client: any) {
    super();
    this.client = client;
  }

  createDataSource(): IDataSource {
    return new IndexerDataSource(this.client);
  }

  createCriteria(): Criteria {
    return new IndexerCriteria();
  }
}

// Service using Repository Pattern
class IndexPatternService {
  private dataSource: IDataSource;
  private factory: DataSourceFactory;

  constructor(factory: DataSourceFactory) {
    this.factory = factory;
    this.dataSource = factory.createDataSource();
  }

  createCriteria(): Criteria {
    return this.factory.createCriteria();
  }

  async fetch(pattern: string, criteria: Criteria): Promise<IndexData> {
    try {
      return await this.dataSource.fetch(pattern, criteria);
    } catch (error) {
      throw new Error(`Error fetching index pattern: ${error.message}`);
    }
  }
}

This implementation:

• Uses Abstract Factory pattern to create data sources
• Implements Repository Pattern to abstract data operations
• Follows SOLID principles:
  • Single Responsibility: Each class has one responsibility
  • Open/Closed: Extensible for new data sources
  • Liskov Substitution: Implementations are interchangeable
  • Interface Segregation: Small, specific interfaces
  • Dependency Inversion: Dependencies based on abstractions

[Machi3mfl]

Added service to register query manager by plugins

• Applied Registry Pattern

export class QueryManagerRegistry {
    private static instance: QueryManagerRegistry;
    private services: Map<string, QueryManagerService>;

    private constructor() {
        this.services = new Map();
    }

    static getInstance(): QueryManagerRegistry {
        if (!QueryManagerRegistry.instance) {
            QueryManagerRegistry.instance = new QueryManagerRegistry();
        }
        return QueryManagerRegistry.instance;
    }

    register(index: string, factory: QueryManagerFactory): void {
        if (this.services.has(index)) {
            throw new Error(`Service for plugin ${index} already registered`);
        }
        this.services.set(index, new QueryManagerService(factory));
    }

    getService(index: string): QueryManagerService {
        const service = this.services.get(index);
        if (!service) {
            throw new Error(`Service for plugin ${index} not found`);
        }
        return service;
    }

    unregister(pluginId: string): void {
        this.services.delete(pluginId);
    }
}

[Machi3mfl]

Created first version

• Added draft PR: Add query manager service  wazuh-dashboard-plugins#7264

[Machi3mfl]

Designing Indexer Query Manager Service (Third iteration)

Filter definitions

interface IFilterCriteria {
  buildFilter(): FilterDefinition;
}

interface ICompositeFilter extends IFilterCriteria {
  addCriteria(criteria: IFilterCriteria): void;
}

class TermFilter implements IFilterCriteria {
  constructor(
    private field: string,
    private value: any
  ) {}

  buildFilter(): FilterDefinition {
    return {
      field: this.field,
      value: this.value,
      operator: 'eq'
    };
  }
}

class RangeFilter implements IFilterCriteria {
  constructor(
    private field: string,
    private from: any,
    private to: any
  ) {}

  buildFilter(): FilterDefinition {
    return {
      field: this.field,
      value: { gte: this.from, lte: this.to },
      operator: 'range'
    };
  }
}

class AndFilter implements ICompositeCriteria {
  private criteriaList: IFilterCriteria[] = [];

  addCriteria(criteria: IFilterCriteria): void {
    this.criteriaList.push(criteria);
  }

  buildFilter(): FilterDefinition {
    return {
      operator: 'and',
      value: this.criteriaList.map(criteria => criteria.buildFilter())
    };
  }
}

class OrFilter implements ICompositeCriteria {
  private criteriaList: IFilterCriteria[] = [];

  addCriteria(criteria: IFilterCriteria): void {
    this.criteriaList.push(criteria);
  }

  buildFilter(): FilterDefinition {
    return {
      operator: 'or',
      value: this.criteriaList.map(criteria => criteria.buildFilter())
    };
  }
}


// Criteria Builder Helper
class IndexerFilterBuilder {
  static term(field: string, value: any): IFilterCriteria {
    return new TermFilter(field, value);
  }

  static range(field: string, from: any, to: any): IFilterCriteria {
    return new RangeFilter(field, from, to);
  }

  static and(...criteria: IFilterCriteria[]): ICompositeCriteria {
    const andCriteria = new AndFilter();
    criteria.forEach(c => andCriteria.addCriteria(c));
    return andCriteria;
  }

  static or(...criteria: IFilterCriteria[]): ICompositeCriteria {
    const orCriteria = new OrFilter();
    criteria.forEach(c => orCriteria.addCriteria(c));
    return orCriteria;
  }
}

Query Manager

// Core Plugin - Interfaces
interface IQueryManagerConfig {
  indexPattern: string;
  predefinedFilters?: IFilterCriteria[];
  customConfig?: Record<string, any>;
}

interface IQueryManager {
  fetch(criteria?: IFilterCriteria[]): Promise<SearchResponse>;
  getIndexPattern(): string;
  getPredefinedCriteria(): IFilterCriteria[];
}

class QueryManagerFactory {
  static create(config: IQueryManagerConfig, searchService: ISearchGeneric): IQueryManager {
    return new BaseQueryManager(config, searchService);
  }
}


class BaseQueryManager implements IQueryManager {
  constructor(
    private config: IQueryManagerConfig,
    private searchService: ISearchGeneric
  ) {}

  async fetch(criteria?: IFilterCriteria[]): Promise<SearchResponse> {
    const allCriteria = [...(this.config.predefinedCriteria || []), ...(criteria || [])];
    const filters = allCriteria.map(criteria => criteria.buildFilter());
    
    return this.searchService.search({
      index: this.config.indexPattern,
      filters,
      ...this.config.customConfig
    });
  }

  getIndexPattern(): string {
    return this.config.indexPattern;
  }

  getPredefinedCriteria(): IFilterCriteria[] {
    return this.config.predefinedCriteria || [];
  }
}

Query manager registry

interface IQueryManagerRegistry {
  register(pluginId: string, queryManagerId: string, queryManager: IQueryManager): void;
  get(pluginId: string, queryManagerId: string): IQueryManager | undefined;
  getAll(): Map<string, Map<string, IQueryManager>>;
}


class QueryManagerRegistry implements IQueryManagerRegistry {
  private registry: Map<string, Map<string, IQueryManager>> = new Map();

  register(pluginId: string, queryManagerId: string, queryManager: IQueryManager): void {
    if (!this.registry.has(pluginId)) {
      this.registry.set(pluginId, new Map());
    }
    this.registry.get(pluginId)?.set(queryManagerId, queryManager);
  }

  get(pluginId: string, queryManagerId: string): IQueryManager | undefined {
    return this.registry.get(pluginId)?.get(queryManagerId);
  }

  getAll(): Map<string, Map<string, IQueryManager>> {
    return this.registry;
  }
}

Examples

Register query manager

const vulsQueryManager = core.queryManager.factory.create(
      {
        indexPattern: 'alerts-*',
        predefinedFilters: [] // add predefined filters
      },
      this.searchService // plugins.data.search service
    );

core.queryManager.registry.register('vuls-plugin', 'vuls-query-manager', alertsQueryManager);

Use registered query manager

const vulsQueryManager =  core.queryManager.registry.get('vuls-plugin', 'vuls-query-manager');

const filters =  IndexerFilterBuilder.and(
    IndexerFilterBuilder.term('agent.id', '0001'),
    IndexerFilterBuilder.or(
      IndexerFilterBuilder.term('rules.level', 'high'),
    ),
    IndexerFilterBuilder.range('@timestamp', 'now-7d', 'now')
  );

await vulsvulsQueryManager.fetch(filters);

[Machi3mfl]

Designing Indexer Query Manager Service (Fourth iteration)

Domain model

type QueryResult = {
  hits: number;
  data: any[];
}

type IndexPattern = {
}

type Filter = {
}

export interface IFilterManagerService {
  getFilters(): Filter[];
  addFilter(filter: Filter): void;
  removeFilter(filter: Filter): void;
  updateFilter(oldFilter: Filter, newFilter: Filter): void;
  clear(): void;
}

export interface IQueryManagerFacade {
  // query service
  executeQuery(): Promise<QueryResult>;
  refreshQuery(): Promise<QueryResult>;
  // index Patterns management
  setDefaultIndexPattern(indexPatternId: string): Promise<void>;
  getCurrentIndexPattern(): Promise<IndexPattern>;
  // Filter management
  addFilter(filter: Filter): Promise<void>;
  removeFilter(filter: Filter): Promise<void>;
  clearFilters(): Promise<void>;
}

export interface ISearchContext {
  selectedPattern: IndexPattern | null;
  supportedPatterns: IndexPattern[]; // create a registry for this
}


export class SearchContext implements ISearchContext {
  private _selectedPattern: IndexPattern | null = null;

  constructor(
    readonly supportedPatterns: IndexPattern[],
  ) { }

  get selectedPattern(): IndexPattern | null {
    return this._selectedPattern;
  }

  async selectPattern(indexPatternId: string): Promise<void> {
    // validate index patterns
    this._selectedPattern = indexPatternId;
  }
}

export interface QueryManagerConfig {
  indexPatterns: IndexPattern[];
}

export interface IQueryService {
  executeQuery(indexPatternId: string, filters: Filter): Promise<QueryResult>;
  refreshQuery(): Promise<QueryResult>;
}

export class QueryService implements IQueryService {
  executeQuery(indexPatternId: string, filters: Filter): Promise<QueryResult> {
    throw new Error('Method not implemented.');
  }

  refreshQuery(): Promise<QueryResult> {
    throw new Error('Method not implemented.');
  }

}

export class FilterManagerService implements IFilterManagerService {
  private filters: Filter[] = [];

  getFilters(): Filter[] {
    return this.filters;
  }

  addFilter(filter: Filter): void {
    this.filters.push(filter);
  }

  removeFilter(filter: Filter): void {
    const index = this.filters.findIndex(f => f === filter);
    if (index !== -1) {
      this.filters.splice(index, 1);
    }
  }

  updateFilter(oldFilter: Filter, newFilter: Filter): void {
    const index = this.filters.findIndex(f => f === oldFilter);
    if (index !== -1) {
      this.filters[index] = newFilter;
    }
  }

  clear(): void {
    this.filters = [];
  }
}


export class QueryManagerService implements IQueryManagerFacade {
  private searchContext: ISearchContext;

  constructor(
    private readonly queryService: IQueryService,
    private readonly filterService: IFilterManagerService,
    supportedPatterns: IndexPattern[],
  ) {
    this.searchContext = new SearchContext(supportedPatterns);
  }

  async executeQuery(): Promise<QueryResult> {

    const filters = await this.filterService.getFilters();
    const indexPatternId = this.searchContext.selectedPattern;

    if (!indexPatternId) {
      throw new Error('No index pattern selected');
    }

    let results = await this.queryService.executeQuery(indexPatternId, filters);
    return results;
  }

  async refreshQuery(): Promise<QueryResult> {
    const result = await this.queryService.refreshQuery();
    return result;
  }

  async setDefaultIndexPattern(indexPatternId: string): Promise<void> {
    if (!indexPatternId) {
      throw new Error('Index pattern is required');
    }
    this.searchContext?.selectedPattern(indexPatternId);
  }

  async getCurrentIndexPattern(): Promise<IndexPattern> {
    return this.searchContext.selectedPattern;
  }

  async addFilter(filter: Filter): Promise<void> {
    this.filterService.addFilter(filter);
  }

  async removeFilter(filter: Filter): Promise<void> {
    this.filterService.removeFilter(filter);
  }

  async clearFilters(): Promise<void> {
    this.filterService.clear();
  }

  async getFilters(): Promise<Filter[]> {
    return this.filterService.getFilters();
  }
}

export class QueryManagerServiceBuilder {
  private config: Partial<QueryManagerConfig> = {};
  private services: {
    queryService?: IQueryService;
    filterService?: IFilterManagerService;
  } = {};

  constructor(private readonly pluginId: string) { }

  withIndexPatterns(patterns: IndexPattern[]): this {
    this.config.indexPatterns = patterns;
    return this;
  }

  withQueryService(service: IQueryService): this {
    this.services.queryService = service;
    return this;
  }

  withFilterService(service: IFilterManagerService): this {
    this.services.filterService = service;
    return this;
  }

  build(): QueryManagerService {
    if (!this.config.indexPatterns) {
      throw new Error('Index patterns are required');
    }

    if (!this.services.queryService) {
      throw new Error('Query service is required');
    }

    if (!this.services.filterService) {
      throw new Error('Filter service is required');
    }

    return new QueryManagerService(
      this.services.queryService,
      this.services.filterService,
      this.config.indexPatterns,
    );
  }
}

Examples

const queryManagerService = new QueryManagerServiceBuilder('plugin-name')
  .withIndexPatterns([{
    id: 'index-pattern-1', fixedFilters: []
  },
  { id: 'index-pattern-2',  fixedFilters: [] }]) // asign one or more index patterns
  .withFilterService(new FilterManagerService()) // add the filter management feature
  .withQueryService(new QueryService()) // add the make fetch queries feature
  .build(); // simplify the creation of the service


// Filters management usage
queryManagerService.addFilter([{ key: 'field', value: 'value' }]);
queryManagerService.removeFilter([{ key: 'field', value: 'value' }]);
queryManagerService.clearFilters();
queryManagerService.getFilters();

// Index patterns management usage
queryManagerService.setDefaultIndexPattern('index-pattern-1');
queryManagerService.getCurrentIndexPattern();

// Query execution
queryManagerService.executeQuery();
queryManagerService.refreshQuery();