Feature Request: Improve Handling of Missing Subperiods During Invalidation Processing

[sgiehl]

Background

Matomo processes invalidation requests for various periods, including day, week, month, and year. The aggregation follows a hierarchical structure:

1. Day periods are processed first.
2. Week periods are built from day data.
3. Month periods are built from week+day data.
4. Year periods are built from month data.

Currently, when processing a higher period (week, month or year), the system may detect that a required subperiod (day, week or month) has not yet been processed. In such cases, Matomo automatically triggers the aggregation of the missing subperiod in the background without creating a corresponding invalidation entry.

Problem Statement

The automatic background processing of missing subperiods introduces several challenges:

• Lack of Visibility: Since these processes are triggered internally without explicit invalidation records, they do not appear in logs or the invalidations table, making it difficult for administrators to track them.
• Unexpected Performance Issues: Normally fast aggregation tasks can experience significant delays due to the unexpected background processing of missing subperiods.
• Potential Duplicate Processing: Multiple archiving jobs might unintentionally trigger processing of the same (sub)period, leading to redundant processing and resource wastage.

Proposed Solution

To provide better control and transparency, we propose the following improvement:

• Pre-Check for Subperiod Data:
  • Before processing an invalidation, the system should verify that all required subperiods (e.g., days for a week, weeks for a month) are already available.
  • If a missing subperiod is spotted, a new invalidation request should be created for that missing period.
  • The current invalidation should then be aborted and reset for later processing.

Special Consideration for Different Archiving Modes

• Archiving Jobs:
  • The proposed behavior should apply when archiving is triggered via scheduled jobs, ensuring that processing follows a structured, step-by-step approach.
• Browser-Triggered Archiving:
  • To maintain the immediate availability of data, browser-triggered archiving should continue automatically processing missing subperiods as it does currently.

Benefits

• Increased Transparency: Ensures that all archiving activities are properly logged and visible through the invalidations table.
• Optimized Processing: Prevents redundant data aggregation, improving overall efficiency and performance.
• Better Debugging: Helps administrators diagnose performance issues by identifying which periods were processed and when.

Implementation Considerations

• Introduce a new validation step before processing higher periods to check for subperiod availability.
• Provide logging to indicate when an invalidation was aborted and rescheduled due to a missing subperiod.