Automatic Cleanup of Duplicate Invalidation Records Upon Reset

[sgiehl]

Background

In Matomo's current archiving process, when an invalidation request is created, the system checks whether an identical invalidation already exists. If an identical entry is found, a new invalidation is not created, as the existing one will be processed anyway.

However, if an invalidation is already in progress and a new identical request is made, the system assumes that the currently processed invalidation might result in outdated data and therefore stores the new request.

A potential issue arises when an in-progress invalidation is interrupted (e.g., due to a crash) but remains marked as "started." While the system is designed to automatically reset such invalidations after 24 hours, it does not currently check for newer identical requests. This can result in duplicate invalidations in the database, leading to unnecessary resource consumption during data aggregation.

Proposed Solution

To optimize resource usage and avoid redundant processing, we propose implementing an automatic cleanup mechanism that performs the following actions when resetting an invalidation after 24 hours:

Check for newer identical invalidation records:

• If a newer identical request exists, the older request should be removed instead of being reset and reprocessed.
• If no newer identical request exists, the invalidation should be reset as usual.

Benefits

• Reduced Resource Consumption: Prevents redundant data aggregation by avoiding duplicate processing.
• Improved System Efficiency: Frees up processing power for other tasks by eliminating unnecessary operations.
• Enhanced Data Accuracy: Ensures that only the most up-to-date invalidation request is processed.

Implementation Considerations

• The cleanup process should be integrated into the existing invalidation reset logic.
• Logging should be added to track cleanup actions for auditing purposes.
• The mechanism should ensure it does not inadvertently remove valid pending invalidations.

---

This request provides a structured and clear outline of the problem, solution, and benefits while ensuring a professional tone suitable for GitHub.