Handle reverted events

[NikolasHaimerl]

Events emitted from the builtin actors can be reverted. It is marked by a reverted bool field in the response of the RawBuiltInActorEvent rpc call. For now the reverted events should not happen since we only try to fetch blocks which have been finalized already.
It is however not guarenteed that they are reverted. We need to address this behaviour and find a solution to events that were reverted.
See the discussion here.

[NikolasHaimerl]

@bajtos @juliangruber
According to the spacemon's implementation fo a deal observer there should not be any reverted events if the finality period is chosen high enough. This is also our current assumption on how the reverted field works.

Note that currently Spacemon will not fetch data past finality, that is 900 epochs prior to the current epoch. So there should not be any "reverted": true events in the data, but the data will also always be at least 4.5 hours old.

To still be aware of any reverted deals (and whether it can be an issue even if we go back 940 epochs to account for block finality) there are two strategies to go about this.

• Check if an incoming event is reverted (reverted field set to true) and throw and error. This has the advantage that we are immediatly pinged if such an event occurs through Sentry. I do not expect this error to occur at all so I would not expect this to increase the cost of error reporting through Sentry significantly. It also has the advantage that deals that are marked reverted will never be written to our database. The downside here is that all deals in a block will not be recorded as we are throwing an error before they are written and we risk the scenario where we are stuck in a loop of fetching the same block over and over again (since we are always starting off from the last written block).

• Add a reverted deal as a telemetry entry and have grafana alert us if any such an event occurs. We could simply filter out deals that have the reverted field set and only write those that are not reverted. I am not sure whether that solves the issue of observing an reverted deal though, because if we observe a deal that is reverted that should mean that we already wrote the same deal previously to our database and we need to edit it.

I think for now the key part is to have a mechanism in place that informs us if there ever are any reverted events. Our current assumption is that they should simply never occur due to the finality epoch but we are not validating that assumption. Handling reverted events on a database level would be a bigger engineering effort and I think it only pays off if our assumption about reverted events was wrong and for that to be the case we first need to know how often reverted events occur despite our assumptions.

This is why I would go with option number 2 and simply add a telemetry alert through grafana whenever a reverted event occurs. Should this be the case often enough to justify the engineering effort we should take action and handle these reverted deals.

Let me know what you think :)