Stations stopped

[ricarper]

All stations have not received any rewards since the 7th. They are working normally without any errors.

[bajtos]

Cross-posting #569 (comment)

After further investigation, I found that Spark had an outage caused by a problem with one of our JSON RPC providers. Unfortunately, our monitoring did not detect this problem. I opened a GH issue to improve our monitoring - see CheckerNetwork/spark-evaluate#312.

The evaluation service runs correctly now.

Unfortunately, the transactions submitting round evaluations to the smart contract are not getting accepted by the network; they have been queueing since yesterday morning. We will investigate this problem further.

Related issues:

• Detect when TXs submitting evaluation scores are not added to the chain spark-evaluate#314
• Cancel pending transactions that are too old spark-evaluate#315

[ricarper]

Cross-posting #569 (comment)

After further investigation, I found that Spark had an outage caused by a problem with one of our JSON RPC providers. Unfortunately, our monitoring did not detect this problem. I opened a GH issue to improve our monitoring - see filecoin-station/spark-evaluate#312.

The evaluation service runs correctly now.

Unfortunately, the transactions submitting round evaluations to the smart contract are not getting accepted by the network; they have been queueing since yesterday morning. We will investigate this problem further.

Related issues:

• Detect when TXs submitting evaluation scores are not added to the chain spark-evaluate#314
• Cancel pending transactions that are too old spark-evaluate#315

Can't we do anything? It's just that they haven't received anything.
Turn off the machines?

[juliangruber]

This should be good again, please reopen if not.

[ricarper]

It's been four hours since the blockage. Stations are again without rewards.

[juliangruber]

We're working on this, currently the Filecoin L1 frequently is becoming very expensive, which makes it hard for messages to go through. We're working on an alternative, using either an L2 or an off-chain data store, which still is transparent and trustless