Onboarding rate and gas optimizations

[irenegia]

The facts:

Since August (nv23) gas used for onboarding methods has been higher than in the past year (link) - though total gas usage now is in-line with avg gas usage in 2021-2023.

Because of this, sector onboarding today is more expensive because of:

1. more gas units are now paid at ProveCommit (due to correct gas fee accounting updated in FIP0084),
2. higher base fee as the chain bandwidth gets more utilized due to network demand, since onboarding gas is a large fraction of the total gas used (~70-80%).

This causes the Filecoin network to have a self-imposed limit to the onboarding rate to avoid increasing basefee.
Currently only ~3-4PiB¹ of data onboarding per day can fit within 5B gas units (block gas target), which is below the demand. 2-3X this value can be acceptable for a short-medium term.

Causes:

Why is the onboarding gas higher than before? We have identified 3 main causes:

• Cause 1 (high gas unit usage): In nv23, FIP0084 was shipped. This FIP removed the cron subsidy for provecommit, making that method 2.7x instantly more expensive in terms of gas units, from ~71M to ~196M (previously this cost was subsidized by the network by virtue of cron).

• Cause 2 (irrational aggregation): The batch gas charge (aka batchBalancer) parameters were introduced in FIP0013 and were set based on gas usage of onboarding methods at the time of the design. Protocol changes that happened after that (in particular FIP32/FIP57 and FIP84) caused a significant change in the average value of gas used for the onboarding method. Because of this, now the batchBalancer parameters are outdated and ProveCommitAggregate is always more expensive to use with respect to non-aggregated provecommit (the original design this method was intended to be cheaper when base fee is higher than batchBalancer *batchDiscount). In short, currently aggregation can not be used as a mechanism to alleviate congestion.

• Cause 3 (high on-chain state storage gas): For a ProveCommit3 of a single sector, 66% of the gas for provecommit methods is related to state storage gas (unrelated to block time). This cost was increased during the FEVM launch (tripled) to try to bring our storage costs closer to Ethereum due to concerns of state growth.

Next Steps :

In the following we list possible next steps that we can take in order to remove or mitigate the causes identified above. Some of these solutions are protocol changes (labeled with “FIP”) others involve shipping new tooling.

Short Term (Jan/Feb)

• DDO gas savings (tooling): FIP-0076 DDO introduced optimization to offset this new cost (eg, we estimate that making a deal with datacap but without storage market notification could save ~1/3 of gas units per sector plus ~2/3 of the gas used for publishStorageDeals). However, not all SPs have switched their onboarding pipeline to DDO due to lack of the ecosystem tooling support, i.e: from FIL+ tooling & Spark, these teams shall prioritize working with the SP implementation teams to add the support. Spark team is planning DDO support for Q1, CC @patrickwoodhead.
• BatchBalancer Param Update (FIP): Recalibrate the parameters for batchBalancer in order to have aggregation become rational again when the base fee is above the threshold batchBalance*batchDiscount. CryptoEcon team is working on this CC @zixuanzh. Note that because of Cause 3 the impact of aggregation would be limited.
• (Added on 17th dec as suggested by @Stebalien ):Batch pre-commit & prove-commit messages, especially the prove-commits. Batching prove-commits will save the cost of updating the deadlines/partitions containing the new sectors multiple times, saving quite a bit of storage gas and will not incur in any aggregate proof fee.

FilOz team is looking in these two:

• Increase block gas limit (FIP): Increasing the block size limit with a reasonable number (eg, add the amount of units that were moved from cron to onboarding by FIP84).
• Adjust gas price list (FIP): Re-parametrize the gas charging system, making state storage gas cheaper. For example, we could give a discount proportional to the amount of compute gas charged.

Note that both these proposals have some security and usability concerns, we are collecting more data to evaluate the pros and cons.

Medium Term (Q3’2025)

• Replace batchBalencer (FIP): To properly address the aggregation incentive issue (cause number2), i.e: always rational for SPs to use aggregation in order to spend less gas, the network should switch from charging an aggregate proof fee based on gas base fee to an onboarding fee based on storage supply & demand. See for example discussion Self-adjusting onboarding fees with demand pricing #587 .

Long Term (2026+)

The Filecoin network needs to scale above 3PiB per day, we want to reach 1EiB per day! To achieve that goal, we have few directions to research. In particular, in order to bring down the gas cost for onboarding method we could:

• Optimize miner actor;
• Neutron Scalable storage onboarding and maintenance (project Neutron) #119 ;
• Delegate some storage state to L2s.

Footnotes

1. The 3PiB value assuming 32GiB sectors with a deal per sector, and it is obtained taking the recent average of gas units used for
   the methods PreCommitSectorBatch2, ProveCommitSectors3 (no batching/aggregation), and PublishStogareDeals. ↩

[Stebalien]

Possible ways to save gas right now:

• Batch pre-commit & prove-commit messages, especially the prove-commits. Batching prove-commits will save the cost of updating the deadlines/partitions containing the new sectors multiple times, saving quite a bit of storage gas.
• Use NI-Porep (FIP0092) to avoid pre-commit entirely. Unfortunately, the hardware required for NI-Porep is different (cheeper, but different) than that required by regular porep.

[jennijuju]

Batching prove-commits will save the cost of updating the deadlines/partitions containing the new sectors multiple times, saving quite a bit of storage gas.

@irenegia can you add this to your "immediate things to do"? This doesn't require any protocol change and should have minimal impact on onboarding pipeline for those who are sealing sectors in parallel

[irenegia]

done

[dannyob]

So it's been suggested that we re-purpose nv25 to incorporate the short-term FIP-based improvements. Currently, it sounds like these FIPs will be relatively straightforward to describe, being mostly changes to network parameters. The plan is to discuss a potential schedule for this on the Filecoin Implementers' sync on 2024-12-17 (tomorrow).

I'm running an every-little-thing update channel for those interested in keeping an eye on the broader workplan of lowering gas costs, but let's keep the discussion centered here.

[Stebalien]

Increasing the block size limit with a reasonable number (eg, add the amount of units that were moved from cron to onboarding by FIP84).

Unfortunately it doesn't quite work that way. The block gas limit mostly exists as to limit the amount of time a tipset can take to execute on average. However, gas is also charged for operations that don't take time (e.g., state storage gas). At the moment over 66% of the prove commit gas is state-storage gas.

When considering the block gas limit, we have to consider the amount of time it would take to execute a tipset in the worst-case scenario: that is, the case where the block is full of messages that do nothing but use compute gas (little to no storage gas).

The problem with simply "donating" the gas available in cron twofold:

1. Most of the gas spent in cron was actually state-storage gas (as far as I can tell, at least) which doesn't affect tipset execution time. However, if this gas is added to the block limit, an attacker would be able to spend it on operations that affect tipset execution time.
2. The pre/prove-commit flow required several operations that cost quite a bit of state storage gas, helping to offset the time taken by cron.

The underlying problem is that we're spending a part of our time budget on limiting the rate at which state increases. Ideally those would be two independent variables, but that's tricky.

[anorth]

I am an advocate for increasing the block gas limit, but I do recognize that it is not as straightforward as just adding back the amount of gas units that was used in cron without being a bit more careful. The worst case outcome increases at a (linearly) faster rate than the average case.

I think we should do some validation of the worst case block time that could result and do measurements on hardware that is more relevant for years later than when the original parameters were set

[Stebalien]

I think we should do some validation of the worst case block time that could result and do measurements on hardware that is more relevant for years later than when the original parameters were set

We have two numbers we can adjust:

1. The block gas limit.
2. The gas charged for different operations.

Gas is equivalent to time (10gas/ns) so adjusting the block gas limit limit is equivalent to adjusting the amount of time we're willing to spend computing a tipset (currently targeting ~2s). So network conditions, pubsub, etc. matter here but current hardware doesn't otherwise have much effect.

We could adjust the gas fees themselves (by benchmarking on modern hardware) but I'm not sure how much we're going to get out of that because 66% of prove commit gas goes towards storage. So any X% reduction in compute (FVM) gas prices is actually 0.44 * X%.

[Stebalien]

Balancing State Storage Gas

Problem

Filecoin charges gas whenever an actor (whether it be a built-in miner actor, the market actor, or an EVM smart contract) stores state. It does so to try to:

1. Limit the rate at which chain state grows.
2. Encourage users to be efficient in what state they store.

Unfortunately:

1. State storage gas eats into chain bandwidth without actually costing any wall-clock time.
2. State storage is actually really cheep on Filecoin, relative to most chains.
3. The Filecoin state tree is huge.
4. State storage gas fees are the majority (66%) of the cost of prove-commit messages.

In other words, we're both charging too little and charging too much for state storage.

We charge so much because our state-storage model has to assume the worst-case:

1. State is forever. When a user stores state we have no way of knowing how long that state will be stored, so we have to assume it'll live forever (quite often it does).
2. All state is new state. The EVM can rebate some state storage fees when updating existing state but Filecoin's data model doesn't allow for this.

We don't charge as much as we should because state gas is already really high.

Proposal

Fixing this in the general case is difficult as it involves things like state "rent" but we we may be able to fix this for specific builtin actors by charging for state at a different layer. That is, we may be able to:

1. Not charge state storage gas to miner actors, the market actor, and/or the power actor in specific cases.
2. Instead charge direct fees proportional to the amount of state storage used by the responsible miner actor.

For example, right now whenever a sector is committed we have to charge state storage fees for the on-chain sector information assuming it sticks around forever. In practice, it generally does stick around forever because there's no incentive to clean it up. Instead, we could charge a per-sector onboarding fee (as has been proposed in the past) and either (a) add a mechanism force the cleanup of expired/terminated sectors and/or add a mechanism that charges a periodic fee as long as the state remains on-chain (i.e., state rent).

Importantly, these fees can be unrelated to gas and therefore won't affect the base fee or use up any chain bandwidth.

If we do this correctly, I'd expect both increased chain bandwidth as we'd be able to spend more of our block limit on compute and reduced chain state as SPs would be directly incentivized to cleanup old sectors. I can't actually promise lower gas fees because gas fees a product of supply and demand: these changes would increase gas supply but there's no guarantee they wouldn't also increase demand (see Braess's paradox).

[anorth]

Optimising things

Not mentioned much in the solution space above is making things more efficient, using less state storage in particular. To my knowledge, very little profiling or optimisation work has been done on this. It may be that there are relatively low hanging fruit. For example, some time ago @ZenGround0 measured that the inputs to the complex retro-active termination fee occupied 20% of all chain state. FIP-0098 may resolve this (at least as far as replacing each input with a null/zero byte) and there's probably more room to squeeze there. I'm not sure this will translate directly into 20% * 66% = 13% more headroom, but it could be decent bang for buck.

I don't have a big list of things to optimise – some profiling work needs to be done first. Just a call for someone to look into it because I have a sense there's decent room for improvement.

Optimising things to give more effective throughput is a short-term strategy, that will only increase the onboarding limit so far. But so are most of the other things here, and pulling a few of them together might easily give us a 50-100% boost.

[Stebalien]

I think we should totally do this, but that's 20% of on-chain sector info objects, not 20% of the state written when adding a new sector (which includes pre-commit stuff, deadline updates, partition updates, power/pledge updates, etc.).

@Kubuxu brought up removing dead miners from the pledge table in the power actor. Doing that could significantly decrease the height of that HAMT, reducing the gas required to update power when onboarding new sectors.

All of these will chip away at the problem, but they probably won't have a massive effect.

[lanzafame]

@jennijuju I see no mention of the increased impact on base fee in that FIP, only the increased activation cost.

[Stebalien]

@lanzafame the base fee is based on chain capacity and demand for gas. The base fee goes up when there's more demand for gas than supply, increasing gas fees for some operations will increase the demand assuming all else stays constant.

The argument made in that section of the FIP is that SPs can (and should) reduce gas usage by using DDO. They can also significantly reduce gas usage by batching prove commit messages (not aggregation, simple batching). Unfortunately, there are other factors (lack of tooling support, Spark depending on f05, lack of expertise) preventing these optimizations from happening organically.

[lanzafame]

To be clear, I am not suggesting that FIP-0084 is the problem here. I am suggesting that the changes of FIP-0084 uncovered issues in other areas of the overall system that resulted in higher than anticipated gas costs for SPs and as such having a snapshotting profile system that allows for the actual before and after measurements of the whole filecoin system would be useful.

[Stebalien]

There's no way to know what will happen after something like this change because that depends on the behavior of the participants in the network.

[lyjmry]

Why not consider increasing TPS

[Fatman13]

The Filecoin network needs to scale above 3PiB per day, we want to reach 1EiB per day!

Is 1EiB the general consensus of the network (as in, part of Filecoin mission / Spec) or author's perspective of how much the network should achieve?

[dannyob]

Can't speak for Irene, but I think it's just an aspiration / turn of phrase. It doesn't affect any of the proposals: it's just that the network shouldn't be limited at a low level.

[Fatman13]

i personally applaud the ambition and have nothing against the proposal itself. But AFAIU i have the impression that there are still no consensus among the "management" about what the numbers should be... I could be wrong tho. I brought this up mainly because having a number with blessings of network consensus would serve as a good guidance for FIP proposals like this one. Maybe having an organizational FIP like FIP0018, which itself can be patched by future FIPs, to provide guard rails for FIP proposals could be a framework to align development efforts towards broader network objectives.

[lyjmry]

Can we roll back the mainnet to the previous version first, and then upgrade it when there is a good solution?

[rvagg]

#1094 is one proposal arising from this discussion, to reduce the batch balancer and remove the aggregate fee from precommit.

[rvagg]

@irenegia would #1094 have any impact on ProveCommitAggregate always being more expensive? I'm interested in this question from a clean-up perspective since it doesn't get any use anymore and I think it would be good to remove it unless we see a path to it having some utility in the future, cost-wise. Its functionality (AFAIK) is a subset of ProveCommitSectors3 and if it's not going to get cheaper then there's not a reason to keep it.

[irenegia]

I think it will makes sense to remove the option of batching and leaving only the option of aggregating in PCS3 only when we will completely remove the batchbalancer mechanism (replacing it with a proof fee for onboarding)

[irenegia]

Let me add a bit more data and a quick recap of ongoing efforts before the holiday break.

• We compared messages sent by the same miner actor in a time window of 48h and they suggest that the current situation is the following:

  • Batching precommit (PrecommitSectorBatch2) -> 3.6x smaller gas units per sector
  • Batching provecommit (ProvecommitSectors3) -> 2.2x smaller gas units per sector
  • Batching provecommit + aggregation -> 1.7x smaller gas units per sector

  Note that currently batching provecommit messages is also always cost-effective, while for the other two cases this depends on the base fee value (because of the batch balancer mechanism). For this reason, we proposed this change FIP draft Remove Batch Fee For Precommit and Lower BatchBalancer #1094.

  If all onboarding messages will be batched, we could go from the current 3-4PiB per day to ~6-7 PiB per day. For this reason, CE and FilOz teams are working on replacing the batch balancer with an onboarding fee, we will post a FIP discussion ad hoc for this in January, cc @AxCortesCubero.

• As regards as DDO savings, we looked at few gas traces and it seems that:

  • Sending a PCS3 with FIL+ deals without notifying the storage market actor save ~⅓ of the PCS3 gas units;
  • Moreover, in this case there is no need of a PSD message. Only a message to transfer datacap back to the verified registry actor is needed. This should save other ~⅔ of gas units.

  So, DDO could bring a ~50% saving per sector, giving an onboarding rate of ~8PiB per day. Looking forward to see more tooling support happening in Q12025 for DDO, cc @magik6k @@patrickwoodhead

• FilOz will research miner actor optimisation and state storage gas in Q12025, cc @rvagg

[rvagg]

I'm proposing in #1098 a minor expansion of scope of this proposal to include some related actors cleanup—the removal of some gas-limited legacy constraints:

• PRE_COMMIT_SECTOR_BATCH_MAX_SIZE
• PROVE_REPLICA_UPDATES_MAX_SIZE
• DECLARATIONS_MAX

This is work that is mostly done and ready to land, but is on a borderline of "does it need a FIP?". It's a protocol change and should go through a process of public consultation. In practice, these constraints are largely unnecessary and are mostly not even used due to the current gas model. Rather than propose a new trivial FIP for this, I'm proposing to bundle the change within this related FIP as a protocol simplification.