Incorrect forge script gas estimation on Sonic

[kasperpawlowski]

Component

Forge

Have you ensured that all of these are up to date?

• Foundry
• Foundryup

What version of Foundry are you on?

forge 0.3.0 (5a8bd89 2024-12-20T08:45:53.204298000Z)

What version of Foundryup are you on?

foundryup: 0.3.0

What command(s) is the bug in?

forge script

Operating System

macOS (Apple Silicon)

Describe the bug

Description

I ran into an issue when trying to run our deployment script on Sonic. This is something that haven't been observed when running it against any other network so far.

The problem seems to be incorrect gas estimation when running forge script. We have very a complex deployment script and, as with any other script, when we run it, foundry first simulates the transactions, estimates gas and saves the results into the broadcast file. Then, it uses that data to broadcast the transactions.

The issue is that after a couple transactions, foundry starts estimating gas as if we were hitting the block gas limit for all the remaining transactions. This can be seen in the broadcast file. The fact that gas gets overestimated makes the script fail because most likely the deployer does not have enough balance to deploy all the contracts according to the incorrectly estimated gas. If the deployer had enough balance to send the transactions, it would appear that only a small amount of gas was used, proving that the gas was heavily overestimated.

To reproduce the issue, I created a very simple example script. The only thing it does is trying to deploy the same contract multiple times. When run against an anvil fork (the same issue can be observed when running against the production network too), foundry estimates gas correctly for the first ~60 transactions and then overestimates the gas for the remaining transactions.

To see that, you can look at the broadcast file in the repo provided below. Gas for the first 60 transactions is equal to ~1.4M. However, for the remaining transactions, the gas estimated is equal to ~1B (roughly 700 times more, where 1B is the block gas limit on Sonic). Because the default private key on Anvil has enough balance to broadcast the transaction, the script passes, but there's no reason the deployment of exactly the same contract should consume vastly different amount of gas.

This is very problematic because on the production network the deployer does not have unlimited amount of gas currency. The trick could be to manually change the gas amount in the broadcast file and rerun the script with the --resume option. However, this seems more like a workaround and not a fix especially because it's not clear what's causing the issue.

Steps to reproduce:

1. Clone the repo

git clone https://github.com/euler-xyz/foundry-sonic-issue.git

2. Install the dependencies and compile the contracts

cd foundry-sonic-issue && forge install && forge build

3. Spin up Anvil fork

anvil --fork-url https://rpc.soniclabs.com

4. Run the script using the default Anvil private key

forge script script/Test.s.sol:Test --broadcast --slow --rpc-url 127.0.0.1:8545 --private-key 0xac0974bec39a17e36ba4a6b4d238ff944bacb478cbed5efcae784d7bf4f2ff80

5. Check the broadcast file (./broadcast/Test.s.sol/146/run-latest.json) and the console logs. According to the console logs, gas estimate will be very high (look at the top) while the real gas consumed will be very low (look at the bottom). The only reason the script succeededs is that the default Anvil account has enough balance to cover the overestimated gas amount. When the same done against Ethereum mainnet or any other network I tested, the gas estimate is the same for all the transactions in the broadcast file.

[zerosnacks]

Was able to reproduce w/ given setup

It appears that this is resolved by forcing gas estimation via the RPC

Would be great if you could check if this resolves it, branch zerosnacks/incorrect-gas-estimation-sonic / #9739

[kasperpawlowski]

Hey @zerosnacks! Thanks for looking into this!

I can confirm that the issue went away after using your branch. However, it only went away for the example repo I prepared. I tried our prod deployment scripts repository and unfortunately nothing has changed. Note that we used this setup on multiple networks already and it has never been an issue. It's only affecting Sonic.

Would you like to take a look at it? If so, these are the steps to reproduce. I prepared it in a way you can easily run the script with no .env file:

git clone https://github.com/euler-xyz/euler-interfaces.git && git clone https://github.com/euler-xyz/evk-periphery.git

cd evk-periphery && git checkout issue-sonic && forge install && forge compile lib/euler-earn/src --force --optimize --optimizer-runs 800 --use 0.8.27 --out out-euler-earn && forge compile

anvil --fork-url https://rpc.soniclabs.com

In a different terminal:

env DEPLOYMENT_RPC_URL=http://127.0.0.1:8545 ADDRESSES_DIR_PATH=../euler-interfaces/addresses FORCE_MULTISIG_ADDRESSES=true forge script script/50_CoreAndPeriphery.s.sol --legacy --slow --rpc-url http://127.0.0.1:8545 --private-key 0xac0974bec39a17e36ba4a6b4d238ff944bacb478cbed5efcae784d7bf4f2ff80

As you can see in ./broadcast/50_CoreAndPeriphery.s.sol/146/dry-run/run-latest.json, after a couple transactions (always 4 in my case; the 5th one has 1B gas) exactly the same can be observed.

[zerosnacks]

Hi @kasperpawlowski

It is still a bit of a mystery but thought I would share preliminary results. It appears that indeed after a couple of transactions it dramatically escalates the gas use. However when running the same test with the proposed fix applied it suddenly recovers.

The first column is the gas usage before.
The second colum is after the initial proposed fix.

I noticed that in the broadcast log it is including a isFixedGasLimit key which seems out of place.

The third column is after disabling this line which seems to yield the desired result:

foundry/crates/script/src/transaction.rs

Lines 78 to 79 in e16a75b

	// If tx.gas is already set that means it was specified in script
	metadata.is_fixed_gas_limit = metadata.tx().gas().is_some();

cc @klkvr I'm missing some context on this, would you know more? It was added in #4219 / #8833

It appears it could be related to the use of {gas: <gas>} inside of scripts (or contracts) triggering this path and causing this to not run

foundry/crates/script/src/broadcast.rs

Lines 98 to 102 in b026f7a

	// Chains which use `eth_estimateGas` are being sent sequentially and require their
	// gas to be re-estimated right before broadcasting.
	if !is_fixed_gas_limit && estimate_via_rpc {
	estimate_gas(tx, &provider, estimate_multiplier).await?;
	}

0x67f87a   => 0x67f87a   => 0x67f87a
0x12d0ce   => 0x12d0ce   => 0x12d0ce
0x4c37d    => 0x4c37d    => 0x4c37d
0x34a3ad   => 0x34a3ad   => 0x34a3ad
0x3b48fa92 => 0x3b48fa92 => 0x24e00f
0x3b2f5693 => 0x3b2f5693 => 0x2aa5ba
0x3b11904b => 0x3b11904b => 0x47fa0e
0x3adf10d6 => 0x3adf10d6 => 0x4b8ce3
0x3aaa14e8 => 0x3aaa14e8 => 0x41c7f8
0x3a7c0cd7 => 0x3a7c0cd7 => 0x2e2733
0x3a5bdc79 => 0x3a5bdc79 => 0xb5523
0x3a543e28 => 0x3a543e28 => 0x4b6182
0x3a1f314c => 0x3a1f314c => 0x64daf3
0x39bf72f7 => 0x19fb91   => 0x19fb91
0x39add03f => 0x6cd5     => 0x6cd5
0x398db396 => 0x61a1ef   => 0x61a1ef
0x39491a78 => 0x1265d5   => 0x1265d5
0x393c5793 => 0x6f715    => 0x6f715
0x39379d77 => 0x6b88a    => 0x6b88a
0x39328f2f => 0x1f7559   => 0x1f7559
0x391c9dd4 => 0x7477a    => 0x7477a
0x391567a2 => 0x677f30   => 0x677f30
0x38c1d9de => 0x96b6c    => 0x96b6c
0x389c9b88 => 0x1e2c05   => 0x1e2c05
0x396db7f7 => 0x6f6f     => 0x6f6f
0x396c3f7a => 0x6f6f     => 0x6f6f
0x396b22cd => 0x6f6f     => 0x6f6f
0x396a0e1b => 0x6f6f     => 0x6f6f
0x38579ef8 => 0x25e0de   => 0x25e0de
0x39224e1a => 0x6f6f     => 0x6f6f
0x3816a78b => 0x233874   => 0x233874
0x37c9ec44 => 0x25646b   => 0x25646b
0x3892d3eb => 0x6f6f     => 0x6f6f
0x3779aeb6 => 0x310fbd   => 0x310fbd
0x3716467d => 0x1e2b61   => 0x1e2b61
0x37013fc1 => 0x1287da   => 0x1287da
0x36f4713a => 0x1287da   => 0x1287da
0x36e7a3ab => 0x11183c   => 0x11183c
0x36dbc624 => 0x1287da   => 0x1287da
0x36ae56ec => 0x12eeaa   => 0x12eeaa
0x377fa4cd => 0x6cd5     => 0x6cd5
0x367610a0 => 0x22d343   => 0x22d343
0x365db734 => 0x6a5e7    => 0x6a5e7
0x364467be => 0xba712    => 0xba712
0x3622d621 => 0x11d6ec   => 0x11d6ec
0x35fa2653 => 0x168677   => 0x168677
0x35b2cd76 => 0x294836   => 0x294836
0x355cfab8 => 0x2a2cbb   => 0x2a2cbb
0x351ae85c => 0xba780    => 0xba780
0x3512f43b => 0x11d6ec   => 0x11d6ec
0x34f33da5 => 0x93aaf    => 0x93aaf
0x34b04664 => 0x495270   => 0x495270
0x3443ce0a => 0x4132fa   => 0x4132fa
0x34046d91 => 0x87009    => 0x87009
0x33d148f6 => 0x2f5ae1   => 0x2f5ae1
0x3363ed30 => 0x58b030   => 0x58b030
0x32f2b6a5 => 0x366ee3   => 0x366ee3
0x339a7f3e => 0xa608     => 0xa608

[zerosnacks]

These are the broadcast logs per column:

1.json
2.json
3.json

[kasperpawlowski]

Interesting! I didn't notice that it actually recovers after a few transactions, I checked only the first few 1B ones assuming it will be screwed until the very end as the last time.

Re isFixedGasLimit I also missed that and it is indeed weird! My scripts do not specify any gas settings. I ran search on all the broadcast files that I retained locally in the last few months. I ran those scripts (and not only them) on at least 10 different networks and the only broadcast files that contain "isFixedGasLimit": true are those targeting Sonic.

@zerosnacks @klkvr Do you have any clue what might be the reason?

[kasperpawlowski]

Interestingly, all 9 contract deployment transactions that hit 1B in case number 2 appear to be from one script (05_EVaultImplementation.s.sol) which gets called by 50_CoreAndPeriphery.s.sol that you ran.

The only thing that comes to my mind is that all those modules deployed in 05_EVaultImplementation.s.sol have address.code.length checks. The addresses on which those checks are performed are contracts deployed in 02_PeripheryFactories.s.sol which are passed in modules' constructors. Could it be the case that while simulating, foundry might be somehow missing the fact that those contracts were deployed in the previous transactions? This would make 05_EVaultImplementation.s.sol modules deployments revert and estimate gas at 1B.

However, this whole script is a chain of contract deployments and interactions with them. If the state change made by the previous transactions is not taken into account in the following one, we would hit many more reverts, not only those 9.

[zerosnacks]

Hi @kasperpawlowski

After further investigation it appears that passing --block-gas-limit 1000000000 (1B, the gas limit of Sonic) to forge script resolves this without any other changes that I had proposed in the draft. It resolves the issue reported in both repro's.

My hunch is that because we have a default gas limit set at 1073741824 (just over >1B) there can be situations where transactions that just exceed the gas limit and the filler still tries to put in the block which would result in this infinite gas behavior. The "recovery" behavior likely stems from when it does reach the (incorrect) limit it automatically rolls over to the next block.

I'll have a look at whether we can make improvements on our end to infer this limit from the RPC during simulation / filling to prevent this going forward but this should unblock you for now.

[kasperpawlowski]

Thank you for your support @zerosnacks! It worked! I finally managed to successfully deploy on Sonic