bytecode is not the same as in Calibration

[AlcibiadesCleinias]

Issue type

Bug

Have you reproduced the bug with the latest dev version?

No

Version

b1dc6c1

Custom code

No

OS platform and distribution

No response

Describe the issue

While adopting tests for IPC subnet (check rpc in the example below) I faced with wrong re-deploy issue (with hardhat-deploy plugin).
hardhat-deploy plugin always redeploys contracts on IPC subnet but it should not.
The reason for that is the difference b/w bytecode saved on local files and bytecode fetched from the ipc subnet network in hardhat-deploy.

PS. I have checked Calibration as well. On the Calibration it runs as expected

Repro steps

checkout this repo: https://github.com/AlcibiadesCleinias/ipc-debugging

1. npm i
2. prepare .env. Only PRIVATE_KEY is needed and subchain ETH value on the account to deploy contracts + WAIT_CONFIRMATIONS=5 for the same tests on the calibration
3. npx hardhat deploy --network ipcsubnet
4. repeat: npx hardhat deploy --network ipcsubnet - deployed again

If you make 3d and 4th with --network calibration - it goes as expected: it will reuse the existed deployment

Relevant log output

No response

[AlcibiadesCleinias]

fyi: I provided the example repo with hardhat example Token contract

[aakoshh]

Here's the link to the investigation done by @snissn:
https://filecoinproject.slack.com/archives/C04JR5R1UL8/p1698790277580059?thread_ts=1698710408.672269&cid=C04JR5R1UL8

It starts with report on hardhat having trouble deciding if a contract needs to be redeployed, followed by looking into how to reproduce the exact same bytecode as on-chain, ending in a command npx hardhat upgrade-gw-diamond that consistently does the right thing, although I don't know where it is and how it's done.

[aakoshh]

I am able to reproduce this locally as of consensus-shipyard/fendermint@0c76ff3

In the first console I started a smoke test environment, no subnets, just a rootnet:

cd fendermint/testing/smoke-test
cargo make setup

I took note of the chain_id from the logs:

$ docker logs smoke-fendermint
...
2023-12-15T11:06:23.176375812Z  INFO fendermint_app::app: init chain height=0 state_root="bafy2bzacebbjpxdtaxzfvbhpi44henf4mre2jzmmbw5vvfhznmedgrjjm722s" app_hash="0171A0E40220655AE24EE64935FF6F27FE08C7636746315FC59F583ECF05FA3699226E4BF69D" timestamp=1680101412 chain_id=2327059176193834

In the second console I cloned the above repo and installed hardhat:

mkdir -p fluence
cd fluence
git clone [email protected]:AlcibiadesCleinias/ipc-debugging.git
cd ipc-debugging
npm install --force --save-dev hardhat

Then I edited the hardhat.config.ts file to point at my smoke test environment and use it the test keys it generated:

cat ../../fendermint/fendermint/testing/smoke-test/test-data/keys/eric-eth.sk

so the config looks like this:

const PRIVATE_KEYS = ["0x8d8eae991de7105f2a8e8f978dc7c21f541550d94666416a562e35d3b7ce802a"];

const config: HardhatUserConfig = {
    ...
    networks: {
        ...
        ipcsubnet: {
            url: 'http://localhost:8545',
            accounts: accounts,
            chainId: 2327059176193834,
            timeout: 1000000,
        },
...
    },

I got the chainId from docker logs smoke-fendermint above.

Then I ran the deployment above:

$ rm -rf deployments/ipcsubnet/ 

$ npx hardhat deploy --network ipcsubnet
Nothing to compile
No need to generate any newer typings.
deploying "Token" (tx: 0xcd38c7d231a77c8941bfe4eb007dc1fd0719b3cb00f914757206359695c59a80)...: deployed at 0xE2e8A2480d8f19a6076b65dC367867E3d343D3EB with 22136934 gas

$ npx hardhat deploy --network ipcsubnet
Nothing to compile
No need to generate any newer typings.
deploying "Token" (tx: 0x723094c6590d1cce1f8ddbbe1af0877699717433d176aae224c914c9d3056bdf)...: deployed at 0x5BAF3F607E8d651ed10d0b1F27a32b32C838915e with 22339301 gas

[aakoshh]

Comparing what code gets returned by the Ethereum API shows that the two contracts have the same deployed bytecode:

$ FIRST=$(curl -s -X POST  -H 'Content-Type: application/json'   -d '{"jsonrpc":"2.0","id":0,"method":"eth_getCode","params":["0x5BAF3F607E8d651ed10d0b1F27a32b32C838915e", "latest"]}'   http://localhost:8545 | jq -r ".result")
$ SECOND=$(curl -s -X POST  -H 'Content-Type: application/json'   -d '{"jsonrpc":"2.0","id":0,"method":"eth_getCode","params":["0xE2e8A2480d8f19a6076b65dC367867E3d343D3EB", "latest"]}'   http://localhost:8545 | jq -r ".result")
$ echo $FIRST
0x608060405234801561001057600080fd5b50600436106100625760003560e01c806306fdde031461006757806318160ddd1461008557806370a082311461009c5780638da5cb5b146100c557806395d89b41146100f0578063a9059cbb146100f8575b600080fd5b61006f61010d565b60405161007c919061037a565b60405180910390f35b61008e60025481565b60405190815260200161007c565b61008e6100aa3660046103b0565b6001600160a01b031660009081526004602052604090205490565b6003546100d8906001600160a01b031681565b6040516001600160a01b03909116815260200161007c565b61006f61019b565b61010b6101063660046103cb565b6101a8565b005b6000805461011a906103f5565b80601f0160208091040260200160405190810160405280929190818152602001828054610146906103f5565b80156101935780601f1061016857610100808354040283529160200191610193565b820191906000526020600020905b81548152906001019060200180831161017657829003601f168201915b505050505081565b6001805461011a906103f5565b336000908152600460205260409020548111156101ff5760405162461bcd60e51b81526020600482015260116024820152704e6f7420656e6f75676820746f6b656e7360781b604482015260640160405180910390fd5b6102236040518060600160405280602481526020016104a8602491393384846102b8565b3360009081526004602052604081208054839290610242908490610445565b90915550506001600160a01b0382166000908152600460205260408120805483929061026f90849061045e565b90915550506040518181526001600160a01b0383169033907fddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef9060200160405180910390a35050565b610301848484846040516024016102d29493929190610471565b60408051601f198184030181529190526020810180516001600160e01b0316638ef3f39960e01b179052610307565b50505050565b61031081610313565b50565b60006a636f6e736f6c652e6c6f679050600080835160208501845afa505050565b6000815180845260005b8181101561035a5760208185018101518683018201520161033e565b506000602082860101526020601f19601f83011685010191505092915050565b60208152600061038d6020830184610334565b9392505050565b80356001600160a01b03811681146103ab57600080fd5b919050565b6000602082840312156103c257600080fd5b61038d82610394565b600080604083850312156103de57600080fd5b6103e783610394565b946020939093013593505050565b600181811c9082168061040957607f821691505b60208210810361042957634e487b7160e01b600052602260045260246000fd5b50919050565b634e487b7160e01b600052601160045260246000fd5b818103818111156104585761045861042f565b92915050565b808201808211156104585761045861042f565b6080815260006104846080830187610334565b6001600160a01b039586166020840152939094166040820152606001529291505056fe5472616e7366657272696e672066726f6d20257320746f20257320257320746f6b656e73a26469706673582212206df5ab4dac5d7547bcaff3bec3420f16bcede6a55544ab65326b5a0a2d78c80b64736f6c63430008120033
$ echo $SECOND
0x608060405234801561001057600080fd5b50600436106100625760003560e01c806306fdde031461006757806318160ddd1461008557806370a082311461009c5780638da5cb5b146100c557806395d89b41146100f0578063a9059cbb146100f8575b600080fd5b61006f61010d565b60405161007c919061037a565b60405180910390f35b61008e60025481565b60405190815260200161007c565b61008e6100aa3660046103b0565b6001600160a01b031660009081526004602052604090205490565b6003546100d8906001600160a01b031681565b6040516001600160a01b03909116815260200161007c565b61006f61019b565b61010b6101063660046103cb565b6101a8565b005b6000805461011a906103f5565b80601f0160208091040260200160405190810160405280929190818152602001828054610146906103f5565b80156101935780601f1061016857610100808354040283529160200191610193565b820191906000526020600020905b81548152906001019060200180831161017657829003601f168201915b505050505081565b6001805461011a906103f5565b336000908152600460205260409020548111156101ff5760405162461bcd60e51b81526020600482015260116024820152704e6f7420656e6f75676820746f6b656e7360781b604482015260640160405180910390fd5b6102236040518060600160405280602481526020016104a8602491393384846102b8565b3360009081526004602052604081208054839290610242908490610445565b90915550506001600160a01b0382166000908152600460205260408120805483929061026f90849061045e565b90915550506040518181526001600160a01b0383169033907fddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef9060200160405180910390a35050565b610301848484846040516024016102d29493929190610471565b60408051601f198184030181529190526020810180516001600160e01b0316638ef3f39960e01b179052610307565b50505050565b61031081610313565b50565b60006a636f6e736f6c652e6c6f679050600080835160208501845afa505050565b6000815180845260005b8181101561035a5760208185018101518683018201520161033e565b506000602082860101526020601f19601f83011685010191505092915050565b60208152600061038d6020830184610334565b9392505050565b80356001600160a01b03811681146103ab57600080fd5b919050565b6000602082840312156103c257600080fd5b61038d82610394565b600080604083850312156103de57600080fd5b6103e783610394565b946020939093013593505050565b600181811c9082168061040957607f821691505b60208210810361042957634e487b7160e01b600052602260045260246000fd5b50919050565b634e487b7160e01b600052601160045260246000fd5b818103818111156104585761045861042f565b92915050565b808201808211156104585761045861042f565b6080815260006104846080830187610334565b6001600160a01b039586166020840152939094166040820152606001529291505056fe5472616e7366657272696e672066726f6d20257320746f20257320257320746f6b656e73a26469706673582212206df5ab4dac5d7547bcaff3bec3420f16bcede6a55544ab65326b5a0a2d78c80b64736f6c63430008120033
$ if [ "$FIRST" = "$SECOND" ]; then echo "ok"; else echo "not ok"; fi
ok

[aakoshh]

Comparing the output from the API to the one hardhat thinks should be deployed shows they also match:

$ HHDBC=$(cat deployments/ipcsubnet/Token.json | jq -r ".deployedBytecode")
$ echo $HHDBC
0x608060405234801561001057600080fd5b50600436106100625760003560e01c806306fdde031461006757806318160ddd1461008557806370a082311461009c5780638da5cb5b146100c557806395d89b41146100f0578063a9059cbb146100f8575b600080fd5b61006f61010d565b60405161007c919061037a565b60405180910390f35b61008e60025481565b60405190815260200161007c565b61008e6100aa3660046103b0565b6001600160a01b031660009081526004602052604090205490565b6003546100d8906001600160a01b031681565b6040516001600160a01b03909116815260200161007c565b61006f61019b565b61010b6101063660046103cb565b6101a8565b005b6000805461011a906103f5565b80601f0160208091040260200160405190810160405280929190818152602001828054610146906103f5565b80156101935780601f1061016857610100808354040283529160200191610193565b820191906000526020600020905b81548152906001019060200180831161017657829003601f168201915b505050505081565b6001805461011a906103f5565b336000908152600460205260409020548111156101ff5760405162461bcd60e51b81526020600482015260116024820152704e6f7420656e6f75676820746f6b656e7360781b604482015260640160405180910390fd5b6102236040518060600160405280602481526020016104a8602491393384846102b8565b3360009081526004602052604081208054839290610242908490610445565b90915550506001600160a01b0382166000908152600460205260408120805483929061026f90849061045e565b90915550506040518181526001600160a01b0383169033907fddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef9060200160405180910390a35050565b610301848484846040516024016102d29493929190610471565b60408051601f198184030181529190526020810180516001600160e01b0316638ef3f39960e01b179052610307565b50505050565b61031081610313565b50565b60006a636f6e736f6c652e6c6f679050600080835160208501845afa505050565b6000815180845260005b8181101561035a5760208185018101518683018201520161033e565b506000602082860101526020601f19601f83011685010191505092915050565b60208152600061038d6020830184610334565b9392505050565b80356001600160a01b03811681146103ab57600080fd5b919050565b6000602082840312156103c257600080fd5b61038d82610394565b600080604083850312156103de57600080fd5b6103e783610394565b946020939093013593505050565b600181811c9082168061040957607f821691505b60208210810361042957634e487b7160e01b600052602260045260246000fd5b50919050565b634e487b7160e01b600052601160045260246000fd5b818103818111156104585761045861042f565b92915050565b808201808211156104585761045861042f565b6080815260006104846080830187610334565b6001600160a01b039586166020840152939094166040820152606001529291505056fe5472616e7366657272696e672066726f6d20257320746f20257320257320746f6b656e73a26469706673582212206df5ab4dac5d7547bcaff3bec3420f16bcede6a55544ab65326b5a0a2d78c80b64736f6c63430008120033
$ if [ "$FIRST" = "$HHDBC" ]; then echo "ok"; else echo "not ok"; fi
ok

[aakoshh]

Trying to understand why it shouldn't deploy twice. The deterministicDeployment option is not used. The deployment logs are coming from the helper which is called by the deployment script, which in turn is called by the runDeploy function of the DeploymentManager.

I can't find any point where it would decide not to deploy. For example I assume it would have to look at deployment.address to check what the bytecode is, but it's only accessed in saveDeployment and export. Similar situation with bytecode and deployedBytecode.

[aakoshh]

Ok, I see some checks happening in fetchIfDifferent.

It calls compareDeploymentTransaction on the DeploymentFactory to check if the data field of the original transaction was the same as the new one would be.

[aakoshh]

Fetching the transactions reveals that they are identical, but that is after being returned by the API, so the question is why the input constructed by Hardhat was different:

$ curl -s -X POST  -H 'Content-Type: application/json'   -d '{"jsonrpc":"2.0","id":0,"method":"eth_getTransactionByHash","params":["0xcd38c7d231a77c8941bfe4eb007dc1fd0719b3cb00f914757206359695c59a80"]}'   http://localhost:8545 | jq -r ".result.input"
0x59070860c0604052601060809081526f26bc902430b9323430ba102a37b5b2b760811b60a05260009061002f9082610139565b5060408051808201909152600381526213521560ea1b60208201526001906100579082610139565b50620f424060025534801561006b57600080fd5b5060025433600081815260046020526040902091909155600380546001600160a01b03191690911790556101f8565b634e487b7160e01b600052604160045260246000fd5b600181811c908216806100c457607f821691505b6020821081036100e457634e487b7160e01b600052602260045260246000fd5b50919050565b601f82111561013457600081815260208120601f850160051c810160208610156101115750805b601f850160051c820191505b818110156101305782815560010161011d565b5050505b505050565b81516001600160401b038111156101525761015261009a565b6101668161016084546100b0565b846100ea565b602080601f83116001811461019b57600084156101835750858301515b600019600386901b1c1916600185901b178555610130565b600085815260208120601f198616915b828110156101ca578886015182559484019460019091019084016101ab565b50858210156101e85787850151600019600388901b60f8161c191681555b5050505050600190811b01905550565b610501806102076000396000f3fe608060405234801561001057600080fd5b50600436106100625760003560e01c806306fdde031461006757806318160ddd1461008557806370a082311461009c5780638da5cb5b146100c557806395d89b41146100f0578063a9059cbb146100f8575b600080fd5b61006f61010d565b60405161007c919061037a565b60405180910390f35b61008e60025481565b60405190815260200161007c565b61008e6100aa3660046103b0565b6001600160a01b031660009081526004602052604090205490565b6003546100d8906001600160a01b031681565b6040516001600160a01b03909116815260200161007c565b61006f61019b565b61010b6101063660046103cb565b6101a8565b005b6000805461011a906103f5565b80601f0160208091040260200160405190810160405280929190818152602001828054610146906103f5565b80156101935780601f1061016857610100808354040283529160200191610193565b820191906000526020600020905b81548152906001019060200180831161017657829003601f168201915b505050505081565b6001805461011a906103f5565b336000908152600460205260409020548111156101ff5760405162461bcd60e51b81526020600482015260116024820152704e6f7420656e6f75676820746f6b656e7360781b604482015260640160405180910390fd5b6102236040518060600160405280602481526020016104a8602491393384846102b8565b3360009081526004602052604081208054839290610242908490610445565b90915550506001600160a01b0382166000908152600460205260408120805483929061026f90849061045e565b90915550506040518181526001600160a01b0383169033907fddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef9060200160405180910390a35050565b610301848484846040516024016102d29493929190610471565b60408051601f198184030181529190526020810180516001600160e01b0316638ef3f39960e01b179052610307565b50505050565b61031081610313565b50565b60006a636f6e736f6c652e6c6f679050600080835160208501845afa505050565b6000815180845260005b8181101561035a5760208185018101518683018201520161033e565b506000602082860101526020601f19601f83011685010191505092915050565b60208152600061038d6020830184610334565b9392505050565b80356001600160a01b03811681146103ab57600080fd5b919050565b6000602082840312156103c257600080fd5b61038d82610394565b600080604083850312156103de57600080fd5b6103e783610394565b946020939093013593505050565b600181811c9082168061040957607f821691505b60208210810361042957634e487b7160e01b600052602260045260246000fd5b50919050565b634e487b7160e01b600052601160045260246000fd5b818103818111156104585761045861042f565b92915050565b808201808211156104585761045861042f565b6080815260006104846080830187610334565b6001600160a01b039586166020840152939094166040820152606001529291505056fe5472616e7366657272696e672066726f6d20257320746f20257320257320746f6b656e73a26469706673582212206df5ab4dac5d7547bcaff3bec3420f16bcede6a55544ab65326b5a0a2d78c80b64736f6c63430008120033
$ curl -s -X POST  -H 'Content-Type: application/json'   -d '{"jsonrpc":"2.0","id":0,"method":"eth_getTransactionByHash","params":["0x723094c6590d1cce1f8ddbbe1af0877699717433d176aae224c914c9d3056bdf"]}'   http://localhost:8545 | jq -r ".result.input"
0x59070860c0604052601060809081526f26bc902430b9323430ba102a37b5b2b760811b60a05260009061002f9082610139565b5060408051808201909152600381526213521560ea1b60208201526001906100579082610139565b50620f424060025534801561006b57600080fd5b5060025433600081815260046020526040902091909155600380546001600160a01b03191690911790556101f8565b634e487b7160e01b600052604160045260246000fd5b600181811c908216806100c457607f821691505b6020821081036100e457634e487b7160e01b600052602260045260246000fd5b50919050565b601f82111561013457600081815260208120601f850160051c810160208610156101115750805b601f850160051c820191505b818110156101305782815560010161011d565b5050505b505050565b81516001600160401b038111156101525761015261009a565b6101668161016084546100b0565b846100ea565b602080601f83116001811461019b57600084156101835750858301515b600019600386901b1c1916600185901b178555610130565b600085815260208120601f198616915b828110156101ca578886015182559484019460019091019084016101ab565b50858210156101e85787850151600019600388901b60f8161c191681555b5050505050600190811b01905550565b610501806102076000396000f3fe608060405234801561001057600080fd5b50600436106100625760003560e01c806306fdde031461006757806318160ddd1461008557806370a082311461009c5780638da5cb5b146100c557806395d89b41146100f0578063a9059cbb146100f8575b600080fd5b61006f61010d565b60405161007c919061037a565b60405180910390f35b61008e60025481565b60405190815260200161007c565b61008e6100aa3660046103b0565b6001600160a01b031660009081526004602052604090205490565b6003546100d8906001600160a01b031681565b6040516001600160a01b03909116815260200161007c565b61006f61019b565b61010b6101063660046103cb565b6101a8565b005b6000805461011a906103f5565b80601f0160208091040260200160405190810160405280929190818152602001828054610146906103f5565b80156101935780601f1061016857610100808354040283529160200191610193565b820191906000526020600020905b81548152906001019060200180831161017657829003601f168201915b505050505081565b6001805461011a906103f5565b336000908152600460205260409020548111156101ff5760405162461bcd60e51b81526020600482015260116024820152704e6f7420656e6f75676820746f6b656e7360781b604482015260640160405180910390fd5b6102236040518060600160405280602481526020016104a8602491393384846102b8565b3360009081526004602052604081208054839290610242908490610445565b90915550506001600160a01b0382166000908152600460205260408120805483929061026f90849061045e565b90915550506040518181526001600160a01b0383169033907fddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef9060200160405180910390a35050565b610301848484846040516024016102d29493929190610471565b60408051601f198184030181529190526020810180516001600160e01b0316638ef3f39960e01b179052610307565b50505050565b61031081610313565b50565b60006a636f6e736f6c652e6c6f679050600080835160208501845afa505050565b6000815180845260005b8181101561035a5760208185018101518683018201520161033e565b506000602082860101526020601f19601f83011685010191505092915050565b60208152600061038d6020830184610334565b9392505050565b80356001600160a01b03811681146103ab57600080fd5b919050565b6000602082840312156103c257600080fd5b61038d82610394565b600080604083850312156103de57600080fd5b6103e783610394565b946020939093013593505050565b600181811c9082168061040957607f821691505b60208210810361042957634e487b7160e01b600052602260045260246000fd5b50919050565b634e487b7160e01b600052601160045260246000fd5b818103818111156104585761045861042f565b92915050565b808201808211156104585761045861042f565b6080815260006104846080830187610334565b6001600160a01b039586166020840152939094166040820152606001529291505056fe5472616e7366657272696e672066726f6d20257320746f20257320257320746f6b656e73a26469706673582212206df5ab4dac5d7547bcaff3bec3420f16bcede6a55544ab65326b5a0a2d78c80b64736f6c63430008120033

[aakoshh]

I can confirm that with the fix in consensus-shipyard/fendermint#467 we get the correct behaviour:

$ rm -rf deployments/ipcsubnet/ 
$ npx hardhat deploy --network ipcsubnet
Nothing to compile
No need to generate any newer typings.
deploying "Token" (tx: 0x5abaa76d9a4b2ed09a35f7453965b345daaf92346f4b9c50ad6f639613831f1e)...: deployed at 0xF73Fbe84e36a23fee270a77b1053AF8975BB9A6d with 22519511 gas
$ npx hardhat deploy --network ipcsubnet
Nothing to compile
No need to generate any newer typings.
reusing "Token" at 0xF73Fbe84e36a23fee270a77b1053AF8975BB9A6d