bug: Jail reason consensus not achieved

[cosmoscats]

What is happening?

Section description

Provide as much context as you can. Give as much context as you can to make it easier for the developers to figure what is happening.

Our node goes to Jail every day few times a day. According to the latest data, it almost always goes to Jail because of ETH RPC.
I have the opportunity to install 10 different RPC providers for testing.

Paloma and pigeon versions and logs

Section description

Write down paloma version. Write down pigeon version. Copy and paste pigeon config file as well as relevant ENV variables.

palomad version
v2.3.2

pigeon version
App version: v2.3.1
Build commit hash: 5f6f4bcaa645e0f9d8530b24cd1a4cd74b79e857

i will attach latest latest jail log
paloma-jail-24-10-2024.txt

How to reproduce?

Section description

Please write detailed steps of what you were doing for this bug to appear.

Unjail, wait for sometime and in 24 hours or less node will be jailed

What is the expected behaviour?

Section description

If you know, please write down what is the expected behaviour. If you don't know, that's ok. We can have a discussion in comments.

[taariq]

Logs added

Oct 24 20:27:28 cc1 palomad[2745431]: 8:27PM INF put message into consensus queue message-id=350030 module=x/palomaconsensus msg="hexAddresses:"0x1177806cD88b0BC0eA363bEd3C64f8314361b162" hexAddresses:"0x0DAaBB4FF60423Eb1F14cC6731394e098ad51bcb" hexAddresses:"0xC1009C72cC2519B0DD595C3E500C1293A862Aa6A" hexAddresses:"0x35c1cC4E3CD4624Ac177c167D2427e9C95336535" hexAddresses:"0x377D23948D41579f2c3cA40308e3bdd53f6dA7B2" hexAddresses:"0x1799D68d773f9E03Baca79cd1d836Ba026B4742A" hexAddresses:"0x43E218f96A567DC26C6e65fCabF1fDE26Af69444" hexAddresses:"0x443266026738061972012e62D5Ecd9D98da8B6F4" hexAddresses:"0x4a89f96fdff3C161937CFBC3e22d2E325612aaEc" hexAddresses:"0xa228292447064D5818BbC80b577C91f5212F9355" hexAddresses:"0xDEea5b069208e0eE37b630A3e7672FC50e8fE24a" hexAddresses:"0xAAc74d38c82c367b3dA7482E3Aecf6DD7A512eF2" hexAddresses:"0xeB784b37365C302C97C9Ec8cB0933cE344e6dE42" hexAddresses:"0x2d91F6C502289eA4d7F9253C45788109377b95bD" hexAddresses:"0x732fBb6018F2cB5b844055C9EB567447a3132833" hexAddresses:"0x19f5911E4cA69E30449aD6BB71De341F01F118BB" hexAddresses:"0x1ad90dB98da083E117D5F62a1673fC0f3A5930ca" hexAddresses:"0x63f55bc560E981d53E1f5bb3643e3a96D26fc635" hexAddresses:"0x6dc59EE4bdFa2C791229004f29b08F783491a934" hexAddresses:"0x14Aa448C2C918C4427c5671028b63BC17f6132d5" hexAddresses:"0xEe21301aF1d9562B5cBEdf520077Ea0a9bC9d535" hexAddresses:"0x7Bd1A3270570b65F264895C2F86E631fDc497545" valAddresses:"\001\276s\021\343\227\312Q_\246G*\224\363(\340K\273\350\006" valAddresses:"\002}0\274a\2108\266M\347f4\001<\236Jp\365bx" valAddresses:"\t\007\3044\210\241\355[\370\347\010W15\206\207(-h." valAddresses:"\t\231\212\337\024\375x\026BE\3600>9\322\357\333\362\316\345" valAddresses:"\021R\201_\333\020\021\310\304-\261\234\366\203F\\313\\363\\344\\322\" valAddresses:\"V5{\\265e\\371\\017\\023\\205\\332J\\335\\205\\245\\275\\342\\306q\\205G\" valAddresses:\"[\\\\\\250\\24435\\325\\313E\\262Z\\363\\177g\\244i64\\242}\" valAddresses:\"\\\\8y\\256\\374\\200sQ~\\331\\222\\345\\026+\\303X\\016\\001x\\032\" valAddresses:\"]:W\\321\\311\\303)o\\274S\\214s)\\260\\255\\266%<\\214\\004\" valAddresses:\"e%\\315\\342\\033\\245\\217\\234\\232H\\022S\\345O\\321\\331\\2446\\340\\332\" valAddresses:\"v\\352C\\222\\341\\t\\270\\335J\\216c\\207\\344\\220\\223\\275\\336U\\355\\224\" valAddresses:\"y\\024\\257H\\255yC\\213d\\304\\212\\230\\244b\\331\\034\\213\\327\\333\\031\" valAddresses:\"y~y\\006\\2421P~7+\\235\\350\\007\\\"\\002\\311\\325\\274\\344\\024\" valAddresses:\"y\\360\\311q\\3445\\352\\333\\313\\314$\\3268\\220\\324(\\375t\\326~\" valAddresses:\"\\200\\242)\200>\033\034\252\305\261\264\332\326\007t\372\304\300\212" valAddresses:"\203\273Ka%\307\350\323(\2013\361A\2274\003S1\004\256" valAddresses:"\203\377\331\332\361\215\202<\215_R\247\331\332\203d9r\332\035" valAddresses:"\217!\373\274\212RS9\264\203\211\242\367\001\247\356\203\220]T" valAddresses:"\242\030KV&\253\364\317[\r\250\322\232\001<X7\211\206Q" valAddresses:"\27471u\333\237xB\200\3260\244\r\253aj\236\035\247\251" valAddresses:"\301\370kXOx&_:\001/6rG\247w\3666G\024" valAddresses:"\3459+\360`_Mf-=\002\020\261\321\334\240!\263@P" fromBlockTime:<seconds:1729801647 nanos:91967665 > " queue-type-name=evm/eth-main/validators-balances
Oct 24 20:27:31 cc1 palomad[2745431]: 8:27PM INF added message evidence. chain-reference-id=eth-main chain-type=evm message-id=350030 module=x/palomaconsensus queue-type-name=evm/eth-main/validators-balances validator=palomavaloper15gvyk43x406v7kcd4rff5qfutqmcnpj3w9wpnm
Oct 24 20:27:31 cc1 palomad[2745431]: 8:27PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=147956415466737
Oct 24 20:27:33 cc1 palomad[2745431]: 8:27PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=147956415466737
Oct 24 20:27:35 cc1 palomad[2745431]: 8:27PM INF added message evidence. chain-reference-id=eth-main chain-type=evm message-id=350030 module=x/palomaconsensus queue-type-name=evm/eth-main/validators-balances validator=palomavaloper1hsmnzawmnauy9qxkxzjqm2mpd20pmfaf5uj30x
Oct 24 20:27:35 cc1 palomad[2745431]: 8:27PM INF added message evidence. chain-reference-id=eth-main chain-type=evm message-id=350030 module=x/palomaconsensus queue-type-name=evm/eth-main/validators-balances validator=palomavaloper12c6hhdt9ly838pw6ftwctfdautr8rp288atqnf
Oct 24 20:27:35 cc1 palomad[2745431]: 8:27PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=379703560579887
Oct 24 20:27:36 cc1 palomad[2745431]: 8:27PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=379703560579887
Oct 24 20:27:38 cc1 palomad[2745431]: 8:27PM INF added message evidence. chain-reference-id=eth-main chain-type=evm message-id=350030 module=x/palomaconsensus queue-type-name=evm/eth-main/validators-balances validator=palomavaloper1v5jumcsm5k8eexjgzff72n73mxjrdcx6xu2e9x
Oct 24 20:27:38 cc1 palomad[2745431]: 8:27PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=560504217079887
Oct 24 20:27:39 cc1 palomad[2745431]: 8:27PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=560504217079887
Oct 24 20:27:41 cc1 palomad[2745431]: 8:27PM INF added message evidence. chain-reference-id=eth-main chain-type=evm message-id=350030 module=x/palomaconsensus queue-type-name=evm/eth-main/validators-balances validator=palomavaloper1qf7np0rp3qutvn08vc6qz0y7ffc02cncwdvdtd
Oct 24 20:27:41 cc1 palomad[2745431]: 8:27PM INF added message evidence. chain-reference-id=eth-main chain-type=evm message-id=350030 module=x/palomaconsensus queue-type-name=evm/eth-main/validators-balances validator=palomavaloper13uslh0y22ffnndyr3x30wqd8a6peqh25m8p743
Oct 24 20:27:41 cc1 palomad[2745431]: 8:27PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=700112818385852
Oct 24 20:27:42 cc1 palomad[2745431]: 8:27PM INF added message evidence. chain-reference-id=eth-main chain-type=evm message-id=350030 module=x/palomaconsensus queue-type-name=evm/eth-main/validators-balances validator=palomavaloper1wm4y8yhppxud6j5wvwr7fyynhh09tmv5fy845g
Oct 24 20:27:42 cc1 palomad[2745431]: 8:27PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=737193758269149
Oct 24 20:27:44 cc1 palomad[2745431]: 8:27PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=737193758269149
Oct 24 20:27:45 cc1 palomad[2745431]: 8:27PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=737193758269149
Oct 24 20:27:46 cc1 palomad[2745431]: 8:27PM INF added message evidence. chain-reference-id=eth-main chain-type=evm message-id=350030 module=x/palomaconsensus queue-type-name=evm/eth-main/validators-balances validator=palomavaloper1s0lankh33kprer2l22nank5rvsuh9ksafx3mze
Oct 24 20:27:46 cc1 palomad[2745431]: 8:27PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=865190075873055
Oct 24 20:27:48 cc1 palomad[2745431]: 8:27PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=865190075873055
Oct 24 20:27:50 cc1 palomad[2745431]: 8:27PM INF added message evidence. chain-reference-id=eth-main chain-type=evm message-id=350030 module=x/palomaconsensus queue-type-name=evm/eth-main/validators-balances validator=palomavaloper1tdw23fpnxh2uk3djtteh7eaydymrfgnaepkpkz
Oct 24 20:27:50 cc1 palomad[2745431]: 8:27PM INF added message evidence. chain-reference-id=eth-main chain-type=evm message-id=350030 module=x/palomaconsensus queue-type-name=evm/eth-main/validators-balances validator=palomavaloper1sz3zjcyq8cd3e2k9kx6d44s8wnavfsy2837nje
Oct 24 20:27:50 cc1 palomad[2745431]: 8:27PM INF added message evidence. chain-reference-id=eth-main chain-type=evm message-id=350030 module=x/palomaconsensus queue-type-name=evm/eth-main/validators-balances validator=palomavaloper108cvju0yxh4dhj7vyntr3yx59r7hf4n7kyzhu4
Oct 24 20:27:50 cc1 palomad[2745431]: 8:27PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=1052826268020914
Oct 24 20:27:51 cc1 palomad[2745431]: 8:27PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=1052826268020914
Oct 24 20:27:53 cc1 palomad[2745431]: 8:27PM INF added message evidence. chain-reference-id=eth-main chain-type=evm message-id=350030 module=x/palomaconsensus queue-type-name=evm/eth-main/validators-balances validator=palomavaloper1pxvc4hc5l4upvsj97qcruwwjaldl9nh9v2k3z8
Oct 24 20:27:53 cc1 palomad[2745431]: 8:27PM INF added message evidence. chain-reference-id=eth-main chain-type=evm message-id=350030 module=x/palomaconsensus queue-type-name=evm/eth-main/validators-balances validator=palomavaloper1z9fgzh7mzqgu33pdkxw0dqmqgm9l8exj6rl5wj
Oct 24 20:27:53 cc1 palomad[2745431]: 8:27PM INF added message evidence. chain-reference-id=eth-main chain-type=evm message-id=350030 module=x/palomaconsensus queue-type-name=evm/eth-main/validators-balances validator=palomavaloper1u5ujhurqtaxkvtfaqggtr5wu5qsmxszs9hrvxe
Oct 24 20:27:53 cc1 palomad[2745431]: 8:27PM INF added message evidence. chain-reference-id=eth-main chain-type=evm message-id=350030 module=x/palomaconsensus queue-type-name=evm/eth-main/validators-balances validator=palomavaloper1pyrugdyg58k4h788pptnzdvxsu5z66pwhu69gs
Oct 24 20:27:53 cc1 palomad[2745431]: 8:27PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={"dc9f5115bc70d34c55aa7a1917f04a73764393c2a66e4d81401a96eb76c758ca":"637500761718165","e953b8678138b429195979e95c1d1badcb8f5ddcadf599fe962370f7f6e21e33":"792824356058927"} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=1430325117777092
Oct 24 20:27:54 cc1 palomad[2745431]: 8:27PM INF added message evidence. chain-reference-id=eth-main chain-type=evm message-id=350030 module=x/palomaconsensus queue-type-name=evm/eth-main/validators-balances validator=palomavaloper1swa5kcf9cl5dx2ypx0c5r9e5qdfnzp9w0yj7uv
Oct 24 20:27:54 cc1 palomad[2745431]: 8:27PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={"dc9f5115bc70d34c55aa7a1917f04a73764393c2a66e4d81401a96eb76c758ca":"637500761718165","e953b8678138b429195979e95c1d1badcb8f5ddcadf599fe962370f7f6e21e33":"912838586458895"} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=1550339348177060
Oct 24 20:27:56 cc1 palomad[2745431]: 8:27PM INF added message evidence. chain-reference-id=eth-main chain-type=evm message-id=350030 module=x/palomaconsensus queue-type-name=evm/eth-main/validators-balances validator=palomavaloper1t5a905wfcv5kl0zn33ejnv9dkcjnerqy9mwutc
Oct 24 20:27:56 cc1 palomad[2745431]: 8:27PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={"dc9f5115bc70d34c55aa7a1917f04a73764393c2a66e4d81401a96eb76c758ca":"637500761718165","e953b8678138b429195979e95c1d1badcb8f5ddcadf599fe962370f7f6e21e33":"918121390061517"} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=1555622151779682
Oct 24 20:27:58 cc1 palomad[2745431]: 8:27PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={"dc9f5115bc70d34c55aa7a1917f04a73764393c2a66e4d81401a96eb76c758ca":"637500761718165","e953b8678138b429195979e95c1d1badcb8f5ddcadf599fe962370f7f6e21e33":"918121390061517"} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=1555622151779682
Oct 24 20:27:59 cc1 palomad[2745431]: 8:27PM INF added message evidence. chain-reference-id=eth-main chain-type=evm message-id=350030 module=x/palomaconsensus queue-type-name=evm/eth-main/validators-balances validator=palomavaloper1tsu8nthuspe4zlkejtj3v27rtq8qz7q6983zt2
Oct 24 20:27:59 cc1 palomad[2745431]: 8:27PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={"dc9f5115bc70d34c55aa7a1917f04a73764393c2a66e4d81401a96eb76c758ca":"637500761718165","e953b8678138b429195979e95c1d1badcb8f5ddcadf599fe962370f7f6e21e33":"985629872353235"} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=1623130634071400
Oct 24 20:28:01 cc1 palomad[2745431]: 8:28PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={"dc9f5115bc70d34c55aa7a1917f04a73764393c2a66e4d81401a96eb76c758ca":"637500761718165","e953b8678138b429195979e95c1d1badcb8f5ddcadf599fe962370f7f6e21e33":"985629872353235"} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=1623130634071400
Oct 24 20:28:02 cc1 palomad[2745431]: 8:28PM INF added message evidence. chain-reference-id=eth-main chain-type=evm message-id=350030 module=x/palomaconsensus queue-type-name=evm/eth-main/validators-balances validator=palomavaloper10y227j9d09pckexy32v2gckerj9a0kcewgf7xy
Oct 24 20:28:02 cc1 palomad[2745431]: 8:28PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={"dc9f5115bc70d34c55aa7a1917f04a73764393c2a66e4d81401a96eb76c758ca":"637500761718165","e953b8678138b429195979e95c1d1badcb8f5ddcadf599fe962370f7f6e21e33":"1053007719853992"} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=1690508481572157
Oct 24 20:28:21 cc1 palomad[2745431]: 8:28PM INF added message evidence. chain-reference-id=eth-main chain-type=evm message-id=350030 module=x/palomaconsensus queue-type-name=evm/eth-main/validators-balances validator=palomavaloper109l8jp4zx9g8udetnh5qwgsze82meeq5yq06ta
Oct 24 20:28:21 cc1 palomad[2745431]: 8:28PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={"dc9f5115bc70d34c55aa7a1917f04a73764393c2a66e4d81401a96eb76c758ca":"637500761718165","e953b8678138b429195979e95c1d1badcb8f5ddcadf599fe962370f7f6e21e33":"1172876672276317"} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=1810377433994482
Oct 24 20:28:23 cc1 palomad[2745431]: 8:28PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={"dc9f5115bc70d34c55aa7a1917f04a73764393c2a66e4d81401a96eb76c758ca":"637500761718165","e953b8678138b429195979e95c1d1badcb8f5ddcadf599fe962370f7f6e21e33":"1172876672276317"} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=1810377433994482
Oct 24 20:28:24 cc1 palomad[2745431]: 8:28PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={"dc9f5115bc70d34c55aa7a1917f04a73764393c2a66e4d81401a96eb76c758ca":"637500761718165","e953b8678138b429195979e95c1d1badcb8f5ddcadf599fe962370f7f6e21e33":"1172876672276317"} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=1810377433994482
Oct 24 20:28:26 cc1 palomad[2745431]: 8:28PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={"dc9f5115bc70d34c55aa7a1917f04a73764393c2a66e4d81401a96eb76c758ca":"637500761718165","e953b8678138b429195979e95c1d1badcb8f5ddcadf599fe962370f7f6e21e33":"1172876672276317"} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=1810377433994482
Oct 24 20:28:27 cc1 palomad[2745431]: 8:28PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={"dc9f5115bc70d34c55aa7a1917f04a73764393c2a66e4d81401a96eb76c758ca":"637500761718165","e953b8678138b429195979e95c1d1badcb8f5ddcadf599fe962370f7f6e21e33":"1172876672276317"} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=1810377433994482
Oct 24 20:28:29 cc1 palomad[2745431]: 8:28PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={"dc9f5115bc70d34c55aa7a1917f04a73764393c2a66e4d81401a96eb76c758ca":"637500761718165","e953b8678138b429195979e95c1d1badcb8f5ddcadf599fe962370f7f6e21e33":"1172876672276317"} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=1810377433994482
Oct 24 20:36:51 cc1 palomad[2745431]: 8:36PM INF jailing a validator jail-time=2024-10-24T20:37:49Z module=x/valset reason="No evidence supplied for contentious message 350030" val-addr=palomavaloper1qxl8xy0rjl99zhaxgu4ffuegup9mh6qxdmcwkg
Oct 24 20:36:51 cc1 palomad[2745431]: 8:36PM INF jailing a validator jail-time=2024-10-24T20:37:49Z module=x/valset reason="No evidence supplied for contentious message 350030" val-addr=palomavaloper1c8uxkkz00qn97wsp9um8y3a8wlmrv3c546jw4q
Oct 24 20:36:51 cc1 palomad[2745431]: 8:36PM INF Removed message from queue block_height=25891550 module=server msg={"bytesToSign":"o1Pnaq+DEtinxt97XiWwf5B4W7QMLOUZj2xEQeq3Xxg=","errorData":null,"gasEstimate":"0","id":"350030","msg":{"@type":"/palomachain.paloma.evm.ValidatorBalancesAttestation","assignee":"","fromBlockTime":"2024-10-24T20:27:27.091967665Z","hexAddresses":["0x1177806cD88b0BC0eA363bEd3C64f8314361b162","0x0DAaBB4FF60423Eb1F14cC6731394e098ad51bcb","0xC1009C72cC2519B0DD595C3E500C1293A862Aa6A","0x35c1cC4E3CD4624Ac177c167D2427e9C95336535","0x377D23948D41579f2c3cA40308e3bdd53f6dA7B2","0x1799D68d773f9E03Baca79cd1d836Ba026B4742A","0x43E218f96A567DC26C6e65fCabF1fDE26Af69444","0x443266026738061972012e62D5Ecd9D98da8B6F4","0x4a89f96fdff3C161937CFBC3e22d2E325612aaEc","0xa228292447064D5818BbC80b577C91f5212F9355","0xDEea5b069208e0eE37b630A3e7672FC50e8fE24a","0xAAc74d38c82c367b3dA7482E3Aecf6DD7A512eF2","0xeB784b37365C302C97C9Ec8cB0933cE344e6dE42","0x2d91F6C502289eA4d7F9253C45788109377b95bD","0x732fBb6018F2cB5b844055C9EB567447a3132833","0x19f5911E4cA69E30449aD6BB71De341F01F118BB","0x1ad90dB98da083E117D5F62a1673fC0f3A5930ca","0x63f55bc560E981d53E1f5bb3643e3a96D26fc635","0x6dc59EE4bdFa2C791229004f29b08F783491a934","0x14Aa448C2C918C4427c5671028b63BC17f6132d5","0xEe21301aF1d9562B5cBEdf520077Ea0a9bC9d535","0x7Bd1A3270570b65F264895C2F86E631fDc497545"],"valAddresses":["palomavaloper1qxl8xy0rjl99zhaxgu4ffuegup9mh6qxdmcwkg","palomavaloper1qf7np0rp3qutvn08vc6qz0y7ffc02cncwdvdtd","palomavaloper1pyrugdyg58k4h788pptnzdvxsu5z66pwhu69gs","palomavaloper1pxvc4hc5l4upvsj97qcruwwjaldl9nh9v2k3z8","palomavaloper1z9fgzh7mzqgu33pdkxw0dqmqgm9l8exj6rl5wj","palomavaloper12c6hhdt9ly838pw6ftwctfdautr8rp288atqnf","palomavaloper1tdw23fpnxh2uk3djtteh7eaydymrfgnaepkpkz","palomavaloper1tsu8nthuspe4zlkejtj3v27rtq8qz7q6983zt2","palomavaloper1t5a905wfcv5kl0zn33ejnv9dkcjnerqy9mwutc","palomavaloper1v5jumcsm5k8eexjgzff72n73mxjrdcx6xu2e9x","palomavaloper1wm4y8yhppxud6j5wvwr7fyynhh09tmv5fy845g","palomavaloper10y227j9d09pckexy32v2gckerj9a0kcewgf7xy","palomavaloper109l8jp4zx9g8udetnh5qwgsze82meeq5yq06ta","palomavaloper108cvju0yxh4dhj7vyntr3yx59r7hf4n7kyzhu4","palomavaloper1sz3zjcyq8cd3e2k9kx6d44s8wnavfsy2837nje","palomavaloper1swa5kcf9cl5dx2ypx0c5r9e5qdfnzp9w0yj7uv","palomavaloper1s0lankh33kprer2l22nank5rvsuh9ksafx3mze","palomavaloper13uslh0y22ffnndyr3x30wqd8a6peqh25m8p743","palomavaloper15gvyk43x406v7kcd4rff5qfutqmcnpj3w9wpnm","palomavaloper1hsmnzawmnauy9qxkxzjqm2mpd20pmfaf5uj30x","palomavaloper1c8uxkkz00qn97wsp9um8y3a8wlmrv3c546jw4q","palomavaloper1u5ujhurqtaxkvtfaqggtr5wu5qsmxszs9hrvxe"]},"nonce":"AAAAAAAFV04=","publicAccessData":"AQ==","signData":[],"valsetID":"0"} msg-id=350030

[cosmoscats]

go version go1.21.0 linux/amd64
maybe this can be the issue?
do i need to update go version on server?

[maharifu]

The jail reason is here:
Oct 24 20:36:51 cc1 palomad[2745431]: 8:36PM INF jailing a validator jail-time=2024-10-24T20:37:49Z module=x/valset reason="No evidence supplied for contentious message 350030" val-addr=palomavaloper1qxl8xy0rjl99zhaxgu4ffuegup9mh6qxdmcwkg

One of the issues here is some pigeons are having RPC issues and returning incomplete balances information, which makes consensus harder to achieve. We can see that here distribution={"dc9f5115bc70d34c55aa7a1917f04a73764393c2a66e4d81401a96eb76c758ca":"637500761718165","e953b8678138b429195979e95c1d1badcb8f5ddcadf599fe962370f7f6e21e33":"918121390061517"}. This means we have two different values for the same request, which in this request means some pigeons failed to get balances for all validators.

However, on the logs, we can see

Oct 24 20:26:50 mainnet-validator palomad[1574145]: {"level":"error","module":"server","module":"x/paloma","msg.args.chain-reference-id":"eth-main","msg.args.error":"failed to broadcast tx: timed out after: 60000000000; timed out after waiting for tx to get included in the block","component":"pigeon-status-update","status":"error attesting messages","sender":"palomavaloper1qxl8xy0rjl99zhaxgu4ffuegup9mh6qxdmcwkg","time":"2024-10-24T20:26:50Z","message":"error attesting messages"}

So this is an issue of failing to broadcast tx to paloma, a duplicate of VolumeFi#2259

@cosmoscats I don't think the go version is the issue but you should still update it, especially if you are compiling your own binary. Are you using pigeon operator keys (like detailed here)?

[cosmoscats]

@maharifu - i will try to update go version.

When we starting to use pigeon operator keys - we go straight to the jail much faster, like in 1 hour or so (situation is same like Nodes Guru described in discord - "out of gas"
We have disabled it for now.

[cosmoscats]

still no info about? VolumeFi#2259

[cosmoscats]

@maharifu i have installed new version of go

go version
go version go1.22.8 linux/amd64

and i have recompiled paloma & pigeon with that new version and made restart.

palomad version
v2.3.2
paloma@cc1:~/pigeon$ pigeon version
App version: v2.3.1
Build commit hash: 5f6f4bcaa645e0f9d8530b24cd1a4cd74b79e857

[taariq]

@maharifu - i will try to update go version.

When we starting to use pigeon operator keys - we go straight to the jail much faster, like in 1 hour or so (situation is same like Nodes Guru described in discord - "out of gas" We have disabled it for now.

This was resolved as user-issue. Please check this comment here: #1312 (comment)

[cosmoscats]

Looks like the same problem still occurs after go update.
Still going to jail.

[cosmoscats]

node jailing every day few times in a day

[taariq]

We're taking the position that this MAY be hardware related as no other validators are dealing with this issue. Are you able to switch hosts?

[cosmoscats]

We're taking the position that this MAY be hardware related as no other validators are dealing with this issue. Are you able to switch hosts?

Yes, i can switch to another hardware server tomorrow.

[cosmoscats]

Today i have moved my Paloma node to another server:

CPU: AMD Ryzen 9 5900X
MB: B550-A PRO (MS-7C56) v: 2.0 (Socket AM4)
Video: GeForce GT 710
RAM: Corsair VENGEANCE® RGB PRO 64GB (2 x 32GB) DDR4 DRAM 3600MHz C18 CMW64GX4M2D3600C18
PSU: Corsair Core GM-650 80 Plus Gold

I will let you know how it goes.

[cosmoscats]

so, after 24h jailed again on new server.

i will include logs, can you check please?

Paloma-Jail-2-11-2024.txt

[cosmoscats]

and again reason: No evidence supplied for contentious message 362591

[cosmoscats]

And again jail, message - 362963

Also please check this jail and let me know if it is related to same problesm as before?

Is it again the same " issue of failing to broadcast tx to paloma" ?
Paloma-Jail-3-11-Night.txt

I need to understand that to dismiss the server problem.
Because we're using a completely different hardware server right now.

[taariq]

@cosmoscats this was a gnosis broadcast problem

Nov 02 22:10:58 server-31-24-56-54 palomad[2946843]: 10:10PM INF added message evidence. chain-reference-id=gnosis-main chain-type=evm message-id=362963 module=x/palomaconsensus queue-type-name=evm/gnosis-main/validators-balances validator=palomavaloper1v5jumcsm5k8eexjgzff72n73mxjrdcx6xu2e9x

Please change your Gnosis endpoint and try again to unjail.

[cosmoscats]

Please check our another jail. Is it again the same " issue of failing to broadcast tx to paloma" ?

Paloma-Jail-4-11-Night.txt

[taariq]

Please check our another jail. Is it again the same " issue of failing to broadcast tx to paloma" ?

Paloma-Jail-4-11-Night.txt

this is now Arbitrum
Nov 04 02:34:43 server-31-24-56-54 palomad[2946843]: 2:34AM INF added message evidence. chain-reference-id=arbitrum-main chain-type=evm message-id=364542 module=x/palomaconsensus queue-type-name=evm/arbitrum-main/validators-balances validator=palomavaloper1tsu8nthuspe4zlkejtj3v27rtq8qz7q6983zt2.

Will you update your ARB RPC endpoint and unjail again?

[cosmoscats]

according to the logs there were no problems with ARB RPC.
Changed from Alchemy to Ankr. I'll check it out

[taariq]

What does palomad q valset get-validator-jail-reason palomavaloper1qxl8xy0rjl99zhaxgu4ffuegup9mh6qxdmcwkg give you?

[cosmoscats]

If I remember correctly from last time:
"No evidence supplied for contentious message 364542"

[taariq]

If I remember correctly from last time: "No evidence supplied for contentious message 364542"

So that's the Arbitrum endpoint. I would test another RPC provider. Yes.

[cosmoscats]

also 4 your info:
i cant change gas-prices in config fewer that 0.01ugrain for paloma.
If i will change this value to 0.001 or to 0.009 I'll instantly get a bunch of errors in my pigeon logs.

[taariq]

also 4 your info: i cant change gas-prices in config fewer that 0.01ugrain for paloma. If i will change this value to 0.001 or to 0.009 I'll instantly get a bunch of errors in my pigeon logs.

Will you open up a new ticket with that and the errors please?

[taariq]

We made no changes in the protocol for this so this should not be an issue, but let's see the errors.

[cosmoscats]

also 4 your info: i cant change gas-prices in config fewer that 0.01ugrain for paloma. If i will change this value to 0.001 or to 0.009 I'll instantly get a bunch of errors in my pigeon logs.

Will you open up a new ticket with that and the errors please?

#1315

[cosmoscats]

another jail consensus not achieved
reason: No evidence supplied for contentious message 365041

[cosmoscats]

another jail
reason: No evidence supplied for contentious message 365797
its again related to ETH RPC (?). but i have changed it to Ankr from Alchemy.

paloma@server-31-24-56-54:~/.pigeon$ sudo journalctl -u pigeond | grep '365797'
Nov 05 01:11:48 server-31-24-56-54 pigeon[3301754]: time="2024-11-05T01:11:48Z" level=info msg="attesting 1 messages" action=attest message-ids="[365797]" messages-to-attest="[365797]" queue-name=evm/eth-main/reference-block x-correlation-id=cskmrv5j182pe4jdat2g
Nov 05 01:13:00 server-31-24-56-54 pigeon[3301754]: time="2024-11-05T01:13:00Z" level=error msg="error attesting messages" action=attest error="rpc error: code = Unknown desc = rpc error: code = Unknown desc = failed to execute message; message index: 0: validator palomavaloper1qxl8xy0rjl99zhaxgu4ffuegup9mh6qxdmcwkg cannot be a pigeon: validator is jailed [cosmos/[email protected]/baseapp/baseapp.go:1023] with gas used: '37828': unknown request" message-ids="[365797]" messages-to-attest="[365797]" queue-name=evm/eth-main/reference-block x-correlation-id=cskmrv5j182pe4jdat2g
Nov 05 01:14:00 server-31-24-56-54 pigeon[3301754]: time="2024-11-05T01:14:00Z" level=error msg="failed to send Paloma status update" action=attest error="failed to broadcast tx: timed out after: 60000000000; timed out after waiting for tx to get included in the block" message-ids="[365797]" messages-to-attest="[365797]" queue-name=evm/eth-main/reference-block x-correlation-id=cskmrv5j182pe4jdat2g

[cosmoscats]

another jail
reason: No evidence supplied for contentious message 368578

its again related to ETH RPC (?)

Nov 07 03:51:06 server-31-24-56-54 pigeon[3301754]: time="2024-11-07T03:51:06Z" level=info msg="attesting 1 messages" action=attest message-ids="[368578]" messages-to-attest="[368578]" queue-name=evm/eth-main/validators-balances x-correlation-id=csm3ialj182pe4jrijk0
Nov 07 03:54:12 server-31-24-56-54 pigeon[3301754]: time="2024-11-07T03:54:12Z" level=error msg="error attesting messages" action=attest error="rpc error: code = Unknown desc = rpc error: code = Unknown desc = failed to execute message; message index: 0: validator palomavaloper1qxl8xy0rjl99zhaxgu4ffuegup9mh6qxdmcwkg cannot be a pigeon: validator is jailed [cosmos/[email protected]/baseapp/baseapp.go:1023] with gas used: '42094': unknown request" message-ids="[368578]" messages-to-attest="[368578]" queue-name=evm/eth-main/validators-balances x-correlation-id=csm3ialj182pe4jrijk0
Nov 07 03:56:12 server-31-24-56-54 pigeon[3301754]: time="2024-11-07T03:56:12Z" level=error msg="failed to send Paloma status update" action=attest error="failed to broadcast tx: timed out after: 60000000000; timed out after waiting for tx to get included in the block" message-ids="[368578]" messages-to-attest="[368578]" queue-name=evm/eth-main/validators-balances x-correlation-id=csm3ialj182pe4jrijk0

[taariq]

Yeah. the message didn't get included in a block and you've updated the gas issue. This is a pigeon issue. We will need some time to get to this. we're blocked for a bit with Paloma L2 work.

[cosmoscats]

jail 6 times in a day.

today i have seen some strange logs.

can you look at it ?

Nov 15 14:21:37 server-31-24-56-54 pigeon[302394]: time="2024-11-15T14:21:37Z" level=error msg="failed to send paloma status update" chain-reference-id=bnb-main error="failed to broadcast tx: timed out after: 60000000000; timed out after waiting for tx to get included in the block" message-id=381788 message-type="*types.Message_SubmitLogicCall" msg-bytes-to-sign="[164 230 4 64 247 15 44 175 246 180 14 69 236 109 112 108 163 66 249 128 179 161 162 12 88 118 186 213 146 62 83 224]" msg-eth-sender=0xEe21301aF1d9562B5cBEdf520077Ea0a9bC9d535 msg-id=381788 msg-msg="turnstoneID:"22497131\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000" chainReferenceID:"bnb-main" submitLogicCall:<hexContractAddress:"0xf5A21C45815b2801B00FdB5E7047BFDE97152040" payload:"F-\000e\000\000\000\000\000\000\000\000\000\000\000\000\305\360\367\266gd\366\354\214\215\377{\246\203\020\"\225\341d\t\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\333\325\266/+\311\272\345\024\315\213k\374Z\006\273K\254\003Ty\3612m\215\221\264A\346A\262/" deadline:1731680290 senderAddress:"\333\325\266/+\311\272\345\024\315\213k\374Z\006\273K\254\003Ty\3612m\215\221\264A\346A\262/" contractAddress:"\333\325\266/+\311\272\345\024\315\213k\374Z\006\273K\254\003Ty\3612m\215\221\264A\346A\262/" executionRequirements:<> > assignee:"palomavaloper1c8uxkkz00qn97wsp9um8y3a8wlmrv3c546jw4q" assigned_at_block_height:"27063750" assigneeRemoteAddress:"0xEe21301aF1d9562B5cBEdf520077Ea0a9bC9d535" " msg-nonce="[0 0 0 0 0 5 211 92]" msg-public-access-data="[]" queue-name=evm/bnb-main/evm-turnstone-message queue-type-name=evm/bnb-main/evm-turnstone-message x-correlation-id=csrlfhdj182mmkhl409g
Nov 15 14:21:37 server-31-24-56-54 pigeon[302394]: time="2024-11-15T14:21:37Z" level=warning msg="Failed to estimate messages" error="failed to find assignee eth address: assignee's eth address not found"
Nov 15 14:21:37 server-31-24-56-54 pigeon[302394]: time="2024-11-15T14:21:37Z" level=info msg="No valid estimates, skipping" action=estimate message-ids="[381788 381789 381790 381791 381799 381800 381801]" messages-to-estimate="[381788 381789 381790 381791 381799 381800 381801]" queue-name=evm/bnb-main/evm-turnstone-message

Jail reason:
reason: No evidence supplied for contentious message 381796

[cosmoscats]

Hello, when i will check this issue?
Our node keep jailing every day

[taariq]

@cosmoscats as this is a one-machine issue, and not validator-wide, we will need to check your setup. We'll attempt to get to that, but are a little behind. Please bear with us.

[cosmoscats]

@cosmoscats as this is a one-machine issue, and not validator-wide, we will need to check your setup. We'll attempt to get to that, but are a little behind. Please bear with us.

Recovering from prison and finding a solution takes so long that holding a node on our part may not be worthwhile.
We lose both time and our own money buying different paid rpc's with no results in two months.

[taariq]

We agree @cosmoscats, but don't give up yet. We need you on the network and will jump into this issue. Please allow us to follow-up on Monday.

[taariq]

Hey there @cosmoscats. We are still here. We'll follow-up with you by Friday. In the Interim, will you please open up a Discord Support ticket for us to bring the team to help check your setup?

[cosmoscats]

For the last four months, our node has been going to jail every day.
We created a ticket on github, we created a ticket on Discord.
Is there any hope that you can help us or not?

[taariq]

@byte-bandit will you take a look here please?

[taariq]

@cosmoscats we'll follow-up here in 24-hours after the next release.

[byte-bandit]

Hey @cosmoscats ,

the message itself is kept fairly ambiguous unfortunately, all we know is that your validator failed to provide evidence for one or more messages and got jailed due to being naughty. It's not an easy issue to debug, but historically we've seen two likely culprits for this behaviour:

• your RPC endpoint: either there's some network communication issue with the provider that lets your calls time out, or the provider node is too slow hasn't caught up to the block in question yet (resulting in a 404). In both cases, you don't get any data to verify and can therefore not participate in consensus.
• your hardware is too slow: not sure if you're running this on a VM or bare metal, but it's possible that you're running out of disk space or your VCPU is simply not beefy enough to run both Paloma & Pigeon fast enough

My suggestion would be:

a) verify your deployment environment, try upgrading to a higher tier for a while and see if the issue persists
b) check the message IDs that your node is being jailed for to find out whether this issue is present on more than one remote chain. Try moving to a different RPC provider as well and see if the issue persist.

[cosmoscats]

Hey @byte-bandit

1. We are using two own hardware servers with following specs:

machine a. ubuntu 22.04

i9-14900K Intel Core
Z790 Gigabyte Aorus Master X
Corsair64GB DDR5 Vengeance (2x32GB) 6600 CL32
4TB Kingston KC3000
Samsung 990 Pro (system)
Corsair RMx Series RM850x 850 W, 80 PLUS Gold
Deepcool LS720S Zero Dark 360mm AiO Liquid CPU Cooler
Thermal Grizzly Intel CPU Contract Frame

machine b. ubuntu 24.04

CPU: AMD Ryzen 9 5900X
MB: B550-A PRO (MS-7C56) v: 2.0 (Socket AM4)
Video: GeForce GT 710
RAM: Corsair VENGEANCE® RGB PRO 64GB (2 x 32GB) DDR4 DRAM 3600MHz C18 CMW64GX4M2D3600C18
PSU: Corsair Core GM-650 80 Plus Gold
NVME Samsung SSD 970 EVO Plus 2TB
SSD: Samsung SSD 970 EVO Plus 2TB

Since it was initially assumed that the problem might be in our hardware, we tried running Paloma on these two different machines - the result is the same = JAIL all the time.

2. I am constantly changing my RPC to different providers. i allready have tried to use:
   Nodies, Quicknode (I used the $10 paid version here for 4 RPC-s),
   Alchemy (I used the $50 paid version here),
   ANKR (I used the $50 paid version here)., Nodereal, Infura, Chainnodes, Blastapi, Drpc, Blockdaemon

maybe you have some specific providers that I can try to use to solve this problem - let me know.

All the latest Jail reason indicated a problem with the ETH RPC, but I kept changing them - the result is the same, after a while I end up back in jail.

Probably you will have a look at my config and can advise something or have any other ideas

[byte-bandit]

Okay, a couple of things to unpack here.

1. Your specs are more than beefy enough to run both Paloma & Pigeon, I think we can rule out weak hardware
2. Given your strategy switching RPCs, we can rule out one faulty endpoint provider

It does happen ever so often when endpoints are not yet fully caught up, but it's weird that it's happening every day, and only for you.

So, a few more questions:

1. Are you running the release binaries, or are you compiling from source?
2. What is the output of go version for the machines?
3. You mentioned two servers. Are you running Paloma on one machine, and Pigeon on another? Or are both binaries running on the same machine?
4. What else are you running on your hardware?
5. What specifically were the machines running during the time you failed to attest? Do you see load spikes on your machines during this time at all?

[cosmoscats]

Okay, a couple of things to unpack here.

1. Your specs are more than beefy enough to run both Paloma & Pigeon, I think we can rule out weak hardware
2. Given your strategy switching RPCs, we can rule out one faulty endpoint provider

It does happen ever so often when endpoints are not yet fully caught up, but it's weird that it's happening every day, and only for you.

So, a few more questions:

1. Are you running the release binaries, or are you compiling from source?
2. What is the output of go version for the machines?
3. You mentioned two servers. Are you running Paloma on one machine, and Pigeon on another? Or are both binaries running on the same machine?
4. What else are you running on your hardware?
5. What specifically were the machines running during the time you failed to attest? Do you see load spikes on your machines during this time at all?

1. i am compiling from source.
2. go version go1.22.5 linux/amd64 on current machine "b"
3. we are running both paloma and pigeon on one machine at the moment
4. we use different services on the machines.
   on machine a, we use cosmos nodes.
   on machine b we run mostly directadmin and a couple of cloud services like nextcloud, cryptdrive, focalboard.
5. we have a configured grafana dashboard, we can see our servers and their stats, and believe me, there should be no weaknesses there. No load spikes at this time on either machine, this has been previously checked by us.

[byte-bandit]

Thanks for the intel.

First I suggest you update your installation of go to 1.23 on both your build agent and the runner.

Secondly, can you share the the commands you're using for building? Are you adding any customizations to the code?

One quick and easy win: can you try running the prebuilt release binaries and check whether the issue persists?

[cosmoscats]

Okey. Updated go on server

go version
go version go1.23.5 linux/amd64

Secondly.

i was using the following commands for building:

sudo systemctl stop palomad
cd $HOME && sudo rm -R paloma
git clone https://github.com/palomachain/paloma.git && cd paloma
git checkout v2.4.3 && palomad version
make build
sudo mv ./build/palomad /usr/local/bin/palomad
palomad version
sudo systemctl restart palomad && sudo journalctl -u palomad -f

now with your suggestion i have run prebuild release via commands (i hope this is correct way to do so):

sudo systemctl stop palomad
sudo rm -rf /usr/local/bin/palomad
wget https://github.com/palomachain/paloma/releases/download/v2.4.3/paloma_Linux_x86_64.tar.gz
tar -xzf paloma_Linux_x86_64.tar.gz
sudo mv palomad /usr/local/bin/palomad
sudo chmod +x /usr/local/bin/palomad
palomad version
sudo systemctl restart palomad && sudo journalctl -u palomad -f

i will let you know how it goes.

[cosmoscats]

Jailed two times in few hours.

reason: No evidence supplied for contentious message 494968

i have checked, seems to be related with eth RPC.

i am using now paid RPC from quicknode with their grow plan for my ETH RPC

[byte-bandit]

@cosmoscats

Thanks for the update, I hope the switch to paid quicknode providers will solve the issue. It's what we've been coasting on for the past couple of months without seeing many issues.

Keep us posted.