failed to decode blob sidecars: xx

[Egozit]

I trying to set up blob-archiver using docker-compose up and connecting it to my beacon node (with Prysm)
I am getting this error, no idea how to fix it:

archiver_1        | t=2024-03-16T13:12:54+0000 lvl=info msg="Initializing Archiver Service"
archiver_1        | {"level":"trace","service":"client","impl":"http","time":"2024-03-16T13:12:54Z","message":"Context done; closing connection"}
archiver_1        | t=2024-03-16T13:12:54+0000 lvl=info msg="starting metrics server"       addr=0.0.0.0 port=7300
archiver_1        | t=2024-03-16T13:12:54+0000 lvl=info msg="started metrics server"        addr=[::]:7300
archiver_1        | t=2024-03-16T13:12:54+0000 lvl=info msg="Archiver API server started"   address=[::]:8000
archiver_1        | {"level":"trace","service":"client","impl":"http","id":"1de5098c","address":"http://MY_BEACON_NODE:PORT","endpoint":"/eth/v1/beacon/headers/head","time":"2024-03-16T13:12:54Z","message":"GET request"}
archiver_1        | {"level":"trace","service":"client","impl":"http","id":"3ac0b1c","address":"http://MY_BEACON_NODE:PORT","endpoint":"/eth/v1/beacon/blob_sidecars/0xb02f2ca9c5892a4e6a31bea43be747f8f1232a419c012b81d306d86e4a92c207","time":"2024-03-16T13:12:54Z","message":"GET request"}
blob-archiver_create-buckets_1 exited with code 0
archiver_1        | t=2024-03-16T13:12:54+0000 lvl=eror msg="failed to fetch blob sidecars" err="failed to decode blob sidecars: xx"
archiver_1        | {"level":"trace","service":"client","impl":"http","id":"2a5b3e44","address":"http://MY_BEACON_NODE:PORT","endpoint":"/eth/v1/beacon/headers/head","time":"2024-03-16T13:12:55Z","message":"GET request"}
archiver_1        | {"level":"trace","service":"client","impl":"http","id":"11b6f22f","address":"http://MY_BEACON_NODE:PORT","endpoint":"/eth/v1/beacon/blob_sidecars/0xb02f2ca9c5892a4e6a31bea43be747f8f1232a419c012b81d306d86e4a92c207","time":"2024-03-16T13:12:55Z","message":"GET request"}
archiver_1        | t=2024-03-16T13:12:55+0000 lvl=eror msg="failed to fetch blob sidecars" err="failed to decode blob sidecars: xx"
archiver_1        | {"level":"trace","service":"client","impl":"http","id":"4a71ead8","address":"http://MY_BEACON_NODE:PORT","endpoint":"/eth/v1/beacon/headers/head","time":"2024-03-16T13:12:58Z","message":"GET request"}
archiver_1        | {"level":"trace","service":"client","impl":"http","id":"498b6717","address":"http://MY_BEACON_NODE:PORT","endpoint":"/eth/v1/beacon/blob_sidecars/0xb02f2ca9c5892a4e6a31bea43be747f8f1232a419c012b81d306d86e4a92c207","time":"2024-03-16T13:12:58Z","message":"GET request"}
archiver_1        | t=2024-03-16T13:12:58+0000 lvl=eror msg="failed to fetch blob sidecars" err="failed to decode blob sidecars: xx"
archiver_1        | t=2024-03-16T13:12:58+0000 lvl=eror msg="failed to seed archiver with initial block" err="operation failed permanently after 3 attempts: failed to decode blob sidecars: xx"
archiver_1        | t=2024-03-16T13:12:58+0000 lvl=crit msg="Application failed"            message="failed to start: operation failed permanently after 3 attempts: failed to decode blob sidecars: xx"
blob-archiver_archiver_1 exited with code 1

I've checked my beacon api endpoint, it returns the data and blob successfully. Any advise?

[wbnns]

@Egozit

Thanks for taking the time to open this issue! We've added this to our backlog, pending to be reviewed.

[danyalprout]

@Egozit just taken a look at this and this code in the library which does the ssz deserialzation is returning an error.

I tested this on Prysm and Lighthouse and it only seems to be an issue for Prysm. I'll open an issue on the library to confirm. However this PR lets you specify some env vars to enforce json encoding instead of ssz which works for both consensus clients. You can enable this by setting the following env vars:

BLOB_ARCHIVER_L1_BEACON_CLIENT_ENFORCE_JSON: true
BLOB_API_L1_BEACON_CLIENT_ENFORCE_JSON: true

[danyalprout]

Going to close this issue, as there is a workaround for this (using JSON via the env vars). I'll follow up on the failed SSZ deserialization in the linked issue.