Zlib error getting tileJSON using Google Cloud Storage

[highway900]

I have put together a POC for running with a google cloud function (typescript) by using the AWS lambda function as a reference. I have managed to get tiles using QGIS using a mvt url pattern.

But when I try to get the tileJSON I run into a zlib error

incorrect header check
errno: -3
code: 'Z_DATA_ERROR'

This seems to imply that the JSON data is not gzipped (or the wrong size? maybe different compression?). If I run the pmtiles file through the viewer I can read meta data in from the header offset range. The internal compression: 2 seems to be correct.
The problem could lie in the storage request? But I do get usable data back from the range request (tiles and some header info)

Here is a working express server that replicates the error. Change the bucketName

import express, { Request, Response, Express } from 'express';
import {
    PMTiles,
    Source,
    RangeResponse,
    ResolvedValueCache,
    TileType,
    Header,
    Compression,
} from 'pmtiles';
import zlib from 'node:zlib';
import { Storage } from '@google-cloud/storage';

const storage = new Storage();

class GSSource implements Source {
    archive_name: string;

    constructor(archive_name: string) {
        this.archive_name = archive_name;
    }

    getKey() {
        return this.archive_name;
    }

    async getBytes(offset: number, length: number): Promise<RangeResponse> {
        const bucketName = "pmtiles-bucket";
        const [buffer] = await storage.bucket(bucketName).file(this.archive_name).download(
            {
                start: offset,
                end: (offset + length - 1),
            }
        );

        return {
            data: buffer.buffer,
        };
    }
    // ...
}

async function nativeDecompress(
    buf: ArrayBuffer,
    compression: Compression
): Promise<ArrayBuffer> {
    if (compression === Compression.None || compression === Compression.Unknown) {
        return buf;
    } else if (compression === Compression.Gzip) {
        return zlib.gunzipSync(buf);
    } else {
        throw Error("Compression method not supported");
    }
}

const tileJSON = (
    header: Header,
    metadata: any,
    hostname: string,
    tileset_name: string
) => {
    console.log("inside tileJSON");
    console.log(" tileJSON header ", header);
    console.log(" tileJSON meta ", metadata);
    let ext = "";
    if (header.tileType === TileType.Mvt) {
        ext = ".mvt";
    } else if (header.tileType === TileType.Png) {
        ext = ".png";
    } else if (header.tileType === TileType.Jpeg) {
        ext = ".jpg";
    } else if (header.tileType === TileType.Webp) {
        ext = ".webp";
    } else if (header.tileType === TileType.Avif) {
        ext = ".avif";
    }

    return {
        tilejson: "3.0.0",
        scheme: "xyz",
        tiles: ["https://" + hostname + "/" + tileset_name + "/{z}/{x}/{y}" + ext],
        vector_layers: metadata.vector_layers,
        attribution: metadata.attribution,
        description: metadata.description,
        name: metadata.name,
        version: metadata.version,
        bounds: [header.minLon, header.minLat, header.maxLon, header.maxLat],
        center: [header.centerLon, header.centerLat, header.centerZoom],
        minzoom: header.minZoom,
        maxzoom: header.maxZoom,
    };
};

const pmtiles_path = (name: string, setting?: string): string => {
    if (setting) {
        return setting.replaceAll("{name}", name);
    }
    return name + ".pmtiles";
};

const TILE =
    /^\/(?<NAME>[0-9a-zA-Z\/!\-_\.\*\'\(\)]+)\/(?<Z>\d+)\/(?<X>\d+)\/(?<Y>\d+).(?<EXT>[a-z]+)$/;

const TILESET = /^\/(?<NAME>[0-9a-zA-Z\/!\-_\.\*\'\(\)]+).json$/;

const tile_path = (
    path: string
): {
    ok: boolean;
    name: string;
    tile?: [number, number, number];
    ext: string;
} => {
    const tile_match = path.match(TILE);

    if (tile_match) {
        const g = tile_match.groups!;
        return { ok: true, name: g.NAME, tile: [+g.Z, +g.X, +g.Y], ext: g.EXT };
    }

    const tileset_match = path.match(TILESET);

    console.log(`${tileset_match}, ${path}`);

    if (tileset_match) {
        const g = tileset_match.groups!;
        return { ok: true, name: g.NAME, ext: "json" };
    }

    return { ok: false, name: "", tile: [0, 0, 0], ext: "" };
};

const app: Express = express();
const port = 3000;

app.get('/tiles/:tileset', async (req: Request, res: Response) => {
    const baseURL = req.protocol + '://' + req.headers.host + '/';

    console.log(`${baseURL} + ${req.url} + ${req.params.tileset}`);

    const { ok, name, tile, ext } = tile_path(`/${req.params.tileset}`);

    if (ok) {
        console.log(`name: ${name}`);
    } else {
        console.log('no name match!');
        res.status(404).send();
        return;
    }

    const source = new GSSource(pmtiles_path(name));
    const p = new PMTiles(source, undefined, nativeDecompress);

    try {
        const p_header = await p.getHeader();
        if (!tile) {
            res.set("Content-Type", "application/json");

            const t = tileJSON(
                p_header,
                await p.getMetadata(), // I cannot seem to execute `getMetadata()`
                "hostname-placeholder",
                name
            );

            res.status(200).send(t);
            return;
        }
    } catch (err) {
        console.log(err);
        res.status(500)
        res.send(err)
        return;
    }
})

app.listen(port, () => {
    console.log(`Example app listening on port ${port}`)
})

There is much omitted code but consider the rest to replicate the AWS lambda function. I hope this is more context.

Mostly I'm at a loss as to what is causing the issue and was hoping someone here might be able to help?

[bdon]

can you share your source code? Moving this to a discussion.

[highway900]

thanks @bdon for looking at this. I have added some code to the original message.

[highway900]

I have now added a working example express server that replicates the issue.

[highway900]

It looks like this might be what is tripping me up https://cloud.google.com/storage/docs/transcoding#range

[highway900]

@bdon is the whole file gzipped? or just some portions of the file outside of the header for example?

[bdon]

The whole PMTiles file cannot be gzipped, because of the limitations on how range requests work described there. The individual contents of tiles can be gzipped, the compression method is specified in the header.

[highway900]

Ok I have solved this!

The Buffer object returned by the nodejs google cloud sdk has access to an ArrayBuffer through the .buffer method. But it is an ArrayBuffer of the Buffer object. So in order to get the ArrayBuffer of the data I needed to do a dance that takes a slice of the arraybuffer offset and the length.
This was not obvious to me as I am not a native node / js / typescript user. Interrogation of the arraybuffer showed it was different to the buffer object. I hope to make a PR for a gcp cloud function version of the pmtiles mvt reader soon. Cheers

[bdon]

Yes, that sounds right. The NodeJS buffer class is different than the one on the web. See this: https://sindresorhus.com/blog/goodbye-nodejs-buffer