Adding generic image-type data chunk(s)

[ProgramMax]

Problem statement:

Currently, PNG only supports RGB or grayscale images with optional alpha. But there is image data which isn't RGB/grayscale (with alpha). For example:

• Satellites images often include infrared data.
• Games use normal map data.
• The proposed gain maps also are not RGB/grayscale (with alpha).

Previously, the restriction of RGB/grayscale (with alpha) was a nuisance which could be worked around by splitting the data across multiple images. Satellite images could have one image for visible colors and another grayscale image for infrared. Games could have one image for albedo colors and another image for normal maps.

This usage has three major drawbacks:

• The image data was split up (into separate files).
• The user/program must know which file contains which data. This is often hard-coded.
• The non-RGB (etc) data is forced into RGB. Sometimes this is mapped like false-color images like satellite data. Sometimes it is not mapped like game normals.

Notice, this breaks down into multiple problems:

• We need to store extra image-type data.
• We also need to mark that data as what it contains (infrared, normals, etc).
• We need to specify visibility mapping (such as false-colors) if this image chooses to use it.
• We need to mark the existing RGB (etc) data as not actually RGB. This will be helpful for backwards compatibility with existing machinery.

Proposed solution:

Since there are multiple separate problems, we need multiple solutions. But since they work together to solve a unified larger problem, some information will be shared.

This GitHub issue will provide a general overview of the approach. It will then link to other GitHub issues with the details of each piece. This will help us maintain discussions at the correct scope.

Shared data

We'll need an enumeration for the type of data. For example, it could be a PNG two-byte unsigned integer (possibly reserving the high X-many values for user defined data) with:

• 1 - Red
• 2 - Green
• ...
• 6 - Infrared
• 7 - Ultraviolet
• 8 - Object Space Normal X
• ...
• 13 - Tangent Space Normal Z
• 14 - Gain map
• etc

Extra image-type data

A new chunk will be added which behaves like IDAT. For now, let's name it iEXt. (If "IDAT" is "image data" then "iEXt" is "image extra"). I'm open to name suggestions.

A deeper discussion about iEXt can happen in #494 .

It will have an ID, which allows other chunks to reference it.
ID 0 could be reserved for the IDAT chunk.

We need to mark the extra image-type data with what it contains

A new chunk will be added to describe an ID. For now, let's name it iEId ("image extra id").

A deeper discussion about iEId can happen in #495 .

For example, an image might contain ID 1, which it defines as gain maps.
A separate image might have ID 1 list of tangent space normal x, y, and z.

The reason for both ID and a list of enum values is because they might be interleaved into the data separately or it is expected that the app will not show it.
For example, the first image could contain IDAT, iEXt, IDAT, iEXt, ... interleaving image & gain map data.
The second example image might contain IDAT, IDAT, ..., iEXt, iEXt, ...

Note that in this example, the tangent space normal x, y, and z are interleaved amongst themselves just like RGB data is in a PNG.

This also allows an image to contain multiple IDs. For example, a satellite image might contain color, infrared, and ultraviolet. ID 1 would be infrared. ID 2 would be ultraviolet. IDAT, IDAT, ..., iEXt (1), iEXt (1), ..., iEXt (2), iEXt (2), ...

We need to specify optional visibility mapping

iCCP might be fine here.
For example, imagine the image that contains only satellite infrared data. It might already be a RGB image using false colors. But perhaps it is a grayscale image using iCCP to change the grayscale values into false color? (If so, iCCP is really only used to visualize the data and not actually alter it. We might need to clarify this.)

We need to mark the existing RGB (etc) data for backwards compatibility.

We might be able to reuse the iEId chunk for this if we reserve ID 0 for IDAT.
For example, that same satellite infrared image mentioned above could contain an iEId chunk that says ID 0 really holds infrared.
That way, existing decoders continue to display the PNG like normal. But aware decoders know to interpret this data as infrared.

[ProgramMax]

Rather than make a generic image-type data chunk, we could just make custom chunks every time we need more data types.

I suggest we carefully research each data type before adding it to the enum. (In #495 I describe an open problem that each additional data type will need its own custom set of extra data.)

If we're carefully adding each new piece anyway, might as well make them bespoke chunks.

However, I think a generic chunk is the better approach. It allows us to reuse existing definitions and understanding. Implementations would similarly benefit.