Formalize Apple's CgBI chunk

[nigeltao]

Apple has an unofficial PNG extension, used by their tools (a modified pngcrush: xcrun -sdk iphoneos pngcrush -iphone) to create iOS app assets.

If you're updating the PNG spec to 3.0, consider making it official. It might not be the extension I would have designed, but if there's millions of iOS devices out there, there's zillions of PNG/CgBI images already out there in the real world.

I don't have official documentation, but here's what others have deduced:

• a CgBI chunk before IHDR (a serialization of a CGBitmapInfo struct).
• an optional iDOT chunk before IDAT.
• BGRA not RGBA.
• premul alpha not non-premul alpha.
• Raw zlib (i.e. deflate), no zlib header and no Adler-32 checksum.

https://iphonedev.wiki/index.php/CgBI_file_format

https://github.com/DHowett/pincrush

https://www.hackerfactor.com/blog/index.php?/archives/895-Connecting-the-iDOTs.html

http://jongware.com/pngdefry.html
This has an example PNG/CgBI image.

https://github.com/jakubknejzlik/cgbi-to-png
This has an example PNG/CgBI image. Not sure if their code does the premul -> non-premul conversion properly.

https://axelbrz.com/?mod=iphone-png-images-normalizer
Not sure if their code does the premul -> non-premul conversion properly.

[DavidBuchanan314]

Note that Apple's variant is patented: https://patents.google.com/patent/US20080177769

Also IIUC any official documentation, if it exists, is under NDA.

[svgeesus]

CgBI is a critical, private, safe-to-copy chunk. If standardized, it would become CGBI.

from 14.1 Additional chunk types:

New public chunks will be registered only if they are of use to others and do not violate the design philosophy of PNG. Chunk registration is not automatic, although it is the intent of the Registration Authority that it be straightforward when a new chunk of potentially wide application is needed. The creation of new critical chunk types is discouraged unless absolutely necessary.

From 14.3.1 Ordering of critical chunks:

Critical chunks may have arbitrary ordering requirements, because PNG editors are required to terminate if they encounter unknown critical chunks.

So as soon as this chunk was standardized, it would be a backwards-incompatible change because all older decoders would terminate on encountering it. (They would now, which is fine because it is a critical private chunk).

From Guidelines for new chunk types

Do not define new chunks that redefine the meaning of existing chunks or change the interpretation of an existing standardized chunk, e.g., do not add a new chunk to say that RGB and alpha values actually mean CMYK.

CgBI literally does that, redefining IDAT to not contain unpremultiplied RGBA but actually, premultiplied BGRA (note that it is not the alpha vale which changes, in premultilied alpha, but the color component values)

Note that Apple's variant is patented: https://patents.google.com/patent/US20080177769

From Introduction

Freedom from legal restrictions: no algorithms should be used that are not freely available.

It does not seem that this PNG-like asset has been designed with a view to possible later standardization.

[ProgramMax]

I contacted some friends at Apple to see if anyone from Apple would like to join the PNG WG.

But for the moment, I think this issue about the CgBI chunk can be closed. @svgeesus is right that if we wanted to add support for premultiplied and/or BGRA, they should be new chunks anyway.

I'll close for now. If we want to revisit BGRA / premultiplied alpha, lets open a new issue.

[nigeltao]

Note that Apple's variant is patented: https://patents.google.com/patent/US20080177769

https://patents.google.com/patent/US20080177769 links to https://patentcenter.uspto.gov/#!/applications/11650712 which says that that patent expired on 06/11/2018.

In any case, I am following up with a professional patent lawyer.

CgBI is a critical, private, safe-to-copy chunk. If standardized, it would become CGBI.

I'm not asking to create a new CGBI chunk (with a capital G) that nobody already uses and nobody would use (e.g. I wouldn't expect Apple to update their tools just because the PNG 3rd edition said so). I'm asking for the 2022 PNG specification to match PNG as used in 2022.

Yes, it might require an asterisk to the "chunk type's second byte denotes private/public status" rule: "an exception was made for the CgBI chunk for historical reasons blah blah blah". This would be similar to formalizing APNG in the spec (#26). Surely the spec will mention acTL chunks, not aCTL chunks, because the spec and reality should match but we can't change reality. We can't retroactively change all of the APNG files and software already out there.

I develop software that processes PNG images (amongst other things). In practical terms, this means things that start with an 8 byte "\x89PNG\x0d\x0a\x1a\x0a" magic string and/or have a .png file name extension. These PNG/CgBI images match that magic string and can be flung around as foo.png. If the PNG specification does not tell me how to handle these, it just means that the PNG specification is less relevant to my software and I will favor unofficial documentation instead.

A decade or two from now, if a digital archaeologist came across a PNG/CgBI image (there's possibly orders of magnitude more PNG/CgBI images than APNG images in existence, because of iOS), their job would be easier if the official PNG specification described how to decode it.

So as soon as this chunk was standardized, it would be a backwards-incompatible change because all older decoders would terminate on encountering it. (They would now, which is fine because it is a critical private chunk).

I don't see the change as backwards-incompatible: your quote from the spec says "encounter unknown critical chunks", not "encounter unknown critical public chunks". Older decoders used to terminate on encountering it. They can still do after PNG/CgBI gets written up in the spec. CgBI support can be optional in PNG 3rd edition (with asterisks where necessary). Same behavior (no compatibility change) either way.

But for the moment, I think this issue about the CgBI chunk can be closed. @svgeesus is right that if we wanted to add support for premultiplied and/or BGRA, they should be new chunks anyway.

I am reminded of APNG's long road to being officially blessed. As you know, the PNG spec authors were reluctant to embrace it for many years. If I remember correctly, APNG folks were officially told (by Glenn??) that PNG isn't animated, so APNG should use a different magic string. But given a critical mass of software support (Firefox first and other browsers later in APNG's case, iOS in CgBI's case), popular implementation trumps specification.

[randy408]

If the PNG specification does not tell me how to handle these, it just means that the PNG specification is less relevant to my software and I will favor unofficial documentation instead.

Is there precedent for standardizing a reverse-engineered (and incomplete) specification? If someone had to tell us how to handle Apple's variant of PNG it would be the company who invented it.

[ProgramMax]

I'm asking for the 2022 PNG specification to match PNG as used in 2022. ... This would be similar to formalizing APNG in the spec (#26).

That's a good argument. You convinced me, actually.

I'll have to reflect for a bit.

Would this be a good candidate for an extension rather than the core spec?
Is the public/private bit no longer important?

[ProgramMax]

I'm re-opening since 1.) the patent may have expired, and 2.) the public/private bit may not actually be a deciding factor.

I still hope we can get input from a Apple.

[nigeltao]

The related iDOT chunk (and the same https://www.hackerfactor.com/blog/index.php?/archives/895-Connecting-the-iDOTs.html link) gets mentioned on #54 (comment).

[nicolas17]

I have been trying to reverse-engineer this chunk properly, since nobody ever figured out what the chunk content means, most third-party tools just look at the chunk's mere presence and assume "raw zlib, premultiplied alpha, BGRA", which isn't correct (some don't even fix the premultiplied alpha which is worse). While experimenting, I found even Apple software doesn't handle it consistently. I easily made a png file that is shown differently on Preview vs Safari on Mac, and another that is shown differently on Mac vs iOS.

The one thing I found for sure is that bit 0x1 in the first byte of the chunk data (ie. bit 0x10000000 if you consider the chunk as containing a big-endian uint32) is what determines whether the file is using raw zlib. Every third-party tool I found to deal with Apple's PNGs assumes the IDAT will have headerless raw zlib if the CgBI chunk is present, without checking if that bit is set.

I also spent days trying to understand bit 0x2 in the first byte; it maps to an internal flag called APPLE_FILTER_SUB_ONLY but I can't figure out what it does, either by decompiling the code or experimenting with setting it.

...do you really want to put this mess into a standard spec?

[randy408]

most third-party tools just look at the chunk's mere presence and assume "raw zlib, premultiplied alpha, BGRA", which isn't correct

This and the fact that not even Apple implements it consistently are more reasons I won't be supporting it in spng.

I also spent days trying to understand bit 0x2 in the first byte; it maps to an internal flag called APPLE_FILTER_SUB_ONLY but I can't figure out what it does, either by decompiling the code or experimenting with setting it.

It could mean some or all scanlines are encoded with filter type 1 (Sub), some tools might tell you the filter type for each scanline (spng exposes this through spng_get_row_info()).

[ProgramMax]

Thank you for your research, @nicolas17 . This is both good to hear and also unfortunate.
I agree with @randy408 .

On one hand, the PNG Working Group has agreed to accept private chunks into the standard in rare occasions where it makes sense, such as existing APNGs. This opens the doors for accepting the private CgBI chunk.

But on the other hand:

1. I'm trying to get Apple involved but haven't heard back yet.
2. If Apple itself is inconsistent with the chunk, it is difficult to argue just how standard-ready it is.
3. While the US patent may have expired, I'm not sure how this works outside of the US. I would need to rope in W3C's lawyers before we could even consider taking steps here. (And again, without Apple's involvement this step requires extra care.)

@nicolas17 , do you happen to know if most (almost all?) CgBI-supporting decoders are consistent and only a handful are off? For example, maybe only a recent version of MacOS changed behavior and it is consistent-enough to not discredit.

If we want to support premultiplied alpha and/or BGRA, it might be best for us to introduce those as a new chunks in a clean-room approach, not considering what Apple has done. Unfortunately, that rather defeats the purpose of accepting widely-adopted, existing chunks.

[Crissov]

Do CgBI and iDOT offer any benefit over existing capabilities of PNG? It does not sound like it, so I do not see a compelling reason to add it to the standard, even without considering the lack of documentation and interoperability.