Imagemagick: identify does not recognize Color Spaces properly

[pauloney]

I have a small PDF file (that one can open with a text editor)

https://www.dropbox.com/s/agjiwei4hga3n7i/im-bug.pdf

and clearly see that it is written in the "CMYK" Color Space, but "identify" reports it as being "sRGB".

$ identify -verbose im-bg.pdf | grep Colorspace
Colorspace: sRGB

The issue has been extensively discussed in the Forum here and one "explanation" given was that IM is reporting a GhostScript rendering color-space of the file - to which a counter example has also been given.

The "Color Space" of a PDF file is an intrinsic part of it and does not depend on the rendering device, so IM should not be using any transformation, via GS, to get this information - because it will inherently wrong and disconnected from the original image. The forum posting contains examples of images that are reported correctly (by accident I believe) and others incorrectly.

[dlemstra]

The detection is not by accident. Your file red-cmyk.pdf contains some parts that we use to detect the colorspace. With that will we can detect that it is cmyk and tell ghostscript to use the cmyk device instead of the rgba device. I checked your file im-bug.pdf and I don't understand how we could detect that it is cmyk and tell ghostscript to use the cmyk device.

[pauloney]

I believe you are referring to the occurrence of

/ColorSpace/DeviceCMY

on line 45 of the file red-cmyk.pdf. Indeed this file has color space CMYK, but many other files that have this string may be something else - because not only you need to mention that line, but you also need to have an object occurring in that defined color space - and if there are NO objects define, that file would NOT be CMYK.

In summary: Detection of the string "CMYK" is not enough to define a color space of a PDF file.

Now there are quite a few things wrong with the logic of using GhostScript to render it and determine the Color Space:

1. It can't really be done, because a PDF file may contain strokes in more than one Color Space to begin with. Conversions to s single color space will not work.
2. The GhostScript rendering will change some of these color spaces at rendering and all you are going to see if what has changed to and not what it came from.

As for your statement:

* I checked your file im-bug.pdf and I don't understand how we could detect that it is cmyk and tell ghostscript to use the cmyk device.*

That is very simple, as per PDF specs, the "K" at the end of line 8 of that file (along with a few others) define that stoke as being CMYK. Since there are no other strokes at any other Color Spaces, this defines this file Color Space as being: CMYK.

[dlemstra]

Do you have a reference to the PDF specs so I can understand it better? And we are not letting Ghostscript determine the Color Space we are telling it which device (pngalpha/cmyk32) / Color Space to use. Do you have an example of a file that has the /ColorSpace/DeviceCMYK line but is not CMYK?

[pauloney]

The ultimate reference is the PDF spec:

http://www.adobe.com/content/dam/Adobe/en/devnet/acrobat/pdfs/pdf_reference_1-7.pdf

starting on Section 4.5.. and the two most important things to keep in mind are:

1. A PDF can have more than one Color Space, in fact it can contain many color spaces and even all 17 defined PDF color spaces. See for example here: https://answers.acrobatusers.com/Acrobat-Page-color-space-q3001.aspx

2. One cannot convert to a single one because there is not a single one to begin with ...

3. Mentioning the color space is not enough, one needs to have an object defined in it. A PDF may mention all possible 11 Color Spaces and not have any strokes defined on it.

I do not have at this moment a file that says /ColorSpace/DeviceCMYK but it is not CMYK, but I can build one. We will need to build several PDF test files to make sure the logic is okay and I guess I should get started on it...

[pauloney]

To have an idea how absurd is this logic to convert using ghostscript and reading the information just run the command

identify -verbose im-bug.pdf

and grab the top 11 lines. It contains the pearls:

Image: im-bug.pdf
Format: PNG (Portable Network Graphics)
Mime type: image/png
Geometry: 28x28+0+0
Resolution: 72x72
Print size: 0.388889x0.388889
Colorspace: sRGB

all of them incorrect!

[pauloney]

The entire logic, if not the code, for this block will have to be changed. Here are the minimal specs to be followed, since the treatment of PDF (and may be PS too) is probably going to have to be completely separate from the other types of images:

1. A PDF file may have any number from 0 to 11 ColorSpaces defined in the file.

2. They are named:

3. Three device color representation ones: Gray, RGB and CMYK
4. Four CIE-based ones: CalGray, CalRGB, Lab and ICCBased.
5. And four specials: Pattern, Indexed, Separation and DeviceN.

(in particular, "sRGB" is NOT a PDF allowed Color Space).

1. Any one of them, as well as NONE, or ALL of them, can occur, simultaneously, in the same page in the same file. They are attached to objects in the file, and not to the file itself.

2. They CANNOT be identified by the occurrence of the words of the type

   /ColorSpace/DeviceCMYK

because the space may be defined, but with NO objects using it.

1. A file may use a ColorSpace CMYK, for example, WITHOUT the words:

   /ColorSpace/DeviceCMYK

in it. This is the core of the example I gave:

https://www.dropbox.com/s/agjiwei4hga3n7i/im-bug.pdf

[pauloney]

Dirk,

I am a bit lost, I guess. Can you elaborate on why it is being closed? What was the solution? I just tried and I get the same exact bug, please elaborate.

[dlemstra]

This was closed due to the topic being stale. And we don't have the resources to resolve this specific use case.

[pauloney]

Okay. Can we remove the error then? It would be tremendously more helpful to just produce a message:

  identify is NOT able to identify Color Spaces on PDF files

if one feeds a PDF file in it.

[dlemstra]

ImageMagick makes an educated guess and this seems to work for most of our users.

[pauloney]

Dirk, it is NOT an educated guess -- it reports sRGB -- to whatever you feed it and sRGB is not even a valid PDF ColorScape. I am sorry, but this does NOT work for ANY user. If we do not have the resources to solve the problem -- I think two things should be done:

1- The manual should be very forthcoming and say the ColorSpace identification does NOT work for PDF.

2- It should be kept into the list of Open Issues, till such time someone with the understanding of the program can do the implementation -- which is very simple, and detailed in my bug report.

Can we please re-open it, this is a very important bug -- there are NO tools to recognize the ColorSpace of a PDF file.

[dlemstra]

When we read a PDF file we pick one of the following output devices: pngalpha, pamcmyk32 or pnmraw. Based on the contents of the file we take a guess which output device would be the best for the file that we are trying to convert to an image. In your case we pick pngalpha instead of pamcmyk32 because we were unable to determine that it would be better to use the pamcmyk32 device. And when we read the PNG file that is created by Ghostscript we are told that the colorspace of the image is sRGB and that is why we tell you that the colorspace of the image is sRGB.

And to avoid any confusion ImageMagick is not a tool that can give info about a PDF file. It is a tool that can be used to convert a PDF file to an image.

[pauloney]

Dirk, the logic is wrong because it assumes that every image has only one colorspace -- an image in PDF can have up to 11 color spaces.

[dlemstra]

The images are transformed to the sRGB colorspace because Ghostscript does that. We only tell Ghostscript what output device it should use to create the images.

[pauloney]

Dirk, the problem is that the logic is completely wrong -- sRGB is not even one of the 11 ColorSpaces allowed in a PDF file. Since this is soo deep ingrained in the program I was hoping that one of the main developers would take it on, but since no one steeped in, I am thinking about taking on the task to fix this. Would you help me out by describing the full process of recognition of a color-space of a GIF, TIFF, ... to begin with? I do not understand why a transformation is necessary, and I need to understand that first. Many thanks.

[Akash91]

@dlemstra @pauloney I have ran into same issue, I also tried using, minecraft(pdfminer based), pdfBox and other pdf parsing libraries, but most of them have the same limitation, I am looking for someone who could help me out with understanding the parsing part of it, I am ready for code contribution.

[pauloney]

I guess the first thing to do would be to produce test files with, for
example, a single page, and a single-square (red) in all colorspaces
possible in PDF format.

Tĥen we would have to agree with the ImageMagick guys on how to change the
framework - so colorspaces of PDF files get properly identified. Right now
the problem is very deep into the code -- ImageMagick only recognizes ONE
colorspace for a file and PDF files are capable of having several
colorspaces in the same file.

Paulo Ney

On Tue, May 21, 2019 at 3:53 AM Akash Sharma [e-mail]
wrote:

@dlemstra https://github.com/dlemstra @pauloney
https://github.com/pauloney I have ran into same issue, I also tried
using, minecraft(pdfminer based), pdfBox and other pdf parsing libraries,
but most of them have the same limitation, I am looking for someone who
could help me out with understanding the parsing part of it, I am ready for
code contribution.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ImageMagick/ImageMagick/issues/431?email_source=notifications&email_token=AAR7WYVY4CXHS54RNACM5XLPWPIDVA5CNFSM4DH5D5HKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODV3Q66A#issuecomment-494342008,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAR7WYXOZBFPF4IWYKHVSWTPWPIDVANCNFSM4DH5D5HA
.

[fmw42]

If a PDF file has many parts with different colorspace, how do you expect that it would be properly rendered? Imagemagick does not read the PDF file. It offloads to Ghostscript, which has limitations. If you have better software for dealing with PDF files, please provide it. What colorspace would you expect when you have a PDF with many colorspaces involved? What would you expect Imagemagick to report for the colorspace? Personally, I don't think your expectation is practical or reasonable. My personal thinking and not that of the IM developers. I will defer to them. If you have such complex PDF files, then perhaps you need to use Adobe tools. Are there any tools that properly tell you want you want?

[urban-warrior]

ImageMagick converts a PDF to a RGB or CMYK image sequence regardless of which mix of colorspaces the PDF encodes. If colorspace fidelity is important to your workflow, ImageMagick is likely not the right tool for your use case.

[pauloney]

Fred,

The answers interspersed in your text:

On Tue, May 21, 2019 at 12:11 PM Fred Weinhaus [e-mail]
wrote:

If a PDF file has many parts with different colorspace, how do you expect
that it would be properly rendered?

As it is usually rendered by many programs/devices -- with each piece being
rendered within its own colorspace. Obviously some conversion is done for
rendering on paper, on a screen, etc ... ghostscript is one such program
that converts to RGB, CMYK, etc ... depending on the need. Observe that
this does NOT change the original file that may have up to 10 colorspaces
-- which are named: "DeviceRGB", "DeviceCMYK", "DeviceGray", "CalRGB",
"CalGray", "Lab", "ICCBased", "Indexed", "Separation" and "DeviceN".

Imagemagick does not read the PDF file. It offloads to Ghostscript, which
has limitations.

Yes! Ghostscript is able to determine the colorspace of an element of the
PDF file and convert it to whatever is the target colorspace... but I do
not think that GhostScript AS-IS can report the colorspace of a PDF file.

If you have better software for dealing with PDF files, please provide it.

I am not sure if I understand this. Do you mean something better than
ghostscript? I don't think there is any Open Source suite that parses a PDF
and reports the colorspaces being used. The easiest way to do this would be
to implement it inside ghostscript and then change IM to make the right
calls and report the correct results.

What colorspace would you expect when you have a PDF with many colorspaces

involved?

That depends on what you mean by "involved".

A PDF file may mention ALL 10 colorspaces and use only ONE of them (all
others can be mentioned but not associated with any non-empty element). In
this case would expect to be informed only of the actively used ones like:

13 elements in DeviceRGB

What would you expect Imagemagick to report for the colorspace?
>
... or even a a more complete report as in:

13 elements in DeviceRGB
2 elements in Lab
DeviceCMYK, DeviceGray, CalRGB mentioned but not used by any elements

but NEVER what is done which is to convert to an image and report the
colorspace of that image ....

Paulo Ney

[pauloney]

That is beside the point! Almost everyone by now know that IM is not able
to determine the colorspace of a PDF file. The issue here is that the BOTH
the program and the manual says it DOES.

IM should be changed, urgently and temporarily to say:

       IM is NOT capable of determining the colospaces being used in a

PDF file yet!

and then work with the GhostScript crowd to add this feature to gs... and
then we implement the re-reporting on IM.

Paulo Ney

On Tue, May 21, 2019 at 4:39 PM ImageMagick [e-mail]
wrote:

ImageMagick converts a PDF to a RGB or CMYK image sequence regardless of
which mix of colorspaces the PDF encodes. If colorspace fidelity is
important to your workflow, ImageMagick is likely not the right tool for
your use case.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ImageMagick/ImageMagick/issues/431?email_source=notifications&email_token=AAR7WYTST2SVYNKKJRDPHYLPWSB3FA5CNFSM4DH5D5HKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODV5PQ2Q#issuecomment-494598250,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAR7WYTF6RF2X26SEASAT7LPWSB3FANCNFSM4DH5D5HA
.

[fmw42]

If you need a change to Ghostscript, I think you should contact that group to make the changes. Then perhaps ImageMagick can make use of those changes to report your colorspace list and make use of it. Again my personal opinion. However, I will defer to the ImageMagick developers.

[pauloney]

That part is being taken care of at this moment, and we hope, some time in the future, to be able to use GS to determine the colors[aces of a PDF file.

What one need here, from IM developers, is for it to state that IM is NOT capable of determining the colorspaces used in a PDF file, maybe with a clear message like:

You are trying to determine the colorspace of a PDF file. Contrary to most image formats, PDF is a programming language and can use up to 10 colorspaces in the same page. IM is not able to determine the colorspaces used in a PDF file at this moment.

[Akash91]

I agree.
Also, I have been going through parsing PDF object stream to understand it better. From my understanding none of the open source projects are able to find colors which are not explicitly present under colorspaces.
I think the only way forward is to actually decode the object stream by decoding (https://blog.didierstevens.com/2008/05/19/pdf-stream-objects/) and parse it properly.

[fxtentacle]

FYI, the identify color space detection also seems to be related to delegates.xml, which is completely not obvious from the documentation.

I encountered this error:

$ identify t.pdf
identify: UnableToOpenConfigureFile `delegates.xml' @ warning/configure.c/GetConfigureOptions/681.
identify: LengthAndFilesizeDoNotMatch `/tmp/magick-NZ52GRY69smteLy1vPNdiWQg-BjkjGQG1' @ error/bmp.c/ReadBMPImage/854.
identify: ImproperImageHeader `/tmp/magick-NZ52GRY69smteLy1vPNdiWQg-BjkjGQG1' @ error/bmp.c/ReadBMPImage/975.

with CMYK PDFs but RGB PDFs were working just fine. The solution was to copy the delegate decode="ps:cmyk" stealth="True" line from a different ImageMagick installation over into the delegates.xml on the affected machine.

[fmw42]

Ghostscript, which is what IM uses to read PDF files cannot handle CMYK. So you need to convert to sRGB by adding -colorspace sRGB before reading the input

identify -colorspace srgb t.pdf

[pauloney]

It is way more than just that .... PDF can have several ColorSpaces on the
same page and IM is not able to handle that either -- it assumes that every
page has a single colorspace.

Paulo Ney

On Tue, Sep 29, 2020 at 8:25 PM Fred Weinhaus [e-mail]
wrote:

Ghostscript, which is what IM uses to read PDF files cannot handle CMYK.
So you need to convert to sRGB by adding -colorspace sRGB before reading
the input

identify -colorspace srgb t.pdf

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ImageMagick/ImageMagick/issues/431#issuecomment-701134356,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAR7WYVU3L5Q3DUJVB2X6XTSIKQIZANCNFSM4DH5D5HA
.