Pdf-lib: Feeding PDFKit into pdf-lib

[mlgoff59]

I'm using PDFKit/Svg-to_PDF to convert SVG maps to a GeoPDF aalong with the sajjad-shirazy/pdfkit.js fork of PDFKit to insert SPOT colors. Works great. But I also have a requirement to output PDF Version 1.4 or greater files. PDFKit only produces v1.3 PDFs.

Your library is outputting v1.7 PDFs. I can work with that, but it doesn't have all of the SVG and SPOT color features.

So what I am trying to do it take the completed PDFKit:PDFDocument, convert it to a byte array and use it with PDFLib.PDFDocumentFactory and PDFLib.PDFDocumentWriter to try to output a v1.7 document.

But every time I try to invoke PDFLib.PDFDocumentFactory it throws an error.

var pdfDoc = PDFLib.PDFDocumentFactory.load(pdfByteArray);

[undefined] - TypeError: e.subarray is not a function

Obviously I'm doing something wrong, I just can't figure out what it is.

I'm testing in Chrome 67.

Any help would be appreciated. I can provide code snippets if you need it.

[Hopding]

Input data to pdf-lib methods/functions must always be in the form of a Uint8Array. I'm guessing that your pdfByteArray variable is not a Uint8Array, but just a normal Array type? Uint8Arrays have a subarray method, while normal (untyped) arrays do not. This would explain the error you are seeing.

Uint8Arrays are typed arrays. This makes them more efficient than just arrays of numbers - especially for large binary data sets (like images or PDFs). As such, it would be ideal if your code never produced a plain array, and just produced the Uint8Array initially. But, if you can't do this, it's possible to convert the plain array to a Uint8Array like so:

var typedPdfByteArray = Uint8Array.from(pdfByteArray);
var pdfDoc = PDFLib.PDFDocumentFactory.load(typedPdfByteArray);

[Hopding]

SVG embedding is a feature I've considered adding to pdf-lib. I think I'll work on it at some point - but you're correct that it's not currently supported.

Also, when you say SPOT colors, are you just referring to drawing vector graphics in the CMYK colorspace, rather than the RGB color space?

[mlgoff59]

Thanks Dillion, I didn’t know that. I’ll try that tomorrow and let you know how it works.

SPOT Colors aren’t just CYMK. they are a method for ensuring that the color you choose is the color that is displayed regardless of the monitor/printer type.

https://helpx.adobe.com/acrobat/using/color-conversion-ink-management-acrobat.html

It is a customer requirement. I just joined this project after most of the team transistioned to another project.

[mlgoff59]

No matter what I do, I can't seem to convert the PDFKit PDFDcoument object into a UInt8Array to initialize the PDFDocumentFactory. I've zxipped up a sample project. It doesn't use real data (the customer would not like it if I distributed their data).

If you can provide any pointers/insight I would be very grateful!
ConvertToPdf.zip

[Hopding]

Yeah, JavaScript has too many ways to handle binary data. Converting between them all is confusing.

I've attached an updated version of your example that:

1. Converts the PDFKit document into a Uint8Array.
2. Loads that Uint8Array into a PDFLib.PDFDocument object.
3. Draws an orange rectangle on the first page (just for funsies 😄).
4. Saves the PDFLib.PDFDocument to a Uint8Array.
5. Prompts the user to download that Uint8Array (with FileReader.saveAs).

ConvertToPdf-Updated.zip

[Hopding]

Here's the JavaScript code for that:

function convertToPdf () {
  var FileSaver = require(['file-saver']);

  let pdf = new PDFDocument({ autoFirstPage: false });

  var stream = pdf.pipe(blobStream());

  pdf.addPage({
    size: [400, 400],
    layout: 'portrait',
    margin: 0
  });

  let svgDiv = document.getElementsByTagName("svg");

  SVGtoPDF(pdf, svgDiv[0].innerHTML.toString(), 0, 0);

  pdf.end();

  stream.on('finish', function () {
    // Convert PDFKit's PDF to a Blob object.
    let blob = stream.toBlob('application/pdf');

    var fileReader = new FileReader();

    // Callback to be invoked when the fileReader finishes converting
    // the Blob object to an ArrayBuffer object.
    fileReader.onload = function(event) {
      // Create a Uint8Array from the ArrayBuffer (event.target.result).
      var origPdfBytes = new Uint8Array(event.target.result);

      // Load the Uint8Array into a pdf-lib PDFDocument object.
      var pdfDoc = PDFLib.PDFDocumentFactory.load(origPdfBytes);

      // Get the first page of the PDFDocument.
      var pages = pdfDoc.getPages();
      var page1 = pages[0];

      // Draw an orange rectangle on the first page.
      var contentStream = pdfDoc.createContentStream(
        PDFLib.drawRectangle({
          x: 25,
          y: 25,
          width: 150,
          height: 50,
          colorRgb: [1, 0.5, 0.25],
        }),
      );
      page1.addContentStreams(pdfDoc.register(contentStream));

      // Save the PDFDocument object to a Uint8Array.
      var updatedPdfBytes = PDFLib.PDFDocumentWriter.saveToBytes(pdfDoc);

      // Convert the Uint8Array to a Blob and save it.
      saveAs(new Blob([updatedPdfBytes]), "test.pdf");
    };

    // Trigger the conversion of the Blob to an ArrayBuffer.
    fileReader.readAsArrayBuffer(blob);
  });
}

And here's the HTML for the webpage:

<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml" lang="en">
<head>
    <meta charset="utf8" />
    <title>Inline SVG within HTML</title>
</head>
<body>
    <svg width="400px" height="400px">
        <rect x="20" y="20" width="100" height="280" fill="blue" stroke="black" stroke-width="3" />
        <circle cx="70" cy="80" r="30" fill="red" stroke="black" stroke-width="2" />
        <circle cx="70" cy="160" r="30" fill="yellow" stroke="black" stroke-width="2" />
        <circle cx="70" cy="240" r="30" fill="green" stroke="black" stroke-width="2" />
    </svg>
    <button type="button" onclick="convertToPdf();">Convert to PDF</button>
</body>

<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.3.1/jquery.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/FileSaver.js/1.3.8/FileSaver.min.js"></script>
<script type='text/javascript' src='./blobstream.js'></script>
<script type='text/javascript' src='./pdfkit.js'></script>
<script type='text/javascript' src='./source.js'></script>
<script type='text/javascript' src='./pdf-lib.min.js'></script>

<script src="./RequireConfig.js"></script>
<script src="./require.js"></script>

<script type="text/javascript">
function convertToPdf () {
  // ... Implementation shown above ...
}
</script>
</html>

[mlgoff59]

That's brilliant! I would never have thought of doing it that way. It works perfectly! If this thing supported SVG and SPOT/GeoPDF, I'd dump PDFKit!

If there is anyway I can assist/aid you. just let me know.

[mlgoff59]

I just noticed a problem (for me). When I compare file that I have generated via your library to one that I generated strictly with PDFKit, except for the versions numbers, they are identical. Which means that pdf-lib is not converting the PDFKit document to a v1.7 PDF. The PDFKit file is 1.922 mb. So is the file generated by pdf-lib. But if I open the file Acrobat Pro and save it, the resulting file is 844kb.
Are you sure you are using the v1.7 compression algorithms?

[Hopding]

pdf-lib doesn't currently compress any newly added objects besides embedded images and fonts. When modifying a PDF, any existing content streams will remain compressed as they were prior to modification.

I'm not aware of any compression-specific features in PDF 1.7. But the PDF spec is massive, so it's certainly possible I'm overlooking something. Do you happen to know in particular which PDF features Acrobat is making use of that pdf-lib is not?

Object Streams are a good candidate for decreasing file size, though they aren't new to 1.7. Here's their description from the specification:

7.5.7 Object Streams

An object stream, is a stream object in which a sequence of indirect objects may be stored, as an alternative to their being stored at the outermost file level.

NOTE 1: Object streams are first introduced in PDF 1.5. The purpose of object streams is to allow indirect objects other than streams to be stored more compactly by using the facilities provided by stream compression filters.

pdf-lib doesn't currently write out files using Object Streams. But this is a feature that could be added. Most of the code for this would likely exist in the PDFDocumentWriter class.

Can you go into any detail as to why the additional compression is a requirement for you? Of course, smaller file size is always good, but I'd like to have more info about the use case for this before deciding whether or not to implement it.

[mlgoff59]

It isn't just the compression. Smaller file size is very important. But what you really get is encryption. v1.4 introduced 128-bit encryption. v1.6 upgraded to AES 256-bit encryption and enhanced the SPOT color support.

The app that generates the maps outputs a much smaller v1.4 file. And Acrobat Pro also generates a much smaller file when I use it to re-save my editor's output. I can't confirm that the larger file is encrypted, so it is hard to convince my boss that it is of equal quality. Our client most likely won't accept it.

https://www.prepressure.com/pdf/basics/version

[Hopding]

I don't plan to do any work with encryption in the near future for pdf-lib, as I suspect it is a pretty niche use case (though I could certainly be wrong).

I'd like to research opportunities for additional file compression, though. Can you share a sample PDF before and after saving it with Acrobat Pro? I'd like to inspect them to see what mechanisms pdf-lib could potentially make use of (in addition to Object Streams).

[Hopding]

Also, you mentioned that one of your requirements is to output a v1.4 or greater PDF file. Are you also required to encrypt the document? It's not clear to me if that is a feature you are looking for, or just a noteworthy feature of v1.4+.

[mlgoff59]

Actually, I thought that encryption was a default feature beginning with v1.4. If it isn't, then it's not an issue. I can understand not wanting to work on encryption. My last job was creating encryption tools for web services. Took me quite a while to come up to speed.

Here is a set of files. The "original" is what we outputted from our editor. The "Acrobat Pro" is the same file saved by Acrobat.

Atlanta.svg-SPOT.zip

[mlgoff59]

I did some quick esearch. Acrobat does not encrypt by default. It is an option available when you password protect the file. So all I really need is the compression/size reduction.

[Hopding]

Thanks for sharing those files. I just took a look at them, and the file output by Acrobat does indeed use object streams. I'm fairly confident that explains the 27 KB between these two files. I'll try to look further into this over the weekend and see how much effort it would take to start using object streams in pdf-lib.

You mentioned in a previous comment that

The PDFKit file is 1.922 mb. So is the file generated by pdf-lib. But if I open the file Acrobat Pro and save it, the resulting file is 844kb.

This size difference is a lot bigger than 27 KB. I assume the files you were working with here had similar contents (SVG graphics), but just a lot more of them?

[mlgoff59]

Yeah.. Our client wouldn't let me give you their copyrighted files, so I pulled a map from OpenStreemaps.org and ran it through my editor. I would have preferred to give you one of the client's files.

Very similar content, much bigger. Customized USGS data.

[Hopding]

That’s fine. Perfectly understandable. The example you provided should be enough to get going with :+1:

[mlgoff59]

Cool beans! Let me know if there is anything I can do to help.

[mlgoff59]

Hopding, have you been able to look at this? Is it doable?

[Hopding]

I think it is doable, yes. I've been pretty busy lately and haven't had time to work on actually implementing this. I'm hoping to have time these next couple of weeks.

[mlgoff59]

Great news!! I really appreciate it. Seriously, if I can help in any way, just let me know what you need.

Too busy? You mean school/life is more important than coding? What kind of crazy talk is that?! lol Whatever you do don't tell my wife I said that...

[Hopding]

@mlgoff59 I'm going to try to work in this over the weekend, but I've got a couple of questions:

I just took another look at the Atlanta.svg-SPOT.zip file you shared. It contains the following PDFs:

| Name | Size |
| ----- | ---- |
| Atlanta.svg-SPOT Acrobat Pro.pdf | 526 KB |
| Atlanta.svg-SPOT Original.pdf | 454 KB |

• Based on our previous conversations, I would have expected the Atlanta.svg-SPOT Acrobat Pro.pdf to be the smaller of the two? Am I missing something?

• Also, you mentioned that The "original" is what we outputted from our editor. How was this file generated in your editor? Using PDFKit and pdf-lib?

[mlgoff59]

The “original” should have been bigger that the Acrobat Pro version. I may have gotten them mixed up.

Hmm. I’m going to have to get you a better set of files. Let me ask our customer for permission to send you their files. If I do that, it will have to be in a private email, so no one else can access the files.

Yes, our editor converts the SVG to PDF using PDFKit and then upgrades them to PDF v1.7 through pdf-lib.

[Hopding]

I've made pretty good progress towards implementing object streams over the weekend: https://github.com/Hopding/pdf-lib/tree/AddObjectStreamSupport. There's still a few things left to do for it to be completed. In particular, I need to start encoding the object streams to see how much space is saved by using them.

Once I get that done, I'm planning to cut a pre-release that you can try out and let me know what you find. If it works, and the file size is comparable to what Acrobat Pro outputs, then there's not much need to send me those files. But if there's still a significant difference, then I'll probably a better set of example PDFs to dig deeper.

So if it's any kind of an inconvenience to ask your customer for permission to share those files, feel free to hold off on that until I get the pre-release cut. I can't make any promises, but I'd guess within a week or two I should have that done.

[mlgoff59]

That’s great!! I look forward to testing it.

It will take a week or two to get permission, so I’ve already asked.

[mlgoff59]

I was rather surprised today. On our weekly WebEx, they told me to go head and give you the data for one of our smaller maps. So I have data for you whenever you want it.

[Hopding]

@mlgoff59 I've just finished a quick and dirty implementation of object stream encoding that should demonstrate the potential space savings (or lack thereof) of object streams.

I published a pre-release: v0.3.1-rc1. Please try it out when you get a chance, and let me know what you find. You can install the pre-release as follows:

npm install --save [email protected]

[Hopding]

The only difference is the method you must call on PDFDocumentWriter to save the PDFDocument:

// To save without using object streams
PDFDocumentWriter.saveToBytes(pdfDoc);

// To save using object streams
PDFDocumentWriter.saveToBytesWithObjectStreams(pdfDoc);

Note that I don't intent to keep this special method permanently. I'll figure out something better before I cut an official release of this feature. But it does the trick for now.

[mlgoff59]

I was getting an error installing this:
npm WARN saveError ENOENT: no such file or directory, open 'D:\Projects\JavaScript\package.json'

I have multiple installations of npm on my system. So I just removed all of them, including the globla copy, and reinstalled npm. Now I don't have any errors installing [email protected].

But several of the files, including pdf-lib.min.js (which is the one I use in my project) have a timestamp of "10/26/1985 4:15 AM"

I think something is wrong. Do I need to browserify the library? If so, which file do I use? index.js?

[Hopding]

Yeah, it looks like unpkg has problems with the "Last Modified" time for files. It looks like all projects have this issue, e.g. redux, so I wouldn't worry about that.

I had thought you were using a package manager, so I gave instructions for installing that way. But you can also get the minified JS file on unpkg (no browserification necessary):

• https://unpkg.com/[email protected]/dist/pdf-lib.js
• https://unpkg.com/[email protected]/dist/pdf-lib.min.js

[mlgoff59]

I ran this on the data I was told I could send to you. Here are the resulting file sizes:
PDFKit: 991kb
pdf-lib: 992kb
Acrobat Pro: 230kb

With Acrobat, all I did was open the pdf-lib file and "Save As".

I've got a zip file with all 3 files in it if you want them. But I would need to send it to you privately.

[Hopding]

You can email them to me at andrew.dillon.[email protected] and I’ll take a look.

[Hopding]

I spent some time analyzing the documents that you emailed me. I found a couple of interesting things.

First, the pdf-lib document that you shared doesn't appear to use object streams. When I re-saved the PDFKit document using PDFDocumentWriter.saveToBytesWithObjectStreams, the document size was reduced by 24 kb:

| Saved By | Document Size |
| -------- | ------------- |
| PDFKit | 1014 KB |
| pdf-lib (plain) | 1015 KB |
| pdf-lib (object streams) | 990 KB |
| Acrobat Pro | 234 KB |

So, object streams definitely save space, but not nearly enough to account for the difference between pdf-lib/PDFKit's output and Acrobat Pro's output.

I then proceeded to crack open the documents with pdf-lib's parser and look around at the objects they contained. I logged the type, object number, and size (in bytes) of each object in the documents. Something quite strange jumped out right away: several of the objects in the pdf-lib and PDFKit documents had objects of the exact same size. But none of the objects in the Acrobat document were the same size. I then compared the byte representations of these objects (as they would appear in the PDF file itself), and found that they weren't just the same size, they were actually identical objects.

These were large objects as well:

| Object Size | Number of Duplicates |
| ----------- | -------------------- |
| 28775 Bytes | 14 |
| 6294 Bytes | 43 |
| 1697 Bytes | 46 |
| 120 Bytes | 106 |

I then computed the amount of space used by these duplicate objects:

> 28775 * 14 + 6294 * 43 + 1697 * 46 + 120 * 106
764274

Or 764 KB.

If you subtract the space occupied by these duplicates from pdf-lib's output, you'll notice the difference is nearly equal to the size of Acrobat's output:

  990 KB (pdf-lib output size)
- 764 KB (space used by duplicates)
---------
  226 KB (Acrobat's output is 234 KB)

So, long story short, I think that:

• Whatever tool you are using to embed these SVGs into PDFs is creating a huge number of duplicate objects (at least in terms of the relative space they occupy).
• Acrobat Pro is automatically removing duplicate objects from the files it saves.

So, interestingly, the particular version of the PDF file is irrelevant here. There are really two possible fixes for this:

• Don't embed duplicate objects in the first place (this would require a change to whatever library you're using to embed these SVGs, I think).
• pdf-lib could detect and remove duplicate objects. Not sure exactly what the API for this would look like. It might be a separate tool you'd have to use, e.g. PDFDocumentCompressor.removeDuplicateObjects.

[Hopding]

Also, here are some CSVs I produced with the data on the objects in the PDF files: pdf_document_objects.zip. There are four columns:

| TYPE | OBJECT_NUM | SIZE | MATCHES |
| --- | --- | --- | --- |
| Type of object | ID of object in document | Size of object in bytes | List of duplicate objects' IDs |

[mlgoff59]

Let me take a look at these and get back to you.

I'm using 3 libraries: PDFkit, Svg-to-Pdf (an extentio of PDFKit that embeds the SVG into the PDF) and pdf-lib. I'm going to open an issue with Svg-to-Pdf and see what they can tell me.

Thank you. I'll keep you posted.

Here's a link for reference purposes:
https://github.com/alafr/SVG-to-PDFKit

[Hopding]

Sounds good! I've subscribed to the issue you created in SVG-to-PDFKit, so I'll keep tabs on whatever discussion arises there.

I'll still go ahead and finish up the object stream implementation that I've started here and get that released. I'm not yet sure about the duplicate object detection and removal feature. I'll need to think about how difficult that would be to implement. But it does seem like it would be a useful feature to provide. (I doubt that SVG-to-PDFKit is the only tool out there producing duplicate objects.)

[mlgoff59]

I agree that is the right approach. I would make it a optional parameter defaulted to false. Then is doesn't need a separate API call.

I'm initiating a review of the SVG design/structure used by our sister app to generate the SVG files my tool edits. I suspect that our structure is causing SVG-to-PDFKit to create the duplicate objects. I'll have to create a tool to edit the data structure to test my theory.

[Hopding]

Without knowing the details of your app(s) or having seen the SVGs themselves, I'd say one possibility is that your SVGs have multiple occurrences of a given graphic that is redrawn multiple times in the SVG. It may be that your app(s) are simplying recreating the elements for that SVG element anew each time, instead of creating a group and referencing that when a redraw is needed.

I also came across svgo, which is a tool for cleaning up and optimizing SVG files. You might be able to run this on the SVGs before embedding them in the PDF and see if that reduces the resulting file's size. Not sure if svgo is capable of auto-grouping elements though. So even if the resulting file is no different, that doesn't necessarily guarantee that the problem doesn't lie in the SVGs.

[mlgoff59]

I'll give that a svgo a shot.

My concern is that when I look at the svg, every path element is wrapped in a group element. Thousands of the them are the only element in the group. And no, those single element groups are not being reused, there aren't any use elements referencing them. These are stand-alone path elements that are only rendered once in the map.

[Hopding]

I've completed and merged the code for object streams. PDFDocumentWriter.saveToBytes now uses object streams by default, but it is possible to disable them with the useObjectStream: false option.

I haven't cut a release with this code just yet. I'm planning to do that later today.

---

I went ahead and closed this issue. If you discover there is more that pdf-lib could do to help you, or that it's working incorrectly, please do open another issue. Regarding the "duplicate object detection and removal" feature: I haven't decided for sure when or if I'll implement it, but it's on the list of features I'm considering .

[Hopding]

Have you made any progress on reducing the size of those SVGs, or otherwise preventing duplicate objects from being embedded in the PDFs you're generating?

[Hopding]

Version 0.3.0 is now published. It contains the object streams code. The full release notes are available here.

You can install this new version with npm:

npm install [email protected]

It's also available on unpkg:

• https://unpkg.com/[email protected]/dist/pdf-lib.js
• https://unpkg.com/[email protected]/dist/pdf-lib.min.js