electron-builder constructs snap squashfs files that are incompatible with the snap store

[jdstrand]

I am a snap store reviewer. As was reported in https://github.com/electron-userland/electron-builder/issues/2803, at a high-level, the 'resquash tests' consist of unsquashing the uploaded snap, then running mksquashfs on it to generate a new snap, then checking the hash of the original and resquashed snaps. The store therefore expects that mksquashfs is invoked with expected options. Issue #2803 was addressed and fixed electron-builder for some incompatible mksquashfs options (thanks!) and we recently re-enabled our resquashfs tests but noticed that (only) electron snaps continue to fail automated review with messages like this:

$ SNAP_ENFORCE_RESQUASHFS=1 snap-review signal-desktop_1.8.0-beta.1_amd64.snap 
Errors
------
 - security-snap-v2:squashfs_repack_checksum
        checksums do not match. Please ensure the snap is created with either 'snapcraft pack <DIR>' (using snapcraft >= 2.38) or 'mksquashfs <dir> <snap> -noappend -comp xz -all-root -no-xattrs -no-fragments'. If using electron-builder, please upgrade to (at least) 20.9.2 stable.
signal-desktop_1.8.0-beta.1_amd64.snap: FAIL

The reporter used electron-builder >=20.9.2 (see the forum URL for specifics). Looking at this with DEBUG=electron-builder, the squashfs is being built in two stages (reusing appimage code). The first builds with incompatible mksquash options and the second with compatible options (that are ignored), but appending to the first stage. Due to how resquash tests work, the resulting snap can never pass snap store review because of the incompatible options and because '-append' is used (the unsquash/resquash will pack them in a different order).

Output (seen with export DEBUG=electron-builder before npm run build-release -- -l):

...
  • execute command           args=[/home/ubuntu/.cache/electron-builder/appimage/appimage-9.1.0/linux-x64/mksquashfs /home/ubuntu/tmp.0mPdLJ0eNz/latest/release/__snap-x64 /home/ubuntu/tmp.0mPdLJ0eNz/latest/release/signal-desktop_1.9.0-beta.1_amd64.snap -no-progress -quiet -all-root -no-duplicates -no-recovery] path=/home/ubuntu/.cache/electron-builder/appimage/appimage-9.1.0/linux-x64/mksquashfs
  • Found a valid exportable SQUASHFS superblock on /home/ubuntu/tmp.0mPdLJ0eNz/latest/release/signal-desktop_1.9.0-beta.1_amd64.snap.
    Compression used xz
    Inodes are compressed
    Data is compressed
    Fragments are compressed
    Xattrs are compressed
    Fragments are not present in the filesystem
    Always-use-fragments option is not specified
    Duplicates are removed
    Xattrs are not stored
    Filesystem size 62373.43 Kbytes (60.91 Mbytes)
    Block size 131072
    Number of fragments 0
    Number of inodes 10032
    Number of ids 1

Scanning existing filesystem...
Read existing filesystem, 10031 inodes scanned
Appending to existing 4.0 filesystem on /home/ubuntu/tmp.0mPdLJ0eNz/latest/release/signal-desktop_1.9.0-beta.1_amd64.snap, block size 131072
All -b, -noI, -noD, -noF, -noX, no-duplicates, no-fragments, -always-use-fragments,
-exportable and -comp options ignored

If appending is not wanted, please re-run with -noappend specified!

No recovery data option specified.
Skipping saving recovery file.

There seem to be a couple of ways forward:

1. support snaps directly rather than using a 2-stage build based on the existing appimage code. This obviously requires more work up front, but you could then use 'snapcraft pack ./dir' which would guarantee that the squashfs is constructed with snap store-compatible options going forward
2. as an immediate, quick and dirty first step you could take the resulting snap at the end of your two stage build, unpack it then repack it. Eg:
   ```
   $ unsquashfs ./path/to/snap
   $ snapcraft pack ./squashfs-root

or using mksquashfs directly:
$ unsquashfs ./path/to/snap
$ mksquashfs ./squashfs-root ./path/to/new/snap -noappend -comp xz -all-root -no-xattrs -no-fragments
```

There are probably other options than the above. Option 2 is going to be slower to build the snap (I suspect if you built the 2 stage without compression this would speed things up).

As it stands, app snaps built with electron-builder are the only known snaps to fail the resquash test. Due to their popularity, we've temporarily disabled the resquash test in the store, but we need to re-enable this soon and would like electron-builder developers to continue to have a great snap store experience. :)

In developing a fix for this, I suggest that you test the resulting snap with the review-tools snap, which is what the store uses. Eg, on any OS that supports snaps:

$ sudo snap install review-tools --beta
$ SNAP_ENFORCE_RESQUASHFS=1 snap-review /path/to/snap
...

[jdstrand]

Developer workaround:

For electron snap publishers using electron-builder, if this bug is not fixed (or you choice to stay on an older electron-builder) when the resquashfs tests are re-enabled, you can always do 'option 2' from above prior to upload. Ie:

1. build your snap in the normal way
2. unsquashfs ./path/to/snap
3. snapcraft pack ./squashfs-root
4. upload the resulting snap from step 3 to the store

[develar]

Easy workaround: set snap.useTemplateApp to false

"snap": {
  "useTemplateApp": false
}

[jdstrand]

@develar - ah, "useTemplateApp": false worked great:

$ export DEBUG=electron-builder
$ npm run build-release -- -l
...
Priming app 
  • map async                 taskCount=10
  • copy files                from=/home/ubuntu/tmp.GT8qZgEesu/latest/release/linux-unpacked isUseHardLinks=true to=/home/ubuntu/tmp.GT8qZgEesu/latest/release/__snap-x64/prime/app
  • copy files                from=/home/ubuntu/tmp.GT8qZgEesu/latest/release/__snap-x64/command.sh isUseHardLinks=true to=/home/ubuntu/tmp.GT8qZgEesu/latest/release/__snap-x64/prime/command.sh
  • execute command           args=[snapcraft pack /home/ubuntu/tmp.GT8qZgEesu/latest/release/__snap-x64/prime --output /home/ubuntu/tmp.GT8qZgEesu/latest/release/signal-desktop_1.10.0-beta.1_amd64.snap] path=/usr/bin/snapcraft
  • Snapping 'signal-desktop' ...

Snapped /home/ubuntu/tmp.GT8qZgEesu/latest/release/signal-desktop_1.10.0-beta.1_amd64.snap
...

$ SNAP_ENFORCE_RESQUASHFS=1 snap-review ./latest/release/signal-desktop_1.10.0-beta.1_amd64.snap
./latest/release/signal-desktop_1.10.0-beta.1_amd64.snap: pass

I noticed this is using 'snapcraft pack' (perfect!). It seems we now have an 'option 3' that is simply: change default to use:

"snap": {
  "useTemplateApp": false
}

I'll update the review-tools to mention this as well, then we can turn on resquashfs in the store soon.

@develar - do you plan to implement a fix for this? If so, in which electron-builder (so I can update the review-tools accordingly).

[develar]

do you plan to implement a fix for this?

Tomorrow morning (CET).

[develar]

@jdstrand

1. electron-builder uses the same flags as snap. (was fixed by your request #2803).
2. electron-builder generates template snap app using snap.
3. The only reason I see — it is fact, that electron-builder instead of unsquashfs template && merge app with template && mksquashfs result simply appends app to template (mksquashfs allows us simply append).

Why electron-builder use template app? Because build time 5-20 minutes on macOS (docker) or 2-10 minutes on Linux is not good. And it is also important for electron-build-service, that used to build snap on Window (https://github.com/electron-userland/electron-build-service, in the readme of service for snap declared "2-3 minutes" but actually now it is only 5-10 seconds).

Ok — as a solution, electron-builder can do unsquashfs template && merge app with template && mksquashfs result instead of appending. And it will be not so dramatical as not using template app at all, maybe only 30seconds (~1 minute on cloud service).

But question to snapcraft team — is it really required? Maybe it is possible to support such snap? Because mksquashfs image is fully working and valid image.

No doubt — if you will say that it is not possible to support this case and modify check/delta algo, ok, will be fixed in electron-builder side.

[jdstrand]

"But question to snapcraft team — is it really required? Maybe it is possible to support such snap? Because mksquashfs image is fully working and valid image.

No doubt — if you will say that it is not possible to support this case and modify check/delta algo, ok, will be fixed in electron-builder side."

Yes. It has long been a Snap Store security requirement that snaps pass the resquashfs tests and we've been working though many reproducible build/mksquashfs issues to get there. This electron issue is the final (known) issue for app snaps.

I'll note that (obviously) this is only for snaps destined for the Snap Store so appimage code doesn't have to change or anything. That said, the unsquash template/merge/mksquash (ideally snapcraft pack) approach should be totally fine from the store's perspective.

[develar]

Ok, now just copied, will be released today.

````
develar@ubuntu:~$ SNAP_ENFORCE_RESQUASHFS=1 snap-review '/home/develar/Desktop/sep_1.1.0_amd64.snap'

Errors

• lint-snap-v2:app_plugs_plug_reference:sep:desktop
  unknown plugs interface name reference 'desktop'

• lint-snap-v2:app_plugs_plug_reference:sep:desktop-legacy
  unknown plugs interface name reference 'desktop-legacy'

  Warnings

• lint-snap-v2:unknown_field
  unknown entries in snap.yaml: 'grade'
  /home/develar/Desktop/sep_1.1.0_amd64.snap: FAIL

```

does it mean success? Uploading to store is successful.

[jdstrand]

"does it mean success? Uploading to store is successful"

What version of the review-tools did you use? This seems like a very old version... Perhaps 'snap-review' on your system is a local alias or script? Note: the store upload won't indicate resquash failure because the test is still temporarily disabled due to this electron-builder issue.

In the meantime, I've downloaded 'sep' revision r7 from the store and verified locally:

$ SNAP_ENFORCE_RESQUASHFS=1 /snap/bin/review-tools.snap-review ./tSyjhB5F2kemCic6hAsyfRxUAhF3hUJu_7.snap 
./tSyjhB5F2kemCic6hAsyfRxUAhF3hUJu_7.snap: pass

Running with '--json' I see:

$ SNAP_ENFORCE_RESQUASHFS=1 /snap/bin/review-tools.snap-review --json ./tSyjhB5F2kemCic6hAsyfRxUAhF3hUJu_7.snap
...
  "snap.v2_security": {
    "error": {},
    "info": {
...
      "security-snap-v2:squashfs_repack_checksum": {
        "manual_review": false,
        "text": "OK"
      }
    },
    "warn": {}
  }
...

So, yes, this indicates it resquashed properly. Thanks!

So I can add a note to the review-tools, what version of electron-builder is this/will this be in?

[develar]

Thanks for verification. My local version of verifier is for Ubuntu 16 (since I have to test at least on this version). I will update.

20.14.7 is published. To avoid such failures I will use mentioned verifier.

Thank you for cooperation :)