Graal: Graal VM Native image UPX Compression

[bajajyog]

Describe the issue
For me, native image size is 337 MB. I want to deploy native image to AWS lambda. there max size allowed is 250 MB. I used UPX to reduce the image size. in linux AMD 64, after compressing and running the image, following error is coming. let me know if you can provide any clue. i already opened ticket with UPX team for this.

"Fatal error: Failed to create a new Isolate. (code 7)". This works fine on mac os.

Please also advise, how to reduce the image size below 250 MB from 337 MB. Using graal vm 20.1.1 EE on amazon linu 2.

[bajajyog]

graal-native-image-hello-world 2.zip

[oubidar-Abderrahim]

@bajajyog Just to be clear, the native-image was working correctly and without error when it was 337MB, but when you used UPX Compression you got an error. Is that correct?
Also when using UPX Compression you only get the error in Linux AMD 64? not in macOS?

[ezzarghili]

@bajajyog What version of UPX are you using ?

[bajajyog]

I am using UPX 3.96 Linux/amd. native image runs fine without compression on linux. After compression by UPX, it crashes on linux. On mac os, even with compression by UPX, it works fine.

[ezzarghili]

Can you try out UPX 3.94, I remember using UPX 3.95 last year for some demo talks but it had some regressions, switching to 3.94 worked for me at the time.

[bajajyog]

I tried with UPX 3.94, 3.93,3.92, 3.91, no success

[ezzarghili]

I don't think we can help much in this as the generated native image works, the issue is definitely related to the compression method.

But with regards to your original issue with size, you could make the image smaller by actually analyzing what is included in the native image snapshot using GraalVM Dashboard you can find how to use it here

[bajajyog]

Please refer the analysis from UPX for this issue.

https://github.com/upx/upx/issues/405 #

[peter-hofer]

Should be solved by the recently introduced fallback when mapping an image fails: https://github.com/upx/upx/issues/405#issuecomment-688342502

[bajajyog]

I was working on graal vm EE 20.1.1 and UPX was not working for me. Now I am using Graal VM CE latest 20.30 DEV build. There UPX is workring fine.

[bric3]

GraalVM 2020.3.0 announced this issue to be fixed, however on macOS, the produced executable fails.

$ build/native-image/slack-upx
[1]    78262 killed     build/native-image/slack-upx

I noticed the issue https://github.com/upx/upx/issues/405 is closed, and the latest release is 3.96 anyway.

I'm using a gradle plugin to generate the archive, but I can reporduce the issue with :

$ native-image --no-fallback --enable-https -cp build/libs/slack-set-profile-0.1.jar slack.set.profile.SlackSetProfileCommand slack-ni

$ upx -o slack-upx slack-ni
                       Ultimate Packer for eXecutables
                          Copyright (C) 1996 - 2020
UPX 3.96        Markus Oberhumer, Laszlo Molnar & John Reiser   Jan 23rd 2020

        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
  33317320 ->  10833936   32.52%   macho/amd64   slack-upx                     

Packed 1 file.

---

$ upx --version
upx 3.96
UCL data compression library 1.03
zlib data compression library 1.2.11
LZMA SDK version 4.43
Copyright (C) 1996-2020 Markus Franz Xaver Johannes Oberhumer
Copyright (C) 1996-2020 Laszlo Molnar
Copyright (C) 2000-2020 John F. Reiser
Copyright (C) 2002-2020 Jens Medoch
Copyright (C) 1995-2005 Jean-loup Gailly and Mark Adler
Copyright (C) 1999-2006 Igor Pavlov
UPX comes with ABSOLUTELY NO WARRANTY; for details type 'upx -L'.

$  sw_vers
ProductName:    macOS
ProductVersion: 11.0.1
BuildVersion:   20B29

$ java --version
openjdk 11.0.9 2020-10-20
OpenJDK Runtime Environment GraalVM CE 20.3.0 (build 11.0.9+10-jvmci-20.3-b06)
OpenJDK 64-Bit Server VM GraalVM CE 20.3.0 (build 11.0.9+10-jvmci-20.3-b06, mixed mode, sharing)

[bric3]

Here's a simple reproducer

$ echo "public class UpxReproducer {
  public static void main(String... args) { System.out.println(\"hi\"); }
}" > UpxReproducer.java
$ javac UpxReproducer.java
❯ java UpxReproducer
hi
$ native-image --no-fallback -cp . UpxReproducer upx-reproducer
[upx-reproducer:80189]    classlist:   1,269.96 ms,  0.96 GB
[upx-reproducer:80189]        (cap):   2,768.96 ms,  0.96 GB
[upx-reproducer:80189]        setup:   4,097.28 ms,  0.96 GB
[upx-reproducer:80189]     (clinit):     253.89 ms,  1.22 GB
[upx-reproducer:80189]   (typeflow):   4,183.00 ms,  1.22 GB
[upx-reproducer:80189]    (objects):   3,726.80 ms,  1.22 GB
[upx-reproducer:80189]   (features):     212.29 ms,  1.22 GB
[upx-reproducer:80189]     analysis:   8,575.00 ms,  1.22 GB
[upx-reproducer:80189]     universe:     388.26 ms,  1.22 GB
[upx-reproducer:80189]      (parse):   1,134.44 ms,  1.68 GB
[upx-reproducer:80189]     (inline):   1,461.46 ms,  1.68 GB
[upx-reproducer:80189]    (compile):   8,331.42 ms,  2.28 GB
[upx-reproducer:80189]      compile:  11,551.24 ms,  2.28 GB
[upx-reproducer:80189]        image:   3,018.31 ms,  2.28 GB
[upx-reproducer:80189]        write:     545.82 ms,  2.29 GB
[upx-reproducer:80189]      [total]:  29,589.04 ms,  2.29 GB
$ ./upx-reproducer
hi
$ upx -o upxed upx-reproducer
                       Ultimate Packer for eXecutables
                          Copyright (C) 1996 - 2020
UPX 3.96        Markus Oberhumer, Laszlo Molnar & John Reiser   Jan 23rd 2020

        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
   8032768 ->   2273296   28.30%   macho/amd64   upxed                         

Packed 1 file.
$ ./upxed
[1]    80280 killed     ./upxed

Using upx on a go binary is working fine

$ ./bbc
Usage:
  bbc [command]

Available Commands:
  changelog           Display bbc changelog
  completion          Generates completion
  dockerfiles         Handle dockerfile lifecycle (build,test,push,...)
  git                 Manage git repositories and configuration
  github              Github
  grafana             Handle Grafana dashboards
  help                Help about any command
  j2l                 Convert a stream of JSON to human-readable logs
  manifests           Handle manifests
  releases            Show release related information
  scan                Scan for things
  secrets             Handle secrets
  update              Update bbc
  versions            All bbc version relative commands

Flags:
  -h, --help               help for bbc
  -H, --home string        bbc home directory (default "/Users/bric3/.config/bbc")
  -L, --log-level string   Set log level

Use "bbc [command] --help" for more information about a command.

$ ls -lh
Permissions Size User  Date Modified Name
.rwxr-xr-x   40M bric3 18 Nov 18:21  bbc

$ upx -o bbc-upx bbc
                       Ultimate Packer for eXecutables
                          Copyright (C) 1996 - 2020
UPX 3.96        Markus Oberhumer, Laszlo Molnar & John Reiser   Jan 23rd 2020

        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
  40690808 ->  11669520   28.68%   macho/amd64   bbc-upx                       

Packed 1 file.

$ ls -lh
Permissions Size User  Date Modified Name
.rwxr-xr-x   40M bric3 18 Nov 18:21  bbc
.rwxr-xr-x   11M bric3 18 Nov 18:21  bbc-upx

$ ./bbc-upx
Usage:
  bbc-upx [command]

Available Commands:
  changelog           Display bbc changelog
  completion          Generates completion
  dockerfiles         Handle dockerfile lifecycle (build,test,push,...)
  git                 Manage git repositories and configuration
  github              Github
  grafana             Handle Grafana dashboards
  help                Help about any command
  j2l                 Convert a stream of JSON to human-readable logs
  manifests           Handle manifests
  releases            Show release related information
  scan                Scan for things
  secrets             Handle secrets
  update              Update bbc
  versions            All bbc version relative commands

Flags:
  -h, --help               help for bbc
  -H, --home string        bbc home directory (default "/Users/bric3/.config/bbc")
  -L, --log-level string   Set log level

Use "bbc-upx [command] --help" for more information about a command.

[bric3]

For reference, upx on linux works well it seems. I tried the same steps inside oracle/graalvm-ce:20.3.0-java11 docker container :

$ docker run --rm -it oracle/graalvm-ce:20.3.0-java11 bash
bash-4.2# yum install -y -q xz
bash-4.2# curl -sL -o - https://github.com/upx/upx/releases/download/v3.96/upx-3.96-amd64_linux.tar.xz | tar xJ
bash-4.2# gu install native-image
Downloading: Component catalog from www.graalvm.org
Processing Component: Native Image
Downloading: Component native-image: Native Image  from github.com
Installing new component: Native Image (org.graalvm.native-image, version 20.3.0)
Refreshed alternative links in /usr/bin/
bash-4.2# echo "public class UpxReproducer {
>   public static void main(String... args) { System.out.println(\"hi\"); }
> }" > UpxReproducer.java
bash-4.2# javac UpxReproducer.java
bash-4.2# java --version
openjdk 11.0.9 2020-10-20
OpenJDK Runtime Environment GraalVM CE 20.3.0 (build 11.0.9+10-jvmci-20.3-b06)
OpenJDK 64-Bit Server VM GraalVM CE 20.3.0 (build 11.0.9+10-jvmci-20.3-b06, mixed mode, sharing)
bash-4.2# native-image --no-fallback -cp . UpxReproducer upx-reproducer
[upx-reproducer:157]    classlist:   2,382.48 ms,  0.96 GB
[upx-reproducer:157]        (cap):     853.98 ms,  0.96 GB
[upx-reproducer:157]        setup:   2,493.77 ms,  0.96 GB
[upx-reproducer:157]     (clinit):     281.27 ms,  1.10 GB
[upx-reproducer:157]   (typeflow):   7,500.36 ms,  1.10 GB
[upx-reproducer:157]    (objects):   6,048.38 ms,  1.10 GB
[upx-reproducer:157]   (features):     312.68 ms,  1.10 GB
[upx-reproducer:157]     analysis:  14,449.89 ms,  1.10 GB
[upx-reproducer:157]     universe:     579.11 ms,  1.10 GB
[upx-reproducer:157]      (parse):   1,961.16 ms,  1.10 GB
[upx-reproducer:157]     (inline):   1,789.60 ms,  1.03 GB
[upx-reproducer:157]    (compile):  12,364.85 ms,  1.08 GB
[upx-reproducer:157]      compile:  16,889.99 ms,  1.08 GB
[upx-reproducer:157]        image:   1,940.13 ms,  1.09 GB
[upx-reproducer:157]        write:     245.27 ms,  1.09 GB
[upx-reproducer:157]      [total]:  39,226.25 ms,  1.09 GB
bash-4.2# ./upx-reproducer 
hi
bash-4.2# ./upx-3.96-amd64_linux/upx -o upxed upx-reproducer
                       Ultimate Packer for eXecutables
                          Copyright (C) 1996 - 2020
UPX 3.96        Markus Oberhumer, Laszlo Molnar & John Reiser   Jan 23rd 2020

        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
   8916928 ->   2590524   29.05%   linux/amd64   upxed                         

Packed 1 file.
bash-4.2# ./upxed 
hi

@peter-hofer Should this issue be re-opened ? Or should I create a new one since the upx issue is persisting with macOs ?

[peter-hofer]

So now it works on Linux, but is broken on OS X? Is that your experience as well @bajajyog? Either way, given that is appears to crash rather than exiting with an error message, the underlying issue seems different, and a new issue seems warranted.

[bric3]

@peter-hofer OK, thanks for the feedback. I will open a new issue (My timezone tells me to go to sleep, so it'll be another day ;))