Nuitka one file standalone option

[kayhayen]

Project description: Nuitka has a mode meant for distribution to another system
that puts everything needed in a single folder.

One complaint often raised is that it's a folder rather than a single file, for alternative packaging methods, e.g. py2exe and pyinstaller, these do actually exist, and this project would be about integrating with that.

In a first stage, you would identify the code of these tools that is doing it and try to port it to Nuitka for one or more platforms.

Skills: Python programming, Linux and/or Windows installs of Python, both would be great.

[kayhayen]

There is already #16 for this, where people have expressed there wish to see this. Being busy with implementing the optimizing compiler, we lack the resources to this ourselves, but it would be a great way to contribute for a student, and we would gladly mentor this.

[GiridharPrasath]

hey @kayhayen i would love to work on this.I'm comfortable with C,x86 and python.
I am going through the developer manual and can you help me on where to start with to completely understand the internals

[kayhayen]

That is great. You definitely need to look at:

https://github.com/Nuitka/Nuitka/blob/cb0e0beb72041adeab1065306d5413a6bd868611/nuitka/freezer/Standalone.py#L1247

https://github.com/Nuitka/Nuitka/blob/cb0e0beb72041adeab1065306d5413a6bd868611/nuitka/freezer/Standalone.py#L1384

On the C side, there is code in Nuitka like this:

https://github.com/Nuitka/Nuitka/blob/cb0e0beb72041adeab1065306d5413a6bd868611/nuitka/build/static_src/MainProgram.c#L68

So one task is collection of the dependencies, that all works very well. What then will be needed is to make some sort of self-extracting binary. This part I really have no idea about myself except that these other projects have solved it, and in part it would be your task to find out how they do it @GiridharPrasath

[JorjMcKie]

I can step in with the self extracting stuff a little bit.

Solution 1: General Background

• The dist folder must first be packed (ZIP format or 7zip)
• at the "front" of the the resulting file a small program needs to be glued, that does the following:
  
  
  
  • take the ZIP part attached to its end and unzip it to a temporary folder
  
  
  • execute the EXE file (there is only one!) in that temporary folder, passing any invocation arguments to it.
  
  
  • upon return from the invoked EXE, delete the temp folder again.
  
  

I have a part solution of the above, based on 7zip on Windows. See attached below. It does have a few issues:

1. 7zip must be installed
2. the "frontending" unzipper ("sfx") must be available
3. The resulting selfextracting archive itself is an EXE and behaves exactly like the compiled Python script, except that it requires authorization (Windows UAC). I don't know yet how to resolve this. Probably, I will have to write a self extractor stub myself.
4. Every execution does the unzip again, before the actual compiled program starts - so startup will be delayed by this.
5. This is the type of solution also implemented in PyInstaller.

self-extract.zip

Solution 2: General Background

• This solution needs more investigation
• the frontend stub for the zipped dist folder does not unzip it but directly executes the compiled program from memory
• the stub also establishes a "redirection" such that dependent binary file calls (DLLs, PYDs) will cause the binary to be read / decompressed and loaded as if being read from disk. This technique (developed by some guy Joachim Bauch) is called MemoryDLL.
• MemoryDLL has been implemented in the script language AutoIT, and apparently also in py2exe.

An elegant solution, but as I said, this need more investigation. E.g. I cannot image how this may work if your dist folder also contains data files!

[deajan]

I am using solution 1 for some of my projects, and have several 7zip sfx codes that require or not Windows UAC depending on the manifest (I actually reshacked them).
I must admit that this solution is quite bad, especially when needing to pass arguments to the SFXed executable, which is a real pain in the *
There is also the fact that several 7z sfx implementations exist, with different syntaxes.
Also, this technique is quite slow depending on the distribution size and compress options.

I am in favor for a SFX implementation since Solution 2 would virtually require a ramdisk like squashfs, but 7zip SFX isn't the best option I think.
I'd love to see an implementation that uses LZ4 or zstd in order to achieve good startup speed, but those don't have ready to use SFX implementations as of today.

[JorjMcKie]

The SFX which I included up there, does hand over parameters no problem. I must say, it took me many hours to find it, though. It is also zero-config: it automatically starts the first executable file in the decompressed folder. Seemed quite cool to me: got my test running in a rush.

And (a gooddie especially for you ;-)) the compression ration is much better than UPX: a 48.8 MB dist folder becomes a 12.8 7z file ...

@deajan, if you could send me your SFX examples, that would be great. The big advantage of using 7x is its superb compression ratio, and its very high decompression speed.

If I could get hold of an SFX source, that might be another starting point, once a solution for how to create ramdisks has been found.

[deajan]

Where does your sfx program come from ? 7zip version 9.20 ?
My 7z sfx require a config.txt file that is copied between the sfx and the archive.
Newer version 18.05 does not seem to "just run" the executable inside the archive.

Why not go the NSIS way, which is cross platform compatible ?

[JorjMcKie]

As I said: took me a while to spot this version:
For some reason, develpment of those SFX guys seems to be draining.

[JorjMcKie]

Downloaded NSIS and tried it a little. Looks promising.
Although I could not figure out yet how to start the EXE as part of the "installation". Not to mention how to pass arguments to it ...

[JorjMcKie]

@deajan Update: NSIS was an excellent hint!
I am now able to do the same as those SFX installers:

1. create a 1-file EXE from the script's standalone dist folder - LZMA compressed.
2. when executing this file, the dist is extracted and the script's EXE is executed - whith any invocation parameters passed in.
3. all this completely silent - no pesky Windows UAC popping up.

Now the whole procedure just needs to be packed into a script and we are done with my "Solution 1" from above.

[JorjMcKie]

Just uploaded a first version of a OneFile creator script.

It is based on the (excellent) installer creation tool NSIS, which was suggested by @deajan.
It currently works on Windows - maybe @GiridharPrasath you want to look at it and try extending it to other platforms?

What it does:

• to be used for creating an installation executable from the dist folder of a standalone compiled script
• has the same name as the exe file of the script
• when invoked on the target system, it will decompress itself on the user's Desktop and then invoke script.exe inside the dist folder. So this installation file looks and behaves like the actual program. This includes any command line parameters, that may have to be passed in.
• after finishing the program inside, the folder again deletes itself.

There are other ways to use this installation file:

• you can just decrompress it and extract its content like with any ZIP file (e.g. use 7zip to do that)
• you can invoke it with an extra command line parameter /D= to specify an alternative installation folder.

To use the script, the installation making software NSIS must first be installed. The onefile-maker.py is actually just a front-end to this software: it creates an installation script, which is being compiled by NSIS.

[deajan]

Seems nice, now using this script entirely via commandline should be a viable option so it gets integrated into Standalone.py
Also, a download routine for NSIS will be necessary, otherwise there has to be a --nsis-path= parameter.
At last, some options could be passed as arguments, including:

• AppName
• Company
• Description
• Version
• Icon file

Also, there should be default values for these.

@JorjMcKie Do you want to handle this ? If not, I can step in.

[GiridharPrasath]

hey @JorjMcKie @deajan ,

Now i could completely understand the issue,as i'm a newbie here.
and i assume that this link discusses the same.

Correct me if i'm wrong.and i will look upon to extend to other platforms.

[kayhayen]

That stackoverflow question @GiridharPrasath doesn't do a lot solution. The build script used will also use options that were removed, e.g. --python-version=3.6.

In my mind, the first step, and this is where your research skills would come in, is to find out, how and why pyinstaller --onefile works. Nuitka can freeze bytecode and contain it all in the binary, and make imports of .pyd (Windows) or .so files (Linux, etc all) from the file system.

From a presentation I saw years ago, there is a shared library loader re-implementation that is being used at least by py2exe. So instead of from disk, shared libraries are loaded from the single file binary, which is normally not allowed, and why it takes a whole new implementation on Windows, but fortunately that would have been done already.

I have no idea, how or if they also how file open, to also give file handles to content contained in the binary, but they might also have that.

So do we design from scratch, or do we look at what others already got first. In my mind, pyinstaller --onefile seems to work pretty well. I have seen it used for tools on Windows for distribution even via pip, not sure if that is good or near necessary, but it works. For now I just want to steal/share from there, and doing that research, and integration into Nuitka will be enough of a challenge if you ask me.

But then I am lazy and most of you are not. :-)

[JorjMcKie]

@GiridharPrasath - yes, goes sort of in that direction.
I believe the issue discussed in that link has been addressed by now via this exe-maker2.py script.

And yes, it would be great if you would look into Linux and/or OSX to see how this thing might work there.

[JorjMcKie]

@kayhayen - maybe you have seen my comments on that py2exe solution. Its basis, MemoryDLL goes very deep into Windows specifics, it effectively mimics the DLL loader of Windows. I am very sceptical that there is a similar thing on Linux etc. py2exe consequently only ever worked on Windows as far as I know.

It does not do other things than loading DLL and PYD files. Which means: if my dist folder also contains data files (easily happens, e.g. certifi data, TCL/TK libraries), I am lost.

PyInstaller does what we now are about to do as well, namely making a file that installs itself and immediately afterwards executes the program.

[JorjMcKie]

@deajan -
I agree: quite a lot still needs to be done.

1. Make sure NSIS indeed also is the viable solution for non-Windows platforms

2. Make design decisions:

   • lots of new parameters, exclusively for standalone! It seems to me, some (the company ones) should be fairly constant - the same across several applications. In addition to your parameters, we need to think about e.g. license and copyright. So we should think about creating a publisher / deployment profile type of thing associated with distro generation, which contains the defaults.

   • not every standalone compile should automatically entail creation of distro material. That should be an option. For instance, in all cases I would first want to make sure, that the compile indeed created an executable app, or if anything went wrong, or if unneeded components can be opted out, etc. I believe we therefore do need a separate step being available, too, which makes the distribution after a level of certainty has been achieved via such tests.

   • I am sure not everyone wants a OneFile distro format - in the sense that deploying, automatically also means execution rightaway. I certainly don't. So we need to decide on adequate possible deployment formats, OneFile being just one of them (and probably not the most professional one, allthough an option of its own right).

3. I do agree: we need to decide where to put the NSIS download / installation decision in the gloabl Nuitka context. I remember, that the "old" dependency walker employment is controlled by an option inside the Nuitka invocation. Something comparable seems required again. Best maybe as part of the Nuitka installation itself ...

[deajan]

@JorjMcKie NSIS isn't multiplatform, but allows cross platform compilation
I don't think we'll have a multi platform compatible solution that does it all, since Win/Unix/Mac are way to different OSes :)
On linux, a shar like solution would be fairly simple to implement.

[JorjMcKie]

You are right. I didn't look closely enough before, sorry.
So our new supporter @GiridharPrasath has a bit more to clarify ...
@GiridharPrasath: what is the platform you are working on normally? Linux?

[GiridharPrasath]

@JorjMcKie Yeah,I am using debian(Kali)

[deajan]

@kayhayen The current PyInstaller --onefile implementation is seen as trojan by 11/61 antivirus scanners of virustotal.com, hence rendering it a pretty unusable solution to me.

@JorjMcKie Thinking about it, on Linux a squashfs solution could be perfect. There is always Python PAR format, and more recently Facebook opensource XAR implementation (which btw does zstd compression), see https://code.fb.com/data-infrastructure/xars-a-more-efficient-open-source-system-for-self-contained-executables/

@GiridharPrasath If @kayhayen thinks this could be a good solution, maybe have a look at a Linux XAR implementation for --onefile ?

[deajan]

I think we need two different --onefile implementations, for Unix and Windows worlds.
Windows 'ramdisk' won't be feasable, because it would require installing a system driver on the fly, making all binaries depending on UAC.
Since we're stuck to Windows archive technique, NSIS does all the polish around the archive.

[JorjMcKie]

@deajan, @kayhayen - agree with Orsiris. I have looked into the resp. source code of PyInstaller. It is written in multiplatform C and does what the NSIS solution does: decompressing / copying what it has in its rucksack (to a temp dir), then executing the main script from there, and delete the whole bunch.

And @deajan, yes we do need a totally different onefile software approach on Linux vs Windows - I see that clearer now.

This also underpins my comments on the limited reach of onefile solutions on Windows. While it is certainly good to have this option, it quickly becomes unhandy when we talk about dozens of MB of deployment size - even for simple applications. Being able to select a fast deployment folder (ramdisk / ssd) is probably the best we can offer to people who want this.

@deajan - let me ask you this:
If I understand the squashfs concept correctly, it is a special filesystem, designed for readonly data and in a highly compressed format.
If I deploy a Nuitka file to such a device, it would still have to be (1) decompressed, and then (2) stored on the squashfs device using that device's compression, then (3) executed from the squashfs device, right?
If yes, that would still entail all this overhead for every OneFile execution (because, as per design of the approach, the installation result is deleted again after program execution).

[JorjMcKie]

I have just looked into that XAR thing description. This indeed looks extremely promising!
It does overcome my points in the previous comment. Certainly worthwhile looking at, @GiridharPrasath.

[JorjMcKie]

XAR obviously does not exist for Windows.
But there are container formats, which are mountable as disks (like wim). I will at least do a few checks around this possibility.

[deajan]

@JorjMcKie The XAR format allows to use FUSE to mount squashfs, hence no need to be root, but on Windows, there's no way to mount a wim without being admin, also, it won't be executable without a script.
Hope you find something better, but as of my understandings, there's no other way than a plain old SFX.

[kayhayen]

I hate to raise it, but if we focused on new Linux OS only, do we consider https://flatpak.org/

[deajan]

Flatpak / snap packages need to be installed in order to run whereas sfx / xar / par solutions are simple click and run. This would be a huge drawback for Linux distros that don't support flatpack / snap packages IMHO.

[kayhayen]

I see @deajan I thought they might just run instantly, provided the OS has flatpak support. And given that standalone is often used to address very old systems, OK, no way.

[deajan]

If someone has enough C knowledge, the solution used in cx_freeze could work too, https://github.com/anthony-tuininga/cx_Freeze

There's also a fairly simple solution (the shar like) https://github.com/megastep/makeself which simply includes an archive as binary blob in the file.

[deajan]

And there's AppImage too.... Which would probably solve GUI applications running with a shell.

[JorjMcKie]

@deajan - yes, WIM is not an option because of the inevitable elevated priviledge requirements. But still ISO might be. There is even a Python library to create an ISO image from arbitrary files.
Mounting ISO is possible on more recent Windows systems, and there is thridparty SW available for older ones.
Actually mounting an ISO does not require priviledges. But there would be research required to find out ways how to automatically do that.
And then bot, WIM and ISO do not compress the files by themselves. A UPX run would be required for a dist folder before ISO-ing it ...
A lot of "if" ... and for what?

[deajan]

@JorjMcKie I personally found find it very strange that a simple app would require mounting a virtual drive... That's a lot of headache for not so much, not talking about initial execution speed.
Why not Zoidberg ? Ehhhm I mean NSIS ? It seems to be a good solution for Windows anyway.
(V)(;,,;)(V)

[JorjMcKie]

@deajan - yes, exactly that I was aiming at.
It will do its jobs and it will do it well. It also is extensible enough to evolve the whole rollout of Nuitka applications to something as professional as we ever want.

[JorjMcKie]

@deajan - coming back to your recommendation re https://github.com/megastep/makeself:
Should we ask @GiridharPrasath to look into this?

@deajan, @kayhayen

• I made another change to onefile-maker.py: if the installation file is invoked with parameter /D=<folder>, it now decompresses the dist folder into that directory and exits. If invoked without this parameter, it will decompress into the user's temp directory and execute script.exe (= the compiled script.py) and pass its own invocation parameters to it. After this, the temp dist folder is removed again.

• I was thinking about @deajan's suggestion to include more parameters in the installation material generator (aka onefile-maker.py). And indeed we also need logic to include the potencial download and parameterization of things like NSIS, ...
  While I believe he is very right, the explosive growth of parameters in every corner should be a concern!
  How do you feel about designing some type of parameter store for all of this? It should come pre-populated with defaults during Nuitka's installation, and be modifyable later by the user.
  
  
  
  • it should support sections for various topics like download approvals (Dependency Walker, NSIS, ...), standard standalone parameters (experimental flags, user/company standards, copyrights, licenses, logos, ...), execution of UPX, etc etc
  
  
  • it could be based on some standard format (INI file, JSON, SQLITE) and reside in the user's home folder
  
  
  • we would need a maintenance component for this
  
  
  • Standalone.py could pick the variables from there, the command line would only contain overwrites, extra scripts we have now (exe-maker, onefile-maker, ...) or may need in future could be invoked (or triggered) automatically based on those values
  
  

[akshai9899]

Hey @kayhayen and @JorjMcKie , I would like to work on this if that's okay :)

So from this comments thread and my searching, this is what I got, correct me if I'm wrong.

• We want to put the output dist folder into a single file that can be executed directly.

• Solution 1 is making a sfx file which can extract itself into a temporary directory and execute the file in it, preferably it'd be good if we can provide arguments to it

• Solution 2 is using MemoryDLL by joachim as in py2exe and this gives a nice overview of it https://sourceforge.net/p/py2exe/mailman/message/1390732/

• The solution 1 in using NSIS which can be compiled in both Linux and windows but the generated installers from NSIS works on windows only.

• XAR seemed to be a nice solution to me but it's not so useful in Windows as suggested in comments

And this is what I did and I am using Debian Ubuntu 16.04, I also have windows 10 if it's required.

1. Using the standalone option, on one of the test files in tests/standalone made a .dist folder.
2. And using this repo https://github.com/megastep/makeself as suggested by @deajan . I was able put everything into a .sh file giving it the option to execute the required executable file after extracting. Also optionally one can provide a script(shell script) during the making of the .sh file to be executed after extracting.
3. And this worked nicely! also one can provide arguments to the .sh file that can be sent as arguments to the script embedded in .sh file.
   Still more tests are to do be done.

What this repo does in detail :

It has a shell script that can generate a self-extractable compressed tar archive from a directory. The resulting file appears as a shell script. Using this file, the archive will then uncompress itself to a temporary directory and an optional arbitrary command or a script(shell script) is executed. Being just a shell script it is portable to most of the systems.

So this can be the solution 1 in Linux

Also I'd be happy if someone points the direction to work on further

[JorjMcKie]

Hi @akshai9899 - this sounds just great!
Taking "your" (makeself) and "my" (NSIS) solution together, we have covered Linux (plus others IX systems) and Windows systems.
This brings us close to fixing the issue.

Commenting the "Solution 1 & 2" discussion:
As I said previously: there will be no solution 2 on Windows any time soon, because data files cannot be covered by something like MemoryDLL (only works with binaries similar to .so libraries on Linux).

If this XAR thing is generally available on Linux (which it isn't yet, I understood), then it indeed represents my "solution-2" definition, because it covers all sorts of data (shared library binaries and data files).
I am not aware of anything close to this on Windows.

So conclusion would be to wrap up what we do have (with Mac OSX still being a missing puzzle piece):

The dist folders, the results of standalone compilations, can be converted into installation files, which can be either deployed to the target system or executed there like normal programs.

What is left to be done is cleaning up some things:

1. decide / implement how to connect compiles with this creation of installation files:
   
   
   
   • via an additional option in the standalone compile
   
   
   • as a separate run
   
   
2. design / implement on additional information to be included in deployment data (see @deajan's comments above): company name, copyright, license, application name, ...
3. investigate what can be done on Mac OSX

If you could pick up points 1 and 2 and come up with suggestions, that would be great.

@deajan - what is your assessment on this?

[deajan]

Seems good so far :)
Now IMO, the --standalone switch should have another subswitch like --singlefile which triggers a separate Python file that does the makeself / NSIS magic.
Since both Linux and Windows methods are completly different, they should be implemented in two different files, ie Singlefile-unix.py and Singlefile-win.py
The windows version could accept a simple JSON file with the required additional information.

@JorjMcKie I'll have to deal with NSIS these days in order to generate some other files (nothing to do with Nuitka), is there any help I can provide ?

[JorjMcKie]

@deajan - I agree ... with clarifications:

• what is the difference between OneFile (SingleFile) and other packed formats? As I wrote somewhere else: if you specify a target directory via the /D=<folder> parameter, it will decompress the dist into it und fertig. Potentially, the NSIS / makeself stub could be marginally modified thus, that it "knows" it must ask for <folder> if it is not OneFile.
• additional parameters (like you named a few):
  
  
  
  • where should they go? As data files copied into the dist folder?
  
  
  • why only Windows - Linux should have the same requirement, shouldn't it? Of course with some exceptions like console (although MAC seems to have that too) and the app icon, but both are already options of Standalone.py.
  
  

As per NSIS: thanks for the offer. I'll come back when questions pop up. I am still digging into it ...

[JorjMcKie]

@akshai9899 - any update on this?
How about making a script similar to onefile-maker.py for Linux?

What I am going to change in this version for Windows:

• include options in the GUI to either create an executable as I have it now, or to create an executable which allows entering a desired folder at the installing site
• include more meta information (see @deajan's and my comments above) like app-name / description, company-name, license, ... It is still unclear how to deal with this information: should they be pointers to text files, which then must be copied into the dist folder ...? Your ideas welcome!

After we have OneFile maker versions for Linux & Windows, I want to extend the standalone portion of Nuitka compile with options to trigger OneFile creation, and potentially more.

[akshai9899]

@JorjMcKie Sorry, I started it but I had to stop with my University semester examinations going on :(
I will send a pull request by the start of the next week, and as my exams will end by then and I can work more freely on Nuitika on extending the standalone and more

This was my plan so far on the onefile for Linux (I will be adding more next week along with the initial pull request) :

• Check if a download of https://github.com/megastep/makeself exists in present dir or if a path is given, if not download it. Including options for the executable and the embedded script (quite a few choices here). I'll mostly try to keep options the same way as Windows version so it won't confuse users.

• I'm thinking of executable --help to get info about publisher and information as this seems more Linux. I don't know if this is a good idea though.

[JorjMcKie]

@akshai9899 - no worries, your education certainly is top priority!

Your plans look good. Displaying help is a good idea, too. How "professional" we want to

In my mind, standalone compilation's (let me abbreviate it: SAC) purpose in life is to produce installation material for other sites (which may have a different configuration than the origin).
This material always is one executable file (with attached-to bundled, compressed stuff).
There are 2-flavor format differences only pertaining to: what happens if that file is executed. Either

1. the embedded script is executed with any arguments passed through to it
2. the executable decompresses its content to a given location and in general completes the installation (platform specific variances).

The decision between the alternatives is made on the creating site - as well as some more sub-alternatives maybe on the Linux side (which IX-variants to support via different comrpession formats and what not). Iwouldn't be too concerned about the last point: people compiling stuff should know their platform and be used to it - subtly different Nuitka behaviour on some other platform is a matter of course.

Please notify me when you have more time again to discuss additional design approaches, e.g. for keeping the Nuitka parameter inflation under control by using configuration profiles.

[JorjMcKie]

@akshai9899 - here is work waiting for you upon return ;-)

I hope your exams went well and everything is fine!

I made a work program which enumerates things already done and open points. Please assume TODOs marked "Linux" are yours.
Feel free to contact me any time with questions. Feel free to use my email jorj.x.[email protected].
Please also let me know if you have no spare time to help. Will not cause bad feelings - I would just have to look around for help then.
standalone-todo.zip

[akshai9899]

@JorjMcKie I did well, Thank you! My exams ended and I am free now.

I have sent a initial pull request of the onefile-linux. Please tell me any changes I have to make.

[feluxe]

Is there any progress on this? I'd love to migrate every command line tool in my org from PyInstaller to Nuitka. This issue is the last thing holding me back.

[JorjMcKie]

The solution in the Nuitka-Utilities repo should be usable. I have successfully used it on Windows and I believe it also works on Linux.

[feluxe]

Thanks for your reply! This looks promising. Are there any plans to deploy the utilities on pypi?

[jayvdb]

I didnt see any mention of staticx anywhere. It is currently a linux only solution, and I havent got it working yet, but it might be worth the effort to see if it can convert a Nuitka one-dir into a one-file.

[byehack]

Solution 3: PyInstaller method

1. compile the python codes in c codes like now.
2. when we use --standalone, we have some .dll and .pyd files in dist folder.
3. we pack all .dll and .pyd files after detect them.
4. convert packed files to string.
5. add a script in c language to above of main app. this script do these:
   a. after convert test.py to test.c put (.dll, .pyd packed files) string in first of test.c.
   b. make random folder in temp with _MEI prefix.
   c. extract all packed files in the _MEI folder via string of (.dll, .pyd packed files).
   d. convert import sys;sys._MEIPASS = <_MEI address> to c code and put it in first of test.c.
   e. convert import sys;sys.path.insert(0,sys._MEIPASS) to c code and put it in first of test.c.
   f. continue of program...

For remove _MEI we have few ways:

1. the best way is don't use _MEI folder. use _MEI zip file instead. as i know the sys.path of python supports zip files too. so in this method we do these:
   a. just use tempfile file module in the top. i tested: test.exe terminated from task manager and zip file removed!
   b. ignore b & c in level 5 and convert import tempfile;x=tempfile.TemporaryFile(prefix="_MEI", suffix=".zip") to c code and put it in first of test.c.
   c. write string of (.dll, .pyd packed files) in x.name address.
   d. instead d in level 5, convert import sys;sys._MEIPASS = x.name to c code and put it in first of test.c.
2. as i said the way 1 is the best way but another way is a another executable runs and if test.exe closed it removes _MEI folder and exit.
3. another way is nuitka don't do anything. Let the user do this.( i use this for pyinstaller. at first kill the parent guard and add script at end to remove _MEI folder)

this is not a sfx! this just make all packages and dependents ready for continue of test app without any problem.

for the Solution 1 i should say the sfx with nsis has a lot of problems. in my project i have these issues with nsis sfx method with your onefile-maker.py script:

1. os.getcwd() problem. current work directory sets to temp folder.
2. antivirus detection.

i don't understand c language, if i know it i was send you a PR to support one-file method.
i just send you a suggestion, i hope this works.

i think doing this work is so easy for who know c language and nuitka project well.
(i think, maximum it gets 1 or 2 hour. please tell me if i'm wrong.)

[kayhayen]

As for extracting to a temp folder, that is not what I want, I would much rather see this:

https://pypi.org/project/py2exe/#the-bundle-files-option-explained

--bundle-files 2 will include all the Python extensions into the appended zip-archive; they are loaded via special code at runtime without being unpacked to the file-system

I don't want to invest effort into this myself, and @JorjMcKie also seems uninclined. We may reconsider this for GSoC 2020 if we get to participate. But either way, what you describe would only be a last resort fallback, because as py2exe docs state, not all programs work when loaded from non-filesystem.

[kayhayen]

@byehack Absolutely nothing in Nuitka is 1h in my experience.

[byehack]

@byehack Absolutely nothing in Nuitka is 1h in my experience.

i wanted to say this method is so easy for nuitka developers.

[byehack]

As for extracting to a temp folder, that is not what I want, I would much rather see this:

https://pypi.org/project/py2exe/#the-bundle-files-option-explained

ok. i understood. but can't you write a script to do this? (without directly add this option to nuitka. like onefile-maker.py script)

[RESP3CT88]

I would love this one file standalone option

[psytron]

+1 for --onefile. This would be amazing! The current --standalone option is nice, but if the app is moved to another directory it doesn't work anymore..

[deajan]

I was initially in favor for the ramdisk option, which would allow virtually any program to run as "onefile", but won't keep any modifications that would be written to that ramdrive after execution, and would need a cleanup script that removes the ramdrive after usage.
Also this would require admin privileges.

The second best solution is a SFX that extracts everything to a temp dir then executes the program.
Zstd would be amazing for decompression speed reasons, but we'd need to find a valid SFX implementation.
This solution would also work with any program, and does not require admin privileges.

The relocate dll calls to zip file option that is the py2exe would not work in all usecases, especially when third party dlls are used.

IMHO the SFX to temp solution is the less hacky one.

RFC :)

[deajan]

I had a quick talk with @mcmilk here https://github.com/mcmilk/7-Zip-zstd/issues/119 about his 7zip fork.
There is an option with 7Zip SFX to support zstd decompression for blazing fast Nuitka SFX files.

If we include two binary blobs of the SFX decompressor in Nuitka (as external downloads ?), we could come up with zstd python bindings to create a zstd compressed file and prepend the SFX decompressor for a full blown SFX archive that would solve both Windows and Linux --one-file option.
The only thing missing would be to find a ressource hacking tool for Windows (pefile ?) and Linux (?) to change the SFX decompressor icon on the fly.
Perhaps antother solution would be to include SFX decompressor compilation to Nuitka, in order to control the icon directly, also getting rid of binary blobs.

I can pledge to add the blob solution, I just don't feel comfortable enough with C to deal with the sfx decompressor compilation in Nuitka myself.

@JorjMcKie @kayhayen Any thoughts ?

[PttCodingMan]

I would love this one file standalone option.

[arrudagates]

I would love the one file option too, currently trying the nuitka tools scripts but i can't seem to get them to work right.

[kayhayen]

@deajan I am OK with including a binary blob for Windows unless it's too big, but if it needs frequent updates, lets put it maybe on the Nuitka website or so. The LGPL license is not a problem for all of that.

Fast decompression is very important, and if you can get that done with good compression, go for it, and I want this for the proper Nuitka core to be present. This will be a Windows only solution though, is that correct?

Nuitka code can update resources of all kinds already, exception are icons, I think. Icons are an issue that I want to work on, mostly because of multi icon support, I think that is a bit tedious, but nothing undoable. I could do it in a day or so I guess. The data structures there are a bit annoying. But also Nuitka should get rid of need for rc.exe which currently is only used for attaching icons, but doesn't support all icon types...

Checkout nuitka.PostProcessing where nuitka.utils.WindowsResources is used. My personal roadmap is filled right now with the constants rework. But I want to get there eventually, but realistically not before many months there is a chance. But I can assist you in that, and explain it.

There is a issue for that #226 and I could add more detail to it, if you are willing to take that one on too.

[cshandley-uk]

I'm not sure the above solutions would work for my use case, which is having a standalone command line executable for Windows (i.e. executed from a Command Prompt, Power Shell or Batch file).

I think there is a POSSIBLE ALTERNATIVE IMPLEMENTATION that would work for my use case. It's likely more complex to implement than the other solutions discussed above, but I thought I'd mention it anyway, just so the possibility it isn't entirely overlooked.

DrDobb's magazine documented something called "DLLpack" for C++ Windows in 2002, which allowed embedding DLLs in an executable, temporarily unpacking & loading them at run-time, and then automagically deleting them when the program finishes:
https://www.drdobbs.com/packing-dlls-in-your-exe/184416443

It doesn't sound cross-platform, but I don't see why a similar solution wouldn't be possible for Linux.

Unfortunately the source code Listings don't seem to be available online any more (as far as I could see), but on the other hand they do give enough of an explanation of the implementation that I imagine someone familiar with low-level Windows coding (not me!) might be able to reimplement it.

[probonopd]

As for Linux, this can be done using AppImage.

Similar to how Python can be distributed as a single-file executable for Linux:
https://github.com/niess/python-appimage/releases/tag/python3.8

[kayhayen]

Looks perfect to me @probonopd and would you allow me to kind of wrap this, downloading your standalone executable of appimage? It seems it would be trivial to integrate that way. I already do something similar for dependency walker. I just would need to have an efficient way of knowing of updates, either at user side, or myself hard coding a version to use.

[probonopd]

Sure, as far as I am concerned, feel free to use anything you need - the code in the AppImage project is under the MIT license. https://github.com/niess/python-appimage is under the GPL,

[kayhayen]

I think I saw your name on the website, assuming it's your code the AppImage. I don't think I will make people build it, as that's never easy, and for python-appimage, Nuitka already has its own implementation of a Python runtime, so I won't use that.

But great, then I expect to do that really soon now. I will not put it in the next release yet, mostly due to be only trying to finalize it as soon as possible, but definitely into one after that. It's a Linux solution only, but that's a lot. Now let me know if you ever migrate it it Windows. ;-)

[kayhayen]

So, this is on the factory branch for Linux, using AppImage for Linux. It downloads the standalone version of AppImage to use it, which kind of is funny thing to have. It's called --onefile there, Windows will follow suite eventually I hope.

[kayhayen]

The help wanted now mainly refers to ideas how to do it for Windows.

[Systemcluster]

I wrote this application for the purpose of creating and distributing single-file applications. I wrote it with the intent to use it with PyInstaller builds (as PyInstallers own onefile mode is very slow), but it's general-purpose. If there is interest I could create a python wheel for it.

[kayhayen]

@Systemcluster I don't quite like that it's not written in Python, also, do you have an SFX link for it, so I can try it out without building anything, which would be what I want for my users, who often have no clue what a C compiler is.

[kayhayen]

This is for Windows only now. I think almost everything that uses 7zip runs a risk of license troubles. The single file case is not helped by having to reproduce the license texts and statements of using the software in the documentation.

However, there is the "LZMA SDK" and that is public domain, and we can use that though: https://www.7-zip.org/sdk.html and create a small bootstrap binary of our own making, which would be used to extract to a temp folder, and run from there the compiled binary. I guess, that means it's not good compression, but such is life. But according to https://en.wikipedia.org/wiki/Lempel%E2%80%93Ziv%E2%80%93Markov_chain_algorithm this aint so bad.

@deajan Nuitka has the code to work with icons, version resources, on the factory branch now. Updating an executable to work with that will not be an issue. When Nuitka creates --onefile binary, it can just decide to work on that one to add icons and version information, rather than on the one in the dist folder, so these things are already sorted now.

[kayhayen]

This is in the current develop version and can be tried out with Linux, might not be all that smooth due to its insistence on icon from a system path, but it sort of works now. Windows will follow later.

[probonopd]

So, this is on the factory branch for Linux, using AppImage for Linux. It downloads the standalone version of AppImage to use it, which kind of is funny thing to have. It's called --onefile there, Windows will follow suite eventually I hope.

Awesome!

[gabrieldp]

Has anyone taken a look at PyOxidizer?
From what I understand, it seems to be capable of creating single executable files, loading stuff from memory (not extracting code to a temporary directory and running from there). It does add Rust as a dependency, though, but I'm not sure how difficult it would be use the same mechanism in something like C or C++.

[JorjMcKie]

Has anyone taken a look at PyOxidizer?

Only after you mentioned it. I do not see any new aspect there to be honest - compared with all the alternatives we have been discussing or even implemented.
The annoying aspect has always been, that the created standalone one-file must be extracted to a temporary folder in the target environment, from where it is then executed.
PyOxidizer adds nothing new here - we have to remember, that the installation material by no means consists of executable code only: there more often than not are configuration files like certifi data and what not else.
What we need is a one-file format that can be mounted as a virtual disk in the target.
There are ways that work like this on Linux et al. - interestingly one is among them that is based on the MS Windows container format WIM (Windows Image) - have to dig out that thing. It works for all platforms we want to support, not only Windows.
Ironically, the ever present stumblestone is the Windows OS: technically everything is possible, but mounting a WIM requires elevated priviledges ...

[franklx]

PyOxidizer adds nothing new here

PyOxidixer doesn't embed a virtal filesystem to be mounted (like appimage) nor transparently unzips modules in a temporary directory.
PyOxidixer authors modified the Python import system loading binary extensions directly from executable without resorting to overmentined techniques.

[muhammeteminturgut]

@JorjMcKie PyOxidizer isn't extracting a temporary folder. Somehow it can create fully statically linked binaries. It also does this in a multiplatform way. Currently the only python distribution tool that manages to do this.
https://pyoxidizer.readthedocs.io/en/latest/packaging_static_linking.html

[JorjMcKie]

@muhammeteminturgut - I do not doubt that. I am also sure that RUST in general has a great future and will replace many C/C++ applications. I also have had some insight on promising RustPython.

But it was not my point. My point was: hundreds of Python packages come with config files, time zone and other data. The most important and prominent ones among them. Just have a look at the data plugin to see all the celebrity guests we have in there.
Can PyOxidizer handle all these data as well in the same one-file?
If yes: _chapeau!_ - I missed a really important point, my fault, let's deep-dive into it!
If not: see my above assessment. Not worth introducing another variable into the game for no actual progress.

[kayhayen]

I am pretty sure that the overview page I just read doesn't imply it can @JorjMcKie

However, using pkgutil and being closer friends with it, one might be able to get data files from there. But last i checked, the interface didn't e.g. allow to get a file handle, just the data, which is horribly inconvenient. And then it's no wonder, that packages, last I saw matplotlib, load their data via __file__ dirname tricks instead.

The Nuitka onefile for Windows is going to use a folder that is cached (in the future), getting you good start up times. The one for Linux does that really nice virtual file system, where no file ever hits a disk. But for Windows, it's unavoidable it seems. At least if you want to be compatible by any stretch.

Also, I have not yet seen any Python on Windows, that doesn't use the DLL and manages to be compatible outside of self compiled stuff. Not sure really why, but static linking is out of the question entirely on this platform. Honestly I never quiet got why, but I think it has to do with extension module loading. Maybe they can't take exported functions from a program, but only a DLL.

Otherwise, there are ways to hook functions. One clearly can intercept the CreateFile Windows API and smuggle results to it, but that would be a hell of effort. Effectly creating a virtual file system without using one. Nobody will ever do that. But lets see, maybe in the future they (Microsoft) are going to add a VFS layer to Windows. They very well could.

[kayhayen]

Released as part of 0.6.10, but mind you, there is no support for adding data files yet, so this is not really usable for everybody yet.