Aws-cli: Windows AWSCLI64PY3 MSI not shipping aws.exe anymore breaks scripts

[DanielLaberge]

The new CLI version on Windows does not install the 'aws.exe' file anymore, instead it's aws.cmd.

This breaks bash scripts running in Windows because bash does not resolve .cmd files as executable.

Would you consider shipping an .exe wrapper?

It's nice being able to execute .sh scripts on Linux and Windows unchanged, but now this forces us to change our calls to 'aws.cmd' rather than just 'aws'

[justnance]

@DanielLaberge - Thank you for reaching out. Please respond with CLI version in use.

[DanielLaberge]

aws-cli/1.16.37 Python/3.6.0 Windows/10 botocore/1.12.27

Installed from this package: AWSCLI64PY3.msi linked from this page

I've noted that the package name changed, it used to be 'AWSCLI64.msi' until very recently.

[angelaman]

we ran into same problem too. Is there any reason the exe is removed?

[bazwilliams]

We're running aws-cli/1.16.40 Python/3.6.0 Windows/10 botocore/1.12.30 and we our Cake build scripts no longer run as a result of this change.

Deploying the cli tools from this link solved our issue for the moment:
https://s3.amazonaws.com/aws-cli/AWSCLI64.msi

Interestingly --version gives:
aws-cli/1.16.40 Python/2.7.15 Windows/10 botocore/1.12.30

[DanielLaberge]

This is also what I did to work around the issue, some of the AWS documentation pages still link to the AWSCLI64.msi package, which includes aws.exe.

Thanks bazwilliams, I should have mentioned this workaround in the OP, my bad.

[justnance]

@DanielLaberge, @angelaman, and @bazwilliams - Thanks for your feedback. I'm curious if you already know about this page? It links to a download to the AWS CLI setup files which includes both the 32-bit and 64-bit MSI installers and will automatically install the correct version. The download links to https://s3.amazonaws.com/aws-cli/AWSCLISetup.exe.

[bazwilliams]

I tried the automated installed you suggested, but that still installs a version of the cli tools without the aws.exe binary.

[no-response[bot]]

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.

[angelaman]

@justnance Like @bazwilliams replied, your solution doesn't seems to work. Not sure why this ticket is closing. Could you reopen it again?

[bazwilliams]

I think we need @DanielLaberge to reply so the bot opens it again.

[DanielLaberge]

Can someone else take ownership of this issue so I don't have to reply every other day?

The nature of the problem doesn't require this bot...

[gfody]

after taking this update our scripts stop executing after the first call to aws. because aws is now a .cmd file it requires the "call" prefix when being called from batch or cmd files, without it the calling script stops executing.

this will also break Windows apps using start process w/o shell execute to shell out to aws.exe - whatever the benefits of dropping aws.exe are please consider them against introducing this breaking change to windows environments. my vote would be to restore aws.exe

[justnance]

All - We have confirmed this is a bug. The behavior is expected but unintentional. The work around at this time is to use the CLI with Python/2.7.15 instead of Python/3.6.0.

[justnance]

To be clear, the CLI MSI version with the issue is our newest MSI versions built against python 3.6. We still publish the older MSI versions which can used as a work around while resolve this issue.

[oliversalzburg]

The work around at this time is to use the CLI with Python/2.7.15 instead of Python/3.6.0.

That workaround is not valid, as the 2.x version suffers from other defects.

[oliversalzburg]

I'd recommend dropping a file names aws into the bin folder with the following contents:

#!/usr/bin/env bash

self=`readlink -f "$0"`
basedir=`dirname "$self"`

"$basedir/../runtime/python.exe" -m awscli "$@"

[victorbiga]

I got this and it seems to be working fine

$vi .bashrc

add the below
alias aws='aws.cmd'

[justnance]

Work has been done to resolve this issue. Please install the latest version of the CLI and let us know if it is working as expected now. Thanks for your patience.

[oliversalzburg]

That is pretty unspecific. Also this:

[JordonPhillips]

Interesting. What Windows version and which installer are you using?

[oliversalzburg]

@JordonPhillips Windows 10 17763.134. For the MSI I only have a revision number {94E88177-499F-4DF5-8D96-25FDD46EA1C6}, created 2019-02-12 01:00

It seems like at the end of the installation, an external application is launched (I briefly see a console window pop open) and then I get the message above. I can create a new ticket if that helps.

[JordonPhillips]

To make the exe work we rely on a file that needs an absolute path to the bundled python, as relative paths are interpreted as relative to the current working directory. (CMD has a syntactic workaround, which is used in aws.cmd, but that's not available here.) So what's happening at the end of the install is we're running shell commands to update that path.

This is what we're doing:

cmd.exe /c echo "[AWSCLI64PY3_RUNTIME]python.exe" > .\bin\tmp.txt
cmd.exe /c type .\bin\aws-script.py >> .\bin\tmp.txt
cmd.exe /c move /y .\bin\tmp.txt .\bin\aws-script.py

With the [AWSCLI64PY3_RUNTIME] bit being a templating thing that just returns the absolute path to the runtime folder. I know that looks a bit hacky, but it's actually what setuptools does and more or less what pip does (pip appends the script to the end of the exe though).

We run tests on these MSIs as part of the release, so I'm a bit surprised that there's an install-time failure. I wonder if there's any material difference between Windows Server and Windows that's causing the issue? My other hunch is that it's some sort of permissions issue causing one of those commands to fail.

[stomplee]

I got the exact same error as above when installing the package on Windows. Something must have changed just this week because last week it was working just fine. I got around the error by not using the .msi package and instead downloading the .exe version. When I ran the .exe version I saw it installing some dotnet dependencies and I'm guessing those same dependencies may be missing in the MSI installer?

[JordonPhillips]

It might be an upgrade issue with the exe changes. Try uninstalling / reinstalling if you see this.

[oliversalzburg]

@JordonPhillips I actually saw the .\bin\aws-script.py during the install. I had investigated a bit, because I assumed my own aws.sh in the bin directory was causing a conflict. I removed it and also tried a couple more things to try to get the installation to pass, but to no avail.

So the file was actually created in the target directory. Not sure what could have gone wrong after that.

[JordonPhillips]

The file exists as part of the normal install - we just prepend the path to it. If you crack it open it should have an absolute path to the python interpreter at the beginning. If the commands failed then it won't be there.

[oliversalzburg]

At the time where I get the error message, the contents of the .\bin\aws-script.py are:

# -*- coding: utf-8 -*-
import sys
from awscli.clidriver import main
if __name__ == '__main__':
    sys.argv[0] = 'aws'
    sys.exit(main())

I'm going to investigate a bit more...

[JordonPhillips]

Yeah we managed to get some logs from another user and it looks like it's failing on the first line of that script I pasted above. I'm leaning more towards a permission issue. Is your user account and administrator account?

[oliversalzburg]

I'm an administrator

[JordonPhillips]

What happens if you run cmd.exe /c echo "foo" > .\bin\tmp.txt in the install directory?

[oliversalzburg]

Weird. When I'm checking the Security tab for the bin folder, I get the following message:

---------------------------
Windows Security
---------------------------
The permissions on bin are incorrectly ordered, which may cause some entries to be ineffective.
---------------------------
OK   
---------------------------

When I run the command you suggested, from an elevated PowerShell, it works fine. This is still in the state where I see the error message during the installation.

[oliversalzburg]

From an unelevated PowerShell instance, it fails with an UnauthorizedAccessException. But I guess that is to be expected.

[JordonPhillips]

I wonder if the issue is calling cmd /c to run the command? Maybe that doesn't keep the admin.

[oliversalzburg]

I think the cmd.exe process itself is not spawned elevated:

The msiexec.exe also is not.

But it does ask for elevation during installation to make changes to the Program Files folder. I would expect that the cmd.exe process is not created with the same privileges.

[oliversalzburg]

Yeah, seems like that's where it switches from the Local System user context into my own. Hope that helps.

Does removing and re-installing really work? It seems like this would also be an issue on fresh installations. I don't want to be left without a CLI

[JordonPhillips]

I don't think so, but you can use either the python 2 msis or older versions of the python 3 msis.

Python 2: https://s3.amazonaws.com/aws-cli/AWSCLI64.msi
Older Python 3: https://s3.amazonaws.com/aws-cli/AWSCLI64PY3-1.16.101.msi

[JordonPhillips]

We're still working on a fix, but in the meantime it seems if you run cmd / powershell as an administrator and then run msiexec from there it works.

[JordonPhillips]

We have a fix in place, which will go out with the next release. These things tend to happen daily so that will probably be tomorrow.

The issue was that by default wix (the toolset used to build msi installers) will run cmd commands using the user privileges of the user that ran the installer rather than the admin role that is assumed to perform the rest of the install. So we just had to tell it to use the same permissions as the rest of the install.

[oliversalzburg]

Awesome. Sounds great :)

The version that is up for download at this time, did not yet contain the fix apparently. I will check again later for a newer revision.

[JordonPhillips]

@oliversalzburg A release just went out with the fix.

[JordonPhillips]

Also to give some context on the original issue (the lack of an exe) since it doesn't seem to be here:

For the python 2 msis we use a tool called py2exe to generate executables. Essentially what that does is it compiles (almost) everything down to the python byte code and then bundles in enough to get an exe working. It more or less works, but the project hasn't seen updates in a long time and doesn't work with Python 3.

For Python 3.6+ the PSF vends a self-contained zip version of python meant to be bundled into applications. So when we went about the work of switching installers to python 3 we decided to use that instead, as it way simpler since we don't have to rely on a third party dependency.

The side effect of doing that was that we no longer had an exe to point to because the CLI doesn't use entry_points. Even if we did use entry_points in general we would still not have a working exe because the exe generated by pip has an absolute path to the python executable baked in. We install all the packages into the bundled python at build time, so that would be whatever the path is on our build box. Unless you had the exact same setup including user name then it wouldn't work.

So what we did was put in a cmd script in a folder that uses a path relative to the file to call the cli with the bundled python. We assumed this would be fine since generally Windows will search for items in the path without looking at the extension if you haven't provided it while working in the terminal. Clearly that was not a correct assumption to make.

So to fix that we looked at what setuptools and pip do for entry_points. Basically they've created exe's that will call out to a special python script. The top of the script is a shebang with an absolute path to the python to use. The problem is that the shebang in that file is not able to take advantage of the cmd script syntax that allows for specifying a path relative to the file on disk.

Since we don't know what path we'll be installing to until install time, we need to be set that shebang during the install. Since we only run the actual pip installs when the msi is built, not when it is run (because for an msi to work right it needs to know every file that will be created), we had to prepend the shebang to the special file at install time. So we do that and include the setuptools version of the wrapper exe (the pip version relies on the file being zipped and appended to the exe, which we weren't comfortable modifying).

This is where we ran into the permissions issues. Files installed with MSIs only grant read/execute permissions to non-admins. Our commands to prepend the shebang did not have those permissions (see previous comment) so they failed. We did not catch this in our automated testing because the user happens to be just an administrator, so it could install no problem.

[oliversalzburg]

I can confirm that the installer issue is resolved now. And the original issue is fixed as well :)

[no-response[bot]]

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.