Azure-functions-core-tools: Changes to publishing pipeline

[asavaritayal]

The following changes are required to the Python publishing workflow for GA:

func publish

1. Check if requirements.txt has changed. If it hasn't, skip to step 4.
2. Run packapp.py to install dependencies using pip.
3. If native dependencies without wheels are required, step 2 should fail with the following error:
   "ERROR: cannot install dependency: binary dependencies without wheels are not supported. Use func publish --build-native-deps to build using a docker container**.
   More information at https://aka.ms/func-python-publish"
4. Zip deploy to Linux

func publish --build-native-deps

1. Check if requirements.txt has changed. If it hasn't, skip to step 3.
2. Builds the dependencies only using docker
   Note: In the short term, we'll retain the PyInstaller path as default along with the --no-bundler option.
3. Zip deploy to Linux

In addition to the above, we'll also require the following commands to support VS Code and DevOps scenarios:

func pack

1. Check if requirements.txt has changed. If it hasn't, no action is required.
2. Run packapp.py to install dependencies using pip.
3. If native dependencies without wheels are required, step 2 should fail with the appropriate error code.

func pack --build-native-deps

1. Check if requirements.txt has changed. If it hasn't, no action is required.
2. Builds the dependencies only using docker
   Note: In the short term, we'll retain the PyInstaller path as default.

func publish --no-build

• Zip deploy to Linux

Issues for this Epic:

• [x] https://github.com/Azure/azure-functions-core-tools/issues/1118
• [x] https://github.com/Azure/azure-functions-core-tools/issues/1117
• [x] https://github.com/Azure/azure-functions-core-tools/issues/1015 (slightly related - goal is to standardize build output across the board)

[asavaritayal]

/cc @ahmelsayed @anirudhgarg

[jeffhollan]

Would ‘build’ do anything outside of Python? I’m not sure I’d want that to be the verb if it only does something unique to 1 language. Maybe “func build” could do also do “func extensions install?” Pretty much it becomes the command to prep and download dependencies for any language.

I’m not saying we need to make it do all that from day 1, but if build only makes sense for Python maybe it’s a flag on func publish like ‘func publish —build-python?” Just thinking out loud

[ColbyTresness]

@asavaritayal can you explain to me how az functionapp devops-build create enables the above?

[asavaritayal]

@ColbyTresness I'd removed the az webapp devops-build instructions from the above since we weren't able to make it work for Python yet but I can explain offline how it integrates into the overall workflow.

[ColbyTresness]

It should work for python - it just might not do what you think it does. Happy to sync offline and update here.

[asavaritayal]

@fabiocav here are the work items corresponding to this Epic:

• https://github.com/Azure/azure-functions-core-tools/issues/1118

• https://github.com/Azure/azure-functions-core-tools/issues/1117

• https://github.com/Azure/azure-functions-core-tools/issues/1015 (slightly related - goal is to standardize build output across the board)

[ankitkumarr]

@asavaritayal Right now, func pack creates a zip as well after performing the steps you outlined. Should I make it so it only downloads, installs and builds the project? That way, A user can do func pack when they want to rebuild dependencies and then do func publish --no-build.

If we do this, do we still want to call it pack?
Or do we not want to do this and let it make a zip file? Then we would have func publish --no-build expect a zipfile to publish.

I personally think having func pack (or whatever we call it) not make a zipfile might be beneficial in case users don't want to reinstall dependencies and just publish. This also enables the scenario where users summon .python_packages (with custom deps) from somewhere and call func publish --no-build.

[asavaritayal]

It makes sense to have func pack simply download, install and build the project and then defer to func publish --no-build for the actual deployment.

If we do this, do we still want to call it pack?

@ahmedelnably will defer to you on the naming decision.

@EricJizbaMSFT any concerns with the plan above? I expect VS Code to be the primary user of this command.

[ahmedelnably]

I thought that VS Code use that command? If yes ... then we shouldn't change the name.

[asavaritayal]

@ahmedelnably yes, added the Code team so we can adapt the changes in the extension, if needed. Just checking in case you'd prefer a different verb for consistency/clarity.

[ejizba]

Yes ideally the func pack command maintains the current behavior of creating a zip file so that we don't break existing users. I suppose now is the time to change it if you want to (aka while it's still in preview), but that would probably need a larger discussion.

This might be a tangent - but I also want to mention that this comment from @jeffhollan stuck out to me:

Would ‘build’ do anything outside of Python? I’m not sure I’d want that to be the verb if it only does something unique to 1 language.

It makes me think - why _am_ I using the func cli to build my project? For C#, I use the dotnet cli. For Java, I use maven. For TypeScript, I use tsc. I'm not super familiar with Python, but can you provide a Python-specific way to build my project? Maybe a python script or something? If you provide a script, that even allows me to customize it for my project.

[asavaritayal]

@EricJizbaMSFT we are okay with keeping func pack as-is and maintaining it as an internal facing command for VS Code only.

but can you provide a Python-specific way to build my project? Maybe a python script or something? If you provide a script, that even allows me to customize it for my project.

Unfortunately, the Python script is not that straight forward. It requires multiple checks, docker-based build and PyInstaller bundling. Given the complexity, we've wanted to centralize and service the logic as part of Core Tools. Good thing is that we know this is problematic and are exploring how the experience can be simplified in the future.

[ankitkumarr]

@asavaritayal, all the associated issues have been closed. This epic should now be completed. As we discussed offline, func pack would not need any changes at the moment. Other than that, the publishing flow should work as described in the issue (available in the next release). 🥂

I am closing this epic.