Haikuports: Snippets system for HaikuPorts

[alaviss]

Background

Currently, for recipes that is based on setuptools_python, we have to copy-paste a snippet like this:

PYTHON_PACKAGES=(python python3)
PYTHON_VERSIONS=(2.7 3.6)
for i in "${!PYTHON_PACKAGES[@]}"; do
pythonPackage=${PYTHON_PACKAGES[i]}
pythonVersion=${PYTHON_VERSIONS[$i]}
eval "PROVIDES_${pythonPackage}=\"\
    ${portName}_$pythonPackage = $portVersion\
    \"; \
REQUIRES_$pythonPackage=\"\
    haiku\n\
    cmd:python$pythonVersion\
    \""
BUILD_REQUIRES="$BUILD_REQUIRES
    setuptools_$pythonPackage"
BUILD_PREREQUIRES="$BUILD_PREREQUIRES
    cmd:python$pythonVersion"
done

INSTALL()
{
    for i in "${!PYTHON_PACKAGES[@]}"; do
        pythonPackage=${PYTHON_PACKAGES[i]}
        pythonVersion=${PYTHON_VERSIONS[$i]}

        python=python$pythonVersion
        installLocation=$prefix/lib/$python/vendor-packages/
        export PYTHONPATH=$installLocation:$PYTHONPATH
        mkdir -p $installLocation
        rm -rf build
        $python setup.py build install \
            --root=/ --prefix=$prefix

        packageEntries  $pythonPackage \
            $prefix/lib/python*
    done
}

Which IMO is:

• Ugly, distract readers from other parts of the recipe.
• Functional updates to the snippet would have to be replicated across recipes.

Proposal

A system where "snippets", which are bash files consist of bash functions which can be used by recipes as a mean to share common patterns

Here's how it works:

1. Recipe A requires a snippet called B
2. HaikuPorter scans the snippets folder located on the top level of the ports tree for a file called B.snippet
3. Contents of B.snippet is made available to recipe A

Guidelines for snippets

• Snippets may not override BUILD(), INSTALL(), PREPARE(), PATCH()
• Snippets may not implictly change the recipe environment (eg. they may not modify *REQUIRES implictly)
• Snippets are advised not to depend on other snippets
• Functions/variables provided by snippets should be documented
• Functions/variables for internal use should be prefixed with <snippet-name>_
• Functions/variables are advised to use camelCase

Implementation notes

• Functions provided by snippets are able to modify package metadata and dependency tree, so they should be made available to recipes during dependency caching
• Snippets will certainly be unversioned, so HaikuPorter should keep track of their hashes, and rebuild dependency cache when snippets are modified

Pros

• Commonly used patterns can now be shared between recipes
• Helper functions can easily be updated/reviewed by HaikuPorts contributors
• Most of HaikuPorter's ShellScriptlets can be moved to HaikuPorts

Cons

• Changes to snippets can break existing recipes if not done carefully

Feedbacks and suggestions are welcome!

[alaviss]

Can I have your feedbacks @fbrosson, @pulkomandy, @waddlesplash, @korli? Thanks beforehand!

[mmlr]

Since the recipes are bash scripts, this would indeed already be possible by simply sourcing other files.

However, since the result of parsing recipes is cached, such dependencies would need to be declared to support the case of automatically invalidating the cache of recipes when one of the included scriptlets is modified. Adding these at the stage of the normal requires would be problematic due to the chicken and egg situation of needing the include to parse, but needing the parsed output to do the includes. So a separate mechanism would be needed for this declaration. A separate requires that is evaluated at an earlier stage would work. A shell function provided by the HaikuPorter scriptlests that fills in specific keys should work too.

The feature as a whole is definitely desirable and could be put to use in other places as well. For example unifying autotools build setups (libtoolize vs. autoreconf and such) might be interesting.

[alaviss]

@mmlr

Since the recipes are bash scripts, this would indeed already be possible by simply sourcing other files.

Sure, my only concern is that since HaikuPorter builds ports within chroot, isolated from the system, source wouldn't work during build unless HaikuPorter copies the file over. That said, I do not know if source is expanded by HaikuPorter when the build script is generated.

Thanks for the feedback!

[fbrosson]

Although I also see these downsides:

• increased dependencies, hence risky snippet updates,
• harder, especially for new (or even occasional) contributors, to get a picture of what is happening behind the scenes,

I have to admit that the idea is interesting and that it could open new possibilities :)

[pulkomandy]

On Mon, May 07, 2018 at 02:09:17PM +0000, Michael Lotz wrote:

The feature as a whole is definitely desirable and could be put to use in other places as well. For example unifying autotools build setups (libtoolize vs. autoreconf and such) might be interesting.

I have a minor complaint about the feature: by doing this, the trivial
recipes get even simpler. But when you need to customize something, you
will have to revert to the "manual" way (as it is now). As a result, we
don't have any simple recipes to show the "manual" way anymore, and we
create a higher step in our learning curve (the first step is lower, but
the second one is as high as before).

If you look at Gentoo ebuild, you will see it is similar, they have
files down to 5-10 lines for the standard autotools thing, but then I
have no idea how to write a proper ebuild for anything else.

Maybe something similar to runConfigure would be a good compromise?

[waddlesplash]

Yes, something similar to runConfigure just for Python, please.

[waddlesplash]

(That also has the advantage that if we ever want to move to Python-based recipe parsing instead of Bash-based, as I've wanted to investigate for some time, we have an easier upgrade path.)

[mmlr]

Sure, my only concern is that since HaikuPorter builds ports within chroot

Scarily enough HaikuPorter does not run the recipe parsing inside the chroot. The inclusion of such snippets would probably happen at that stage to be able to produce/modify the parsed recipe keys.

Even when the recipe parsing is moved into a chroot, it would probably be one chroot for all recipe parsing and not one chroot per recipe due to performance concerns. Recipe parsing is already the slowest part of the whole process which is why it is aggresively cached.

IMO making something like runConfigure for these cases does not really help transparency. These scriptlets are part of HaikuPorter, which puts them a lot farther out of sight than a file in the HaikuPorts repo alongside the recipes. It also makes modifying them more difficult for the HaikuPorts contributor that may not be a HaikuPorter contributor. So in my opinion we should implement this feature and then move the existing scriptlets out of HaikuPorter.

[waddlesplash]

Even when the recipe parsing is moved into a chroot, it would probably be one chroot for all recipe parsing and not one chroot per recipe due to performance concerns. Recipe parsing is already the slowest part of the whole process which is why it is aggresively cached.

Like I noted above, a Python recipe parser would make sense here, considering the subset of Bash we use is not that substantial...

[korli]

I would switch to function based python subpackages.

definePythonSubpackage 2.7 python
definePythonSubpackage 3.6 python3

INSTALL()
{
    installPythonSubpackage python
    packageEntries python $prefix/lib/python*
    installPythonSubpackage python3
    packageEntries python3 $prefix/lib/python*
}

Otherwise the current dependency between HaikuPorter and the recipe format for scriptlets is not ideal.
So moving in the HaikuPorts tree such scriptlets would be a good idea.

[alaviss]

After spending sometime working with gentoo's portage and studying contributors concerns, I've updated the proposal

Changes:

• Snippets are now provided in the top level of HaikuPorts tree instead of being provided by ports. This will make finding the snippets easier
• An elaborative guidelines has been proposed for snippets. This is based on my pain points when working with portage's eclass, and based on various contributors concerns

Misc. changes:

• Reworded, simplified lots of text

Again, feedbacks and suggestions are welcome!