Nuitka: Generating specialize C helper code automatically

[kayhayen]

This issue is about generating specialized code helpers automatically. A code helper is considered specialized, if it doesn't work on the most general form, e.g. two PyObject *, but instead e.g. on two PyUnicode_Object instead, where the task can be identified more immediately as PyUnicode_Concat, just concatenating the two strings.

In Nuitka as a result of compile time analysis during optimization we have increasingly more shape knowledge. Often with relatively concrete types. A shape is a set of characteristics that the uses of a expose, e.g. could mean it's "iterable", "indexable", as example shapes. Many times however, shapes are known to be one or two types specifically, int, bool, unicode, etc. but often also one of the other, e.g. unicode or str.

Currently in Nuitka there are helper functions used for code that does all sort of operations. There are very general ones, and then the more specialized ones. The general ones use OBJECT as a type indicator in the function names (where there is specialization). As of writing this, this is only the case for '+' and += (_INPLACE) are optimized for a subset of types and shapes. And there are comparison operations, <, <=, ==, !=, >=,>, but they are only specialized forINT`, i.e. Python2 integers.

These helpers are hand optimized versions of more general code that represents what e.g. PyNumber_Add does. It looks at type slots and implements special behavior like the concatenation that + also does in Python. Many of their forms are not optimized, or not yet fully optimized. The current form is more of a demonstration of feasibility than a practical approach. This will only be sufficient to get few operations optimized, but it will never scale and doing this by hand is error prone and could lead to bugs.

Instead, what we need is a code generator with sufficient type knowledge about the two arguments, so that it can make decisions automatically. For instance, having a nb_add is a given in for long, float, but not so for a list. So instead of making checks, and executing branched code, the code generator would not generate that check, or do so in a way that the C compiler will know it's not happening.

Other examples are checks like type1 == type2. Many times, this comes down to branch decisions. And the different shapes allow some of these, some do not.

[kayhayen]

The slots in the Python type system, e.g. nb_add are related to, but still somewhat different to slots in the Python language itself:

http://nuitka.net/doc/developer-manual.html#python-slots-in-optimization

[kayhayen]

Consider e.g. this very general function used for +:

https://github.com/Nuitka/Nuitka/blob/063a370cf224f860fc1a288b907d32d60772c02b/nuitka/build/static_src/HelpersOperationBinaryAdd.c#L2345

[kayhayen]

This is the functionally hopefully identical version in CPython 2.7:

https://github.com/Nuitka/Nuitka-references/blob/f6671eabb00c551f9a49422896c910fa53fe2c69/final/python2.7-2.7.6/Objects/abstract.c#L1183

[kayhayen]

In Nuitka, we also get to make differences based on Python versions and strive to generate code for all versions. This is currently achieved with the C pre-processor, but could be done also in the code generator we are talking of here.

[kayhayen]

The goal would be to code for all operations and comparisons for all the built-in type shapes and very common combinations, e.g. int or long.

A stretch goal would to integrate the struct types, like this one:

https://github.com/Nuitka/Nuitka/blob/063a370cf224f860fc1a288b907d32d60772c02b/nuitka/build/include/nuitka/helper/ints.h#L40

Here you have to deal with dispatching based on indicator flags to functions. Lets call it NLONG in helper names, and CLONG for the long C type. So ADD_NLONG_NLONG would be code that uses e.g. ADD_CLONG_CLONG in the best case, but also ADD_LONG_CLONG, ADD_CLONG_LONG, and ADD_LONG_LONG etc. and their relatioship is probably very fixed. The ADD_LONG_LONG would be generated as described above, and for the mixed parts, we have to come up with a way to generate them too.

[kayhayen]

Hello @riya-17 this is what we discussed, I suggest you aim at implementing the example + for a first PR and to use one Python templating engine.

I suggest you start with Jinja2, as it is generally the most adopted all around, and check out how it likes curly braces. It says on http://jinja.pocoo.org/ that

Configurable syntax. For instance you can reconfigure Jinja2 to better fit output formats such as LaTeX or JavaScript.

The later should me that C will also be good with the right set of options.

My preference is for this to be its own binary. You can put it in bin and import nuitka and esp. the type shapes, adding needed information to these classes as you go.

We will later need to decide if we want to always create things ahead of time, i.e. #102 or if we only generate used stuff, or if we have a hybrid mode, where programs generate only some special forms.

Yours,
Kay

[kayhayen]

Some pairs of operation will make sense, e.g. float and int make sense for most operations, but float and list will normally not reach code generation and should of course not be generated. In the beginning, I am OK with us providing static input lists, and code generation warning about things it would have liked to use, but that didn't exist.

[riya-17]

Hi @kayhayen Thanks for giving a brief overview about the task, I will generate a PR for '+' and would come back to you if I have any doubt regarding this and I started with Jinja2 as well.

I would also like to ask how can I run and check this code on my laptop. I am using Eclipse IDE for C/C++ Developers and included PyDev for Python

[kayhayen]

@riya-17 So for the tests, there is a description in the developer manual how to run general tests, but they are really bad for you right now, as you are going to want to have coverage of your code as opposed to running random code.

Therefore I propose to use Jinja2 to also generate tests, but to start out, I would advise you to got with Python2 and int, and use this kind of function:

def f(cond1, cond2):
    if cond1:
        x = 1
    else:
        x = 2

    if cond2:
        y = 4
    else:
        y = 7

    return x+y

print("IntAdd")
print(f(True,True))
print(f(True,False))
print(f(False,True))
print(f(False,False))

The optimization doesn't yet inline these function calls of f, after which we would have to come up with a way to prevent that. The cond1/2 variables allow to excercise branches. Instead of a boolean, depending on the type, if more than one value combination is interesting, you could do. You should enhance my example to use something that will over oder underflow sys.maxint/2+1 would e.g. give you going into that branch, where the end result is a long. And for negative values, a similar trick should apply.

Then the C function is used.

A similar trick can be done with float (easier for you in that both Python2 and Python3 will work). I think ultimately you should come with a clever way of providing a list of values and generating code like above that makes all the cases happen.

For the combination FLOAT_OBJECT (one argument is float, one is unknown), I suggest using an argument to the function for the right side (left side for OBJECT_FLOAT), and then you can think of a set of objects that is passed into every kind of function like that. You then would there e.g. define classes that overload __coerce__, but don't go there just yet, start out with simpler candidates, one int, one long, one float value, etc.

[kayhayen]

As for formatting of C code, I wanted to point out, that there is a description how to use clang-format, so don't worry about that at all yet. In case you don't know how to install it on your Ubuntu, I am going to add instructions about that to the manual later this weekend.

[kayhayen]

Also, once you got something that sort of works, it would be nice, if you integrated jinja2 into Travis build, or even made it (for your PR temporarily) only execute your test.

In order to execute your test, I would advise you to do on your local machine:

python2 bin/nuitka-run --debug MyTestFileLikeAbove.py

You do not have to setup a PYTHONPATH at all in that mode. The result is automatically ran.

[kayhayen]

Also, for Nuitka testing, we have kind of developed a testing method, where functions, and classes do stuff, and print() result values. Then it's run with CPython and the output is captured. And then it's also run with Nuitka and the output is captured. No differences -> We are good.

To make up for Python2 compatibility, a future import of print_function is used. The test runner will also tun 2to3 if a syntax error would happen for it. And if it's at run time, then I typically do things like this:

    try:
        print("long", repr(x), long(x), long(x,2))
    except (TypeError, ValueError) as e:
        print("caught", repr(e))
    except NameError:
        print("Python3 has no long")

So we just catch the exception and ignore it, still executing other parts, where int is used. In my mind, this works very well for transparency. If your generated test also includes information about what it does next, for what type, etc. all of which should be easy to generate, that's perfect.

[kayhayen]

There is also bin/compare_with_cpython which does the comparison automatically and ignores irrelevant differences. So types are named compiled_function vs. function and if you output one of those, that needs to be corrected, the tool does this. But it's amazingly bad, using lazy command line parsing, and not being all that well documented.

[kayhayen]

As for integration of course generated code, I would recommend you to add it right here:

https://github.com/Nuitka/Nuitka/blob/ba9e2d49dca932fa52a755eba89e552a98b2c1bd/nuitka/build/static_src/CompiledCodeHelpers.c#L2006

Make another #include with the generated code, and remove from HelpersOperationAdd.c the ones you generate. Just an idea, but with #if 0 .. #endif guard, it's easier to read it still later on for your reference. And if you consistently used a #if _NUITKA_RIYA and #if !_NUITKA_RIYA there and in the newly generated C code, you can enable and disable, switching back and forth easily. Feel free to come up with a better name though @riya-17 . :)

[kayhayen]

And if you are stuck on anything, let me know.

[riya-17]

Ok Sure

[riya-17]

Hi @kayhayen I want to ask regarding implementation in jinja2. I would like to know whether the implementation should be purely in jinja2 or it can be a mix of jinja2 and python like the code given below.

from jinja2 import Template

def calc(a, b):
    c = a+b
    return c

template = Template("Add a + b = {{ calc(4, 2) }}")

out = template.render(calc=calc)
print(out)

[kayhayen]

So @riya-17 as you are going to do that, I added this chapter to the developer manual the other day:

http://nuitka.net/doc/developer-manual.html#adding-dependencies-to-nuitka

That is for you to add Jinja2, obviously you can wait with that until we get closer to merging stuff.

[kayhayen]

@riya-17 I think that approach can work pretty well, of course you will be using triple quoted strings, like raw strings r''' ... '''' best. I think PyDev has some support for Jinja templates, not sure how good it is though. Does it work for you this way?

[riya-17]

@kayhayen , Jinja2 works with Python 2.6.x, 2.7.x and >= 3.3 So, I don't think there should be any problem with dependencies.

[riya-17]

@kayhayen I have tried both on terminal as well as eclipse it is working fine.

[kayhayen]

@riya-17 I am even willing to make it a run time dependency, so far templates have been used that didn't contain any logic for the standard C generation, but if we like Jinja for making C code, we might change these things.

https://github.com/Nuitka/Nuitka/blob/856b38bf863c7a21472374f647dc653227e7af5a/nuitka/codegen/templates/CodeTemplatesCalls.py#L18

Much like operations, e.g. function calls could take call side parameters of various types, and try to be fast about them. I am dreaming of providing e.g. a call that gets handed a C long and enters a specialized code of a function body that never had a PyObject *, but goes with a long from the start, making function call overhead of course drastically slower, or at least to move conversion overheads into one place instead of many.

Not sure what you refer to when you say it's working in the terminal, @riya-17

[riya-17]

@kayhayen I understood what you mean, I will work over it and send a PR soon.

I meant, Terminal i.e. command line of Fedora as wells as Eclipse both are producing same output for the above code.

[kayhayen]

@riya-17 When you use Eclipse and with it PyDev, you are using an implementation of Python in Java for the JVM, called Jython. However, what you typically do is to configure an "interpreter". The newly pushed PyDev Nuitka project to current develop branch refers to Python3.6, but you are very free in reality.

But what this does is to run "python" which is the same you use on the command line. Note that Nuitka will only work with CPython and not Jython, or IronPython, but is tied to CPython due to its use of the Python C/API.

So differences would only happen if you ended up using different binaries somehow.

[riya-17]

Hey @kayhayen , you are also using eclipse so you must also be using pydev?

[kayhayen]

Yes @riya-17 I am currently using Eclipse. I have added project files to the git repo recently, in hope they will be useful, as they add ignores of stuff, similar, but not identical to .gitignore, you want PyDev to ignore many file types, e.g. build directories.

I do not use Eclipse to run Nuitka though, for that I typically use the command line. It may however not be all that wise to do so, because I am missing out on e.g. the Python debugger abilities.

I also added a small begin of the description what to install for Eclipse, but not many of the recommended settings to use. I hope to grow that though as we go, finding e.g. sensible values for line length indicator, etc.

[kayhayen]

If you have troubles with Eclipse @riya-17 it may be best to have a small Hangouts, where we share your Eclipse, I will gladly assist you, some of these things are best shown live.

[riya-17]

@kayhayen thanks for your help, I have set up eclipse and I work with command line for jinja2. I am moving to implement basic C addition, substraction functionalities in jinja2

[ManPython]

Hi @kayhayen
I have question.
Due to situation that I not analized depthly the nuitka around the methods in nuitka on python dynamic variables and static for c++ with optimalisation, and the first post.

Did you made some thoughts around decorators for variables? I know that python allow this for functions and classes only, but maybe we can do this "artificaly" for nuitka only? Eg. with comment? In future python can allow decorators for variables maybe.. don't know is sense.. just thinking laterally (can be easy deleted # in front of @ in refactoring process in future).

So.. eg:

@string

foo = ' foo '

@int

var_int= 1

@float

var_float= 1.1

In case that python is front-end (FE), then maybe this way can be more friendly for middle-end (ME) for compialtor with internal representation (RE)?

Soo.. if python runing vars are dynamic. For other lang static? Or maybe universal as directly indication for lang eg cpp_, go_, eg.

@cpp_string

foo = ' foo '

@cpp_int

var_int= 1

@cpp_float

var_float= 1.1

I started time ago thinking with https://github.com/ManPython/PythonGo but after many critical opinions (irc discuss) about this problem (static/dynamic) I leave this. But in this way in future can open some door for "universal driver" from python to other compilers.

[kayhayen]

Hello @ManPython,

for newer Python there is the infamous PEP, that adds annotations to everything. An it does allow for a clean syntax here, albeit only with newest Pythons:

#@string
foo = ' foo '

->

foo : str
foo = ' foo '

That is what the PEP allows. Notice how Python will however never be enforcing. What I don't have yet, but look forward to, is getting this to be properly understood:

foo = something()
assert type(foo) is str

The benefit is going to be, works for everything, and works as a guard, and later code then can safely assume str and work fast, and if you make a mistake, well, it's going to be caught.

Right now though, assert becomes a conditional statement (re-formulation as in Developer Manual):

if type(foo) is not str:
   raise AssertionError

But past that conditional statement, the falseness of the condition is not used yet at all. That is going to be a huge improvement for optimization when it comes, and then we can recommend this for type annotations instead, as it's going to create good and fast code with Nuitka.

[DMRathod]

hello @kayhayen i am dharmraj and i would like to contribute to Nuitka for Gsoc2019. how can get started with this issue ?

[kayhayen]

@DMRathod It's hard to say, I think that #294 might be a good idea too, or #229 as well. Please be aware that your application should contain code, and I think #294 is your best bet to being able to actually get something demonstrable.

[riya-17]

Hey @kayhayen In jinja2 file, what have to be moved to environment and why?

[kayhayen]

So @riya-17 in the trial that we made, we passed a bunch of things, one being "left" and "right", and they are supposed to be some sort of object representing "object" (nothing known really), "int", "long", "str", "bytes", "unicode", etc.

Then there are also passed free functions, i.e. functions which took that kind of object as an argument, but that was obviously non-sense. We could and should attach those to the objects from above, and e.g. use left.isIntTypeExact() to detemine if the left value has PyInt_CheckExact(operand1) predictable as true, and use that in left.getCheckIntTypeExactCode("operand1") and make that return 0, 1, and PyInt_CheckExact(o"perand1") if that is the case.

Obviously, putting getCheckIntTypeExactCode in a base class, will make it accessible via the passed arguments.

This leaves us with no need for an environment at all at this time.

[kayhayen]

However, @riya-17, should we e.g. want to expose Nuitka configuration, e.g. python_version in the desire to turn the C level #if PYTHON_VERSION < 300 ... #endif into Jinja {% if python_version < 300 } ... {% endif } (unsure about the number of braces there), then we could use it.

I am not sure, this is going to be a lot though, and one can still pass one or two items as an argument, not sure if and how an environment is helping at all for our use case.

For what I am thinking, most of the information will be attached to and come from left, right. There might be debug_mode, or you may want to have some sort of optional tracing, about how you are used, a trace_mode. And maybe eventually we could have some kind of stats_mode, where you e.g. count the number of invocations of a method, so we would know what gets use and how often.

But reserve that as a stretch goal.

[kayhayen]

Right now I am most interested in seeing you try to turn more of those C if`` statement conditions into dynamically computed ones, which degrade forobjectto the implementation of currentADDOBJECT_OBJECT*`, but make them have better tricks for other types. Let me see what you can come up with, then update the PR, by just adding the files.

May I suggest you put your tool under nuitka.tools.gen_helper_code for now? Check out the other runners in bin, to see how a __main__ there is supposed to be launched. I guess in the #! you can put python3.6 (I think your Ubuntu should that?) or python3.7, it wouldn't matter.

I am keen to see code. :)

[riya-17]

@kayhayen thanks for clearing my doubt, i was struck with the thought of using environment(env). I will follow your suggestion and put the code under gen_helper_code. My exams got over today so i will get back to it and will send the code till 19th & update you soon..

[kayhayen]

Good to hear @riya-17 and it's easy to get stuck with things like this.

In an attempt at mentoring, let me tell you how I see these things. Many times, I find people discover a tool, let's say a hammer. And then they desperately search for nails. And they come up with things that are not really nails, and hammer them in things, where they don't belong. This can be a great learning exercise, but I generally am too lazy for that. And I consider laziness a virtue.

It is very easy to get lost and spend a lot of time on non-problems this way. Contrast this with searching a problem, then looking for the tool that matches it. A hammer seems to be intended to to attach things, so lets use it. Depending on what it is, it may happen that we don't need no nails, because e.g. the way the parts we want to attach are constructed. They may just fit stiff together and hold. Maybe they get loose after some time, or we want to put more on it, then we need to search for a concept like nails, and will find it.

So my philosophy about things like Jinja is to simply put it to the intended use, and see where I hit limits. When I feel this is way to verbose, there ought to be some to save repetitive tasks, or there ought to be a way to make sure that all templates are created identical, or share the same values, then I will look into it.

This explains why first thing I do, is to paste a mock up (in our case an actually really good version that Nuitka currently uses) of one thing, see what I can do with it, that brings me closer to solving the problem.

if a need for an environment, doesn't enter the picture, then they probably are not for us.

This approach has served me well. Also, I have had embarrassing failures of course, where reading up ahead of time, could be made me avoid big mistakes. But once I discover the need for nails, I am sure, I understand what I want to use them for. I really don't like sitting with a hammer and a nail, and wondering what they are good for. :)

As always, there is a balance. But experience has taught me to always dive in rather immediately, and then look at the documentation after a while, to see if I missed anything important, and then being able to understand how it relates to stuff.

There are these tools, py.test is also a fine example of those, which carry a lot of flexibility, and a lot of baggage with them. So naturally they will have solutions for problems I never encountered, and simply cannot understand by looking at their nails without seeing the hammer, nor the problem they are intended for. In these cases I tend to just ignore it, until I realize that I have a problem that is not well solved with the part of the tool set I used so far. Then I can revisit it, because now I see probably all the things necessary to understand how it helps me.

[riya-17]

@kayhayen I learned a lot from this approach, thanks for sharing such a nice thought :)

[kayhayen]

Unfortunately this has not received an application this time around, therefore removing the tag.