Python: Disable semantic interposition?

[blopker]

Red Hat recently released an article about how they improved Python's performance by 30% in Python 3.8 by disabling semantic interposition. Article: https://developers.redhat.com/blog/2020/06/25/red-hat-enterprise-linux-8-2-brings-faster-python-3-8-run-speeds/

Looks like they added the -fno-semantic-interposition flag to GCC.

Would this option make sense for this image?

[blopker]

This Fedora wiki has some benchmarks from the change: https://fedoraproject.org/wiki/Changes/PythonNoSemanticInterpositionSpeedup

[tianon]

This is neat! Do you know if there's been any attempt to bring this suggestion upstream? (and/or any official recommendation from upstream to set this flag?)

[tianon]

It would appear that https://bugs.python.org/issue38980 is the appropriate upstream discussion. :+1:

[blopker]

Nice. There's some good points in that python.org thread in favor of the option, like Clang disabling it by default already. I wonder if adding the option to the Docker images would help upstream make a decision?

[yosifkit]

adding the option to the Docker images would help upstream make a decision?

We follow their lead as to how they would like to Python built. We aspire to represent how Python maintainers desire Python to be built as purely as possible. When they reach a decision to change the default, we will definitely use it.

[blopker]

I think that's fair, being conservative here is important so this image doesn't break things for people. My hope was that the performance gains would make for an exception in this case.

Shot in the dark, but since @gpshead had some great input on #160, are you familiar with the risks here as well?

[gpshead]

i'll focus on the upstream issue. But the simple answer is that if you test this with your interpreter's build + configure setup and it makes a demonstrable improvement that you can measure. I'd go ahead and adopt it.

[blopker]

That's great, thank you for the feedback. You're right, we should do an actual test on this specific case.

I ran a quick benchmark. baseline is https://github.com/docker-library/python/blob/master/3.8/buster/Dockerfile with no changes. 3.8-interposition is baseline with just -fno-semantic-interposition. 3.8-interposition-lto is 3.8-interposition with also the --with-lto flag.

| Benchmark       | 3.8-baseline | 3.8-interposition-lto        | 3.8-interposition          |
+=================+==============+==============================+============================+
| django_template | 108 ms       | 90.5 ms: 1.19x faster (-16%) | 104 ms: 1.04x faster (-4%) |
+-----------------+--------------+------------------------------+----------------------------+
| nbody           | 237 ms       | 210 ms: 1.13x faster (-11%)  | 227 ms: 1.04x faster (-4%) |
+-----------------+--------------+------------------------------+----------------------------+
| raytrace        | 895 ms       | 723 ms: 1.24x faster (-19%)  | 871 ms: 1.03x faster (-3%) |
+-----------------+--------------+------------------------------+----------------------------+

Not quite the numbers that Red Hat was seeing, but adding LTO does seem to have a significant impact.

[blopker]

Getting similar results with Alpine 3.12 with LTO and -fno-semantic-interposition:

| Benchmark       | alpine | alpine-optimized             |
+=================+========+==============================+
| django_template | 110 ms | 95.9 ms: 1.15x faster (-13%) |
+-----------------+--------+------------------------------+
| nbody           | 242 ms | 207 ms: 1.17x faster (-14%)  |
+-----------------+--------+------------------------------+
| raytrace        | 914 ms | 717 ms: 1.27x faster (-22%)  |
+-----------------+--------+------------------------------+

[tianon]

Interesting results, although I'm not sure this is very convincing WRT -fno-semantic-interposition being worth specifying explicitly/manually -- it seems to me those results point more to LTO making a fairly large difference, and -fno-semantic-interposition making a very small difference. Would you be willing to perform your tests again with just LTO and without -fno-semantic-interposition to ensure we have a "control" to compare the LTO+-fno-semantic-interposition results against?

Additionally, I want to make sure we're careful with the build-time implications of this, which is why we resisted --enable-optimizations for so long (see conversation in https://github.com/docker-library/python/issues/160).

[tianon]

Additionally, I want to make sure we're careful with the build-time implications of this, which is why we resisted --enable-optimizations for so long (see conversation in #160).

To illustrate better what I mean, the build of python:3.9-buster in #502 appears to have taken roughly 14 minutes -- the same build on master took roughly 8 minutes. That ~6 minute difference doesn't seem like much until you multiply it by ~30 supported variations, at which point it becomes a roughly 2.5 - 3 hour increase in total build time (and on the "official" build servers, the builds do not happen in parallel, so this really is pretty heavy for only a ~10% speed increase).

[blopker]

Fair point. I actually did do an LTO-only test when I did the other tests. I don't have the results now, but they were interesting. Mostly no changes in performance, some tests were even slower. I believe LTO only works when function calls don't go through the lookup table, like when the -fno-semantic-interposition is set. In the Fedora wiki they don't mention this since they already had --with-lto enabled and likely didn't test it.

I remember also testing --with-lto in #160 and didn't propose we add it because it didn't seem to do anything.

[blopker]

OK, I took some time to run more tests: https://gist.github.com/blopker/87153749077fc2ca233310627a564e42
These results are preliminary. I'm currently rerunning the tests with the --rigorous flag since some of the faster tests showed pretty inconsistent results. However, it looks like the performance improvements are as high as ~30%. Tests that rely more on calling C extensions show the least changes. The more Python-heavy tests show the most improvement.

[blopker]

Ok, the results with rigorous didn't change too much, there was still a lot of variance with the quick tests. I think the only way to get those more consistent is to get a dedicated host to run on, but that's more $$.

Seems like most of the performance gains are more than 10%. Is there any other information that would be useful?

[blopker]

Some good news on this! Looks like -fno-semantic-interposition was added to --enable-optimizations in Python 3.10 (https://docs.python.org/3.10/whatsnew/3.10.html#optimizations). Thanks @gpshead for raising that upstream.

To get the full performance benefits we'll still need to add --with-lto though.