Tfjs: this TensorFlow binary was not compiled to use: SSE4.2 AVX AVX2 FMA

[happy-machine]

Hi!

How do i get rid of this warning? Found help for the python wrapper but not for JS.
We're about to deploy a production model API and trying to bring down inference and response speed.

Installing the GPU binding doesn't seem to have any effect

Thanks!

Dan

[gunan]

Not sure how I can help with tfjs? we do not distribute anything for tfjs.
Python bindings are actually built with sse 4.2 and avx.

[happy-machine]

I'll put an issue in the node repo... sorry!

[caisq]

@gunan Sorry - I didn't realize this issue is for tfjs, not tensorflow.

@happy-machine This issue seems legit and I think we should re-open it.

cc @nkreeger Is it possible for us to bind to a version of libtensorflow built with SSE 4.2 and AVX?

[nkreeger]

Hi @happy-machine - We don't have support for that just yet - see this issue for more details: https://github.com/tensorflow/tfjs/issues/571

Would love to know more about that production use-case - are you deploying to a cloud Node.js service? Real-world server side uses cases will help us prioritize feature work on Node.js :-)

[happy-machine]

Hey @nkreeger I am ML Engineer at @ht2labs https://www.ht2labs.com/
a leading R&D company in the field of E-Learning.

Learning Locker is an open source xAPI conformant e-learning data store available under GPL 3.0, and is the most installed open source LRS with over 12k installations worldwide and corporate clients such as Tes, Jisc, IHG group, City and Guilds

We’ve been applying various academic models (such as the Community of Inquiry framework) to help us build models and analyse text for the last couple of years. We're now exploring large scale Node.js deployment of tensorflow models to add ML inference that will hopefully benefit our clients and the global e-learning community as a whole.

We use AWS across the board for deployment. We can't squeeze the dependencies into AWS Lambda so are currently using EC2.

if you're interested in learning more please drop me a line at dan.[email protected], we are big fans of the work you guys are doing and would love to talk in more detail.

[happy-machine]

we're now up to average of less than 50ms per request on 1000 concurrent requests from node on aws ec2 with no extra gpu thanks to the new update! (small model)

[dsmilkov]

I want to up the priority on this one. I just tested the latest tfjs-node (1.0.1) by installing @tensorflow/tfjs-node and I still get the following warning:

2019-03-13 15:47:50.393657: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA
2019-03-13 15:47:50.422756: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3700000000 Hz
2019-03-13 15:47:50.423809: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x49da6c0 executing computations on platform Host. Devices:
2019-03-13 15:47:50.423850: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): <undefined>, <undefined>
(node:47438) Warning: N-API is an experimental feature and could change at any time.

I think given the ubiquitous support of AVX and AVX2, we should ship our pre-compiled binaries with that optimization. This will give us significant performance wins for free.

[nkreeger]

@dsmilkov yeah we should - however we're using TensorFlow hosted binaries now. We'll have to either update those build scripts or start rolling them again ourselves.

[dsmilkov]

Happy to help. I think it's important to align with whatever Python does.

[josei]

What would be a manual way to recompile the binaries once the package is installed? A starting point would be appreciated in order to take advantage of the performance improvements.

[dsmilkov]

Hi, I just sent a PR to update the main README file with more detailed instructions on how to build your own TF from source: https://github.com/tensorflow/tfjs-node/pull/237/files?short_path=04c6e90#diff-04c6e90faac2675aa89e2176d2eec7d8

[ShadabFaiz]

Any progress on it?

[caisq]

@ShadabFaiz When libtensorflow 1.14 comes out, we'll release a new version of tfjs-node, which will have AVX and SSE optimization included.

[ShadabFaiz]

Ok. Thanks for the information.

[caisq]

The latest release of @tensorflow/tfjs-node (1.2.3) depends on libtensorflow 1.14.0, which is built with SSE and AVX support. I hesitate to build the default distribution of libtensorflow with AVX2, AVX512 or FMA, as those are more recent and hence supported by fewer CPUs.

As such, I'll mark this bug as fixed for now.

If anyone has remaining questions, please feel free to comment.

[josephrocca]

@caisq Would one expect a significant speedup with AVX2 and FMA? This SO user says that you can expect up to 300% improvement (which seems too good to be true, but it got me interested). Both my home machine and cloud machines seem to have those instructions, so if it would provide a decent speedup, it might be worth compiling with support for those? Mainly for the cloud use case.

Here's a Glitch repo (pretty sure they run on AWS machines) where you can see these warnings (click tools > logs in the bottom-right):

node-pre-gyp info This Node instance does not support builds for N-API version 5
node-pre-gyp info This Node instance does not support builds for N-API version 5
2020-01-19 02:25:32.110991: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-01-19 02:25:32.129597: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2300075000 Hz
2020-01-19 02:25:32.129911: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x39ead40 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-01-19 02:25:32.129949: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version

[qwertangel]

i still have several machines which do not have avx/avx2. now tfjs node doesn't work anymore. Only option for me is to downgrade but that means all other software which are dependent on new version will break. can we still have support for older cpu's without reverting back to old version ?

[sbrl]

I've decided to use Tensorflow.js as my primary deep learning framework for my PhD. I'm just starting the AI part of my project now, and run into this message. It's a shame this issue has essentially been closed as a wontfix, as the potential performance increase is pretty huge.

I have a pair of suggestions:

• Would it be possible for the Tensorflow binary to auto-detect as to whether the instruction sets are present, and dynamically pick the right function to call at runtime? Or perhaps an environment variable TENSORFLOW_AVX_ENABLED?
• If that's too challenging, what about a separate npm package? e.g. @tensorflow/tfjs-node-avx or something that's built to include the optimisations.

@caisq

Edit: That documentation update to the README seems to have now been lost when the tfjs-node repository was merged with the tfjs repo. Perhaps it could be re-added?