Python-language-server: TensorFlow 2.0 support

[MichalPospech]

With Microsoft Python Language Server version 0.2.30.0 does not offer keras with TF2.0.0a0 installed while with TF1.1.13 it does.

TensorFlow 1.1.13:

TensorFlow 2.0.0a0:

[zhen8838]

hi , maybe you can try use import keras like this:

from tensorflow.python.keras.api import keras
from tensorflow.python.keras.api.keras import datasets, layers, models, optimizers, metrics

I test in TensorFlow2.0Tutorials :

[thomasgassmann]

It seems tf 2.0 imports the keras API in a different way than it did previously.

Now it's imported like:
_component_api_helper.package_hook( parent_package_str=__name__, child_package_str=('tensorflow.python.keras.api._v2.keras'))

After some debugging, it seems like the language server has some problem resolving this kind of import. It will always result in an unknown expression.

[jakebailey]

@zhen8838 That import technically shouldn't work, and I fixed it in #886 (though it is not yet in stable, but will be very soon). You can try the beta build to test:

"python.analysis.downloadChannel": "beta"

@thomasgassmann The language server doesn't (generally) execute the Python code it analyzes. We'd probably need to special case more things if this is how TensorFlow is going to be importing its own modules.

[jakebailey]

That being said, if @zhen8838 is testing with Tensorflow 2, then I'm not sure my fix will help, since it was targeted at unbreaking non-2.0 missing items in #881

[thomasgassmann]

@jakebailey I tested it just before on the current master with TensorFlow 2 and the import does not seem to work. So changing the downloadChannel to betawon't fix this issue too, I'd say.
So the language server has no way to check for modules that are imported dynamically?

[jakebailey]

It depends on the definition of "dynamic". If it's written somewhere in the code, then we may be able to treat those as imports, or some other solution.

[zohaad]

@jakebailey where did you put this line, in settings.json?

"python.analysis.downloadChannel": "beta"

[jakebailey]

At the top level, but my comment earlier about trying the beta was to fix tensorflow 1.0 after a change we made hid its members. Switching to the beta wouldn't affect this issue (which is still open).

[dynamicwebpaige]

Are there any updates on this issue -- or any way that the TensorFlow engineering team can help? 🙂

[jakebailey]

No updates to this issue at the moment.

The lack of "real" import statements breaks our analysis immensely. We need import statements to understand which modules depend on each other. Without that, we don't know which modules need what, or in what order.

As far as I can tell, the way imports are being done in tensorflow 2.0 is to call a function, which then calls into Python's import system directly, and then modifies the scope of the caller with the module as a variable... which is what the import statement already does. I'm not really sure how _any_ tooling is supposed to be handling things with that sort of trickery.

[MikhailArkhipov]

As @jakebailey said, analysis needs to build graph of dependencies so module sources can be loaded and analyzed in order. The analysis is static, it does not run Python code. Running code is theoretically possible by loading libraries into a Python process but it is actually a security violation since we don't know what the code does and yet it has to be executed in user privileges.

Basically, the problem is not specific to Python per se. It is possible to build dynamic object/code loading system in pretty much any language, but then tools most probably won't be providing much help to the user with completions or navigation.

[MikhailArkhipov]

Alternative is to have stubs for the library :-). It would be great anyway since TF is large and analysis of it takes quite some time. Even with analysis persistence and caching there is still a first time hit.

[alextp]

@pkch we're adding pytype stubs to TF, right?

[dynamicwebpaige]

Looping back to check if pytype stubs have been added and are working as expected. I'm getting autocomplete when using VS Code + a conda environment with TF 2.0, but it's not semantically aware and the referenced docs leave quite a bit to be desired. 😢

[miffyrcee]

The stable version of tf2.0 has come out, but I found that this problem still occurs.

[RensDimmendaal]

Same for me. Hope this info helps a bit.

• tensorflow.__version__ == "2.0.0"
• I use the intellicode plugin

[MikhailArkhipov]

TF 2.0 has custom import system. We can't run the custom code, LS relies on import statements to figure out dependencies. Therefore there is little we can do without TF developers providing module stubs.

[zhen8838]

Can we execute the imported program and store the dependency as a cache and use it?

[jakebailey]

I'm not sure what you mean. The language server does static analysis and cannot run your code to get any info out of it (by design), nor does it persist the analysis of user code.

[RensDimmendaal]

TF 2.0 has custom import system. We can't run the custom code, LS relies on import statements to figure out dependencies. Therefore there is little we can do without TF developers providing module stubs.

FYI, I created an issue at tensorflow for this: https://github.com/tensorflow/tensorflow/issues/32982

[Mulns]

TF2.0 has published, it applies "lazy import" so that intellisense in vscode cannot handle...
Try this solution which is not a long term plan but works fine currently.

[pkjq11]

Here is a better solution

[normanmatrix]

I just switch the import paths for now, which works well otherwise. See https://github.com/tensorflow/tensorflow/issues/32982#issuecomment-554289087

Rather waiting for a fix than messing with my conda environment.

[haimat]

So I got the same issue here. But frankly I don't want to mess with my TF package installation or anything like that, as suggested in those to workarounds. Is there at least a way to disable that warning itself?

[JirenJin]

@MikhailArkhipov Thanks for the explanation! I'm just wondering how autocomplete for TF 2.0 works in jupyter notebook and Pycharm preview version. Can VS Code do the same thing?

[DaKup]

any updates?

[kpguru20001]

Hey the solution worked but as you said it's not longterm and we cant play with the code files of TensorFlow which is wrong is there any update to show function parameters on hover

[theonlyfoxy]

Any Update or ETA to the Solution?

[RensDimmendaal]

The issue is fixed in the tensorflow repo. VSCode should be able to support intellisense for tf2 after this update: https://github.com/tensorflow/tensorflow/issues/32982#issuecomment-589767901 . I guess it's available in the nightly version already. I'm waiting for the next real update.

[hansmichaels]

Hi, maybe you can try it like this. I recently tried using tensorflow_core, not tensorflow, and it works for me. By the way, i install tensorflow using Anaconda.

[RensDimmendaal]

Nice. This issue has been resolved with tensorflow 2.2:

you can check it by pip installing tf-nightly==2.2.0.dev20200422.

[jakebailey]

Tensorflow 2.2.0 is out on PyPI; I've tested this myself and completion appears to work as expected without any weird hacks.

[Siddhant-ops]

Okay so I just imported tensorflow and waited for like 5-10 mins max didn't touch my laptop and then It started analyzing in the background after that's done intellisense worked like a charm.
PS
tensorflow version : 2
my processor is i5 9th gen