Jupytext: QUESTION: What needs to be updated for format_Version

[lwasser]

Hey there @mwouts !! still enjoying jupytext a lot!

I have a question about the format_version . I am working with a colleague @nkorinek and each time he pushes a notebook + md or py, and i try to open a notebook i get the error below:

File python-fundamentals-variables.py is in format/version=light/1.5 (current version is 1.4). It would not be safe to override the source of python-fundamentals-variables.ipynb with that file. Please remove one or the other file.

the notebook can not be opened until i delete said .py or md file. We have worked around it but i see I have format_version 1.4 and he has 1.5 (looking at the .py file metadata at the top). He is on linux i believe and I am mac.

ive tried to reinstall from conda-forge (was previously installed via pip) but that does not help. how do i upgrade whatever format_Version is calling? i've gone through the docs and see discussion of the format version but don't see how i can align version (upgrade to 1.5!) with my colleagues for seamless collaboration! Many thanks for any suggestions.

a bit more information - here is my environment.

(base) CIRES-EL-LM-020:edsc-summer-2020 leahwasser$ conda list
# packages in environment at /Users/leahwasser/miniconda3:
#
# Name                    Version                   Build  Channel
alabaster                 0.7.12                   pypi_0    pypi
appnope                   0.1.0                    pypi_0    pypi
argh                      0.26.2                   pypi_0    pypi
asn1crypto                1.3.0                    py37_0    conda-forge
async-generator           1.10                     pypi_0    pypi
attrs                     19.3.0                     py_0    conda-forge
babel                     2.8.0                    pypi_0    pypi
backcall                  0.1.0                    pypi_0    pypi
bleach                    3.1.5                    pypi_0    pypi
brotlipy                  0.7.0           py37h9bfed18_1000    conda-forge
bzip2                     1.0.8                h0b31af3_2    conda-forge
ca-certificates           2020.4.5.1           hecc5488_0    conda-forge
certifi                   2020.4.5.1       py37hc8dfbb8_0    conda-forge
cffi                      1.14.0           py37h356ff06_0    conda-forge
chardet                   3.0.4           py37hc8dfbb8_1006    conda-forge
click                     7.1.2                    pypi_0    pypi
colorama                  0.4.3                    pypi_0    pypi
conda                     4.8.3            py37hc8dfbb8_1    conda-forge
conda-package-handling    1.6.0            py37h9bfed18_2    conda-forge
cryptography              2.8              py37he655712_2    conda-forge
decorator                 4.4.2                      py_0    conda-forge
defusedxml                0.6.0                    pypi_0    pypi
docutils                  0.16                     pypi_0    pypi
entrypoints               0.3                      pypi_0    pypi
gitdb                     4.0.5                    pypi_0    pypi
gitpython                 3.1.2                    pypi_0    pypi
icu                       64.2                 h6de7cb9_1    conda-forge
idna                      2.9                        py_1    conda-forge
imagesize                 1.2.0                    pypi_0    pypi
importlib-metadata        1.6.0            py37hc8dfbb8_0    conda-forge
importlib_metadata        1.6.0                         0    conda-forge
ipykernel                 5.3.0                    pypi_0    pypi
ipython                   7.14.0                   pypi_0    pypi
ipython-genutils          0.2.0                    pypi_0    pypi
ipython_genutils          0.2.0                      py_1    conda-forge
ipywidgets                7.5.1                    pypi_0    pypi
jedi                      0.17.0                   pypi_0    pypi
jinja2                    2.11.2                   pypi_0    pypi
jsonschema                3.2.0            py37hc8dfbb8_1    conda-forge
jupyter-book              0.7.0b3                  pypi_0    pypi
jupyter-cache             0.2.1                    pypi_0    pypi
jupyter-client            6.1.3                    pypi_0    pypi
jupyter-sphinx            0.2.4                    pypi_0    pypi
jupyter_core              4.6.3            py37hc8dfbb8_1    conda-forge
jupytext                  1.4.2              pyh9f0ad1d_0    conda-forge
latexcodec                2.0.0                    pypi_0    pypi
libarchive                3.3.3             h02796b4_1008    conda-forge
libcxx                    10.0.0                        1    conda-forge
libcxxabi                 4.0.1                hcfea43d_1    conda-forge
libedit                   3.1.20170329      hcfe32e1_1001    conda-forge
libffi                    3.2.1             h4a8c4bd_1007    conda-forge
libiconv                  1.15              h0b31af3_1006    conda-forge
libsass                   0.20.0                   pypi_0    pypi
libxml2                   2.9.10               h53d96d6_0    conda-forge
livereload                2.6.1                    pypi_0    pypi
lz4-c                     1.9.2                h4a8c4bd_0    conda-forge
lzo                       2.10              h1de35cc_1000    conda-forge
markdown-it-py            0.4.5                    pypi_0    pypi
markupsafe                1.1.1                    pypi_0    pypi
mistune                   0.8.4                    pypi_0    pypi
more-itertools            8.2.0                    pypi_0    pypi
myst-nb                   0.8.1                    pypi_0    pypi
myst-parser               0.8.1                    pypi_0    pypi
nbclient                  0.3.1                    pypi_0    pypi
nbconvert                 5.6.1                    pypi_0    pypi
nbdime                    2.0.0                    pypi_0    pypi
nbformat                  5.0.6                      py_0    conda-forge
ncurses                   6.1               h0a44026_1002    conda-forge
nest-asyncio              1.3.3                    pypi_0    pypi
notebook                  6.0.3                    pypi_0    pypi
openssl                   1.1.1g               h0b31af3_0    conda-forge
oset                      0.1.3                    pypi_0    pypi
packaging                 20.3                     pypi_0    pypi
pandocfilters             1.4.2                    pypi_0    pypi
parso                     0.7.0                    pypi_0    pypi
pathtools                 0.1.2                    pypi_0    pypi
pexpect                   4.8.0                    pypi_0    pypi
pickleshare               0.7.5                    pypi_0    pypi
pip                       20.0.2                     py_2    conda-forge
pluggy                    0.13.1                   pypi_0    pypi
port-for                  0.3.1                    pypi_0    pypi
prometheus-client         0.7.1                    pypi_0    pypi
prompt-toolkit            3.0.5                    pypi_0    pypi
ptyprocess                0.6.0                    pypi_0    pypi
py                        1.8.1                    pypi_0    pypi
pybtex                    0.22.2                   pypi_0    pypi
pybtex-docutils           0.2.2                    pypi_0    pypi
pycosat                   0.6.3           py37h9bfed18_1004    conda-forge
pycparser                 2.20                       py_0    conda-forge
pydata-sphinx-theme       0.2.2                    pypi_0    pypi
pygments                  2.6.1                    pypi_0    pypi
pyopenssl                 19.1.0                     py_1    conda-forge
pyparsing                 2.4.7                    pypi_0    pypi
pyrsistent                0.16.0           py37h9bfed18_0    conda-forge
pysocks                   1.7.1            py37hc8dfbb8_1    conda-forge
pytest                    5.4.1                    pypi_0    pypi
python                    3.7.6           h90870a6_5_cpython    conda-forge
python-dateutil           2.8.1                    pypi_0    pypi
python-libarchive-c       2.9                      py37_0    conda-forge
python.app                1.3              py37h9bfed18_1    conda-forge
python_abi                3.7                     1_cp37m    conda-forge
pytz                      2019.3                   pypi_0    pypi
pyyaml                    5.3.1            py37h9bfed18_0    conda-forge
pyzmq                     19.0.1                   pypi_0    pypi
readline                  8.0                  hcfe32e1_0    conda-forge
requests                  2.23.0             pyh8c360ce_2    conda-forge
ruamel-yaml               0.16.10                  pypi_0    pypi
ruamel-yaml-clib          0.2.0                    pypi_0    pypi
ruamel_yaml               0.15.80         py37h9bfed18_1001    conda-forge
send2trash                1.5.0                    pypi_0    pypi
setuptools                46.1.3           py37hc8dfbb8_0    conda-forge
six                       1.14.0                     py_1    conda-forge
smmap                     3.0.4                    pypi_0    pypi
snowballstemmer           2.0.0                    pypi_0    pypi
sphinx                    2.4.4                    pypi_0    pypi
sphinx-autobuild          0.7.1                    pypi_0    pypi
sphinx-book-theme         0.0.17                   pypi_0    pypi
sphinx-copybutton         0.2.11                   pypi_0    pypi
sphinx-togglebutton       0.1.1                    pypi_0    pypi
sphinxcontrib-applehelp   1.0.2                    pypi_0    pypi
sphinxcontrib-bibtex      1.0.0                    pypi_0    pypi
sphinxcontrib-devhelp     1.0.2                    pypi_0    pypi
sphinxcontrib-htmlhelp    1.0.3                    pypi_0    pypi
sphinxcontrib-jsmath      1.0.1                    pypi_0    pypi
sphinxcontrib-qthelp      1.0.3                    pypi_0    pypi
sphinxcontrib-serializinghtml 1.1.4                    pypi_0    pypi
sqlalchemy                1.3.17                   pypi_0    pypi
sqlite                    3.30.1               h93121df_0    conda-forge
terminado                 0.8.3                    pypi_0    pypi
testpath                  0.4.4                    pypi_0    pypi
tk                        8.6.10               hbbe82c9_0    conda-forge
tornado                   6.0.4                    pypi_0    pypi
tqdm                      4.45.0             pyh9f0ad1d_0    conda-forge
traitlets                 4.3.3            py37hc8dfbb8_1    conda-forge
urllib3                   1.25.9                     py_0    conda-forge
watchdog                  0.10.2                   pypi_0    pypi
wcwidth                   0.1.9                    pypi_0    pypi
webencodings              0.5.1                    pypi_0    pypi
wheel                     0.34.2                     py_1    conda-forge
widgetsnbextension        3.5.1                    pypi_0    pypi
xz                        5.2.5                h0b31af3_0    conda-forge
yaml                      0.2.4                h0b31af3_0    conda-forge
zipp                      3.1.0                      py_0    conda-forge
zlib                      1.2.11            h0b31af3_1006    conda-forge
zstd                      1.4.4                h4b3e974_3    conda-forge

but this is what my .py file metadata look like

# ---
# jupyter:
#   jupytext:
#     formats: ipynb,py:light
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.4'
#       jupytext_version: 1.2.4
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

i am working in jupyter locally. but can't seem to identify how the jupytext version is 1.2.4. maybe this is an envt issue so please tell me if this is out of scope to ask here. many thanks

[lwasser]

ok i'm closing this @mwouts sorry to bother you. but creating the issue i realized it WAS indeed an envt issue and i didn't understand how the terminal in jupyter notebook references environments. I was looking at one envt and my notebook was running in another. all resolved! i updated to the newest version. Many thanks for your patience!! closing!

[mwouts]

Hello @lwasser , thanks for sharing this! Yes as you found out, it was the version of jupytext _used by Jupyter_ that was outdated.

Still your example is very interesting. It should be feasible to make the error message more informative, so that anyone is this situation can find the resolution more easily. Let me see if I can add

• the version of Jupytext that wrote the file
• the version of Jupytext used in Jupyter (and maybe its path?)

Do you see anything else that would have helped you?

[lwasser]

@mwouts that would be great. Thank you! My terminal (in a jupyter window) used the default base environment. but i launched jupyter notebook in another environment. so the terminal was actually using the base environment as any terminal would be default. but the notebook in the same jupyter launch was using the kernel for environment that it was launched in (if that makes sense).

I think adding the version of jupytext and the path as you mentioned would be immensely helpful.
I think the thing about this error message that threw me was that it did not mention the jupytext version. So really the thing that you want to update as a user is jupytext. Maybe mentioning "you may need to update jupytext" could be useful?

File python-fundamentals-variables.py is in format/version=light/1.5 (current version is 1.4). It would not be safe to override the source of python-fundamentals-variables.ipynb with that file. Please remove one or the other file.

thank you again!! i'm glad this was an easy, user error item to resolve! :)

[mwouts]

Hi @lwasser , I have prepared the error message below. This is a bit long, but at least all the information is there... what do you think?

The text file notebook.py (generated with Jupytext 42.1.1) is in the light format in version 55.4, but Jupytext 1.5.0.rc4+dev installed at /home/marc/GitHub/jupytext can read the light format in versions 1.1 to 1.5. It would not be safe to override the source of notebook.ipynb with that file. Please either install Jupytext 12.1.1, or remove one or the other file.

[lwasser]

hi there @mwouts this looks great!! I have a question. I have jupytext version 1.4.2 which i see as the most current version on pypi and conda forge. i'm just curious is 42.1.1 a version that you have on your computer locally? what is confusing in the above message that i received are the versions. Having the path to the install is super helpful for environment issues! and then i'm just wondering if you want to say something simpler like --

your file was generated with jupytext version 1.4.2 but you have 1.2.4 installed. Please consider upgrading jupytext to version 1.4.2 or remove one or the other file.

42.1.1, 1.5.0 and 12.1.1 are all versions that seem different. this may just be because you have a dev interactive install but i though i would ask. what was confusing for me about that error message was i couldn't figure out what version 1.4 and 1.5 was for because there is no 1.5 version of jupytext right now!

thank you for being so responsive and willing to enhance your package!! and please know this is just my two cents on simplifying the message just a bit!

[mwouts]

Haha... Yes sure version numbers 42 / 12 etc are fake numbers, inserted in a unit test... I just wanted to make sure the test would always raise the error!

And yes, there is one version number for Jupytext, and another one for each file format. The correspondance is documented in jupytext.formats.

[lwasser]

ahhh ok got it :) then what you have is perfect! there are just a lot of different numbers up there that i was wondering... where did they come from. maybe you have been working nonstop day and night to release jupytext version 42??!! :)

thank you again. this will be a great and useful update!!

[mwouts]

maybe you have been working nonstop day and night to release jupytext version 42??!! :)

:joy:

Thank you @lwasser ! I've followed your suggestion at #534 so that the first part of the message is indeed simpler (no mention of the file format version in the first sentence). So we will have something like this:

The file notebook.py was generated with jupytext version x.x.x but you have y.y.y installed. Please upgrade jupytext to version x.x.x, or remove either notebook.py or notebook.ipynb. This error occurs because notebook.py is in the light format in version z.z, while jupytext version y.y.y installed at /home/marc/GitHub/jupytext can only read the light format in versions 1.1 to 1.5.'

[lwasser]

sounds great @mwouts !! thank you again for being so responsive!!