Vedo: fails Test reverse on arm64

[drew-parsons]

Running on the alternative CPU arm64, Test reverse in test_actors.py fails:

Processing test_actors.py script..
---------------------------------
vtkVersion 7.1.1
---------------------------------
getArrayNames: {'PointData': ['parr', 'Normals'], 'CellData': ['carr', 'Normals']}
Test getPointArray
Test __add__
Test points
Test faces (2112, 3)
Test texture
Test deletePoints
Test reverse
Traceback (most recent call last):
  File "test_actors.py", line 159, in <module>
    assert sr.N() == 576
AssertionError

see https://ci.debian.net/data/autopkgtest/testing/arm64/v/vedo/6399125/log.gz

The test passes on amd64.

[marcomusy]

Oh! :-(
that's particularly weird because reverse() doesn't even touch the mesh vertices.. just flipping face orientations.
How may i test what's happening with the arm64 system?
Though I very much suspect this is a vtk issue not a vedo one, as reverse is just a thin wrapper of vtkReverseSense.
(btw, vtk7.1.1 seems quite old already.. is there a specific reason for not having the more modern 8.1.2 in debian?)

[drew-parsons]

In regards to the version of vtk in Debian, kitware have made their software extremely difficult to package. Difficult in the sense that they embed 3rd-party libraries inside their builds rather than using the versions installed on the system (paraview contains it's own copy of vtk, for instance), which makes it nearly impossible to maintain the packages with normal quality control. Several attempts have been made to package vtk-8, and stalled because of these problems. We've asked kitware to straighten out their builds, but they haven't replied.

We can get you a guest account on an arm porterbox. The information needed is summarised at https://dsa.debian.org/doc/guest-account/ (non-DMs section). I can sponsor you, the main thing the application needs is your PGP key fingerprint for authentication. My email is [email protected]

[marcomusy]

Thanks Drew for the detailed explanation. Before trying to get a guest account I would love to see if anything else fails in the latest release: so I temporarily commented out the reverse() test and added more printout to have more info of what could have gone wrong, although I doubt that it could be something fixable in vedo... i think it must be a problem of the upstream vtk.

Besides, I moved all the data files to vedo.embl.es/examples/data, they are downloaded and cached on the fly when needed. This simplifies the licence file which changed accordingly (I also replaced all non-free textures with CC0-licensed ones). Fonts licensed under OFL needed a name change - as explained here at sec.5.2 - because i created a polygonized version of them.
Let me please know if you think it's ok or if I missed anything! Thanks again for your support, M.

[drew-parsons]

Hi Marco, nice to see the CC0 images in 2020.4.0

I'd still count vedo/fonts/Inversionz.ttf as DFSG-non-free since it's commercial donationware (free only for personal use), so not public-domain. The other fonts look good (name-change is fine, the original name and author is still acknowledged).

I've got a build problem with vedo/examples, presumeably related to swapping the symlink around.
The Debian build (pybuild, invoking python3 setup.py build) reports

   dh_auto_build -O--buildsystem=pybuild
   I: pybuild base:217: /usr/bin/python3 setup.py build 
   running build
   running build_py
   package init file 'vedo/examples/__init__.py' not found (or not a regular file)
   running egg_info
   creating vedo.egg-info
   writing vedo.egg-info/PKG-INFO
   writing dependency_links to vedo.egg-info/dependency_links.txt
   writing requirements to vedo.egg-info/requires.txt
   writing top-level names to vedo.egg-info/top_level.txt
   writing manifest file 'vedo.egg-info/SOURCES.txt'
   reading manifest template 'MANIFEST.in'
   writing manifest file 'vedo.egg-info/SOURCES.txt'
   error: can't copy 'vedo/examples': doesn't exist or not a regular file
   E: pybuild pybuild:352: build: plugin distutils failed with: exit code=1: /usr/bin/python3 setup.py build

Can you reproduce that problem? I'm not sure if the vedo/examples/__init__.py' warning is responsible or if it can be ignored.

[marcomusy]

I'd still count vedo/fonts/Inversionz.ttf as DFSG-non-free since it's commercial donationware (free only for personal use), so not public-domain.

oh, you're right... i overlooked that, thanks for spotting it. I'll simply remove it.

Can you reproduce that problem? I'm not sure if the vedo/examples/__init__.py' warning is responsible or if it can be ignored.

I guess it's because examples is not a separate package, and it has no __init__.py, so it can be ignored. If you remember some time ago I merged examples back into the main module, as it's simpler to maintain and it gives the possibility to run them directly from command line e.g.:
vedo -r some_example_script
and
vedo --list

I liked to have the directory examples at the top (along with setup.py) in the git repo, so I added the symlink vedo/examples which points to it so that it gets detected and installed by pip.
I'm not sure what is the best practice to achieve this.

[drew-parsons]

It makes sense for the examples to stand separate. In fact dh_installexamples installs them into the debian doc dir, and I have to make a symlink in the python install dir just as you've done. The question is simply what needs to be done with the config files so that the install error doesn't happen. I don't understand the error well enough to be certain what needs to be done (I tried hacking MANIFEST.in but that didn't seem to help)

One option is to keep the examples out of the python module altogether. For instance have them installed in /usr/share/vedo/examples with that path known to the python module code.

[marcomusy]

what happens if you add an empty file examples/__init__.py ?

have them installed in /usr/share/vedo/examples with that path known to the python module code.

that makes a lot of sense! ..but I'm not sure how to do it.. I guess one needs to add something in the setup.py (?), analogous to the line:
scripts=["bin/vedo", "bin/vedo-convert"],

---

Maybe I know... I have in setup.py:
packages=["vedo", "vedo.examples"],
that's why it looks for a init.py ? if so an empty file might solve the problem..

[drew-parsons]

That sounds right. If it's packages=["vedo", "vedo.examples"] causing the complaint, then that explains why editing MANIFEST.in does not fix it.

[drew-parsons]

I added an empty __init__.py, but the build error still occurs.
Probably the problem is the packages=["vedo", "vedo.examples"] entry, but the behaviour suggests adding __init__.py alone is not enough to avoid the error.

[drew-parsons]

But removing the "vedo.examples" entry doesn't help either. In fact, removing all references to examples from setup.py and MANIFEST.in does not fix it. The error message remains, error: can't copy 'vedo/examples': doesn't exist or not a regular file.

The only action I've found so far that allows a successful build is to delete the symlink in the vedo dir itself. Then the build proceeds (but with vedo/examples not installed).

[marcomusy]

Indeed it looks like it doesn't like the symlink as you say, what if you had some patch script which removes the link and moves examples to vedo/examples?

The error is pretty weird though - why don't i get it in my ubuntu ...

[drew-parsons]

I needed two parts to the patch

• remove "vedo.examples" from the packages entry in setup.py
• delete the vedo/examples symlink

Both of these steps are needed, but I don't need to add an __init__.py

Strange you're not seeing the same problem on Ubuntu. Usually Ubuntu tracks Debian closely on these kind of problems. Maybe the problem comes from some update in python 3.8 or setuptools which hasn't got to your Ubuntu version yet.

I've now uploaded 2020.4.0+dfsg2-1 to debian unstable.

[marcomusy]

I see.. it's strange that it doesn't like vedo.examples I think that's quite standard practice to organize code into subfolders. No big deal though.. thanks for taking care of this!
M.

[drew-parsons]

There are some reports of setuptools getting finicky over folders vs files (and certainly the symlink is "not a regular file"),
e.g. using the package_data field in https://stackoverflow.com/questions/3712033/python-distutils-error-directory-doesnt-exist-or-not-a-regular-file
The underlying problem might remain a mystery!