Chapel: Make test-venv is very fragile and undocumented.

[buddha314]

There are a few issues around make test-venv and chpldoc that shows the compilation scripts are too sensitive to Python versions. If nothing can be done about that, perhaps a top level building with Python page should be set up.

[buddha314]

On a whim, I created a virtual environment to run make test-venv with

virtualenv -p python2.6 env
source env/bin/activate
make test-venv

And that succeeded (after a lot of complaining from PyYaml). I was able to run start_test on my project until I deactivated the virtualenv. At that point, I got the error

[Error: Chapel test virtualenv not available, run a top-level `make test-venv`]

[ben-albrecht]

Historically, the testing framework has been considered a developer feature, which means documentation and compatibility aren't treated at the same standard as user-facing features. As a result, documentation would probably be in developer documentation if it existed and the testing framework isn't expected to support any python versions beyond CPython 2.6+, which is what our nightly testing runs on. This means there are no guarantees for CPython 3 or Anaconda.

That's not to say we don't want developer features to be well documented and to support other Python versions, but effort could generally be better spent elsewhere.

Having said that, chpldoc is a user-facing feature, so I think it'd be reasonable to support Python 3 for chpldoc builds if we don't today. Could you clarify whether or not that is working for Python 3 for you?

I'm inclined to take down "gating issue" here, because the work-around is "use CPython 2"

[buddha314]

Looks like chpldoc will not build with python 3 either.

Without documentation on how to use start_test and no test suite, this really is a blocking issue. As of this moment, I have to rebuild functionality that start_test is supposed to handle. So we need either a test suite that works for applications (and I think Mason is supposed to play a role in this, right?) or we need to improve start_test in some way.

[ben-albrecht]

Looks like chpldoc will not build with python 3 either.

Could you provide more info on this?

I aliased python to python3 and was able to build and run chpldoc successfully.

[buddha314]

Okay, so chpldoc does build, I took the flood of pip errors to mean it did not build and I hadn't properly removed it.

[buddha314]

What does start_test do? I'm guessing it looks for a virtual env and sources that. Now I have to source a virtual env inside a virtual env to get it to work. Very "Inception"-like and not especially practical.

[buddha314]

Ah, poop, that trick only works in the CHPL_HOME directory. So I'm back to not being able to test outside of that dir.

[ben-albrecht]

Without documentation on how to use start_test and no test suite, this really is a blocking issue. As of this moment, I have to rebuild functionality that start_test is supposed to handle. So we need either a test suite that works for applications (and I think Mason is supposed to play a role in this, right?) or we need to improve start_test in some way.

I see. Would it be more accurate to say that #7495 is a blocking issue rather than this issue then? It sounds like the lack of a user-facing testing framework is a blocking issue for you, where beefing up start_test to be a user-facing test framework would be one way to get around that.

Also note, start_test is documented, but the documentation does not cover 100% of the features as of now.

[ben-albrecht]

As of this moment, I have to rebuild functionality that start_test is supposed to handle.

Again, you should be able to work around this issue by not using python as python3, which isn't a great idea for many OSes currently [1], [2].

Maybe try alias python=python2 and rebuild the test-venv to see if that works for you.

[buddha314]

I admit there are several issues here.

First, the alias did not work.

> alias python=python2.6
macbuddha:chapel buddha$ python --version
Python 2.6.9
macbuddha:chapel buddha$ make test-venv
cd chpl-venv && /Applications/Xcode.app/Contents/Developer/usr/bin/make test-venv
export PYTHONPATH=/Users/buddha/github/chapel/third-party/chpl-venv/install/darwin/py3.6:$PYTHONPATH && \
    /usr/local/bin/easy_install --install-dir=/Users/buddha/github/chapel/third-party/chpl-venv/install/darwin/py3.6 virtualenv==12.0.5
Searching for virtualenv==12.0.5
Best match: virtualenv 12.0.5
Processing virtualenv-12.0.5-py3.6.egg
virtualenv 12.0.5 is already the active version in easy-install.pth
Installing virtualenv script to /Users/buddha/github/chapel/third-party/chpl-venv/install/darwin/py3.6
Installing virtualenv-3.4 script to /Users/buddha/github/chapel/third-party/chpl-venv/install/darwin/py3.6

Using /Users/buddha/github/chapel/third-party/chpl-venv/install/darwin/py3.6/virtualenv-12.0.5-py3.6.egg
Processing dependencies for virtualenv==12.0.5
Finished processing dependencies for virtualenv==12.0.5
install/darwin/py3.6/virtualenv --python=/usr/local/bin/python /Users/buddha/github/chapel/third-party/chpl-venv/install/darwin/py3.6/chpl-virtualenv
Running virtualenv with interpreter /usr/local/bin/python
Using base prefix '/usr/local/Cellar/python/3.6.4_3/Frameworks/Python.framework/Versions/3.6'
New python executable in /Users/buddha/github/chapel/third-party/chpl-venv/install/darwin/py3.6/chpl-virtualenv/bin/python3.6
Not overwriting existing python script /Users/buddha/github/chapel/third-party/chpl-venv/install/darwin/py3.6/chpl-virtualenv/bin/python (you must use /Users/buddha/github/chapel/third-party/chpl-venv/install/darwin/py3.6/chpl-virtualenv/bin/python3.6)
Installing setuptools, pip...
  Complete output from command /Users/buddha/github...ualenv/bin/python3.6 -c "import sys, pip; sys...d\"] + sys.argv[1:]))" setuptools pip:
  Traceback (most recent call last):

[buddha314]

Second, yes, start_test is documented but it the issue I am having is building it, not using it. This appears to be because of, let's just call it "Pythonic inflexibility". I don't see any work-around noted for that.

Third, probably the optimal path is building a new, application-oriented test suite as you noted in https://github.com/chapel-lang/chapel/issues/7495 I'm not entirely sure what "application-centric" means, but it probably includes something like mvn test and is python agnostic.

[ben-albrecht]

First, the alias did not work.

It looks like test-venv is still being built with python3. Did you clobber the venv before rebuilding?

i.e.

cd third-party/chpl-venv && make clobber

If that doesn't work, there may be something else still pointing to python3 like easy_install or pip.

[buddha314]

Sad to say, that still does not work. I saw that too and I'm looking for ways around it.

[buddha314]

Do I really have to set my entire system Python to be 2.6 just to build test-venv? Although I can't seem to get that to work, either.

[ben-albrecht]

Do I really have to set my entire system Python to be 2.6 just to build test-venv?

Python 2.6 or 2.7. Yes, test-venv expects python to be Python 2. Setting python to Python 3 will break many applications (see the linked references above on why this is not preferred on many OSes).

[buddha314]

How dependent is start_test on Python? It's beginning to seem like I'm going to have to abandon that framework and throw my complaining behind developing a true test framework ;)

[ben-albrecht]

How dependent is start_test on Python?

On a scale of 1-10, about a 10. It's written in python

It's beginning to seem like I'm going to have to abandon that framework and throw my complaining behind developing a true test framework ;)

I think that would be a better use of your complaining efforts :)

[buddha314]

You're sure it's not a 9.8 or 9.9 ?

Well then, poop. How is the the reflections package? Is there one of those parliamentary tickets you guys love around the testing framework?

[buddha314]

It has already begun: https://github.com/chapel-lang/chapel/issues/8724

[bradcray]

@ben-albrecht : With all your recent test-venv work, can this be closed now?

[ben-albrecht]

make test-venv is certainly less fragile now, but I think https://github.com/chapel-lang/chapel/pull/8784 would need to be merged in order to resolve the issue @buddha314 encountered here.

I'd opt for closing this issue for being duplicate with https://github.com/chapel-lang/chapel/issues/8976 though.

[bradcray]

I'd opt for closing this issue for being duplicate with #8976 though.

That makes sense to me. @buddha314, are you OK with that?

[buddha314]

izok=true;

[ben-albrecht]

Python 3 support for make test-venv tracked in https://github.com/chapel-lang/chapel/issues/8976