Px4-autopilot: Ubuntu toolchain installation script: Improvement suggestions

[TheLegendaryJedi]

First of all, awesome work with all the install scripts for all the platforms. I just tested the ubuntu one in 18.04 and it worked like a charm.

@jinger26 and me want to improve the user experience regarding the installation of PX4 toolchain and the first "take-off" like a "hello sky" for the geeks :smile:

We can't address every installation script at the same time so I suggest we start by making some improvements to the ubuntu.sh one.

Problem solved by the proposed feature:

• Improve the installation script UI/UX and number of commands to run a simple example.

Preferred solution:

• Modify the scripts to echo the percentage of the installation showing only the title of what is installing in the moment. For the error debugging, print the installation [errors], [warnings] and [info] to a log file.
• Include the requirements in the installation files to reduce the number of commands inserted by the user to run PX4 for the first time. For this we could just do something like this:

1. sudo wget -O - https://raw.githubusercontent.com/PX4/Firmware/master/Tools/setup/ubuntu.sh | bash
2. git clone https://github.com/PX4/Firmware.git --recursive
3. cd Firmware
4. make px4_sitl gazebo
5. commander takeoff

I made some changes to the script just for testing if it was possible:
ubuntu.txt

The script is still printing the some install info that can be removed possible using this:
sudo DEBIAN_FRONTEND=noninteractive apt-get install -qq git < /dev/null > /dev/null

But we need to think of a way to print this output a log file instead of just disabling it.

Thanks

[jinger26]

@julianoes @hamishwillee can you take a look here?

[hamishwillee]

Thanks very much. Always great to see people trying to improve the workflow.

Can you submit this as a PR so we can see what has actually changed in the script. On quick inspection, all I can see is the addition of some colour markup.

Include the requirements in the installation files to reduce the number of commands inserted by the user to run PX4 for the first time.

What does this mean?

The example is more or less the same as what you do now to get setup. Though this won't work because you also need to pull in the file requirements.txt (not just ubuntu.txt). The shortest way to do installation on ubuntu is to not download the script separately:

git clone https://github.com/PX4/Firmware.git --recursive
cd Firmware
bash ./Tools/setup/ubuntu.sh
make px4_sitl gazebo
commander takeoff

I show the wget approach in the docs because many people do not want to have to get the Firmware tree to install the dependencies (perhaps they already have various trees present).

I don't see any particular value in removing the errors, warnings etc to logs, but perhaps I am a minority (I guess it is less scary to users when things "work"). When something fails I like to see it then an there. But I would be "OK" with this.

What is important though is that this script must still run in docker following any changes - this allows the script to be automatically tested in parallel with changes to the toolchain.

[TheLegendaryJedi]

I didn't think of that, but yes, is better to clone the repo first and then run the installation script. I think that is the way to go.

What we are trying to do is to make it easier for people to run it for the first time. Like copy/paste the commands and give the user some better UX/UI. For example, it takes like 5min to clone plus 10min to install everything. It was interesting to give the user some message alerting that it will take about 10min and that he should go grab a coffee or something.

I can make a PR but I was just trying to add some messages to the user. The problem is that during install all the logs rapidly hide all starting messages. We can also forget this and give the message to the users on the website and in the docs.

[hamishwillee]

@TheLegendaryJedi Thank you for the explanation.

I didn't think of that, but yes, is better to clone the repo first and then run the installation script. I think that is the way to go.

I had reasons for the way I did it (mostly because "getting source" feels like part of "building" not "setting up env"). But I think you're right that the "80%" will are better off getting source first. Will update docs for that.

What we are trying to do is to make it easier for people to run it for the first time. Like copy/paste the commands and give the user some better UX/UI. For example, it takes like 5min to clone plus 10min to install everything. It was interesting to give the user some message alerting that it will take about 10min and that he should go grab a coffee or something.

Certainly the docs should be updated to say this. Again, I will do that.

I'm not sure that it is a good idea to make all the other proposed changes to the scripts (that's not "no" but "I don't know" and I forsee breaking other things").

How about a small change. When not in docker, pause 5 seconds with your proposed messages before starting?

[TheLegendaryJedi]

Thanks a lot.

I think your idea is the way to go. I never wanted to change what is working already.

I'll add the sleep with the message to the script and make a PR.

[TheLegendaryJedi]

@hamishwillee I made a fresh install of ubuntu 18.04 and tried to run:

git clone https://github.com/PX4/Firmware.git --recursive
cd Firmware
bash ./Tools/setup/ubuntu.sh
make px4_sitl gazebo
commander takeoff

but I got the error

-- Found PythonInterp: /usr/bin/python (found version "2.7.17") 
CMake Error at /usr/share/cmake-3.10/Modules/FindPackageHandleStandardArgs.cmake:137 (message):
  couldn't find python module jinja2:



  for debian systems try: 

        sudo apt-get install python-jinja2 

  or for all other OSs/debian: 

        sudo -H pip install jinja2

   (missing: PY_JINJA2) 
Call Stack (most recent call first):
  /usr/share/cmake-3.10/Modules/FindPackageHandleStandardArgs.cmake:378 (_FPHSA_FAILURE_MESSAGE)
  cmake/px4_base.cmake:144 (find_package_handle_standard_args)
  CMakeLists.txt:347 (px4_find_python_module)

Is there a step I'm missing?

[julianoes]

@TheLegendaryJedi thanks for your suggestions and your PR.

Something is wrong:

-- Found PythonInterp: /usr/bin/python (found version "2.7.17") 

It should select Python3, not Python2, so that's confusing.

[TheLegendaryJedi]

Can we force the code to use python 3?

[julianoes]

It should already do that:

https://github.com/PX4/Firmware/blob/9edb1179d09bbfabf103b86d47fcf70174ac0c15/CMakeLists.txt#L270-L276

[TheLegendaryJedi]

But python 3 is installed in Ubuntu 18.04 by default...I don't get that specific error.

I was able to solve all the problems manually installing the missing modules but not using only sudo pip install ...
I need to use sudo -H pip install or, sometimes, sudo -H python pip install ...

Do you have any idea how can we solve this? Maybe use virtual environments for python to make it OS agnostic.

[julianoes]

What is?

which python
python --version
which python3
python3  --version
which pip
pip --version
which pip3
pip3 --version

[TheLegendaryJedi]

/usr/bin/python Python 2.7.17 /usr/bin/python3 Python 3.6.9 /usr/local/bin/pip pip 20.0.2 from /usr/local/lib/python3.6/dist-packages/pip (python 3.6) /usr/local/bin/pip3 pip 20.0.2 from /usr/local/lib/python3.6/dist-packages/pip (python 3.6)

[julianoes]

Ok, something is wrong here.
I'm not sure how you installed pip3 into /usr/local but I would suggest to remove all that and only use pip and pip3 from the system. You can install it by doing sudo apt-get install python-pip and sudo apt-get install python3-pip.

[TheLegendaryJedi]

Thanks @julianoes. Probably that solves the problem for my case. But my case was a vanilla Ubuntu 18.04 installation. A lot of people will get this error I think. Me and @jinger26 are trying to make the toolchain installation commands easy to run. The objective is to have a easy entry point for PX4 new developers :smile:

[julianoes]

But my case was a vanilla Ubuntu 18.04 installation.

A vanilla installation does not have pip installed in /usr/local!

[julianoes]

The only way I can think of that this happens is if you used an older ubuntu.sh script which installs some updated pip there.

[TheLegendaryJedi]

But my case was a vanilla Ubuntu 18.04 installation.

A vanilla installation does not have pip installed in /usr/local!

If you make a fresh install of Ubuntu 18.04 in a VM and run:

git clone https://github.com/PX4/Firmware.git --recursive
cd Firmware
bash ./Tools/setup/ubuntu.sh
make px4_sitl gazebo
commander takeoff

You don't get that error?

[julianoes]

I just tried exactly your steps in a fresh 18.04 docker image:

root@e9072779c6bd:~/Firmware# which python
/usr/bin/python
root@e9072779c6bd:~/Firmware# python --version
Python 2.7.17
root@e9072779c6bd:~/Firmware# which python3
/usr/bin/python3
root@e9072779c6bd:~/Firmware# python3 --version
Python 3.6.9
root@e9072779c6bd:~/Firmware# which pip
root@e9072779c6bd:~/Firmware# which pip3
/usr/bin/pip3
root@e9072779c6bd:~/Firmware# pip3 --version
pip 9.0.1 from /usr/lib/python3/dist-packages (python 3.6)

And the build works:

-- Found PythonInterp: /usr/bin/python3 (found suitable version "3.6.9", minimum required is "3") 

[hamishwillee]

FYI, I just tried in in VM using freshly downloaded Ubuntu 18.04.4 and today's firmware. All worked fine.

[hamishwillee]

FYI Updated ubuntu script to suggest install source and run ubuntu.sh in place as first option: https://github.com/PX4/Devguide/blob/master/en/setup/dev_env_linux_ubuntu.md

[TheLegendaryJedi]

I just tried exactly your steps in a fresh 18.04 docker image:

root@e9072779c6bd:~/Firmware# which python
/usr/bin/python
root@e9072779c6bd:~/Firmware# python --version
Python 2.7.17
root@e9072779c6bd:~/Firmware# which python3
/usr/bin/python3
root@e9072779c6bd:~/Firmware# python3 --version
Python 3.6.9
root@e9072779c6bd:~/Firmware# which pip
root@e9072779c6bd:~/Firmware# which pip3
/usr/bin/pip3
root@e9072779c6bd:~/Firmware# pip3 --version
pip 9.0.1 from /usr/lib/python3/dist-packages (python 3.6)

And the build works:

-- Found PythonInterp: /usr/bin/python3 (found suitable version "3.6.9", minimum required is "3") 

Strange, I tried it two times in two different VMs. I guess I'll try it again today.

[TheLegendaryJedi]

Just tried it again today with a fresh install of Ubuntu 18.04 and it worked. The only thing I did different was not installing vscode and it's extensions. I don't know what happened before.

Thanks a lot for all the support.

[hamishwillee]

@TheLegendaryJedi Thanks for letting us know. FYI I didn't try VSCode /extensions. I guess it is possible installing their suggested cmake might break something.

[TheLegendaryJedi]

@TheLegendaryJedi Thanks for letting us know. FYI I didn't try VSCode /extensions. I guess it is possible installing their suggested cmake might break something.

Yes, probably was that. Because using the recommended extensions (suggested automatically by vscode) for the Project and sub modules installs cmake extension and python extensions that always tries to compile using cmake as soon as you open the project.

[julianoes]

@TheLegendaryJedi so vscode extensions ask for sudo in order to install to /usr/local?