Platformio-core: REFUSAL to compile if api.platformio.org is down!
What kind of issue is this?
[ ] Question.
This issue tracker is not the place for questions. If you want to ask how to do something,
or to understand why something isn't working the way you expect it to,
use Community Forums or Premium Support[ ] PlatformIO IDE.
All issues related to PlatformIO IDE should be reported to appropriate repository:
PlatformIO IDE for Atom or
PlatformIO IDE for VSCode[ ] Development Platform or Board.
All issues (building, uploading, adding new boards, etc.) related to PlatformIO development platforms
should be reported to appropriate repository related to your hardware
https://github.com/topics/platformio-platform[ ] Feature Request.
Start by telling us what problem you鈥檙e trying to solve. Often a solution
already exists! Don鈥檛 send pull requests to implement new features without first getting our
support. Sometimes we leave features out on purpose to keep the project small.[x ] PlatformIO Core.
If you鈥檝e found a bug, please provide an information below.
You can erase any parts of this template not applicable to your Issue.
Configuration
Operating system:
PlatformIO Version (platformio --version):
4.1.0
Description of problem
If api.platformio.org is down then platformio refuses to compile.
See this issue: https://github.com/esphome/issues/issues/948
NOTE: This is after the first compile ran successfully; after having downloaded all needed libraries (older versions than the current ones online).
Steps to Reproduce
3.
Actual Results
Prints an exception and bails out. Example log:
esphome plug_1.yaml compile
INFO Reading configuration plug_1.yaml...
INFO Generating C++ source...
INFO Compiling app...
INFO Running: platformio run -d plug_1
Processing plug_1 (board: esp8285; framework: arduino; platform: [email protected])
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
HARDWARE: ESP8266 80MHz, 80KB RAM, 1MB Flash
Looking for ESPAsyncTCP-esphome library in registry
APIRequestError: [API] Invalid response: b'<html>\r\n<head><title>404 Not Found</title></head>\r\n<body bgcolor="white">\r\n<center><h1>404 Not Found</h1></center>\r\n<hr><center>nginx/1.14.0 (Ubuntu)</center>\r\n</body>\r\n</html>\r\n':
File "/home/hassdev/.local/lib/python3.8/site-packages/platformio/builder/main.py", line 151:
env.SConscript("$BUILD_SCRIPT")
File "/home/hassdev/.platformio/packages/tool-scons/script/../engine/SCons/Script/SConscript.py", line 605:
return _SConscript(self.fs, *files, **subst_kw)
File "/home/hassdev/.platformio/packages/tool-scons/script/../engine/SCons/Script/SConscript.py", line 286:
exec(compile(scriptdata, scriptname, 'exec'), call_stack[-1].globals)
File "/home/hassdev/.platformio/platforms/espressif8266/builder/main.py", line 203:
target_elf = env.BuildProgram()
File "/home/hassdev/.platformio/packages/tool-scons/script/../engine/SCons/Environment.py", line 224:
return self.method(*nargs, **kwargs)
File "/home/hassdev/.local/lib/python3.8/site-packages/platformio/builder/tools/platformio.py", line 140:
_build_project_deps(env)
File "/home/hassdev/.local/lib/python3.8/site-packages/platformio/builder/tools/platformio.py", line 48:
project_lib_builder = env.ConfigureProjectLibBuilder()
File "/home/hassdev/.platformio/packages/tool-scons/script/../engine/SCons/Environment.py", line 224:
return self.method(*nargs, **kwargs)
File "/home/hassdev/.local/lib/python3.8/site-packages/platformio/builder/tools/piolib.py", line 1027:
project.install_dependencies()
File "/home/hassdev/.local/lib/python3.8/site-packages/platformio/builder/tools/piolib.py", line 859:
lm.install(uri)
File "/home/hassdev/.local/lib/python3.8/site-packages/platformio/managers/lib.py", line 289:
name = "id=%d" % self.search_lib_id(
File "/home/hassdev/.local/lib/python3.8/site-packages/platformio/managers/lib.py", line 184:
result = util.get_api_result(
File "/home/hassdev/.local/lib/python3.8/site-packages/platformio/util.py", line 349:
result = _get_api_result(url, params, data)
File "/home/hassdev/.local/lib/python3.8/site-packages/platformio/util.py", line 81:
return func(*args, **kwargs)
File "/home/hassdev/.local/lib/python3.8/site-packages/platformio/util.py", line 323:
raise exception.APIRequestError("Invalid response: %s" % r.text.encode("utf-8"))
==================================================================================== [FAILED] Took 0.87 seconds ====================================================================================
Expected Results
Print the exception and continue compiling with locally available libraries.
erlanger
All 10 comments
ivankravets
on 27 Dec 2019
@ivankravets thanks for your reply, and for the fact that the website is up!
However, I don't think it is really resolved, because the client still depends on the website being up in order to compile. What I would humbly suggest is to do the following in the code:
- Try to contact
api.platformio.org, if the reply does not include the expected json* then: - Print a warning and use the locally available dependencies to compile: "WARNING: Using locally available dependencies because of lack of response from api.platformio.org"
- If there are no locally available dependencies because it is the first time a compilation is done, then print an error and exit saying: "ERROR: first time compilation requires api.platformio.org to be accessible"
This will make the client resilient and stable.
Hopefully you will consider this, and thank you so much for your great work!
(*) This includes any kind of error, not just 4XX or 5XX or lack of network connectivity but even getting some unexpected json reply without the proper data
erlanger
on 27 Dec 2019
Yeah, I don't see how forcing online checks is acceptable.
brandond
on 28 Dec 2019
PlatformIO does not depend on api.platformio.org if you don't use packages from the registry. You can use lib_deps and set direct URLs to your deps, for example, to Github. See https://docs.platformio.org/en/latest/librarymanager/index.html
Sorry for this issue. This is the first big outage since the 2014 year. We didn't expect. We used own bare metal server and the hardware issue arose (problem with power control on the motherboard).
The second problem, it has happened during the Christmas holidays. We tried to order a new one but no one wanted to do this in "next hours". As a result, we moved to Cloud VPS and restored everything from backup. It took some time. These are gigabytes of data.
We still monitor current VPS and it works well. In any case, we plan to upgrade our cloud infrastructure which be ready for the similar issues when some nodes goes offline.
ivankravets
on 28 Dec 2019
@ivankravets, thanks for your reply.
There is something strange happening here. This same issue came up in 2018 (when @OttoWinter and @seksity reported it) , and the client was left dependent on online access. You took the same position: "the server is up now, don't worry about it". This is very strange.
There is absolutely no technical reason why the client needs to force online access even for packages in the registry; it can always fall back to locally installed packages.
So the real reason you don't want to make the client more independent must not be technical. This leads me to think that some company or group of companies is giving you substantial donations, which allow you to run the website and continue development.
Maybe some company or group of companies is saying to you that they will reduce their donations if they can't gather usage statistics or something similar.
I am totally okay with that, and I am sure many others will be okay with it. If that is the case simply say: "I collect usage statistics which allow me to fund the hosting and the development costs, so I can't make the client more independent from online access".
Don't be afraid to be honest, the community will always respond well to honesty, and many, including myself will support you.
SIDE NOTE: just an aside on the possible dangers of funding for open source projects: Why I forked my own project and my own company ownCloud to Nextcloud
erlanger
on 28 Dec 2019
There is absolutely no technical reason why the client needs to force online access even for packages in the registry; it can always fall back to locally installed packages.
This is what PlatformIO Core does if you installed all dependencies before. It does not use on-line registry for each incremental build. The only if you declare dependencies in platformio.ini and PlatformIO Core LDF can not find them.
Maybe some company or group of companies is saying to you that they will reduce their donations if they can't gather usage statistics or something similar.
What do you mean? Everything is open source. See https://github.com/platformio/platformio-api
Check this source code, the only one stats which we use is DL LOG which we need to provide TRENDING POPULAR libraries. See https://platformio.org/lib
If you don't want to use PlatformIO Regstry - just copy libraries to lib folder or use other source URIs in lib_deps, such as, tarball, github, etc. See docs https://docs.platformio.org/en/latest/userguide/lib/cmd_install.html
I collect usage statistics which allow me to fund the hosting and the development costs
There is no me, there is PlatformIO Labs company behind PlatformIO project. We provide commercial services for semiconductor and OEM companies that helps to maintain this project. We are fully independent and are neutral to any architecture, software, IDE, operating system.
Today, PlatformIO is THE ONLY neutral and fully open source ecosystem for embedded development. Just take a look on Arduino.cc which was sold to ARM. There is no freedom, all new tools and services are closed source, and each new board is the only ARM based. We do not want to have the same with PlatformIO. Our users should have a choice which architecture, operating system, IDE to use.
If you have any ideas on how we can improve our tools, please open a new issue. Thanks!
ivankravets
on 28 Dec 2019
I see that you are being honest, and that you honestly think that the client is not forcing unneeded online access. I apologize for suggesting there may be other reasons.
api.platformio.org is down again and I can not compile.
erlanger
on 30 Dec 2019
ivankravets
on 30 Dec 2019
@ivankravets thanks for your prompt reply!
This time it was my fault, I had changed /etc/hosts to test the issue and I forgot to change it back.
It is working now.
Test method
I think it would be useful to setup a test scenario in the following manner:
- Change the IP address of
api.platformio.orgin/etc/hoststo the IP address of the local machine, a line like this in/etc/hostswill work:
api.platformio.org 127.0.0.1
- Setup a python web server by running
python -m SimpleHTTPServer <port used by api.platformio.org> - Try to compile and see if it fails (because the webserver will not reply with the expected json)
This test scenario will allow us to test that platformio is truly resilient to server outages.
erlanger
on 30 Dec 2019
BTW, see this comment at esphome to see that platformio does not make it easy to work offline:
https://github.com/esphome/issues/issues/948#issuecomment-569919940
erlanger
on 31 Dec 2019
Related issues
Navis-Raven
路
3Comments
weberw
路
4Comments
RayEbb
路
4Comments
positron96
路
5Comments
HamzaHajeir
路
5Comments
Most helpful comment
@ivankravets thanks for your reply, and for the fact that the website is up!
However, I don't think it is really resolved, because the client still depends on the website being up in order to compile. What I would humbly suggest is to do the following in the code:
api.platformio.org, if the reply does not include the expected json* then:This will make the client resilient and stable.
Hopefully you will consider this, and thank you so much for your great work!
(*) This includes any kind of error, not just 4XX or 5XX or lack of network connectivity but even getting some unexpected json reply without the proper data