Octoprint: [1.4.0rc3] Exception uploading a gcode file (using Python 3)

[gdombiak]

E_PETG M3 test v1.gcode.zip

What were you doing?

While I was trying to create a PR to help run the CancelObject plugin with Python 3 and also newest OctoPrint, I ran into this problem.

1. Start OctoPrint :D
2. Go to Upload and upload attached gcode file
3. See error
   > Upload failed
   Could not upload the file. Make sure that it is a readable, valid file with one of these extensions: .g, .gco, .gcode, .msf

What did you expect to happen?

To be able to upload the gcode file

What happened instead?

Got the error

2019-12-13 20:01:21,754 - tornado.application - ERROR - Uncaught exception POST /api/files/local (::1)
HTTPServerRequest(protocol='http', host='localhost:5000', method='POST', uri='/api/files/local', version='HTTP/1.1', remote_ip='::1', headers={'Host': 'localhost:5000', 'Connection': 'keep-alive', 'Accept': 'application/json, text/javascript, /; q=0.01', 'Cache-Control': 'no-cache', 'Origin': 'http://localhost:5000', 'X-Requested-With': 'XMLHttpRequest', 'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.79 Safari/537.36', 'Sec-Fetch-Site': 'same-origin', 'Sec-Fetch-Mode': 'cors', 'Accept-Encoding': 'gzip, deflate, br', 'Accept-Language': 'en-US,en;q=0.9,es;q=0.8', 'Cookie': '_ga=GA1.1.1429836465.1558858415; cookieconsent_status=dismiss; session_P5000=.eJxdzrtuwzAMheF30RwUoi4U6TFu_BoGJVONgcAJLHkIir57g3Ypup7_G86nmeuu7WqGvh96MvO6mMEsmFRtUfUFOC8-QlHnOHjkDJG8lCzVc6iIxYtFqJq1gLgklpakVJFKJLAZfYixuJcWyZRDrEmlFiEGLokpRU41BeeQSSsJWc2czMmsi2597c83Ofp17s-HmmE7brc_5efqh_T7ix9N9_n_0LS19b79wounaQRvecIYxokI3ic3sgOOeInns_n6BkFQUA0.XfReiw.hWw--NOKL9uTh--LHhqCAVYT4lI'})
Traceback (most recent call last):
File "/Users/gaston/repo/OctoPrintPython3/venv/lib/python3.7/site-packages/tornado/web.py", line 1510, in _execute
result = method(self.path_args, *self.path_kwargs)
File "/Users/gaston/repo/OctoPrintPython3/src/octoprint/server/util/tornado.py", line 506, in _handle_method
self._fallback(self.request, body)
File "/Users/gaston/repo/OctoPrintPython3/src/octoprint/server/util/tornado.py", line 619, in __call__
WsgiInputContainer.environ(request, body), start_response)
File "/Users/gaston/repo/OctoPrintPython3/venv/lib/python3.7/site-packages/flask/app.py", line 1985, in wsgi_app
response = self.handle_exception(e)
File "/Users/gaston/repo/OctoPrintPython3/venv/lib/python3.7/site-packages/flask/app.py", line 1540, in handle_exception
reraise(exc_type, exc_value, tb)
File "/Users/gaston/repo/OctoPrintPython3/venv/lib/python3.7/site-packages/flask/_compat.py", line 33, in reraise
raise value
File "/Users/gaston/repo/OctoPrintPython3/venv/lib/python3.7/site-packages/flask/app.py", line 1982, in wsgi_app
response = self.full_dispatch_request()
File "/Users/gaston/repo/OctoPrintPython3/venv/lib/python3.7/site-packages/flask/app.py", line 1614, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/Users/gaston/repo/OctoPrintPython3/venv/lib/python3.7/site-packages/flask/app.py", line 1517, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/Users/gaston/repo/OctoPrintPython3/venv/lib/python3.7/site-packages/flask/_compat.py", line 33, in reraise
raise value
File "/Users/gaston/repo/OctoPrintPython3/venv/lib/python3.7/site-packages/flask/app.py", line 1612, in full_dispatch_request
rv = self.dispatch_request()
File "/Users/gaston/repo/OctoPrintPython3/venv/lib/python3.7/site-packages/flask/app.py", line 1598, in dispatch_request
return self.view_functionsrule.endpoint
File "/Users/gaston/repo/OctoPrintPython3/src/octoprint/server/util/flask.py", line 1297, in decorated_view
return func(args, *kwargs)
File "/Users/gaston/repo/OctoPrintPython3/src/octoprint/vendor/flask_principal.py", line 199, in _decorated
rv = f(args, *kw)
File "/Users/gaston/repo/OctoPrintPython3/src/octoprint/server/api/files.py", line 371, in uploadGcodeFile
display=canonFilename)
File "/Users/gaston/repo/OctoPrintPython3/src/octoprint/filemanager/__init__.py", line 501, in add_file
path_in_storage = self._storage(destination).add_file(path, file_object, links=links, printer_profile=printer_profile, allow_overwrite=allow_overwrite, display=display)
File "/Users/gaston/repo/OctoPrintPython3/src/octoprint/filemanager/storage.py", line 765, in add_file
file_object.save(file_path)
File "/Users/gaston/repo/OctoPrintPython3/src/octoprint/filemanager/util.py", line 92, in save
shutil.copyfileobj(source, dest)
File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/shutil.py", line 79, in copyfileobj
buf = fsrc.read(length)
File "/Users/gaston/repo/OctoPrintPython3/src/octoprint/filemanager/util.py", line 205, in read
result += processed_line
TypeError: can't concat str to bytes
2019-12-13 20:01:21,754 - tornado.access - ERROR - 500 POST /api/files/local (::1) 40.81ms

Did the same happen when running OctoPrint in safe mode?

Not tested

Version of OctoPrint

Latest from staging/devel. commit 74f3bc9ae6bcffcc86a6ad80f636d13e1fa3efda

Operating System running OctoPrint

2019-12-13 19:51:20,501 - octoprint.environment - INFO - Detected environment is Python 3.7.4 under Macos (darwin). Details:
| hardware:
| cores: 12
| freq: 3200
| ram: 17179869184
| os:
| id: macos
| platform: darwin
| python:
| pip: 19.3.1
| version: 3.7.4
| virtualenv: /Users/gaston/repo/OctoPrintPython3/venv

Printer model & used firmware incl. version

Browser and version of browser, operating system running browser

Link to octoprint.log

https://gist.github.com/gdombiak/476306a0108b2e402ef97744c9d23d19

Link to contents of terminal tab or serial.log

Link to contents of Javascript console in the browser

Screenshot(s)/video(s) showing the problem:

I have read the FAQ.

[gdombiak]

I'm more than a newbie with Python but found this comment on some Python forum that I think might be the reason for this error. Will keep investigating and who knows ... maybe contribute a PR

In Python 3, there's no implicit conversion between unicode (str) objects and bytes objects. If you know the encoding of the output, you can .decode() it to get a string, or you can turn the \n you want to add to bytes with "\n".encode('ascii')

[gdombiak]

Fixed created and tested both on Python 2 and 3. PR coming in a bit.

[gdombiak]

Created PR https://github.com/foosel/OctoPrint/pull/3393. Remember I'm not a Python dev so I hope fix is correct.

[gdombiak]

While testing the PR, I noticed that to reproduce the error I had to have a plugin like CancelObject installed so that src/octoprint/filemanager/util.py#LineProcessorStream#read() was used. It's possible that ExcludeRegion plugin helps test this too but haven't tested that. I think it extends same method that causes the execution of that method.

[foosel]

I'm just wrapping up things for 2019 and will take a proper look at both the issue and the PR when I'm back at work. Thank you!

[foosel]

So I'm a tad confused. You closed the PR as "fixed in the plugin". So was that a plugin issue only or what?

[gdombiak]

Hi Gina, welcome back!

Correct. The plugin was converting str into unicode and returning them. This was then causing an exception in OctoPrint since the variable should have been a String when it wasn't.

Gaston