Calibre-web: Issue with Upload, books with covers error out and are not imported

[eubanks718]

Describe the bug/problem
Book reports an error on upload. Upon checking directory folder, directory folder is created, both epub and cover are in folder. Epub is not imported into calibre database and is not shown when browsing calibre-web. Calibre-web is running from Python 3.7 as root. Permissions of the folder are rwx for user root, group 1000. If I run the command 'calibredb add -r "/mnt/calibre/Jack London/" --with-library /mnt/calibre' afterwards, then the book gets added to the library successfully and I am able to read it. This is both with direct IP access of calibre-web as well as reverse proxy access.

To Reproduce
Upload an epub from project Gutenberg.
So far, what I have noticed that uploading anything with a cover causes the issue. PDFs aren't affected.

Logfile
Add content of calibre-web.log file or the relevant error, try to reproduce your problem with "debug" log-level to get more output.

[2020-05-03 08:48:12,692] ERROR {cps:1892} Exception on /upload [POST]
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/shutil.py", line 566, in move
    os.rename(src, real_dst)
OSError: [Errno 18] Cross-device link: '/tmp/calibre_web/23d5b735e436b0e37180b0209fce5607.@public@vhost@g@gutenberg@html@files@215@215-h@[email protected]' -> '/mnt/calibre/Jack London/The Call of the Wild/cover.jpg'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/calibre-web/vendor/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/calibre-web/vendor/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/calibre-web/vendor/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/calibre-web/vendor/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/local/calibre-web/vendor/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/calibre-web/vendor/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/calibre-web/cps/web.py", line 165, in decorated_view
    return login_required(func)(*args, **kwargs)
  File "/usr/local/calibre-web/vendor/flask_login/utils.py", line 272, in decorated_view
    return func(*args, **kwargs)
  File "/usr/local/calibre-web/cps/web.py", line 236, in inner
    return f(*args, **kwargs)
  File "/usr/local/calibre-web/cps/editbooks.py", line 628, in upload
    move(meta.cover, os.path.join(filepath, "cover.jpg"))
  File "/usr/local/lib/python3.7/shutil.py", line 580, in move
    copy_function(src, real_dst)
  File "/usr/local/lib/python3.7/shutil.py", line 267, in copy2
    copystat(src, dst, follow_symlinks=follow_symlinks)
  File "/usr/local/lib/python3.7/shutil.py", line 211, in copystat
    lookup("chmod")(dst, mode, follow_symlinks=follow)
PermissionError: [Errno 1] Operation not permitted: '/mnt/calibre/Jack London/The Call of the Wild/cover.jpg'

Expected behavior
Expect the epub to be uploaded and imported into library

Screenshots
If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):
Program library Installed Version
Platform FreeBSD calibre 11.3-RELEASE-p7 FreeBSD 11.3-RELEASE-p7 #0 r325575+ca0f1a6ba25(HEAD): Tue Apr 21 20:46:20 UTC 2020 [email protected]:/freenas-releng/freenas/_BE/objs/freenas-releng/freenas/_BE/os/sys/FreeNAS.amd64 amd64 amd64
Python 3.7.7 (default, Apr 2 2020, 01:23:28) [Clang 8.0.0 (tags/RELEASE_800/final 356365)]
WebServer Tornado 6.0.4
Flask 1.1.2
Flask_Login 0.5.0
Flask_Principal 0.4.0
Werkzeug 1.0.1
Babel 2.8.0
Jinja2 2.11.2
Requests 2.23.0
SqlAlchemy 1.3.16
pySqlite 2.6.0
SQLite 3.30.1
iso639 0.4.5
pytz 2019.3
Unidecode installed
Flask_SimpleLDAP not installed
Goodreads not installed
Image Magick not installed
PyPdf v1.26.0
lxml v4.5.0.0
Wand not installed
Pillow v7.0.0
Comic_API not installed
ebook converter ebook-convert (calibre 4.13.0)

Additional context

[nicoolaro]

same error here when uploading epub, i teste 3 different files

[OzzieIsaacs]

One information is missing? Which version of Calibre-Web are you using? Looks like 0.6.6?

@nicoolaro: Are you also using FreeBSD? Which version of Calibre-Web are you using? Could you please doublecheck if the errormessage is the same. Have you also the following requirements installed: Pillow (yes) and Imagemagick (no), Wand (no), lxml (yes)?

[nicoolaro]

One information is missing? Which version of Calibre-Web are you using? Looks like 0.6.6?

@nicoolaro: Are you also using FreeBSD? Which version of Calibre-Web are you using? Could you please doublecheck if the errormessage is the same. Have you also the following requirements installed: Pillow (yes) and Imagemagick (no), Wand (no), lxml (yes)?

I am using 0.6.6. with Linuxserver docker. Not sure if it may be informative: I could upload PDF no problem, but when i changed policy.xml per instructions to export cover i got same error on PDF upload too.

Original error message here:

Internal Server Error

500 Internal Server Error: The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.

Traceback (most recent call last):

File "/usr/lib/python3.6/shutil.py", line 550, in move

os.rename(src, real_dst)

OSError: [Errno 18] Invalid cross-device link: '/tmp/calibre_web/b12fa20cb1951b4d4b00d5db22b1acfc.cover.png' -> '/books/Waldman, Diane & Lichtenstein, Roy, 1923-1997 & Solomon R. Guggenheim Museum & Museum of Contemporary Art (Los Angeles, Calif.) /Roy Lichtenstein/cover.jpg'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 2447, in wsgi_app

response = self.full_dispatch_request()

File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1952, in full_dispatch_request

rv = self.handle_user_exception(e)

File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1821, in handle_user_exception

reraise(exc_type, exc_value, tb)

File "/usr/local/lib/python3.6/dist-packages/flask/_compat.py", line 39, in reraise

raise value

File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1950, in full_dispatch_request

rv = self.dispatch_request()

File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1936, in dispatch_request

return self.view_functionsrule.endpoint

File "/app/calibre-web/cps/web.py", line 164, in decorated_view

return func(args, *kwargs)

File "/app/calibre-web/cps/web.py", line 236, in inner

return f(args, *kwargs)

File "/app/calibre-web/cps/editbooks.py", line 628, in upload

move(meta.cover, os.path.join(filepath, "cover.jpg"))

File "/usr/lib/python3.6/shutil.py", line 564, in move

copy_function(src, real_dst)

File "/usr/lib/python3.6/shutil.py", line 264, in copy2

copystat(src, dst, follow_symlinks=follow_symlinks)

File "/usr/lib/python3.6/shutil.py", line 205, in copystat

follow_symlinks=follow)

PermissionError: [Errno 1] Operation not permitted

Please report this issue with all related information: Create issue

Back to home

[eubanks718]

I'm using calibre-web 0.6.6

[OzzieIsaacs]

Calibre-Web is extracting the cover from the file and storing it into the temp folder. In the 1st example into file:
/tmp/calibre_web/23d5b735e436b0e37180b0209fce5607.@public@vhost@g@gutenberg@html@files@215@215-h@[email protected]
After creating everything else from the book(the database entries), the cover shall be copied to the final destination. this is done via python standard lib shutil move function. This function starts a call to the rename function, this call fails, because rename can only rename files on the same mount (the documentation talks about a foreign file system. Your temp folder is a different mount point from the books folder?) This is handled by the lib function and a workaround by copy and delete is started. The copy function now tries to copy also file metadata (...permission bits, last access time, last modification time, and flags from src to dst. [..] also copies the "extended attributes" where possible) and this fails with a permission error.
This is the explanation, but not the solution. Could you tell me something about both filesystems, is the books folder a smb mount or something else?

[eubanks718]

yeah. My books folder is mounted. It isn't quite smb mount. I use FreeNAS. I'm not sure exactly how familiar you are with FreeNas but my books folder is a separate dataset from the calibre-web dataset. All the datasets use zfs and all are contained under the same pool (which is the spread of hard drives). calibre-web runs in its own iocage jail (which uses the calibre-web dataset). I'm not quite sure the protocol but the jail can mount the books dataset into a folder of the calibre-web dataset in order to allow calibre-web to access and modify those files. I do know that these still are treated as two separate file systems since you can't hard-link between datasets but you can within an individual dataset. I never have any issues with copy or move functions.

Here is the documentation on the jail system if that helps any.
https://www.ixsystems.com/documentation/freenas/11.3-U2/jails.html

[nicoolaro]

In my case it is smb mount , do you think its user permission related?

[OzzieIsaacs]

Thanks for all the information.
I think I have an idea, by answering the question: Why is moving the book from the temp folder not causing a problem, only the cover file.
I think I will have a fix finished tomorrow.

[OzzieIsaacs]

I fixed it and also created a new release for it.