Beets: FetchArt crashes with TypeError for particular album

[bartkl]

I've imported hundreds of albums just fine using fetchart in auto mode, and then this one came along and crashed the import.

The issue in short:

  File "/usr/local/lib/python3.7/dist-packages/beetsplug/fetchart.py", line 416, in get
    self.API_ALBUMS + album.mb_releasegroupid,
TypeError: can only concatenate str (not "int") to str

So it seems album.mb_releasegroupid is an int. This seems like a bug.

Problem

Running this command in verbose (-vv) mode:

$ beet -vv im --set=genre=Vaporwave Vektroid\ -\ Color\ Ocean\ Road/

Led to this problem:

user configuration: /media/droppie/libraries/music/.meta/beets/config.yaml
data directory: /media/droppie/libraries/music/.meta/beets
plugin paths: /opt/whatlastgenre/plugin/beets/beetsplug
Sending event: pluginload
library database: /media/droppie/libraries/music/.meta/beets/library.db
library directory: /media/droppie/libraries/music
Sending event: library_opened
Sending event: import_begin
Sending event: import_task_created
Sending event: import_task_start
Looking up: /media/droppie/data/music/Vektroid - Color Ocean Road
Tagging Vektroid - Color Ocean Road
No album ID found.
Search terms: Vektroid - Color Ocean Road
Album might be VA: False
Searching for MusicBrainz releases with: {'release': 'color ocean road', 'artist': 'vektroid', 'tracks': '6'}
Requesting MusicBrainz release acfcb884-a136-44d3-a537-65194f07bf59
primary MB release type: album
Sending event: albuminfo_received
Candidate: Vektroid - Color Ocean Road (acfcb884-a136-44d3-a537-65194f07bf59)
Computing track assignment...
...done.
Success. Distance: 0.19
Requesting MusicBrainz release 42fa86c0-c28c-4850-ba6a-b2696808dabf
primary MB release type: album
Sending event: albuminfo_received
Candidate: Vektroid - RE•SET (42fa86c0-c28c-4850-ba6a-b2696808dabf)
Computing track assignment...
...done.
Success. Distance: 0.65
Requesting MusicBrainz release 6a832aac-a80b-4db2-a1d7-dc65d92a982d
primary MB release type: album
Sending event: albuminfo_received
Candidate: Vektroid - Telnet Complete (6a832aac-a80b-4db2-a1d7-dc65d92a982d)
Computing track assignment...
...done.
Success. Distance: 0.76
Requesting MusicBrainz release 043e9481-f864-4676-9888-924f026fa3dd
primary MB release type: album
Sending event: albuminfo_received
Candidate: Vektroid - Seed & Synthetic Earth (043e9481-f864-4676-9888-924f026fa3dd)
Computing track assignment...
...done.
Success. Distance: 0.67
Requesting MusicBrainz release 12ec78f1-169a-408c-9d9e-a699901d133c
primary MB release type: broadcast
secondary MB release type(s): dj-mix
Sending event: albuminfo_received
Candidate: Vektroid - FACT Mix 619: Vektroid (Sept '17) (12ec78f1-169a-408c-9d9e-a699901d133c)
Computing track assignment...
...done.
Success. Distance: 0.65
discogs: Searching for master release 1520897
discogs: hit rate limit, waiting for 0.9732460975646973 seconds
discogs: Searching for master release 1520897
discogs: hit rate limit, waiting for 0.9869840145111084 seconds
discogs: Searching for master release 1520897
discogs: hit rate limit, waiting for 0.9847149848937988 seconds
Sending event: albuminfo_received
Candidate: Vektroid - Color Ocean Road (4820006)
Computing track assignment...
...done.
Success. Distance: 0.22
Sending event: albuminfo_received
Candidate: Vektroid - Color Ocean Road (13385176)
Computing track assignment...
...done.
Success. Distance: 0.05
Sending event: albuminfo_received
Candidate: Vektroid - Color Ocean Road (13395427)
Computing track assignment...
...done.
Success. Distance: 0.22
Evaluating 8 candidates.

/media/droppie/data/music/Vektroid - Color Ocean Road (6 items)
Sending event: before_choose_candidate
Tagging:
    Vektroid - Color Ocean Road
URL:
    https://www.discogs.com/Vektroid-Color-Ocean-Road/release/13385176
(Similarity: 95.2%) (source) (Discogs, File, 2012, US, PrismCorp)
[A]pply, More candidates, Skip, Use as-is, as Tracks, Group albums,
Enter search, enter Id, aBort, eDit, edit Candidates? 
Sending event: import_task_choice
Sending event: import_task_apply
Replacing item 2040: /media/droppie/data/music/Vektroid - Color Ocean Road/01 Shalom.mp3
Sending event: database_change
Sending event: item_removed
Replacing item 2041: /media/droppie/data/music/Vektroid - Color Ocean Road/02 Color Ocean.mp3
Sending event: database_change
Sending event: item_removed
Replacing item 2042: /media/droppie/data/music/Vektroid - Color Ocean Road/03 Seafoam Island.mp3
Sending event: database_change
Sending event: item_removed
Replacing item 2043: /media/droppie/data/music/Vektroid - Color Ocean Road/04 Sushi Plaza.mp3
Sending event: database_change
Sending event: item_removed
Replacing item 2044: /media/droppie/data/music/Vektroid - Color Ocean Road/05 Mango _ Fuji.mp3
Sending event: database_change
Sending event: item_removed
Replacing item 2045: /media/droppie/data/music/Vektroid - Color Ocean Road/06 Om Namo Ocean Road.mp3
Sending event: database_change
Sending event: database_change
Sending event: item_removed
6 of 6 items replaced
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Reimported album: added 1602800480.7681532, flexible attributes [] from album 185 for /media/droppie/data/music/Vektroid - Color Ocean Road
Reimported item added 1602800480.7766094 from item 2040 for /media/droppie/data/music/Vektroid - Color Ocean Road/01 Shalom.mp3
Reimported item flexible attributes ['track_alt', 'data_source', 'track_alt', 'data_source'] from item 2040 for /media/droppie/data/music/Vektroid - Color Ocean Road/01 Shalom.mp3
Sending event: database_change
Reimported item added 1602800480.7842014 from item 2041 for /media/droppie/data/music/Vektroid - Color Ocean Road/02 Color Ocean.mp3
Reimported item flexible attributes ['track_alt', 'data_source', 'track_alt', 'data_source'] from item 2041 for /media/droppie/data/music/Vektroid - Color Ocean Road/02 Color Ocean.mp3
Sending event: database_change
Reimported item added 1602800480.7913542 from item 2042 for /media/droppie/data/music/Vektroid - Color Ocean Road/03 Seafoam Island.mp3
Reimported item flexible attributes ['track_alt', 'data_source', 'track_alt', 'data_source'] from item 2042 for /media/droppie/data/music/Vektroid - Color Ocean Road/03 Seafoam Island.mp3
Sending event: database_change
Reimported item added 1602800480.7984622 from item 2043 for /media/droppie/data/music/Vektroid - Color Ocean Road/04 Sushi Plaza.mp3
Reimported item flexible attributes ['track_alt', 'data_source', 'track_alt', 'data_source'] from item 2043 for /media/droppie/data/music/Vektroid - Color Ocean Road/04 Sushi Plaza.mp3
Sending event: database_change
Reimported item added 1602800480.8071747 from item 2044 for /media/droppie/data/music/Vektroid - Color Ocean Road/05 Mango _ Fuji.mp3
Reimported item flexible attributes ['track_alt', 'data_source', 'track_alt', 'data_source'] from item 2044 for /media/droppie/data/music/Vektroid - Color Ocean Road/05 Mango _ Fuji.mp3
Sending event: database_change
Reimported item added 1602800480.8171701 from item 2045 for /media/droppie/data/music/Vektroid - Color Ocean Road/06 Om Namo Ocean Road.mp3
Reimported item flexible attributes ['track_alt', 'data_source', 'track_alt', 'data_source'] from item 2045 for /media/droppie/data/music/Vektroid - Color Ocean Road/06 Om Namo Ocean Road.mp3
Sending event: database_change
Set field genre=Vaporwave for /media/droppie/data/music/Vektroid - Color Ocean Road
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
fetchart: trying source filesystem for album Vektroid - Color Ocean Road
fetchart: trying source coverart for album Vektroid - Color Ocean Road
fetchart: downloading image: https://coverartarchive.org/release/13385176/front
fetchart: not a supported image: image/x-None
fetchart: trying source coverart for album Vektroid - Color Ocean Road
fetchart: downloading image: https://coverartarchive.org/release-group/1520897/front
fetchart: not a supported image: image/x-None
fetchart: trying source itunes for album Vektroid - Color Ocean Road
fetchart: getting URL: https://itunes.apple.com/search?term=Vektroid+Color+Ocean+Road&entity=album&media=music&limit=200
fetchart: iTunes search for 'Vektroid Color Ocean Road' got no results
fetchart: trying source amazon for album Vektroid - Color Ocean Road
fetchart: trying source albumart for album Vektroid - Color Ocean Road
fetchart: trying source wikipedia for album Vektroid - Color Ocean Road
fetchart: getting URL: https://dbpedia.org/sparql?format=application%2Fsparql-results%2Bjson&timeout=2500&query=PREFIX+rdf%3A+%3Chttp%3A%2F%2Fwww.w3.org%2F1999%2F02%2F22-rdf-syntax-ns%23%3E%0A+++++++++++++++++PREFIX+dbpprop%3A+%3Chttp%3A%2F%2Fdbpedia.org%2Fproperty%2F%3E%0A+++++++++++++++++PREFIX+owl%3A+%3Chttp%3A%2F%2Fdbpedia.org%2Fontology%2F%3E%0A+++++++++++++++++PREFIX+rdfs%3A+%3Chttp%3A%2F%2Fwww.w3.org%2F2000%2F01%2Frdf-schema%23%3E%0A+++++++++++++++++PREFIX+foaf%3A+%3Chttp%3A%2F%2Fxmlns.com%2Ffoaf%2F0.1%2F%3E%0A%0A+++++++++++++++++SELECT+DISTINCT+%3FpageId+%3FcoverFilename+WHERE+%7B%0A+++++++++++++++++++%3Fsubject+owl%3AwikiPageID+%3FpageId+.%0A+++++++++++++++++++%3Fsubject+dbpprop%3Aname+%3Fname+.%0A+++++++++++++++++++%3Fsubject+rdfs%3Alabel+%3Flabel+.%0A+++++++++++++++++++%7B+%3Fsubject+dbpprop%3Aartist+%3Fartist+%7D%0A+++++++++++++++++++++UNION%0A+++++++++++++++++++%7B+%3Fsubject+owl%3Aartist+%3Fartist+%7D%0A+++++++++++++++++++%7B+%3Fartist+foaf%3Aname+%22Vektroid%22%40en+%7D%0A+++++++++++++++++++++UNION%0A+++++++++++++++++++%7B+%3Fartist+dbpprop%3Aname+%22Vektroid%22%40en+%7D%0A+++++++++++++++++++%3Fsubject+rdf%3Atype+%3Chttp%3A%2F%2Fdbpedia.org%2Fontology%2FAlbum%3E+.%0A+++++++++++++++++++%3Fsubject+dbpprop%3Acover+%3FcoverFilename+.%0A+++++++++++++++++++FILTER+%28+regex%28%3Fname%2C+%22Color+Ocean+Road%22%2C+%22i%22%29+%29%0A++++++++++++++++++%7D%0A+++++++++++++++++Limit+1
fetchart: wikipedia: album not found on dbpedia
fetchart: trying source fanarttv for album Vektroid - Color Ocean Road
Traceback (most recent call last):
  File "/usr/local/bin/beet", line 10, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/dist-packages/beets/ui/__init__.py", line 1266, in main
    _raw_main(args)
  File "/usr/local/lib/python3.7/dist-packages/beets/ui/__init__.py", line 1253, in _raw_main
    subcommand.func(lib, suboptions, subargs)
  File "/usr/local/lib/python3.7/dist-packages/beets/ui/commands.py", line 955, in import_func
    import_files(lib, paths, query)
  File "/usr/local/lib/python3.7/dist-packages/beets/ui/commands.py", line 925, in import_files
    session.run()
  File "/usr/local/lib/python3.7/dist-packages/beets/importer.py", line 329, in run
    pl.run_parallel(QUEUE_SIZE)
  File "/usr/local/lib/python3.7/dist-packages/beets/util/pipeline.py", line 445, in run_parallel
    six.reraise(exc_info[0], exc_info[1], exc_info[2])
  File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise
    raise value
  File "/usr/local/lib/python3.7/dist-packages/beets/util/pipeline.py", line 312, in run
    out = self.coro.send(msg)
  File "/usr/local/lib/python3.7/dist-packages/beets/util/pipeline.py", line 194, in coro
    func(*(args + (task,)))
  File "/usr/local/lib/python3.7/dist-packages/beets/importer.py", line 1511, in plugin_stage
    func(session, task)
  File "/usr/local/lib/python3.7/dist-packages/beets/plugins.py", line 143, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/beetsplug/fetchart.py", line 854, in fetch_art
    candidate = self.art_for_album(task.album, task.paths, local)
  File "/usr/local/lib/python3.7/dist-packages/beetsplug/fetchart.py", line 920, in art_for_album
    for candidate in source.get(album, self, paths):
  File "/usr/local/lib/python3.7/dist-packages/beetsplug/fetchart.py", line 416, in get
    self.API_ALBUMS + album.mb_releasegroupid,
TypeError: can only concatenate str (not "int") to str

Setup

• OS: Raspbian Buster
• Python version: 3.7.3
• beets version: 1.4.9
• Turning off plugins made problem go away (yes/no): n/a

My configuration (output of beet config) is:

library: /media/droppie/libraries/music/.meta/beets/library.db
directory: /media/droppie/libraries/music
original_date: yes
per_disc_numbering: no
va_name: "\U0001F465 Various Artists"

import:
    write: no
    copy: no
    move: no
    link: yes

    resume: ask
    incremental: yes
    quiet_fallback: skip
    none_rec_action: ask
    timid: yes
    default_action: apply
    log: /var/log/beets/import.log
    languages:
        - en
    detail: no
    group_albums: no
    autotag: yes
    duplicate_action: ask
    incremental_skip_later: no
    set_fields:
        genre: "\U0001F3A7 New"

pluginpath:
    /opt/whatlastgenre/plugin/beets/beetsplug

plugins: discogs edit fetchart bucket replaygain info follow wlg types

types:
    rating: int

edit:
    itemfields: track artist title genre albumartist
    albumfields: albumartist album genre rating

fetchart:
    sources:
        - filesystem
        - coverart: release releasegroup
        - itunes
        - '*'
    auto: yes
    art_filename: cover.jpg
    store_source: yes

bucket:
    bucket_year: ['1930s', '1940s', '1950s', '1960s', '1970s', '1980s', '1990s', '2000s', '2010s', '2020s']

paths:
    default: %bucket{$year}/$year - $albumartist - $album/$track - $title
    comp: %bucket{$year}/$year - $albumartist - $album/$track - $artist - $title

replaygain:
    backend: gstreamer
    auto: no
    overwrite: no
    targetlevel: 89

follow:
    auto: no
    email: [email protected]
    password: XXX
    userid: XXX

wlg:
    auto: no
    force: no
    count: 3
    separator: ', '

web:
    host: 0.0.0.0
    cors: '*'

discogs:
    user_token: XXX

[sampsyo]

Huh, this is quite odd. It should be impossible for album.mb_releasegroupid to be anything other than a string. The beets data model is supposed to automatically convert data so we get consistent types—and that field is declared as a string.

Is there any way you could either share your database file or narrow down what's going on with that particular album—to check whether its SQLite entry looks different from all other albums, for some reason?

[bartkl]

Okay, this is what I've come up with so far:

• Most of my items have actual MusicBrainz release group IDs, while this one has one from Discogs (see log above). What's particularly interesting is that those consist only of digits, no letters. That sounds relevant.
• When I open a manual session to the Beets library in a REPL, the mb_releasegroup_id field is a string as you'd expect, also for that album.

So what this looks like to me is that somewhere down the line in this import process, the album object has been modified in such a way that the type of that field has changed.

Any ideas?

[bartkl]

I've dug deep and after lots of debugging print statements I think I've found the issue :). Along the way I monitored the type of the releaseinfo_id field this way, and it seemed to have always been int.

Here's my explanation.

During the import process, the matched info is copied to the library model, which is then added to the library.
In particular, for this field (mb_releasegroupid), I believe this line (140) in beets.autotag tells us where it comes from:

        item.mb_releasegroupid = album_info.releasegroup_id

So I thought: since Discogs was the data source here (which uses all-digits IDs as I said), I might want to see how that plugin builds up that metadata.

Please pay attention to this part of beetsplug.discogs (starting at line 332) which I believe is responsible for this:

        # Retrieve master release id (returns None if there isn't one).
        master_id = result.data.get('master_id')
        # Assume `original_year` is equal to `year` for releases without
        # a master release, otherwise fetch the master release.
        original_year = self.get_master_year(master_id) if master_id else year

        return AlbumInfo(album, album_id, artist, artist_id, tracks, asin=None,
                         albumtype=albumtype, va=va, year=year, month=None,
                         day=None, label=label, mediums=len(set(mediums)),
                         artist_sort=None, releasegroup_id=master_id,
                         catalognum=catalogno, script=None, language=None,
                         country=country, albumstatus=None, media=media,
                         albumdisambig=None, artist_credit=None,
                         original_year=original_year, original_month=None,
                         original_day=None, data_source='Discogs',
                         data_url=data_url)

Now, what I did is log the type of master_id (which is used for the releasegroup_id as you can see), which turned out to be int!

So, mystery solved I guess? I'm a but uncertain because I don't know this code too well, and I also don't know how to judge this situation exactly. But it feels like this is where the issue occurs.

[sampsyo]

Hmm, this is definitely pointing in the right direction, and perhaps we should just change this here to stringily the ID. But it is still somewhat confusing because the Album object (like any dbcore.Model object) is supposed to normalize the types of the values it gets assigned. So this line, for instance:

item.mb_releasegroupid = album_info.releasegroup_id

should be setting the field to be a string, even if the RHS of the assignment is an integer. Clearly that's not happening here, but I'm still sort of stumped about why.

Anyway, like I said, maybe we should just stick our heads in the sand and do the conversion at that point, which would definitely be nice to do anyway. But I remain slightly disturbed that the automatic type normalization that's supposed to go on at that assignment is apparently not happening. 😱

[bartkl]

Yes, that is unsettling. I don't like not understanding it properly, so I dug a little deeper.

>>> i = lib.get_item(2)
>>> print(i.mb_releasegroupid, type(i.mb_releasegroupid))
c2841928-501f-37f7-a59e-cb1629471258 <class 'str'>
>>> i.mb_releasegroupid = 1
>>> print(i.mb_releasegroupid, type(i.mb_releasegroupid))
1 <class 'int'>
>>> i.store()
>>> i = lib.get_item(2)
>>> print(i.mb_releasegroupid, type(i.mb_releasegroupid))
1 <class 'str'>

So, the Item model object does allow the assignment this way, but after storing, it is normalized, right? Not sure if this finding contributes or is obvious. At least I myself am learning ;).

Is it possible that this has something to do with the problem? That maybe somehow FetchArt uses the direct model instance, instead of retrieving it after storing?

[sampsyo]

Wow, awesome (and scary re. the implications of the problem)—thank you for digging more deeply into this. Is there any chance you might be able to give the proposed fix in #3774 a shot to see if it resolves the problem?

[bartkl]

Do you still want me to test this? Or should we wait for the discussion in #3774 to progress further first?

[sampsyo]

If you wouldn't mind testing it, that would be awesome. If it works I think we should merge it as-is and revisit the broader change later.

[bartkl]

Okay. It won't be before Sunday evening at the earliest, but I'll let you know!

[bartkl]

I just patched my Beets installation with this change, and it resolved this issue :).

[sampsyo]

Yay!! I'll merge it.