Beets: Updates "r128_album_gain: 000000 -> 000000" With or without replaygain plugin

[MDCEY]

Problem

Running beets update leeds to the r128 tags been updated to 000000 even if its already at 000000. If I run the replaygain plugin manually this tag is updated correctly, only to be wiped again when beet update is run.

Running this command in verbose (-vv) mode:

$ beet -vv update

Led to this problem:

skipping /mnt/data/music/Pearl Jam/Vitalogy/06 Whipping.ogg because mtime is up to date (1537958068.0)
Pearl Jam - Ten - Alive
  r128_album_gain: 000000 -> 000000
Sending event: before_item_moved
Sending event: item_moved
Sending event: database_change
Pearl Jam - Ten - Why Go
  r128_album_gain: 000000 -> 000000
Sending event: before_item_moved
Sending event: item_moved
Sending event: database_change

(Happens to all my music)

Setup

• OS: Ubuntu bionic
• Python version: 3.6.5
• beets version: 1.4.8
• Turning off plugins made problem go away (yes/no): No

My configuration (output of beet config) is:

# Physical Location of your *Processed* music
directory: /mnt/data/music
# Where to store your beets database
library: /home/admin/.config/beets/musiclibrary.db

import:
  # write metadata to music files
  write: yes

  # move imported files from source to the music directory
  move: yes
  copy: no
  delete: no

  # use auto-tagging where possible
  # do not require confirmation on strong matches
  autotag: yes
  timid: no

  resume: ask
  incremental: no
  none_rec_action: ask
  log: /home/admin/logs/beets.log

  quiet: no
  quiet_fallback: skip
  default_action: apply


  singletons: no
  detail: no
  flat: no

#  duplicate_action: skip

# use the release-date of the original (first) release of an album?
original_date: yes

# on multi-disk releases, assign track numbers for the whole album.
# If "per disk", make sure tracknames do not collide ("paths" setting).
per_disc_numbering: no

# files matching these patterns are deleted from source after import
clutter: ["Thumbs.DB", ".DS_Store", "*.m3u", ".pls", "*.jpg"]

# files/directories matching one of these patterns are ignored during import
ignore: [".*", "*~", "System Volume Information"]

# Paths and filenames for music files
# relative to music directory
paths:
    default: $albumartist/$album%aunique{}/$track $title
    singleton: Non-Album/$artist/$title
    comp: Compilations/$album%aunique{}/$track $title

replace:
    '[\\/]': _
    '^\.': _
    '[\x00-\x1f]': _
    '[<>:"\?\*\|]': _
    '\.$': _
    '\s+$': ''

path_sep_replace: _


# filename for the album art
art_filename: cover  # results in "cover.jpg"


# General --------------------------------------------------------------------

# use mutliple threads during import
threaded: yes
timeout: 5.0
verbose: no

# User Interface -------------------------------------------------------------

color: yes
list_format_item: %upper{$artist} - $album - $track. $title
list_format_album: %upper{$albumartist} - $album
time_format: '%Y-%m-%d %H:%M:%S'
terminal_encoding: utf8

ui:
    terminal_width: 80
    length_diff_thresh: 10.0


# Auto Tagger ----------------------------------------------------------------

match:
    strong_rec_thresh: 0.15      # match 90% or better for auto import
    medium_rec_thresh: 0.25
    rec_gap_thresh: 0.25
    distance_weights:
        source: 2.0
        artist: 3.0
        album: 3.0
        media: 1.0
        mediums: 1.0
        year: 1.0
        country: 0.5
        label: 0.5
        catalognum: 0.5
        albumdisambig: 0.5
        album_id: 5.0
        tracks: 2.0
        missing_tracks: 0.9
        unmatched_tracks: 0.6
        track_title: 3.0
        track_artist: 2.0
        track_index: 1.0
        track_length: 2.0
        track_id: 5.0
    preferred:
        countries: []
        media: []
        original_year: no
    ignored: missing_tracks unmatched_tracks
    track_length_grace: 10
    track_length_max: 30


# Plugins -------------------------------------------------------------------

plugins: beatport chroma convert discogs fetchart ftintitle info lastgenre lyrics missing replaygain scrub web acousticbrainz

acousticbrainz:
  auto: yes

chroma:
  auto: yes

convert:
  auto: yes
  format: opus
  formats:
    opus:
      command: ffmpeg -i $source -y -vn -acodec libopus -ab 96k $dest
      extension: ogg

fetchart:
  auto: yes

ftintitle:
  auto: yes

lastgenre:
  auto: yes
  canonical: yes

lyrics:
  auto: yes

missing:
  format: $albumartist - $album - $title
  count: no
  total: no

replaygain:
  auto: yes
  backend: bs1770gain
  method: ebu
scrub:
  auto: yes
web:
  host: 127.0.0.1
  port: 8337
  reverse_proxy: true

[sampsyo]

Hi! Can you please try turning off the scrub plugin?

Does this only happen to ogg files, or do other files (FLAC, MP3, etc.) have the same problem?

[MDCEY]

I've tried turning all the plugins off, I still get the same issue.

My whole library is ogg/opus, I've just added a couple mp3s but cant update yet as it just skips because mtime is up to date (1539261233.0). Any idea how to force an update?

[wisp3rwind]

My whole library is ogg/opus, I've just added a couple mp3s but cant update yet as it just skips because mtime is up to date (1539261233.0). Any idea how to force an update?

I don't actually know how beet update works, but I guess you could just touch(1) some of the files in question, i.e. set mtime to the current time.

[MDCEY]

Managed to get there, just added a few more tracks and it managed to update. So first off it updated as expected adding fingerprints etc. However it didnt touch the replaygain tags according to the logs.

On second run though the same happens and this is with mp3

This is with plugins btw (original config minus the convert plugin"

 library directory: /mnt/data/music
Sending event: library_opened
Journey - Arrival - Higher Place
  r128_album_gain: 000000 -> 000000
Sending event: before_item_moved
Sending event: item_moved
Sending event: database_change
Journey - Arrival - All the Way
  r128_album_gain: 000000 -> 000000
Sending event: before_item_moved
Sending event: item_moved
Sending event: database_change
Journey - Arrival - Signs of Life
  r128_album_gain: 000000 -> 000000
Sending event: before_item_moved
Sending event: item_moved
Sending event: database_change

Plugins off

Journey - Arrival - All the Things
  r128_album_gain: 000000 -> 000000
Sending event: before_item_moved
Sending event: item_moved
Sending event: database_change
Journey - Arrival - Loved by You
  r128_album_gain: 000000 -> 000000
Sending event: before_item_moved
Sending event: item_moved
Sending event: database_change

[sampsyo]

Weird! I'm having trouble reproducing anything like this on my machine.

It seems like this is only happening for the album version of the gain fields. Are the track-level RG fields correctly set? (Are you doing per-album or per-track analysis?)

[djl]

This has been happening for me, too, but with beet write instead of beet update.

Re-importing or running beet mbsync results in any updated files needing to be re-written:

$ beet mbsync helios remembrance
... [snip] ...
$ beet write
Helios - Remembrance - Vainglory
  r128_album_gain: 000000 -> 000000
Helios - Remembrance - Awe
  r128_album_gain: 000000 -> 000000
Helios - Remembrance - Land Father
  r128_album_gain: 000000 -> 000000
Helios - Remembrance - Isostacy
  r128_album_gain: 000000 -> 000000
Helios - Remembrance - Facing West
  r128_album_gain: 000000 -> 000000
Helios - Remembrance - All You Are
  r128_album_gain: 000000 -> 000000
Helios - Remembrance - Ebb/Flow
  r128_album_gain: 000000 -> 000000

I've never used the replaygain plugin so this is fairly unexpected.

[0siribix]

Happens to me with beet import using gstreamer as backend. Removed all RG tags before import, imported flac have 0 for R128_albumand R128_track while all four REPLAYGAIN_ tags contain values

[WatermelonDriveway]

This just started happening to me too. Every single 'beet update' leads to 'r128_album_gain: 000000 -> 000000' for a ton of music. Doesn't do it to every file, but it changes which ones it does it to each update. It's doing it on a song level for me, not just album.

[djl]

The problem seems to be the NullPaddedInt type and it's default null value of None. There doesn't seem to be a way to fix this problem with breaking #2757 again :confused:

[sampsyo]

Hmm… do you think you could expand on that a little more? When we detect the difference between the value in the tags and in the database, are they not both None?

[djl]

One is None and the other is an int. I modified _field_diff to print the old and new values:

$ beet write -p
r128_album_gain:
  oldval: None (NoneType)
  newval: 0 (int)

[jackwilsdon]

@djl, it's kind of weird that newval is an integer — what version of beets are you on? It looks like the new NullPaddedInt type is only on master at the moment.

[djl]

Ah I didn't notice that it's master only right now. I always run directly from master. It helps catch bugs like this :smile:

[jackwilsdon]

So you're still seeing the issue even on master?

[djl]

Correct. Although I'm seeing this on beet write instead of beet update, but I believe it's the same bug (or at least very similar).

[sampsyo]

That's odd. So the database is storing 0, while it should store None instead… what happens if you try to "null out" that field with beet modify, as in beet mod r128_album_gain! [some query]?

[djl]

That fixes the value in the database - it's now null instead of 0. So something isn't

A couple of days ago I ran beet mbsync on my entire library and haven't yet run beet write. Out of curiosity, I checked my beets database to see how many tracks have 0 instead of null in the r128_album_gain field:

sqlite> select count(*) from items where r128_album_gain=0;
32892
sqlite> select count(*) from items where r128_album_gain is null;
29093
sqlite> select count(*) from items;
61985

Just over half.

So I have a ton of test cases available for debugging :smile:

[sampsyo]

Wow; that's odd! I think the most useful thing to do with your trove of test cases is to ask how they got that way—can you see a pattern that would explain the difference between the two groups? Or perhaps even more relevantly, do you know what happens to new tracks that you import?

[djl]

Wow; that's odd! I think the most useful thing to do with your trove of test cases is to ask how they got that way—can you see a pattern that would explain the difference between the two groups?

There's defintely a pattern. Like I said previously anything that modifies existing tracks, such as beet import -L or beet mbsync, writes r128_album_gain=0 to the database.

Here's an example from just now. I just ran beet mbsync on an album I knew had changes available in MusicBrainz:

# Nothing
$ beet write [some query]

# This command changed one track in an album
$ beet mbsync [some query]
[...snip...]

$ beet write [some query]
# this repeats for every track in the album, not just the modified track
$artist - $album - $title
  r128_album_gain: 000000 -> 000000
...

Or perhaps even more relevantly, do you know what happens to new tracks that you import?

Nothing. Newly imported files are not affected.

[sampsyo]

Got it; sorry I missed that!

I am still pretty mystified about why something as benign-seeming as mbsync would do that. I suppose the right way forward to debug it would be to step through that (or some other) plugin's metadata-updating process, checking at each step to see where the 0 value appears in the database, so we can blame a particular step.

[djl]

Oh, I think I see the problem with mbsync. The plugin doesn't call item.try_sync(), only item.try_write(), so the database is updated but the tags are not. Modifying the plugin to call item.try_sync() does seem to fix my particular issue.

Before I submit a pull request, can someone confirm that this makes sense?

[djl]

Scratch that.

try_sync() doesn't fix the problem, which seems to be deeper than just mbsync. Even after switching mbsync to use try_sync(), some albums still end up with r128_album_gain=0 instead of null.

When I get some time, I'm going to run beet mod r128_album_gain! against my whole library and then manually track all modifications. That should (hopefully) make it easier to debug this.

[djl]

I may have figured it out for real this time. The r128_album_gain field on Item was a NullPaddedInt while the same Album field is a regular PaddedInt. Changing the latter field seems to do the trick.

I've opened a pull request for review and testing.

[sampsyo]

Oh wow!! That would indeed be quite a problem. :scream: Thank you for tracking that down!

[djl]

PR merged.

If someone else who was having this issue could check the latest master, I'd appreciate it!

[sampsyo]

Let's call this done unless evidence emerges to the contrary!

[ilohmar]

Is this supposed to work in beets 1.4.9? I am getting the 0-value r128_* tags, and beet modify -Ma "r128_album_gain!" <query> does not remove anything..

EDIT: Sorry, too hasty.. it outputs the familiar 000000 -> 000000, but does remove the tag from the file.