Azuracast: Custom Fields auto assignable fields feature does not work for Unique File Identifier (and several others)

[nickrobillard]

Using Docker installation method
Yes

AzuraCast version
Rolling Release #8416e97 (2020-10-02 3:24)

Host Operating System
MacOS Catalina

Why do I care about this?
I want to be able to uniquely identify tracks playing in AzuraCast so that I can create a "hard" association in an external system that I control.

Describe the bug
I have dug into the Custom Fields "Automatically Set from ID3v2 Value" feature and have found that it does not function for many of them. While simple tags like Artist, Album, Title, Genre, etc are auto assigned as expected, other more complex frames are not. In my case, I am attempting to auto assign values from the Unique File Identifier / UFID (https://id3.org/id3v2.3.0#Unique_file_identifier) field. In src/Entity/Repository/StationMediaRepository.php public function loadFromFile, the reason is clear. When finding auto assignments, only the values from getID3 analyze() result top-level "tags" ($file_info['tags']) are iterated over. I have found that the more complex id3v2 frames are not added to "tags" by getID3 lib. I've separately set up the same version of getID3 lib that AzuraCast is using (v1.9.20) and have confirmed this. Only the basic field values are added to $getID3->analyze() result top-level "tags". The more complex frames are added to a top level associative array named "id3v2".

Observe the following result from $getID3->analyze():

Analyze result

```
Array
(
[GETID3_VERSION] => 1.9.20-202006061653
[filesize] => 208233159
[filepath] => /Users/redacted
[filename] => redacted.mp3
[filenamepath] => /Users/redacted.mp3
[avdataoffset] => 419673
[avdataend] => 208233159
[fileformat] => mp3
[audio] => Array
(
[dataformat] => mp3
[channels] => 2
[sample_rate] => 44100
[bitrate] => 320000
[channelmode] => stereo
[bitrate_mode] => cbr
[lossless] =>
[encoder_options] => CBR320
[compression_ratio] => 0.22675736961451
[streams] => Array
(
[0] => Array
(
[dataformat] => mp3
[channels] => 2
[sample_rate] => 44100
[bitrate] => 320000
[channelmode] => stereo
[bitrate_mode] => cbr
[lossless] =>
[encoder_options] => CBR320
[compression_ratio] => 0.22675736961451
)

            )

    )

[tags] => Array
    (
        [id3v2] => Array
            (
                [title] => Array
                    (
                        [0] => This is the track title
                    )

                [artist] => Array
                    (
                        [0] => This is the track artist
                    )

            )

    )

[id3v2] => Array
    (
        [header] => 1
        [flags] => Array
            (
                [unsynch] => 
                [exthead] => 
                [experim] => 
            )

        [majorversion] => 3
        [minorversion] => 0
        [headerlength] => 419673
        [tag_offset_start] => 0
        [tag_offset_end] => 419673
        [encoding] => UTF-8
        [UFID] => Array
            (
                [0] => Array
                    (
                        [frame_name] => UFID
                        [frame_flags_raw] => 0
                        [data] => thisismyuuid
                        [datalength] => 33
                        [dataoffset] => 10
                        [framenamelong] => Unique file identifier
                        [framenameshort] => unique_file_identifier
                        [flags] => Array
                            (
                                [TagAlterPreservation] => 
                                [FileAlterPreservation] => 
                                [ReadOnly] => 
                                [compression] => 
                                [Encryption] => 
                                [GroupingIdentity] => 
                            )

                        [ownerid] => http://www.id3.org/dummy/ufid.html
                    )

            )

        [comments] => Array
            (
                [title] => Array
                    (
                        [0] => This is the track title
                    )

                [artist] => Array
                    (
                        [0] => This is the track artist
                    )

            )

        [TIT2] => Array
            (
                [0] => Array
                    (
                        [frame_name] => TIT2
                        [frame_flags_raw] => 0
                        [data] => This is the track title
                        [datalength] => 13
                        [dataoffset] => 53
                        [framenamelong] => Title/songname/content description
                        [framenameshort] => title
                        [flags] => Array
                            (
                                [TagAlterPreservation] => 
                                [FileAlterPreservation] => 
                                [ReadOnly] => 
                                [compression] => 
                                [Encryption] => 
                                [GroupingIdentity] => 
                            )

                        [encodingid] => 0
                        [encoding] => ISO-8859-1
                    )

            )

        [TPE1] => Array
            (
                [0] => Array
                    (
                        [frame_name] => TPE1
                        [frame_flags_raw] => 0
                        [data] => This is the track artist
                        [datalength] => 8
                        [dataoffset] => 76
                        [framenamelong] => Lead performer(s)/Soloist(s)
                        [framenameshort] => artist
                        [flags] => Array
                            (
                                [TagAlterPreservation] => 
                                [FileAlterPreservation] => 
                                [ReadOnly] => 
                                [compression] => 
                                [Encryption] => 
                                [GroupingIdentity] => 
                            )

                        [encodingid] => 0
                        [encoding] => ISO-8859-1
                    )

            )

        [padding] => Array
            (
                [start] => 94
                [length] => 419579
                [valid] => 1
            )

    )

[mime_type] => audio/mpeg
[mpeg] => Array
    (
        [audio] => Array
            (
                [raw] => Array
                    (
                        [synch] => 4094
                        [version] => 3
                        [layer] => 1
                        [protection] => 1
                        [bitrate] => 14
                        [sample_rate] => 0
                        [padding] => 1
                        [private] => 0
                        [channelmode] => 0
                        [modeextension] => 0
                        [copyright] => 0
                        [original] => 0
                        [emphasis] => 0
                    )

                [version] => 1
                [layer] => 3
                [channelmode] => stereo
                [channels] => 2
                [sample_rate] => 44100
                [protection] => 
                [private] => 
                [modeextension] => 
                [copyright] => 
                [original] => 
                [emphasis] => none
                [padding] => 1
                [bitrate] => 320000
                [framelength] => 1045
                [bitrate_mode] => cbr
            )

    )

[playtime_seconds] => 5195.33715
[tags_html] => Array
    (
        [id3v2] => Array
            (
                [title] => Array
                    (
                        [0] => This is the track title
                    )

                [artist] => Array
                    (
                        [0] => This is the track artist
                    )

            )

    )

[bitrate] => 320000
[playtime_string] => 1:26:35
[comments] => Array
    (
        [title] => Array
            (
                [0] => This is the track title
            )

        [artist] => Array
            (
                [0] => This is the track artist
            )

    )

[comments_html] => Array
    (
        [title] => Array
            (
                [0] => ERROR: Character set "" not supported in MultiByteCharString2HTML()
            )

        [artist] => Array
            (
                [0] => ERROR: Character set "" not supported in MultiByteCharString2HTML()
            )

    )

)
```

Note that the UFID data "thisismyuuid" and ownerId "http://www.id3.org/dummy/ufid.html" are inside the ['id3v2']['UFID'] structure.

_The minimal fix_
For my immediate needs, adding support for UFID data value to be auto assigned to my custom field is straightforward. I don't care about ownerId. I have this working locally.

_The full solution_
I would like to address this in a way that is most useful to others. And most importantly, I would like to contribute my work back to AzuraCast. There are several things to consider here - from how to handle multiple entries of the same tag, how to handle multiple values within each entry, and how to handle 1-off differences (like field names) in the frames we choose to support. Let's consider multiple values within a single tag entry. For example, with UFID we have data and ownerId. Looking at src/Entity/StationMediaCustomField.php, it appears that custom fields are only meant to take a single text value. So this would require some changes. I realize that I am "opening a can of worms" by going into these details, and that maybe from AzuraCast's perspective, simply removing the non-functional select option values from the list may be the most appropriate (and completely understandable) solution, especially considering the need to support this going forward.

Expected behavior
I expect that auto assignment for all options in the "Automatically Set from ID3v2 Value" select list to be functional to some degree.

Screenshots

Additional context
I have a fully functioning local dev environment and I have done the initial work to make the UFID field data value auto assignment work. But I would like to consider this from an AzuraCast community perspective. @SlvrEagle23 - what do you consider to be the appropriate resolution here? I will work on a PR for whatever you decide. Thank you!

[SlvrEagle23]

@nickrobillard Thank you for the detailed report, and for looking into the underlying cause in our source code.

It seems the solution is as simple as just checking inside the id3v2 array for auto-assignable values in the loadFromFile call if the expected value isn't found in the main metadata array.

In this case, we could take advantage of the null coalesce operator like so, on StationMediaRepository.php line 238:

$customFieldValue = $tag_data[$tag][0] ?? $tag_data['id3v2'][$tag][0] ?? null;
if (!empty($customFieldValue)) {
    $tagValue = $this->cleanUpString($customFieldValue);

(Update: It's not quite as simple as this because of the loop through file_info['tags'] above this, but it's not much more complicated than that, just a matter of moving the auto-assignment loop out of the tags foreach.)

I could implement this myself, but in the Hacktoberfest spirit if you'd like to submit a PR for it, you're welcome to. :)

[SlvrEagle23]

Also, this is the first time I've ever seen:

This kind of thing

Logs

used before...and it's pretty awesome and I'm going to use it a lot now.

[nickrobillard]

I've done a lot of analysis and ended up having to go through each tag one by one (see https://github.com/JamesHeinrich/getID3/issues/271) and here is where I'm at. Yes, unfortunately this is quite complicated due to the multi-valued and unique nature of the "un-common" ID3 tag values that are not included under [tags][id3v2]. Most, if not all, raw field values under [id3v2] would need one-off handling and I really don't think it's worth the effort. To give you a sense of what I am referring to, here is an example of the SYLT / synchronised_lyric raw frame data:

Complex frame data example:

[SYLT] => Array
    (
        [0] => Array
            (
                [frame_name] => SYLT
                [frame_flags_raw] => 0
                [datalength] => 46
                [dataoffset] => 350
                [framenamelong] => Synchronised lyric/text
                [framenameshort] => synchronised_lyric
                [flags] => Array
                    (
                        [TagAlterPreservation] => 
                        [FileAlterPreservation] => 
                        [ReadOnly] => 
                        [GroupingIdentity] => 
                        [compression] => 
                        [Encryption] => 
                        [Unsynchronisation] => 
                        [DataLengthIndicator] => 
                    )

                [timestampformat] => 2
                [contenttypeid] => 1
                [contenttype] => lyrics
                [encodingid] => 0
                [encoding] => ISO-8859-1
                [language] => eng
                [languagename] => English
                [lyrics] => Array
                    (
                        [0] => Array
                            (
                                [data] => 
                            )

                        [1] => Array
                            (
                                [data] => 
lyric 1
                                [timestamp] => 10
                            )

                        [2] => Array
                            (
                                [data] => lyric 2
                                [timestamp] => 3072010
                            )

                        [3] => Array
                            (
                                [data] => lyric 3
                                [timestamp] => 40000
                            )

                    )

            )

    )

Here is what I suggest and I will leave it up to you. I think we should continue along your path, and in order to avoid a maintenance nightmare, only offer for auto assignment the values under [tags][id3v2]. Because there was no definitive list, I have gone through the entire list of tag options in config/forms/custom_field.php and I have come up with the following. I made these determinations by a combination of looking at getID3 source code, and in many cases actually writing the tag to an MP3 file and checking the result of analyze().

Fields confirmed to be included under [tags][id3v2]:

'album' => __('Album'), // TAL, TALB
'album_artist_sort_order' => __('Album Artist Sort Order'), // TS2, TSO2
'album_sort_order' => __('Album Sort Order'), // TSA, TSOA
'artist' => __('Artist'), // TP1, TPE1
'band' => __('Band'), // TP2, TPE2
'bpm' => __('Bpm'), // TBP, TBPM
'comment' => __('Comment'), // COM, COMM
'commercial_information' => __('Commercial Information'), // WCM, WCOM
'composer' => __('Composer'), // TCM, TCOM
'composer_sort_order' => __('Composer Sort Order'), // TSC, TSOC
'conductor' => __('Conductor'), // TP3, TPE3
'content_group_description' => __('Content Group Description'), // TIT1, TT1
'copyright' => __('Copyright'), // WCOP, WCP
'copyright_message' => __('Copyright Message'), // TCOP, TCR
'encoded_by' => __('Encoded By'), // TEN, TENC
'encoder_settings' => __('Encoder Settings'), // TSS, TSSE
'encoding_time' => __('Encoding Time'), // TDEN
'file_owner' => __('File Owner'), // TOWN
'file_type' => __('File Type'), // TFLT, TFT
'genre' => __('Genre'), // TCO, TCON
'initial_key' => __('Initial Key'), // TKE, TKEY
'internet_radio_station_name' => __('Internet Radio Station Name'), // TRSN
'internet_radio_station_owner' => __('Internet Radio Station Owner'), // TRSO
'involved_people_list' => __('Involved People List'), // IPL, IPLS, TIPL
'isrc' => __('ISRC'), // TRC, TSRC
'language' => __('Language'), // TLA, TLAN
'length' => __('Length'), // TLE, TLEN
'linked_information' => __('Linked Information'), // LINK, LNK
'lyricist' => __('Lyricist'), // TEXT, TXT
'media_type' => __('Media Type'), // TMED, TMT
'mood' => __('Mood'), // TMOO
'music_cd_identifier' => __('Music CD Identifier'), // MCDI, MCI
'musician_credits_list' => __('Musician Credits List'), // TMCL
'original_album' => __('Original Album'), // TOAL, TOT
'original_artist' => __('Original Artist'), // TOA, TOPE
'original_filename' => __('Original Filename'), // TOF, TOFN
'original_lyricist' => __('Original Lyricist'), // TOL, TOLY
'original_release_time' => __('Original Release Time'), // TDOR
'original_year' => __('Original Year'), // TOR, TORY
'part_of_a_compilation' => __('Part Of A Compilation'), // TCMP, TCP
'part_of_a_set' => __('Part Of A Set'), // TPA, TPOS
'performer_sort_order' => __('Performer Sort Order'), // TSOP, TSP
'playlist_delay' => __('Playlist Delay'), // TDLY, TDY
'produced_notice' => __('Produced Notice'), // TPRO
'publisher' => __('Publisher'), // TPB, TPUB
'recording_time' => __('Recording Time'), // TDRC
'release_time' => __('Release Time'), // TDRL
'remixer' => __('Remixer'), // TP4, TPE4
'set_subtitle' => __('Set Subtitle'), // TSST
'subtitle' => __('Subtitle'), // TIT3, TT3
'tagging_time' => __('Tagging Time'), // TDTG
'terms_of_use' => __('Terms Of Use'), // USER
'title' => __('Title'), // TIT2, TT2
'title_sort_order' => __('Title Sort Order'), // TSOT, TST
'track_number' => __('Track Number'), // TRCK, TRK
'unsynchronised_lyric' => __('Unsynchronised Lyric'), // ULT, USLT
'url_artist' => __('URL Artist'), // WAR, WOAR
'url_file' => __('URL File'), // WAF, WOAF
'url_payment' => __('URL Payment'), // WPAY
'url_publisher' => __('URL Publisher'), // WPB, WPUB
'url_source' => __('URL Source'), // WAS, WOAS
'url_station' => __('URL Station'), // WORS
'url_user' => __('URL User'), // WXX, WXXX
'year' => __('Year'), // TYE, TYER

Special:
'text' => __('Text'), // TXX, TXXX - NOTE: Under "text" key. Sub-keyed by corresponding "description" value instead of numerically.

Definitely not included under [tags][id3v2]:

'attached_picture' => __('Attached Picture'), // APIC, PIC
'audio_encryption' => __('Audio Encryption'), // AENC, CRA
'audio_seek_point_index' => __('Audio Seek Point Index'), // ASPI
'commercial_frame' => __('Commercial Frame'), // COMR
'date' => __('Date'), // TDA, TDAT
'encrypted_meta_frame' => __('Encrypted Meta Frame'), // CRM
'encryption_method_registration' => __('Encryption Method Registration'), // ENCR
'equalisation' => __('Equalisation'), // EQU, EQU2, EQUA
'event_timing_codes' => __('Event Timing Codes'), // ETC, ETCO
'general_encapsulated_object' => __('General Encapsulated Object'), // GEO, GEOB
'group_identification_registration' => __('Group Identification Registration'), // GRID
'mpeg_location_lookup_table' => __('MPEG Location Lookup Table'), // MLL, MLLT
'ownership_frame' => __('Ownership Frame'), // OWNE
'play_counter' => __('Play Counter'), // CNT, PCNT
'popularimeter' => __('Popularimeter'), // POP, POPM
'position_synchronisation_frame' => __('Position Synchronisation Frame'), // POSS
'private_frame' => __('Private Frame'), // PRIV
'recommended_buffer_size' => __('Recommended Buffer Size'), // BUF, RBUF
'relative_volume_adjustment' => __('Relative Volume Adjustment'), // RVA, RVA2, RVAD
'reverb' => __('Reverb'), // REV, RVRB
'seek_frame' => __('Seek Frame'), // SEEK
'signature_frame' => __('Signature Frame'), // SIGN
'size' => __('Size'), // TSI, TSIZ
'synced_tempo_codes' => __('Synced Tempo Codes'), // STC
'synchronised_lyric' => __('Synchronised Lyric'), // SLT, SYLT
'synchronised_tempo_codes' => __('Synchronised Tempo Codes'), // SYTC
'unique_file_identifier' => __('Unique File Identifier'), // UFI, UFID

Might not be included under [tags][id3v2] / could not get to write:

'featured_artist' => __('Featured Artist'), // TFEA
'recording_dates' => __('Recording Dates'), // TRD, TRDA
'recording_studio' => __('Recording Studio'), // TSTU
'replay_gain_adjustment' => __('ReplayGain Adjustment'), // rgad
'time' => __('Time'), // TIM, TIME

So I am suggesting that we remove from config/forms/custom_field.php all entries that are not in "Fields confirmed to be included" list and to leave your existing logic in tact. (And also remove the Text option, unless you want to support User Defined Text - see below.)

_User Defined Text handling_

I found that Text, being a User Defined Text tag, would need a bit more work if you want to support it. To do so, you could replace the Text option with a User Defined Text option. Selecting it could trigger visibility of a free form text field where a user would enter the "description". The "description" is used as the key in the [tags][id3v2][text] associative array. See:

[text] => Array
    (
      [custom text] => some text
      [something else] => some other text
    )

You also could just go with some pre-selected User Defined Text options (where the "description" is baked into the select option value) and avoid the extra UI work. That being said, supporting User Defined Text in any form may not be worthwhile but I wanted to mention it.

I will also mention that I no longer need the UFID mapping (I had missed the "List all files" API - /api/station/1/files - which allows me to create associations based off file name). But I would still like to see this issue through to a logical conclusion. Please let me know what direction you want to take. Thank you!

[SlvrEagle23]

@nickrobillard I agree fully with your assessment; if the mapping isn't a direct or obvious one that's possible by our current logic (or a fairly minor modification to it), we don't want to include it unless there is a very compelling reason to do so, and given the tags that aren't included I don't expect any of them to fulfill that criteria.

I am okay with having the hard/impossible-to-map fields removed; if you'd like to submit a PR for that go ahead. :)

[nickrobillard]

PR merged. Closing. Thank you for your assistance @SlvrEagle23