Mediaelch: Incompatibility of generated nfo files with new Kodi unique IDs

[modem]

Describe the bug
Kodi 18 has introduced unique IDs in NFO files.
Some plugins such as trakt.tv fails to syncronize kodi library with trakt.tv collection, including watched status.
https://kodi.wiki/view/NFO_files/Movies
https://kodi.wiki/view/NFO_files/TV_shows

To Reproduce
Steps to reproduce the behavior:

1. Get a new movie with a new nfo file generated by MediaElch
2. Add the new movie to Kodi library
3. Install trakt.tv plugin from the official kodi repository, activate and configure it
4. Mark the movie watched in trakt.tv website
5. Run trakt.tv
6. Check trakt.tv website that the new movie was not added to the library
7. Check movie in Kodi library that it is not marked watched
8. Repeat the process without the nfo file present and verify that trakt.tv and kodi library are correctly synced

Expected behavior
It is expected that the movie is well syncronized between Kodi library and trakt.tv.

Screenshots
If applicable, add screenshots to help explain your problem.

MediaElch Version:

• [ ] 2.6.1-dev (nightly)
• [X] 2.6.0 stable

Operating System:

• [X] Windows
• [ ] macOS
• [ ] Linux (distribution: ___)

Additional context
Add any other context about the problem here.

[bugwelle]

Related issue: #516

Thanks you for creating this issue. I'll try to reproduce and fix this issue. It may take a while, though, as I don't have a lot of time left for developing MediaElch (at the moment at least). :-)

Regards,
Andre

[bugwelle]

Apparently <imdbid>, <tmdb>, ... have been renamed to <uniqueid type="xxxx" default="true"></uniqueid>. I'll look into this and how we can still have backwards compatibility. :)

[KarellenX]

Unique id's has been a requirement in kodi prior to v18 and for a number of versions.

A lack of id's affects TV Shows in a negative way as it results in the mixing of episodes from different tv shows into other tv shows.

Another example is here... https://forum.kodi.tv/showthread.php?tid=331429&pid=2730734#pid2730734

The TV Show, Episode and Movie nfo files use the same unique ID format.

[bugwelle]

Thank you. I didn't know of uniqueid as a field. I thought it was added in Kodi v17. Do you know since when this tag has been added?
Or are you talking about IDs that are unique?

I plan on fixing this issue in #738 (at least for movies at the moment; other NFO files will come after that)

[KarellenX]

I just found the original PR and you are correct, it was introduced in v17.
https://github.com/xbmc/xbmc/pull/10000

As for movie problems, I am not aware of them, and am inclined to say that TV Shows should receive attention first as the result is disastrous and requires rebuilding the library. Radarr/Sonarr unfortunately also has this issue as do some other, lesser known media managers that claim Kodi compliance, but do not. I am then left to help users fix the mess of media managers not creating fully compliant nfo files.

Feel free to ping for information :)

[bugwelle]

Thank you @KarellenX . You're the hero we need :)

738 now uses the new <ratings> syntax for movies and TV shows if v17 or v18 is wanted. Also <uniqueid>.

But I've got a question, though: Your post mentioned that <id> can be used as well (instead of <uniqueid> but we already support that and it should be set. So, is the problem that <id> may be empty? Or should both fields be set (this is done in the example on https://kodi.wiki/view/NFO_files/TV_shows )?
Thank you in advance! :)

[KarellenX]

Those NFO wiki pages need an update, and I will get around to it soon, just need to find the time.

The <id><id> is legacy and it will be set as default if used only when the full format ID is not available, but no identifier is attached (imdb, tvdb etc) so functions and addons that require an identifier will fail. But my experience is that there is a bug using the <id> tag only, and it is not set as default.

See... https://github.com/xbmc/xbmc/pull/10774

You should use <uniqueid type="imdb" default="true">uniqueid> format for all ID's so the identifier is available to those add-ons that need it. I believe Trakt is one, as well as WatchedList and a few others. This applies to all classes of media- Movies, TV Shows, Episodes.

When determining correct function I always find it useful to scrape a popular title using the Kodi scrapers then running an Export to Separate Files and analysing the Kodi created NFO file for correct syntax.

[bugwelle]

Hi,

thank you! :)
I'll look into it and will hopefully fix this issue on the weekend.

Regards,
Andre

[bugwelle]

I can't believe that pull request is from 2016. Well, I came to this project last year and haven't looked that much at the NFO output and what's the current syntax. This is definitely a no-go for media managers like MediaElch.

[bugwelle]

Hi,

the PR is nearly finished. Here are a few NFO files created with kodi which we now use as references.

https://github.com/bugwelle/MediaElch/blob/master/test/resources/show/kodi_v18_show_Game_of_Thrones.nfo
https://github.com/bugwelle/MediaElch/blob/master/test/resources/movie/kodi_v18_Toy_Story_3_2010.nfo
https://github.com/bugwelle/MediaElch/blob/master/test/resources/movie/kodi_v18_Alien_1979.nfo

Here are the current differences in the NFO files from Alien (1979)

10a11
>     <userrating>0</userrating>
20a22
>     <lastplayed></lastplayed>
28d29
<     <sorttitle></sorttitle>
30,31c31,32
<     <watched>false</watched>
<     <credits>Dan O'Bannon</credits>
---
>     <status></status>
>     <credits>Dan O&apos;Bannon</credits>
33a35,36
>     <code></code>
>     <aired></aired>
332c335,339
---
>     <resume>
>         <position>0.000000</position>
>         <total>0.000000</total>
>     </resume>

(note that I've removed the diff of the thumb tags. Their attributes are different and need updating)

And for Game of Thrones:

7c7
<     <uniqueid type="tvdb" default="true">121361</uniqueid>
---
>     <uniqueid default="true" type="tvdb">121361</uniqueid>
13a14
>     <userrating>0</userrating>
15,18c16,22
<     <season>0</season>
<     <episode>0</episode>
<     <plot>Seven noble families fight for control of the mythical land of Westeros. Friction between the houses leads to full-scale war. All while a very ancient evil awakens in the farthest north. Amidst the war, a neglected military order of misfits, the Night's Watch, is all that stands between the realms of men and the icy horrors beyond.</plot>
<     <outline>Seven noble families fight for control of the mythical land of Westeros. Friction between the houses leads to full-scale war. All while a very ancient evil awakens in the farthest north. Amidst the war, a neglected military order of misfits, the Night's Watch, is all that stands between the realms of men and the icy horrors beyond.</outline>
---
>     <season>3</season>
>     <episode>30</episode>
>     <displayseason>-1</displayseason>
>     <displayepisode>-1</displayepisode>
>     <plot>Seven noble families fight for control of the mythical land of Westeros. Friction between the houses leads to full-scale war. All while a very ancient evil awakens in the farthest north. Amidst the war, a neglected military order of misfits, the Night&apos;s Watch, is all that stands between the realms of men and the icy horrors beyond.</plot>
>     <outline></outline>
>     <tagline></tagline>
24a29,30
>     <playcount>0</playcount>
>     <lastplayed></lastplayed>
26c32
<         <url>https://api.thetvdb.com/login?{"apikey":"439DFEBA9D3059C6","id":121361}|Content-Type=application/json</url>
---
>         <url post="yes" cache="auth.json">https://api.thetvdb.com/login?{&quot;apikey&quot;:&quot;439DFEBA9D3059C6&quot;,&quot;id&quot;:121361}|Content-Type=application/json</url>
31,346c37,468
---
>     <code></code>
>     <aired></aired>
>     <trailer></trailer>
828c950,955
< </tvshow>
---
>     <resume>
>         <position>0.000000</position>
>         <total>0.000000</total>
>     </resume>
>     <dateadded>2019-07-05 17:35:08</dateadded>
> </tvshow>

Again, I removed the diff of thumbs and fanart tags. Also the <order> tag of actors is different. @KarellenX , can different actors have the same <order> tag? The Kodi wiki just says that it determines the order shown in the GUI.

Oh, and <season> and <episode> are wrong because MediaElch simply reads NFO files and doesn't store those two numbers. We set them depending on the actually scanned seasons/episodes.

This should fix the issue reported by users. <imdb>, etc. are only written to NFO files when the user explicitly wants v16 NFO files. :)

Regards,
Andre

[KarellenX]

can different actors have the same <order> tag?

Yes

Oh, and <season> and <episode> are wrong because MediaElch simply reads NFO files and doesn't store those two numbers. We set them depending on the actually scanned seasons/episodes.

Actually Kodi completely ignores these two tags during import. They are there just for the sake of completeness of information. Kodi reads the Season and Episode number from the Filename. It is another issue I deal with every now and then, when users believe they can name their files however they want "because they have nfo files". Not true for TV Shows and episodes.

I could not open your linked nfo files, but do you also have...
<namedseason number="4">Children of Earth</namedseason>
This allows users to name Seasons. eg Torchwood S04 can be renamed to Children of Earth.

[KarellenX]

Are you aware of changes at TVDB from 15 July? New v3 API followed by v4 API a few months later.

https://beta.thetvdb.com/blog/something_new

The endpoints are...
V1: https://beta.thetvdb.com/api
V2: https://api.beta.thetvdb.com

The new frontend is located at https://beta.thetvdb.com.

[bugwelle]

I fixed the URLs above. The change was merged and the old branch deleted.
New links:

https://github.com/bugwelle/MediaElch/blob/master/test/resources/show/kodi_v18_show_Game_of_Thrones.nfo
https://github.com/bugwelle/MediaElch/blob/master/test/resources/movie/kodi_v18_Toy_Story_3_2010.nfo
https://github.com/bugwelle/MediaElch/blob/master/test/resources/movie/kodi_v18_Alien_1979.nfo

Actually Kodi completely ignores these two tags during import.

Good to know. Thank you :)

This allows users to name Seasons. eg Torchwood S04 can be renamed to Children of Earth.

Interesting. No, that wasn't included. I'll add Torchwood to our fake-data-set that I use for testing.

I won't close this issue for now because there are still quite a few tags that we need to support. But at least those errors shouldn't happen anymore.

Are you aware of changes at TVDB from 15 July? New v3 API followed by v4 API a few months later.

No. Thank you for pointing that out. We're still on the XML-based version 1. And we have a lot of issues open regarding TvDb (see search).

Thank you :)

[KarellenX]

Interesting. No, that wasn't included. I'll add Torchwood to our fake-data-set that I use for testing.

See wiki page... https://kodi.wiki/view/Season_rename

We're still on the XML-based version 1

The URL in your NFO file above is v2 though.

[bugwelle]

The NFO files were created using Kodi and the TheTVDb plugin :)
TheTvDb API v1 isn't even documented anymore. That will be the next item on my todo list.

[KarellenX]

The NFO files were created using Kodi and the TheTVDb plugin :)

Ok, understand.
That means that if you are using v1 URL's, then Kodi won't read them and the episode nfo files wont be scanned into the library.

[bugwelle]

@KarellenX May I ask you another question? We add a <watched>true</watched> tag even though it is implied if playcount > 0. I also can't find any mentioning of it on https://kodi.wiki/index.php?title=NFO_files/Movies :smiley:

[KarellenX]

<watched>true</watched>

This is not used by Kodi. The Watched status is determined by the <playcount> tag.

Playcount 0 = Unwatched

Playcount =>1 = Watched

[bugwelle]

Oh, didn't even ask a question in my previous post...
Thanks. That's what I wanted to know :)
Will remove that tag then.

[bugwelle]

Concerts will be v17 compliant after #769 is merged

[bugwelle]

Should have looked at Kodi's source...
https://github.com/xbmc/xbmc/blob/master/xbmc/video/VideoInfoTag.cpp#L97
Helps a lot :-)

[KarellenX]

@bugwelle

Are you asking me questions or are they general comments or reminders for yourself?

[bugwelle]

@KarellenX More like general comments and reminders where I have to look for NFO related stuff.
I just add comments here for others that are interested in this issue or were wondering as well how Kodi creates NFO files.

I still need to have a look at episode and music (album/artist) NFO files. I'll close this issue when those are fixed.

[KarellenX]

Ok, no problems. FYI I am updating the nfo wiki pages for v18. Should be done in a week or so, have already completed the main page and Movies, but I will upload all at once. TV Shows and music are the difficult ones that will take a bit of time and patience :(

[bugwelle]

Thanks! :smiley:

FYI, technically episode NFO files for multi-episode files are invalid (see https://forum.kodi.tv/showthread.php?tid=346370 ). I know that this is probably intended (?) or at least won't get fixed anytime soon but that may be worth a note there. :)

775 makes episode NFO files that are generated by MediaElch compliant with Kodi. There are still quite a few unsupported tags but at least MediaElch now supports <uniqueid> for episode NFO files.

Thanks for keeping the wiki up-to-date. It's really helpful. :+1:

[bugwelle]

FYI, artists will be Kodi v17 compatible with #782
Only difference is that multiple <genre> tags are now supported.

[bugwelle]

Just for reference:
MediaElch is case-sensitive in regards to XML tags. That's important because Kodi switched from musicBrainzAlbumID to musicbrainzalbumid for albums in 2017: https://github.com/xbmc/xbmc/pull/12120/files#diff-1e00d710fc5792210b07b13cf8819315
MediaElch has not yet updated these tags.

[bugwelle]

Oh, I just noticed that musicBrainzArtistID is still CamelCase which makes it a bit inconsistent... :/
I'll merge #782 this evening and it should make MediaElch's music export compatible with Kodi :)

[KarellenX]

I did not realise you also had music library capabilities in MediaElch.

I strongly urge you to contact @DaveTBlake on the forum in the Music section for any questions before making any major changes if you are unsure. He is quite approachable and welcomes questions from developers, especially as he explains what he is trying to ultimately achieve and you can prepare for it.

I know he would like greater compatibility with the new Artist Information Folder that was introduced for v18.

[bugwelle]

I did not realise you also had music library capabilities in MediaElch.

Only basic support, i.e. MediaElch does not even read mp3 metadata.

I strongly urge you to contact DaveTBlake on the forum in the Music section for any questions before making any major changes if you are unsure.

Thank you. I haven't done any major changes only smaller ones (e.g. renaming XML tags, using multiple genre tags instead of one). I will keep this in mind. 👍

I know he would like greater compatibility with the new Artist Information Folder that was introduced for v18.

I've read about this. I will have a look at that next month and will contact Dave. :)

As always, thank you letting me know all of this. :)

[bugwelle]

I think we're now compatible. I'll release a new version soon. GUI elements will be updated in v2.6.4. :-)