Pyatv: Support for artwork in MRP

[postlund]

Short feature/function description
Artwork is tricky in MRP and is currently not supported at all. It seems to be pushed to the client at some point, but not sure when.

What needs to be done?
Investigate when artwork is pushed and try to make pyatv handle that. Artwork should be available in the same way as in the DMAP implementation. As I haven't looked too much into this I don't have any more details.

Is this a breaking change?
No

Anything else worth knowing?
No

[postlund]

DMAP returns artwork as PNG whilst MRP uses jpeg, so I need to add mime type or similar to the interface for that.

[postlund]

Maybe I have to request artwork initially (if available) with PLAYBACK_QUEUE_REQUEST_MESSAGE?

[postlund]

Sniffed traffic from the app and couldn't see any artwork there, despite it being visible in the app. Starting to believe it's our-of-band signaling going on...

[postlund]

I'm gonna delay this from 0.4.0 as I haven't found a way to get it working. It just delays other work that is more important. If someone wants to investigate, that would be great.

[postlund]

Ok, so I think I cracked this now. I was right about having to request the artwork using PLAYBACK_QUEUE_REQUEST_MESSAGE. If available, the device will respond with a SET_STATE_MESSAGE containing artwork for the requested item. My hacky test code looks like this:

from pyatv.mrp.protobuf import ProtocolMessage
msg = messages.create(ProtocolMessage.PLAYBACK_QUEUE_REQUEST_MESSAGE)
inner = msg.inner()
inner.location = 0
inner.length = 1
inner.artworkWidth = -1.0
inner.artworkHeight = 368.0
resp = await protocol.send_and_receive(msg)
item = resp.inner().playbackQueue.contentItems[0]
with open("output.jpg", "wb") as fh:
    fh.write(item.artworkData)

In my previous attempts I believe that the image probably was cached on my phone, so it was never requested. To know if artwork is available or not, one can look at the initial SetStateMessage:

[setStateMessage] {
  supportedCommands {
  }
  playbackQueue {
    location: 0
    contentItems {
      identifier: "com.apple.avkit.1640.dc78db9a"
      metadata {
        title: "S3: A9 \342\200\235The Doldrums\342\200\235"
        duration: 3453.8250833
        artworkAvailable: true
        infoAvailable: true       <--------------- THIS ONE IS INTERESTING
        languageOptionsAvailable: true
        numberOfSections: 0
        artworkMIMEType: "image/jpeg"    <------------ THIS ONE AS WELL
        elapsedTime: 3334.888795751
        isAlwaysLive: false
        playbackRate: 1.0
        contentIdentifier: "80191829"
        mediaType: Video
        nowPlayingInfoData: "bplist00\324\001\002\003\004\005\006\007\nX$versionY$archiverT$topX$objects\022\000\001\206\240_\020\017NSKeyedArchiver\321\010\tTroot\200\001\245\013\014\025\026\027U$null\323\r\016\017\020\022\024WNS.keysZNS.objectsV$class\241\021\200\002\241\023\200\003\200\004_\0200AVMediaRemoteManagerNowPlayingInfoHasDescription\020\001\322\030\031\032\033Z$classnameX$classes_\020\023NSMutableDictionary\243\032\034\035\\NSDictionaryXNSObject\000\010\000\021\000\032\000$\000)\0002\0007\000I\000L\000Q\000S\000Y\000_\000f\000n\000y\000\200\000\202\000\204\000\206\000\210\000\212\000\275\000\277\000\304\000\317\000\330\000\356\000\362\000\377\000\000\000\000\000\000\002\001\000\000\000\000\000\000\000\036\000\000\000\000\000\000\000\000\000\000\000\000\000\000\001\010"
        elapsedTimeTimestamp: 596061068.205099
        inferredTimestamp: 596059238.350599
        legacyUniqueIdentifier: -596059238
      }
...

I need to fix naming of the message as my protobuf autogen script doesn't pick it up (so I get no nice convenience stuff for it). My first attempt will be to fetch artwork on demand and cache further request, until something else is playing (thus invalidating it).

[postlund]

Pretty cool:

[postlund]

I believe that #317 is good enough to get initial artwork in place (I have local, un-pushed changes with som refactoring laying around too). But I need to change the API a bit because Home Assistant (and other users as well, probably) require the image mimetype. This is not provided by pyatv today as DMAP always provided a PNG image. So it has not been necessary (with MRP other types can be used).

My intentions now are to return a new type (namedtuple) containing the raw image as well as mimetype. In the future, this could be extended to include for instance image width, height and other things. It is worth noting that this is a breaking change and require changes in DMAP as well. I will also extend the API to support requesting a particular size of the image.

[glassbase]

Great work on getting artwork working again.

... Totally unrelated but I just discovered those 3 Toy Story shorts a few weeks ago and my son loves them, especially “Small Fry”.

[postlund]

Thanks! 😊

Honestly, I haven't seen them. I got them through iTunes 12 days or Christmas or something a few years back. But I haven't watched them yet, maybe my son will appreciate them too when he gets a little older.

[postlund]

So, PR (#317) is merged, so I guess this can be closed now 🎉