Mastodon: Media Attachments for Remote Accounts "Not Available"

[a-dows]

A seemingly random assortment of image previews from followed remote accounts are failing to generate previews. The issue persists in all browsers and apps.

No errors appear in the console when viewed in a desktop browser; the images are available on clickthrough. Opening the status in its own local url renders all previews normally.

Loading the same remote status in two different instances produces different missing image previews, though at least one preview remains missing (see second photo).

Problem observed in Mastodon 3.1.2 as well as 3.1.3. Users first reported this issue to me as an admin about two weeks ago.

Reported remote instances displaying the issue include mastodon.social, skull.website, and godforsaken.website.

[Moth-Tolias]

i've run into this several times on my instance, chitter.xyz.

[fcashman]

Yep, reported this to my admin last week on cooler.mom. And the occurrence is inconsistent. The picture will be unavailable in a post and then later from the same account a new post's picture is unaffected. The picture must be clicked on and opened in separate tab to be viewed. I use Mastodon through the browser on both my desktop (MacOS Mojave and on my iPhone (pinned to my homescreen). I have not seen this issue in Amaroq, which is on my phone (but I hardly use it except now to see if pictures are loading). I need to keep checking Amaroq to see if this claim is 100% true. Also Amaroq hasn't been updated in 11 months, so maybe that is the reason.

[lawremipsum]

I've seen this, and have not noticed any pattern to its occurence.

[davidwolfpaw]

This happens to me regularly on tech.lgbt, and is not a new occurrence

[snoozysoft]

Can confirm it's occurring on cybre.space also.

[bclindner]

mastodon.technology also has this issue - glad to hear it isn't just us.

[ThibG]

This happens when the remote media couldn't be fetched/saved by Mastodon. It may either be an issue in the underlying media handling code, or more probably, just the originating instance being overwhelmed by follower instances trying to fetch the media at the same time.

Normally, Mastodon should retry a few times for each failed fetch, meaning the media can “unbreak” later.

[a-dows]

@ThibG It's been increasing in frequency lately, per user reports, and it doesn't appear possible to force a retry. The documentation for preview cards in the CLI specifically advises against doing anything with card data less than 7 days old.

[ThibG]

Note that those aren't preview cards, but media attachments. Preview cards are the thing where Mastodon crawls a link present in the toot text to display a preview of such link.

[a-dows]

@ThibG Thanks—fixed the title.

[ThibG]

Also it should be possible to re-fetch the media attached to a specific status by running tootctl media refresh --status=id (where id is the numeric identifier of the status on the receiving instance, such as you can see in the URL when viewing the status in the WebUI).
Of course this isn't a viable solution to fetch missing media on an everyday basis, but it should at least help figuring out whether the issue is transient (e.g., originating instance too slow to serve the media) or a reproducible issue with the media handling itself.

[a-dows]

@ThibG So I ran media refresh ID on an affected post. It did nothing.

Local url:
https://dads.cool/web/statuses/104150363538006622
Remote url:
https://skull.website/@Pixley/104150362454628540

mastodon@localhost:~/live$ RAILS_ENV=production bin/tootctl media refresh --status=104150363538006622 --verbose
/home/mastodon/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/bundler-1.17.2/lib/bundler/rubygems_integration.rb:200: warning: constant Gem::ConfigMap is deprecated
Processing 181872                                
Processing 181873                |  ETA: ??:??:??
2/2 |============================| Time: 00:00:00
Downloaded 2 media attachments (approx. 0 Bytes)

Then I ran it with the --force tag, just in case:

mastodon@localhost:~/live$ RAILS_ENV=production bin/tootctl media refresh --status=104150363538006622 --verbose --force
/home/mastodon/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/bundler-1.17.2/lib/bundler/rubygems_integration.rb:200: warning: constant Gem::ConfigMap is deprecated
Processing 181872                                                                                       
Processing 181873                                                                                       
2/2 |===================================================================================| Time: 00:00:00
Downloaded 2 media attachments (approx. 213 KB)

This, too, did nothing.

I reloaded the status in multiple browsers, including private windows without cache.

[ThibG]

The fact that it doesn't help is interesting. I cannot reproduce it, though :/

If you run bundle exec rails c RAILS_ENV=production and, within it, the following commands: MediaAttachment.find(181872).pluck(:remote_url, :file_file_name, :type, :processing, :file_meta), what does it return?

[a-dows]

mastodon@localhost:~/live$ RAILS_ENV=production bundle exec rails console
/home/mastodon/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/bundler-1.17.2/lib/bundler/rubygems_integration.rb:200: warning: constant Gem::ConfigMap is deprecated
/home/mastodon/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/bundler-1.17.2/lib/bundler/rubygems_integration.rb:200: warning: constant Gem::ConfigMap is deprecated
Chewy console strategy is `urgent`
Loading production environment (Rails 5.2.4.1)
irb(main):001:0> MediaAttachment.find(181872).pluck(:remote_url, :file_file_name, :type, :processing, :file_meta)
Traceback (most recent call last):
        1: from (irb):1
NoMethodError (undefined method `pluck' for #<MediaAttachment:0x000056155a25e318>)
irb(main):002:0> 

[ThibG]

Ah sorry, I meant
MediaAttachment.where(id: [181872, 181873]).pluck(:remote_url, :file_file_name, :type, :processing, :file_meta])

[a-dows]

irb(main):009:0> MediaAttachment.where(id: [181872, 181873]).pluck(:remote_url, :file_file_name, :type, :processing, :file_meta)
Traceback (most recent call last):
        2: from (irb):9
        1: from (irb):9:in `rescue in irb_binding'
ActiveRecord::StatementInvalid (PG::UndefinedColumn: ERROR:  column "processing" does not exist)
LINE 1: ...ts"."file_file_name", "media_attachments"."type", "processin...
                                                             ^
: SELECT "media_attachments"."remote_url", "media_attachments"."file_file_name", "media_attachments"."type", "processing", "media_attachments"."file_meta" FROM "media_attachments" WHERE "media_attachments"."id" IN ($1, $2) ORDER BY "media_attachments"."id" ASC
irb(main):010:0> 

[ThibG]

Ah, you're still on v3.1.2… try the same thing without :processing then

[a-dows]

irb(main):001:0> MediaAttachment.where(id: [181872, 181873]).pluck(:remote_url, :file_file_name, :type, :file_meta)
=> [["https://cf.mastohost.com/v1/AUTH_91eb37814936490c95da7b85993cc2ff/skullwebsite/media_attachments/files/000/173/356/original/6286945ae4532280.jpg", "35d84615834cde6f.jpeg", "image", {"original"=>{"width"=>1129, "height"=>1451, "size"=>"1129x1451", "aspect"=>0.7780840799448656}, "small"=>{"width"=>352, "height"=>453, "size"=>"352x453", "aspect"=>0.7770419426048565}}], ["https://cf.mastohost.com/v1/AUTH_91eb37814936490c95da7b85993cc2ff/skullwebsite/media_attachments/files/000/173/357/original/541e181a043c2f98.jpg", nil, "unknown", nil]]
irb(main):002:0> 

so nil, unknown, nil.

[ThibG]

what does the following do?

attachment = MediaAttachment.find(181873)
attachment.reset_file!
attachment.save!

[Restioson]

Opening it as a link works usually.

[a-dows]

irb(main):002:0> attachment = MediaAttachment.find(181873)
=> #<MediaAttachment id: 181873, status_id: 104150363538006622, file_file_name: nil, file_content_type: nil, file_file_size: nil, file_updated_at: nil, remote_url: "https://cf.mastohost.com/v1/AUTH_91eb37814936490c9...", created_at: "2020-05-11 14:46:57", updated_at: "2020-05-11 14:47:08", shortcode: nil, type: "unknown", file_meta: nil, account_id: 19719, description: "The black and orange bird is leaning down to nibbl...", scheduled_status_id: nil, blurhash: "UANKFv~p4o-;00fSRkbI00xuD%V?NGR*WAad">
irb(main):003:0> attachment.reset_file!
=> nil
irb(main):004:0> attachment.save!
=> true
irb(main):005:0> 

[ThibG]

Hmm… did that download the file?

[a-dows]

No, it did not.
On May 11, 2020, 6:04 PM -0400, ThibG [e-mail], wrote:

Hmm… did that download the file?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or unsubscribe.

[ThibG]

Ok. Can you retry the reset_file! within RAILS_ENV=production RAILS_LOG_LEVEL=debug bundle exec c? Hopefully, if something fails in the download, we'll get more info.

[BalooUriza]

This is also occurring on social.tulsa.ok.us

[a-dows]

still nil.

mastodon@localhost:~/live$ RAILS_ENV=production RAILS_LOG_LEVEL=debug bundle exec rails console
/home/mastodon/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/bundler-1.17.2/lib/bundler/rubygems_integration.rb:200: warning: constant Gem::ConfigMap is deprecated
/home/mastodon/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/bundler-1.17.2/lib/bundler/rubygems_integration.rb:200: warning: constant Gem::ConfigMap is deprecated
Chewy console strategy is `urgent`
Loading production environment (Rails 5.2.4.1)
irb(main):001:0> attachment = MediaAttachment.find(181873)
  MediaAttachment Load (1.6ms)  SELECT  "media_attachments".* FROM "media_attachments" WHERE "media_attachments"."id" = $1 ORDER BY "media_attachments"."id" ASC LIMIT $2  [["id", 181873], ["LIMIT", 1]]
=> #<MediaAttachment id: 181873, status_id: 104150363538006622, file_file_name: nil, file_content_type: nil, file_file_size: nil, file_updated_at: nil, remote_url: "https://cf.mastohost.com/v1/AUTH_91eb37814936490c9...", created_at: "2020-05-11 14:46:57", updated_at: "2020-05-11 14:47:08", shortcode: nil, type: "unknown", file_meta: nil, account_id: 19719, description: "The black and orange bird is leaning down to nibbl...", scheduled_status_id: nil, blurhash: "UANKFv~p4o-;00fSRkbI00xuD%V?NGR*WAad">
irb(main):002:0> attachment.reset_file!
=> nil
irb(main):003:0> attachment.save!
   (0.5ms)  BEGIN
  Account Load (1.9ms)  SELECT  "accounts".* FROM "accounts" WHERE "accounts"."id" = $1 LIMIT $2  [["id", 19719], ["LIMIT", 1]]
   (0.8ms)  COMMIT
=> true
irb(main):004:0> 

[ThibG]

Thank you! I believe @noellabo has found the issue and it will soon be fixed (see #13741).

Basically, what is happening is that the re-download function is broken for media attachments, so temporary failures are not properly corrected.

[ThibG]

Fixed by #13741