Netlify-cms: Pull media library images through API
Currently, we use GitHub's raw.githubusercontent.com URLs, instead of pulling the media through the API. This doesn't really allow caching, and won't work with other new backends, like GitLab, since they don't necessarily have a download URL that will work for private repos.
tech4him1
All 20 comments
@Quicksaver mentioned in https://github.com/netlify/netlify-cms/pull/994 that the tokens are included in the download_urls, so it's not an issue on private GitHub repos.
tech4him1
on 9 Jan 2018
Pulling thumbnails through the API is still worthwhile because it allows caching.
erquhart
on 19 Jan 2018
This will be necessary for private GitLab repos.
tech4him1
on 30 Jan 2018
1092 will need to be addressed as part of this.
tech4him1
on 3 Apr 2018
I just about have a PR ready for this. When first loading images, however, they must all be downloaded through the API before showing anything. Currently, using URLs instead of through the API, we show all the image filenames, and then load the thumbnails when ready. I'm thinking that is a good idea to do here as well. Maybe we can just load thumbnails with a lazy-loader when they come into view?
@erquhart @Benaiah @talves Any input on this?
tech4him1
on 5 Jun 2018
Sorry, thought I commented here a while ago - lazy loading makes sense, agreed.
erquhart
on 11 Jun 2018
This has been implemented in the main GitLab PR. Instead of using a lazy-loader, Benaiah and I ended up just loading and caching all images in the background the first time the media library was opened, using a semaphore to prevent overloading the network connection (images are large). I think it is fairly common to scroll (or search) through a media library, so this may be a better idea than using a lazy-loader. We'll have to see how well it works in real life, and then backport it to the GitHub backend.
tech4him1
on 12 Jun 2018
As we finish rounding out this improvement, we should ensure image caching is happening now that we're pulling from API's. We could even cache the local file object to avoid the round trip for the user that does the initial upload.
erquhart
on 12 Jun 2018
In my use-case I have 6000+ (~1.5gb) worth of image assets in the library, so loading them all upfront and caching seems overwhelming, although I am not familiar with how a semaphore would be put to use in this case.
owenhoskins
on 12 Jun 2018
@owenhoskins That is a valid point. @erquhart Maybe we should wait to port the code back to the GitHub backend until thumbnail creation is in place?
tech4him1
on 12 Jun 2018
Hmm there's actually no need to download all asset blobs ever, thumbnails or not. We do this with entries because their contents are subject to search, but only the titles of blobs are searched and we get those en masse.
@Benaiah is working on virtualization (I think you did too @owenhoskins just not sure where it landed) so that only visible images would load their blobs - we can move forward with that even before getting thumbnail generation in place.
@tech4him1 to answer your exact question, though, yes, we should wait on either thumbnail generation or virtualization before porting images-via-API to GitHub, agreed.
erquhart
on 12 Jun 2018
@Benaiah I thought this was done, no?
erquhart
on 23 Oct 2018
Correct me if I'm wrong, but as far as I can see only the media library uses the API, the editor uses getAsset(), which does not have any API involvement. It also does not recognize when the image has in fact already been loaded when visiting the media library.
selaux
on 7 Dec 2018
@selaux that sounds about right, our primary concern was loading a billion images in the library. We should be pulling via API everywhere, agreed.
erquhart
on 11 Dec 2018
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
![stale[bot] picture](https://avatars3.githubusercontent.com/in/1724?v=4&s=40)
stale[bot]
on 29 Oct 2019
This was fixed here https://github.com/netlify/netlify-cms/pull/2817
erezrokah
on 12 Nov 2019
This looks like it is a specific fix for the github backend. Are you sure this fixes it for for other backends as well?
selaux
on 12 Nov 2019
This was already done for the other backends:
https://github.com/netlify/netlify-cms/blob/9978769ece9262265d3efa77357f9e8b46ad9a1e/packages/netlify-cms-backend-bitbucket/src/implementation.js#L236
erezrokah
on 12 Nov 2019
Sorry, my question was incomplete, I wanted to refer to both backends and areas where media is displayed. Basically I wanted to emphasize my previous comment here where this issue is also there when editing an entry.
selaux
on 12 Nov 2019
Sorry @selaux , I should have linked to https://github.com/netlify/netlify-cms/issues/1750.
I think that's what left of this issue.
erezrokah
on 12 Nov 2019
Related issues
papandreou
路
3Comments
tech4him1
路
3Comments
calavera
路
3Comments
dur41d
路
3Comments
andresalerno
路
3Comments
Most helpful comment
This has been implemented in the main GitLab PR. Instead of using a lazy-loader, Benaiah and I ended up just loading and caching all images in the background the first time the media library was opened, using a semaphore to prevent overloading the network connection (images are large). I think it is fairly common to scroll (or search) through a media library, so this may be a better idea than using a lazy-loader. We'll have to see how well it works in real life, and then backport it to the GitHub backend.