Colly: How to get the content of iframe?

Created on 23 Aug 2018 · 6Comments · Source: gocolly/colly

I want to acquire comment data from a music website, but all comments content are loaded in the iframe and no way I could find to access these data, so what am I missing?

question

Source

killernova

All 6 comments

I haven't had direct experience with iframes, but I would imagine that you can setup a collector to look for the iframe tag, then take the src attribute and call collector.Visit() on the source URL. From there you would need another callback to collect the information you are looking for.

Let me know if that works!

vosmith on 28 Aug 2018

👎2 👍1

@killernova I'm going to close this issue for now. Feel free to reopen it if you feel it is not resolved.

vosmith on 5 Sep 2018

This doesn't work if the iframe uses srcdoc where the HTML is inline. There's no exposed method in Colly to feed this through as a synthetic request so the regular set of scrapers one might have configured can run.

wrouesnel on 30 Apr 2020

@wrouesnel could you provide an example HTML code?

asciimoo on 11 May 2020

@asciimoo Sure - so something like this - is what you end up with. It's just a single element with a srcdoc element containing full HTML. When scraping, you'd like to be able to register srcdoc as being a full HTML document so it can be parsed. My workaround is to construct this manually, which leads to a lot of code duplication.

<iframe x-ref="iframe" class="w-full" srcdoc="<!doctype html><meta charset=&quot;utf-8&quot;><meta name=&quot;viewport&quot; content=&quot;width=device-width, initial-scale=1, shrink-to-fit=no&quot;><link rel=&quot;stylesheet&quot; href=&quot;https://rsms.me/inter/inter.css&quot;><link rel=&quot;stylesheet&quot; href=&quot;/css/iframe.css?id=a05cfc4bbf6c64e1422e&quot;><script src=&quot;/js/iframe.js?id=1564701a250e51c06261&quot;></script><script src=&quot;https://cdn.jsdelivr.net/gh/alpinejs/[email protected]/dist/alpine.js&quot; defer></script><body class=&quot;antialiased font-sans bg-gray-200&quot;><div class=&quot;&quot; style=&quot;&quot;><div class=&quot;relative bg-gray-50 overflow-hidden&quot;></iframe>

The goal when I'm scraping is to pick up those script and hrefs inside the iframe, since they're all static content necessary to display the site (and with a workaround, work perfectly).

wrouesnel on 11 May 2020

Thanks! This raises a lot of questions. E.g. what if the original html and the content of the srcdoc contains an element with the same id attribute? I'm not sure if this feature should be supported by colly. Perhaps we should add a new function instead which creates HTMLElement from strings to avoid code duplication in these cases.
What do you think?

asciimoo on 19 May 2020

Was this page helpful?

0 / 5 - 0 ratings