When I execute time.sleep in a parse of scrapy, does it block other concurrent requests? And what if I execute Request (from urllib)in a parse? On what occasions are all the concurrent requests blocked?
(Sorry for my lousy English!)
Hecate2
All 3 comments
不会卡掉的
sukaaaa
on 26 Dec 2018
In my experience the time.sleep method blocks the full application, so also all requests. What I know about scrapy is that it doesn't do concurrent requests (since its build on twisted which is not concurrent), and time.sleep blocks the current thread so this will block your full scraper.
Matthijsy
on 2 Jan 2019
👍4
@Hecate2 if @Matthijsy reply, anwer your question you could close this issue :smile:
joaquingx
on 8 Jan 2019
😄1
Was this page helpful?
0 / 5 - 0 ratings
Related issues
Gallaecio
·
3Comments
redapple
·
3Comments
Dainius-P
·
3Comments
mohmad-null
·
4Comments
osmenia
·
3Comments
Most helpful comment
In my experience the time.sleep method blocks the full application, so also all requests. What I know about scrapy is that it doesn't do concurrent requests (since its build on twisted which is not concurrent), and time.sleep blocks the current thread so this will block your full scraper.