Streetmerchant: Amazon giving captchas, not getting to item page

[stevezub]

Description

I tried disabling headless and noticed Amazon links are hitting a captcha page and not actually making it to the item page. Is there any way around this?

Running on Ubuntu 20.04

[jxjftw]

open a browser to amazon and pass the captcha manually.

[stevezub]

If I open Chromium and go to one of the Amazon product pages manually, I don't get prompted for a captcha. It just takes me to the product page.

[jxjftw]

Also, try custom user agents.

[jef]

Also, try custom user agents.

This is most likely the case. Amazon will tend to give more CAPTCHA requirements when the browser doesn't match the system.

[batcheck]

Did some quick googling and found this Stack Overflow discussion

https://stackoverflow.com/questions/55678095/bypassing-captchas-with-headless-chrome-using-puppetee

I added the following line to my USER_AGENT env variable

USER_AGENT=5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36

Watched it run through amazon with headless = false and captchas were gone.

Give that a shot.

[stevezub]

USER_AGENT=5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36

Awesome, thanks! That worked for me, too.