Streetmerchant: Screenshot

[DerHOgi]

For a few tests:
Is it possible to take a screenshot everytime and not only on success?
What do I have to change in the code?

[jef]

This should do it for you.

diff --git a/src/store/lookup.ts b/src/store/lookup.ts
index 275557a..8082177 100644
--- a/src/store/lookup.ts
+++ b/src/store/lookup.ts
@@ -284,13 +284,13 @@ async function lookupCard(
                inStock[link.url] = false;
            }, 1000 * config.page.inStockWaitTime);
        }
+   }

-       if (config.page.screenshot) {
-           logger.debug('ℹ saving screenshot');
+   if (config.page.screenshot) {
+       logger.debug('ℹ saving screenshot');

-           link.screenshot = `success-${Date.now()}.png`;
-           await page.screenshot({path: link.screenshot});
-       }
+       link.screenshot = `success-${Date.now()}.png`;
+       await page.screenshot({path: link.screenshot});
    }

    return statusCode;

Basically move this code:

    if (config.page.screenshot) {
        logger.debug('ℹ saving screenshot');

        link.screenshot = `success-${Date.now()}.png`;
        await page.screenshot({path: link.screenshot});
    }

below the if statement in lookup.ts:267-294.

[DerHOgi]

Thanks @jef for you reply!
Now I can see what's going on and why I allways get false negative results. Item is available but bot result is OUT OF STOCK!
What can we do against this. There is a new captcha-check that concerns all Europe amazon-*.ts sites!

[jef]

I actually just fixed this bug too! Not sure how it ever happened, but #1209 should make this more apparent.

[gigi2006]

Thanks @jef for you reply!
Now I can see what's going on and why I allways get false negative results. Item is available but bot result is OUT OF STOCK!
What can we do against this. There is a new captcha-check that concerns all Europe amazon-*.ts sites!

hey @jef

how do you think we can solve the problem with the captcha? i don't think there will be a perfect solution, i already set and tried PAGE_BACKOFF_MIN and PAGE_BACKOFF_MAX as well as PAGE_SLEEP_MIN and PAGE_SLEEP_MAX. But it did not help much.

now a consideration, couldn't we just let the browser open when the captcha warning appears and then the real human could solve the captcha from time to time? so i would do it.

@DerHOgi
how could you solve the problem?
with me it is just the most important contact points, like amazon.de etc.

//edit:

by the way, on the same computer where the script runs i can use the amazon.de site and the playstation 5 etc... there is no captcha... so i thought of USER_AGENT.

so maybe you could do something with random USER_AGENT?

I have already set INCOGNITO and so on.

//edit2:

OK i found this from you, but I have not testet yet.

https://github.com/jef/streetmerchant/wiki/Troubleshoot:-General:-Captcha

//edit3:

OK so I tested it, unfortunately without success.
a cromium festern opens with about:blank and streetmerchant gets stuck.

[09:21:05] info :: [amazon-uk] Running linksBuilder...
[09:21:06] error :: Protocol error (Performance.enable): Target closed.
C:\streetmerchant\node_modules\puppeteerlib\cjs\puppeteer\common\Connection.js:208
this._callbacks.set(id, { resolve, reject, error: new Error(), method });
^

Error: Protocol error (Performance.enable): Target closed.
at C:\streetmerchant\node_modules\puppeteerlib\cjs\puppeteer\common\Connection.js:208:63
at new Promise ()
at CDPSession.send (C:\streetmerchant\node_modules\puppeteerlib\cjs\puppeteer\common\Connection.js:207:16)
at Page._initialize (C:\streetmerchant\node_modules\puppeteerlib\cjs\puppeteer\common\Page.js:181:26)
at Function.create (C:\streetmerchant\node_modules\puppeteerlib\cjs\puppeteer\common\Page.js:168:20)
at C:\streetmerchant\node_modules\puppeteerlib\cjs\puppeteer\common\Target.js:72:88
at processTicksAndRejections (node:internal/process/task_queues:93:5) {
level: 'x1B[31merrorx1B[39m',
timestamp: undefined,
metadata: {},
[Symbol(level)]: 'error',
[Symbol(message)]: 'x1B[90m[09:21:06]x1B[39m x1B[31merrorx1B[39m x1B[90m::x1B[39m Protocol error (Performance.enable): Target closed.'
}

//edit4:

ok i dont know why, but the second time the browser started normally... it was a bit too fast for me, but i still managed to get it working and could solve the captchas... but the problem came back right after that... so again captchas...

oh yeah and then again the same mistake as above and streetmerchant is interrupted...

edit5:

@jef

i have a question, would it make any difference if i log in to the chromium browser with my real data? so for all amazon, mediamarkt, saturn etc sites including google?

can it be that I as a logged in user don't have to do captchas anymore?

[DerHOgi]

@gigi2006 No, I have no solution for this problem until now. I tested the bot with an item that is on stock and I allways got false negative results from the bot. First I thought this is a problem with the container, but I can't find any errors. This is why I asked @jef if it is possible to make a screenshot everytime and not only on success. Then I saw, whats going on and why the bot is not working correctly. The captcha check.

I allready tried different things but with no success until now.

I run the bot in a virtual machine with ubutu. The strage thing: If I open my "normal" desktop browser in Windows or macOS, I can reload the website with the same ip without any captcha checks. Why?---- What is the difference? The operating system? The chromium brwoser? Allready tried different user agent strings, but without success. Maybe Amazon use techniques like browser fingerprinting and can notice immediately that a bot is running. Maybe because there are too many people using this bot...

[gigi2006]

@DerHOgi i think, but I am Not Sure.
You have no captchas Whit Login User in your normal Browser.

[DerHOgi]

@gigi2006 No, I tried this with no user logged in! Maybe Amazon foud a way to identify

[DerHOgi]

@gigi2006 I just wrote my own little bot only for amazon.de which uses the "normal" windows browser. No user is logged in. This is running now since 1 hour and there are no captchas. If this works fine, I will add some other european amazon websites.

[jef]

Try using PUPPETEER_EXECUTABLE_PATH. This will use your regular browser.

This is what it looks like on my machine:

PUPPETEER_EXECUTABLE_PATH=/usr/bin/google-chrome-stable npm run start

You can add before running or add to your .env. Up to you.

[DerHOgi]

@jef My ubuntu is running in a virtual machine only with command line interface - no gui...

[SAMLCSE]

It's works on Raspberry Pi. No issues. Running with Chromium browser[not
the Chromium].

On Sun, Dec 6, 2020, 8:37 AM DerHOgi [e-mail] wrote:

@gigi2006 https://github.com/gigi2006 I just wrote my own little bot
only for amazon.de which uses the "normal" windows browser. No user is
logged in. This is running now since 1 hour and there are no captchas.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/jef/streetmerchant/issues/1194#issuecomment-739503893,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AKCRMIJPUKY76SO5KE2TMQ3STOCKXANCNFSM4UON6N4Q
.

[SAMLCSE]

On Sun, Dec 6, 2020, 9:51 AM Madhusanka Jayawardhana <
[email protected]> wrote:

>

On Sun, Dec 6, 2020, 9:50 AM Madhusanka Jayawardhana <
[email protected]> wrote:

It's works on Raspberry Pi. No issues. Running with Chromium browser[not
the Chromium].

On Sun, Dec 6, 2020, 8:37 AM DerHOgi [e-mail] wrote:

@gigi2006 https://github.com/gigi2006 I just wrote my own little bot
only for amazon.de which uses the "normal" windows browser. No user is
logged in. This is running now since 1 hour and there are no captchas.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/jef/streetmerchant/issues/1194#issuecomment-739503893,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AKCRMIJPUKY76SO5KE2TMQ3STOCKXANCNFSM4UON6N4Q
.

[DerHOgi]

@SAMLCSE Will this work on ubuntu console only version? What do i have to do?

[SAMLCSE]

I'm not sure I'm running HEADLESS = "true" on my .env

On Sun, Dec 6, 2020, 10:00 AM DerHOgi [e-mail] wrote:

@SAMLCSE https://github.com/SAMLCSE Will this work on ubuntu console
only version? What do i have to do?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/jef/streetmerchant/issues/1194#issuecomment-739514565,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AKCRMIOUHSWM3LNUZDQYZX3STOL7BANCNFSM4UON6N4Q
.

[gigi2006]

Try using PUPPETEER_EXECUTABLE_PATH. This will use your regular browser.

This is what it looks like on my machine:

PUPPETEER_EXECUTABLE_PATH=/usr/bin/google-chrome-stable npm run start

You can add before running or add to your .env. Up to you.

hey @jef how can i test it on a windows machine?

i have this error here:

[19:26:39] error :: ✖ something bad happened, resetting streetmerchant in 5 seconds Tried to use PUPPETEER_EXECUTABLE_PATH env variable to launch browser but did not find any executable at: /usr/bin/google-chrome-stable npm run start {
"stack": "Error: Tried to use PUPPETEER_EXECUTABLE_PATH env variable to launch browser but did not find any executable at: /usr/bin/google-chrome-stable npm run start\n at ChromeLauncher.launch (C:\streetmerchant\node_modules\puppeteer\lib\cjs\puppeteer\node\Launcher.js:79:23)\n at async PuppeteerExtra.launch (C:\streetmerchant\node_modules\puppeteer-extra\dist\index.cjs.js:129:25)\n at async main (C:\streetmerchant\build\index.js:36:15)\n at async Timeout.loopMain [as _onTimeout] (C:\streetmerchant\build\index.js:70:9)"

[prettyboyy]

Try using PUPPETEER_EXECUTABLE_PATH. This will use your regular browser.

This is what it looks like on my machine:

PUPPETEER_EXECUTABLE_PATH=/usr/bin/google-chrome-stable npm run start

You can add before running or add to your .env. Up to you.

i tried this and it still pops up with captcha on my browser, even on incongnito mode it will still ask for captcha (both headless and no headless mode)

[SWEETDREAMSIV]

Try using PUPPETEER_EXECUTABLE_PATH. This will use your regular browser.
This is what it looks like on my machine:

PUPPETEER_EXECUTABLE_PATH=/usr/bin/google-chrome-stable npm run start

You can add before running or add to your .env. Up to you.

hey @jef how can i test it on a windows machine?

i have this error here:

[19:26:39] error :: ✖ something bad happened, resetting streetmerchant in 5 seconds Tried to use PUPPETEER_EXECUTABLE_PATH env variable to launch browser but did not find any executable at: /usr/bin/google-chrome-stable npm run start {
"stack": "Error: Tried to use PUPPETEER_EXECUTABLE_PATH env variable to launch browser but did not find any executable at: /usr/bin/google-chrome-stable npm run start\n at ChromeLauncher.launch (C:\streetmerchant\node_modules\puppeteerlib\cjs\puppeteer\node\Launcher.js:79:23)\n at async PuppeteerExtra.launch (C:\streetmerchant\node_modules\puppeteer-extra\dist\index.cjs.js:129:25)\n at async main (C:\streetmerchant\build\index.js:36:15)\n at async Timeout.loopMain [as _onTimeout] (C:\streetmerchant\build\index.js:70:9)"

you should use correct exe file location for your operating system with windows 10 is like:

C:\Program Files (x86)\Google\Chrome\Application\Chrome.exe

keep detecting captcha even with this fix I can't understand, for now the bot is unusable for 3 amazon stores.