Magento2: Concurrent (quick reload) requests on checkout cause cart to empty - related to session_regenerate_id

[minlare]

Preconditions

1. Magento 2.2.0
2. PHP7

Steps to reproduce

1. Add product to cart
2. Navigate to checkout
3. Reload the page as quick as possible (F5 multiple times)

Expected result

1. Reload the checkout as usual
2. Cart contains session items

Actual result

1. Redirects to empty cart
2. session_id() is old session id (previous request regenerated a new session_id, but this request has old session_id)

http://php.net/manual/en/function.session-regenerate-id.php

In https://github.com/magento/magento2/blob/2.2-develop/lib/internal/Magento/Framework/Session/SessionManager.php#L507 you reference http://php.net/manual/en/function.session-regenerate-id.php#53480 however I believe on concurrent requests this does not work correctly...

[minlare]

I see you have re-opened the issue yourselves... Does that mean you were able to reproduce?

[magento-engcom-team]

@minlare, thank you for your report.
We've created internal ticket(s) MAGETWO-84160 to track progress on the issue.

[gewaechshaus]

Very good finding @minlare - Respect, this is a real conversion killer. Everybody can test it in one of our m2.2.1 live systems on https://www.makerdise.com... It's easy to reproduce.

@minlare - are you using varnish?

[josh-nh]

Following..

[minlare]

Thanks @gewaechshaus - it was a bit of a shock to find no-one else had raised the issue! I just tested on a base install of Magento 2.1.9 & 2.2.0 with sample data, so you'd have to test with varnish yourself...

[southerncomputer]

i've seen this on 2.1.10 with classylama_defaultbillingaddress plugin with custom OPC! It causes billing-information to loop 4 times - which then looses the session cookie!

[gewaechshaus]

@southerncomputer - we are also using a custom OPC. But you can test the regular checkout by just attaching /checkout/ instead of /onestepcheckout/ to the url - it does behave the same way.

However, I did test it on E3D website which doesn't seem to have this bug, but is also using a custom checkout.
https://e3d-online.com/

You can also easily reproduce this in
http://demo-acm2.bird.eu
https://magento.nublue.co.uk

@magento-engcom-team - This bug should be handled with the highest possible priority!

[acetronaut]

I have implemented some changes that seem to resolve the issue but if anyone can look over this and feedback that would be great

vendor/magento/framework/Session/SessionManager.php

The regenerateId function needs replacing with this

    public function regenerateId()
    {
        if (headers_sent()) {
            return $this;
        }

        if ($this->isSessionExists()) {
            $oldSessionId = session_id();            
            session_regenerate_id();   //regen the session
            $new_session_id = session_id();

            $_SESSION['new_session_id'] = $new_session_id;

            // Set destroy timestamp
            $_SESSION['destroyed'] = time();

            // Write and close current session;
            session_commit();
            $oldSession = $_SESSION;   //called after destroy - see destroy!
            // Start session with new session ID
            session_id($new_session_id);
            ini_set('session.use_strict_mode', 0);
            session_start();
            ini_set('session.use_strict_mode', 1);
            $_SESSION = $oldSession;
            // New session does not need them
            unset($_SESSION['destroyed']);
            unset($_SESSION['new_session_id']);  
        } else {
            session_start();
        }
        $this->storage->init(isset($_SESSION) ? $_SESSION : []);

        if ($this->sessionConfig->getUseCookies()) {
            $this->clearSubDomainSessionCookie();
        }
        return $this;
    }

and also the start function needs replacing with this

    public function start()
    {
        if (!$this->isSessionExists()) {
            \Magento\Framework\Profiler::start('session_start');

            try {
                $this->appState->getAreaCode();
            } catch (\Magento\Framework\Exception\LocalizedException $e) {
                throw new \Magento\Framework\Exception\SessionException(
                    new \Magento\Framework\Phrase(
                        'Area code not set: Area code must be set before starting a session.'
                    ),
                    $e
                );
            }

            // Need to apply the config options so they can be ready by session_start
            $this->initIniOptions();
            $this->registerSaveHandler();
            if (isset($_SESSION['new_session_id'])) {
             // Not fully expired yet. Could be lost cookie by unstable network.
             session_commit();
             session_id($_SESSION['new_session_id']);
             }
            // potential custom logic for session id (ex. switching between hosts)
            $this->setSessionId($this->sidResolver->getSid($this));
            session_start();
            if (isset($_SESSION['destroyed'])) {
               if ($_SESSION['destroyed'] < time()-300) {
                   $this->destroy(['clear_storage' => true]);

               }
            }
            $this->validator->validate($this);

            register_shutdown_function([$this, 'writeClose']);

            $this->_addHost();
            \Magento\Framework\Profiler::stop('session_start');
        }
        $this->storage->init(isset($_SESSION) ? $_SESSION : []);
        return $this;
    }

[gewaechshaus]

@acetronaut - give us a few hours...

[gewaechshaus]

@acetronaut

[acetronaut]

Further notes R.E this issue

public function regenerateId() was updated in 2.1.9 and most likely the reason this isnt apparent on all M2 instances

https://github.com/magento/magento2/blob/2.1.8/lib/internal/Magento/Framework/Session/SessionManager.php

https://github.com/magento/magento2/blob/2.1.9/lib/internal/Magento/Framework/Session/SessionManager.php

So a revert to the previous code may resolve the issue - however possibly re-introduce a different bug this was changing anyway

*If anyone knows why this was changed & what bug it was meant to fix, please let me know!*

[southerncomputer]

The above patch does not work. you guys have redis locking on sessions enabled?? there was a time when the config read locking inverted!!

[acetronaut]

its not a redis issue - you can disable redis and replicate the same thing

however the problem does kind of get worse - because session_regenerate_id actually calls the read function in redis as well as the session start

https://github.com/magento/magento2/issues/12362
This is a change to how PHP 7 works over 5.6

The above fix works - but im trying to confirm with magento status on their internal ticket for looking at this issue with more detail, ill be chasing them hard on this one

so basically if you use M2.1.9+ and the redis inverted fix to turn locking on, it causes a loop in the locking function

@keithbentrup identified the issue on the admin section - but this stretches much deeper as the checkout calls regenerateId and adds a lot of overhead onto the checkout process if you have locking turned on, enough to kill your conversions and cause customer dropouts anyway

so you kind of have to weigh up i guess whether instead of updating the function above, whether you should leave the redis locking fix in place for the benefits this will give you and remove the call to $this->_customerSession->regenerateId(); in this file

/vendor/magento/module-checkout/Controller/Index/Index.php

im trying at the moment to justify any security issues that could arise from this by session hijacking, especially as the regenerateId is also called on the login function itself

there are a few seperate but major issues at play here, ill feedback what i can on this, but at the same time if anyone else is debugging either of these issues in detail, please feel free to share your thoughts!

[southerncomputer]

Why would you call session regenerate from there? There a recent patch I applied that extends sessions on frontend to not let them timeout.

I'm getting a bug in my OPC where multiple dispatches starting at xhr billing-address are being called 3-simultaneosly that starts this issue!

[acetronaut]

you would have to ask magento for more details on why specifically they are calling it from there, but most likely its to prevent CSRF / XSS attacks, but i think its also called on login too anyway.

it bugs me a little anyway if there is a legitimate reason for it being there anyway such as possible session hijacking - that there isnt a note linking to doc that explains this, just so people dont remove it from the code

but anyway this wouldnt cause an issue anyway if the regenerate function wasnt causing the basket to drop, and also wouldnt cause an issue if the redis module didnt have the bug causing the session locking and overhead

these issues above affect core behaviour of magento, so it would be worth you stripping back, removing any modules or changes, replicating the core issues, fixing the core issues and building up from there so you can isolate if your issues are either the ones described here or new ones outside the scope of this problem

[southerncomputer]

Since the session expiration is only set on login/checkout - i suspect the regenerate is there to make sure you don't timeout during checkout!

[elevateweb]

The regeneration of the session id is not there to make sure you don't timeout

If you google session_regenerate_id or session fixation or session hijacking

https://stackoverflow.com/a/22965580

You will find a load of good explanations on what the session regeneration is done for.

the session timeout should only occur after period of inactivity, so this should increase on every page not only for example when you only visit certain pages like the checkout.

[southerncomputer]

https://www.google.com/search?client=firefox-b-1&ei=060YWu2bJoOIggeIoqqACA&q=session_regenerate_id+at+checkout&oq=session_regenerate_id+at+checkout

Regenerate session id is part of checkout success as part of clearing the session(contents) as far as *.php store code seems to discuss..

[Bobstar040]

we updated to m2.2.2 and when adding a product to cart and refresh checkout the checkout is empty when using incognito mode. We have a multi site, sometimes its working mostly not.

Any idea what's the fix for this? We did not have this issue in M2.1.9

[thaddeusmt]

This is still an issue in 2.2.2, which on slow development machine is a killer (in addition to being a conversion killer). I have opened a bug with Magento support (17 days ago), and they say they are working on it...

Sounds like it may be related to the redis session locking issue that caused 30 seconds of sleep waits and was "resolved" by turning session locking off.

[gewaechshaus]

@thaddeusmt - sadly it is, we are still patching all related systems. I can't understand how they handle priorities. This thing and the paypal express doesn't appear thing are the biggest conversion killers in our case.

[bpoiss]

Same problem here, everytime reloading the checkout or even navigate from checkout back to another page. @acetronaut fix does not change anything. Using local session storage and varnish, no redis. Any ideas how to fix this?

[thaddeusmt]

We have reverted (via a patch) to the 2.1.8 version of the SessionManager, and it seems to fix the issue:
https://github.com/magento/magento2/blob/2.1.8/lib/internal/Magento/Framework/Session/SessionManager.php

We have not yet noticed any side effects. (Are using Redis session caching with "locking" disabled).

[erikhansen]

This appears to be related to #4301. I've experienced this issue recently on a Magento 2.2.2 installation.

[yohanespradono]

happening in our 2.1.9.
The codes that @acetronaut posted above works well. I put those methods in a plugin.

[udovicic]

Confirmed on 2.2.1 EE

[elioermini]

The solution of @acetronaut is working on 2.1.12 is also solving this issue #4301 the only change on top I added in start() is $this->renewCookie($sid); after $this->validator->validate($this);

[aeu]

Thank you for reporting this @minlare and thanks for the solution @acetronaut . I'll try your fix and see if that help us. It is good to know that this is related to PHP7, because I had been trying figure out how come some stores that I was seeing online had this issue and some did not.

This is definitely a conversion killer and we received many client reports saying that users were complaining that their cart was mysteriously cleared.

[ghost]

The fix from acetronaut works.

[LockonZero]

Hello everyone! I also want to report that we have Magento Commerce clients complaining about the issue above. It really is a conversion killer, and it should have been solved by now.

It still occurs in version 2.2.3

[davidwindell]

Probably also related to #12742, switching off memcache session storage suddenly boosts checkout performance by ~15s

[sunilit42]

Hello,

Any Update on that?

[erfanimani]

Just offering my thoughts here — this is obviously a serious bug causing lost sessions, empty carts and lower conversions (and no offence, but the older generation probably still double-clicks web links!)

What are your alternatives though? Wait for this fix to be included in the next version? What if that's 2.3? What if you can't do a major upgrade and need to wait for the back-port? Your best bet is to patch your current installation and move on :)

There's two PRs open that are a workaround which you can use, but it's part of a larger issue; regenerateId is also called in other places (on every POST request in the frontend) and losing your cart could still happen elsewhere.

[gewaechshaus]

Link to most relevant PR
https://github.com/magento/magento2/pull/14487

[shyam100379]

Any updates? 2.2.3 got the same issue and we need to fix asap. We are currently in development process.
For me a single refresh in checkout page takes me to cart page.

[shyam100379]

It would be better to provide a quick fix to the solution rather than a whole upgrade i think. Like a patch which includes a file change, that kind of thing will suites here since it seems to be a small bug.

[southerncomputer]

you know programmatic fast loading of the backend causes failures as well. might be something lower level like redis causing trouble!

[shyam100379]

@southerncomputer no, as i mentioned above, after i went to checkout and a single refresh takes me to empty cart. The session is getting emptied for sure.

[aeu]

@southerncomputer as far as we know it's a PHP7 issue, not Redis. There was a fix provided by @acetronaut on Nov 27, 2017 that several posters have tested and works.

Stores are losing sales, I have seen this first hand and gotten more emails than I care to count from people saying they have tried to checkout and their cart is empty. Not only are conversions are being lost, but they are being lost for good. There's no abandoned cart, nothing to follow up on, the sale is just straight up gone.

I am amazed that a fix for this isn't being released as an emergency patch for all branches. I have no idea what could possibly be a more important issue to fix than a broken checkout.

[udovicic]

Not that it helps, but there is also a priority ticket opened on EE support for EE client with Magento ticketing system. But there is no response, same as here. We patched it for now ourselves, using the snippets provided in this thread.

[aeu]

Thanks @udovicic, good to know.

[shyam100379]

the solution provided by @acetronaut works. However, we would love Magento to release a patch or something for Magento 2.2.3 version.

[southerncomputer]

https://github.com/magento/magento2/pull/14484 has a pr for 2.2.x and works like @acetronaut 's solution!

but i think magento is going to conditionally apply session_regenerate_id only if you are coming from unsafe http store - if your store is 100% SSL - removing regerenate_session_id works just as well.

see https://github.com/magento/magento2/pull/14428#issuecomment-377156663 for condition solution

[aleksejtsebinoga]

Can confirm that in our case - the solution by @acetronaut is not working at Magento 2.1.1 and the session is still dying after 5-10 fast refreshes of the page.

[aeu]

@aleksejtsebinoga Are you on a SSL front end, and if so, have you tried the fix in this PR https://github.com/magento/magento2/pull/14428/commits/f4435fdf986eeb4ff16228aef27d8ee2bd90e342 ?

This is a change to remove the call to regenerateId(), which as I understand it, is the cause of the problem - there is a race condition and any requests that come in from the browser before the session ID is regenerated clear the session. Removing this call supposedly fixes the issue, but is only viable if the front end is on SSL already.

[aleksejtsebinoga]

@aeu ,

Thank you for the comment.
But it is not working on the http / https both.

[elioermini]

Hi all,

at the Imagine event I've spoken with several other developers that are having the same issue.
I've finally created the PR of what is actually working for us https://github.com/magento/magento2/pull/14973

Regards

[aleksejtsebinoga]

@acetronaut ,

Well, the fix is working perfectly after some magic. As your fix was done into the core files, it may be used by the Magento to include it into the build, but not by others to use the code, and keep the installation upgradable.

I've developed the around plugin approach for each of this functions (as regular rewrite was not working due to extending, not implementing the original interface - not perfect and obvious code structure) and executing all the logic in prior to the parent functions.

So may confirm the fix's working on https / http. Also, the #14973 looks like the same with a kind of improvement. And also some Magento's team movements happen in that PR, so maybe soon there will be an official patch or whatever.

[magento-engcom-team]

Hi @minlare. Thank you for your report.
The issue has been fixed in magento/magento2#14428 by @adrian-martinez-interactiv4 in 2.2-develop branch
Related commit(s):

• dc687c6730ad87189e86867a30e5349388a10777

The fix will be available with the upcoming 2.2.5 release.

[magento-engcom-team]

Hi @minlare. Thank you for your report.
The issue has been fixed in magento/magento2#14429 by @adrian-martinez-interactiv4 in 2.3-develop branch
Related commit(s):

• f82a17507acfa821e2bea543a99f072e1de1252f
• 4d23b8b3c64e4e0276c0f562ceba36f2c5688e4d

The fix will be available with the upcoming 2.3.0 release.

[magento-engcom-team]

Hi @minlare. Thank you for your report.
The issue has been fixed in magento/magento2#14973 by @elioermini in 2.2-develop branch
Related commit(s):

• 5e66f1b21b8741f9a4d54c6a31c7372e44780551
• 8f033ac342d8343e931a2104f0e2ba8f21156469
• 6ee737bb43adcd9b40c170170c5ea8e5d5ad6731
• d63c2c3899e9f91d6afd0a9438a0ecf1de7842e9
• 5dbce2a48e875cb38d82ba6648b5c04578bf9852
• b1865674bff3c03b92fcc3a2c85f4446735054de
• eabccc83c9ca4e4f55afe6451a93dee3f09115b9

The fix will be available with the upcoming 2.2.7 release.

[erikhansen]

Here is a patch that should fix the issue, until 2.2.7 is released. This patch contains the relevant contents of the f82a17 commit referenced in the comment from May 12th.

github-12362-composer.patch.zip

[ishakhsuvarov]

Hi @minlare. Thank you for your report.
The issue has been fixed in magento/magento2#17608 by @jignesh-baldha in 2.3-develop branch
Related commit(s):

• eee587bc647c053d9bd4eb18c699b00a19535b61
• 3828ac51d0185419c0463cda40734c19d50f186b
• ac42b70d9332cb848ca639362d4d4136e9385283
• 5b6e78ea23a7ca6d7149ea63788fd2debe98fdd0
• e688b8a5ed1539f48056d850fb2c7c0529ec43d3
• f53cfd7a8098896e32210b35e1c90ec3f5df1008
• bc15e1ab0df077996e09d11257474adcc0e14899

The fix will be available with the upcoming 2.3.0 release.

[saxsax1995]

Quick fixed:
https://magento.stackexchange.com/a/246493/70258
Hope it help some one.
Thanks :)

[gabrieljadeau]

hi @saxsax1995 : your patch isn't working for m2.2.5. Do you have link for this one ?

Best,

[saxsax1995]

hi @gabrieljadeau , maybe you will have to custom by yourself, that's what i did back there :( because each magento version is difference :(

[winterk80]

Confirmed this is occurring in m2.2.5 EE

[tuyennn]

Same issue on server running redis and magento 2.1.9

15641 #13427

[Alexander-Pop]

Did someone try https://github.com/integer-net/magento2-session-unblocker ?

[hmphu]

@Alexander-Pop have you try this magento2-session-unblocker module? Is it good

[Alexander-Pop]

@Alexander-Pop have you try this magento2-session-unblocker module? Is it good

Nope. Asked myself

[aleksejtsebinoga]

@hmphu @Alexander-Pop @winterk80 @tuyennn @gabrieljadeau ,

Guys, you can try this composer package, which is wrapping all the knowledge found on the web.
It was originally developed by me for Scandiweb, and now it became public -

https://bitbucket.org/scandiweb/sessionfix/src/master/

It would be great to know if it worked for you.

[hmphu]

Thanks @aleksejtsebinoga I will try it

[marco7319]

@aleksejtsebinoga

Hello,
can you explain to me in detail the steps please?
I use magento 2.3.2
cart empty when i add product in the cart

thanks

aleksejtsebinoga> @hmphu @Alexander-Pop @winterk80 @tuyennn @gabrieljadeau ,

Guys, you can try this composer package, which is wrapping all the knowledge found on the web.
It was originally developed by me for Scandiweb, and now it became public -

https://bitbucket.org/scandiweb/sessionfix/src/master/

aleksejtsebinoga

It would be great to know if it worked for you.

[hammockvienna]

i can confirm report of @marco7319 , i have the same issue on 2.3.2.

When i add something to the cart, the cart is not empty from a frontend user experience. BUT the small cart icon does not show (1). I have also seen occurrences that the small icon had (1) but after the page completed loading the (1) was gone. When i proceed to checkout the cart is already empty.
In the bug description above it only mentions that the issue happens on the checkout page with multiple reloading.

I also have seen that we dont have that issue on an testenvironment where is no traffic. The issue appears only in production with traffic? Is that logical?
We even dont reach the checkout page.
@aleksejtsebinoga : Could the reason be the same like the description above (race conditions?)
@hmphu did you try it? Did it solve the issue?

thanx

[aleksejtsebinoga]

@hammockvienna, @marco7319

Ah, sorry. I should've mentioned, that the patch was developed for 2.2.x.
Never tested it with 2.3.x.

But I suppose that they've updated session management. So it is not compatible anymore.

But you still have the empty cart problem in 2.3.2 without the patch?

[hammockvienna]

@aleksejtsebinoga
yes i have that issue as discribed above. The product is in the cart, but the cart i can changes after some ajax requests from (1) to (0). After that if i reload, the product is gone.
In Magento 2.2.8 i didnt have that issue, but we would like to upgrade to 2.3.2 and there we have it :-(

@Alexander-Pop which Magento version do you use? And did you check the magento2-session-unblocker in the meantime?

[marco7319]

see ticket #23923 . Is it an operating system problem? to install linux on his computer? or have a linux server?

[hammockvienna]

@marco7319 Its a Vultr linux server (In #23923 they mention windows)

[erfanimani]

For everyone that is experiencing empty cart issues, a good thing to check is to see if your session is actually different. So compare the PHPSESSID cookie value before and after the cart becomes empty. If it's the same session ID, it might not actually be a session loss issue, but something else that messes up your cart.

[aleksejtsebinoga]

@erfanimani , @marco7319 , @hammockvienna ,

Guys, please note that this thread is for concurrent session issue in checkout.
This bug happens, when you are landing into the checkout when it is under high load.

It is not related to the empty cart on the product adding.

[Alexander-Pop]

@hammockvienna still on M1 having all issues described here. In process of migration to M2 hopping that session problem will go away but as i see this "ghost" still appear from time to time on some stores...

[wajahatbashir]

I'm facing same issue on my live server, when i'm adding product to cart the ajax requests from (1) to (0). i'm getting this message (Product added successfully) But mini cart, cart page and checkout are empty. I'm using Magento 2.3.0.

On localhost add to cart is working fine but on live server this is not working.

[pmathbliss]

For me, there was an error tangent to this with the wishlist section had products that no longer exists.

Try changing the Magento\Customer\Controller\Section\Load:execute()
response
from
$response = ['message' => $this->getEscaper()->escapeHtml($e->getMessage())];
to
$response = ['message' => $this->getEscaper()->escapeHtml($e->getTraceAsString())];

then reload the frontend to see the stack trace.