Magento2: Product export gets "Message is added to queue, wait to get your file soon"

[mesnitu]

Just updated magento from 2.3.1 to 2.3.2, all looked fine until I wanted to export a csv file for products, etc... Is this a new featured, or how can I simple download / export the products as it was doing a couple hours ago?
The only thing I get is "Message is added to queue, wait to get your file soon".

Preconditions (*)

1. version 2.3.2
   2.

Steps to reproduce (*)

1. update from 2.3.1 to 2.3.2
2. try to export csv files (products, etc)

Expected result (*)

1. Get the files

Actual result (*)

1. "Message is added to queue, wait to get your file soon"
2. a temporary file is created at var/ , like importexport_5d14dcb332d84
3. No error log
4. No job is added to the cron_schedule table

[m2-assistant[bot]]

Hi @mesnitu. Thank you for your report.
To help us process this issue please make sure that you provided the following information:

• [ ] Summary of the issue
• [ ] Information on your environment
• [ ] Steps to reproduce
• [ ] Expected and actual results

Please make sure that the issue is reproducible on the vanilla Magento instance following Steps to reproduce. To deploy vanilla Magento instance on our environment, please, add a comment to the issue:

@magento give me 2.3-develop instance - upcoming 2.3.x release

For more details, please, review the Magento Contributor Assistant documentation.

@mesnitu do you confirm that you were able to reproduce the issue on vanilla Magento instance following steps to reproduce?

• [ ] yes
• [ ] no

---

• Join Magento Community Engineering Slack and ask your questions in #github channel.

[m2-assistant[bot]]

Hi @engcom-Bravo. Thank you for working on this issue.
In order to make sure that issue has enough information and ready for development, please read and check the following instruction: :point_down:

• [ ] 1. Verify that issue has all the required information. (Preconditions, Steps to reproduce, Expected result, Actual result).
  Details

  If the issue has a valid description, the label Issue: Format is valid will be added to the issue automatically. Please, edit issue description if needed, until label Issue: Format is valid appears.

• [ ] 2. Verify that issue has a meaningful description and provides enough information to reproduce the issue. If the report is valid, add Issue: Clear Description label to the issue by yourself.

• [ ] 3. Add Component: XXXXX label(s) to the ticket, indicating the components it may be related to.

• [ ] 4. Verify that the issue is reproducible on 2.3-develop branch

  Details

  - Add the comment @magento give me 2.3-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.3-develop branch, please, add the label Reproduced on 2.3.x.
  - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and _stop verification process here_!
  

• [ ] 5. Verify that the issue is reproducible on 2.2-develop branch.

  Details

  - Add the comment @magento give me 2.2-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.2-develop branch, please add the label Reproduced on 2.2.x

• [ ] 6. Add label Issue: Confirmed once verification is complete.

• [ ] 7. Make sure that automatic system confirms that report has been added to the backlog.

[engcom-Bravo]

Hello @mesnitu. Thanks for applying. To get those files now You need Your cron jobs be running. Will You try please to execute php bin/magento cron:run.

[mesnitu]

ok, I'm still waiting for the cron to end ( 5 minutes now...) , so, meanwhile, how can I disable this? I didn't do anything, except the update. This csv export, is a operation that expects a immediate conclusion to get the work done.

[mesnitu]

Nothing.
If I clear the cron_schedule table, and try to export, I get the same message, but there's no schedule job or id created at that table. Not sure how this works...

[agbdxb]

I had the same issue.

1. Got a sample csv for import >>System>>Import>>select Products>>click "Download Sample File",
2. Created 10 records of data for my products and imported successfully >> System >> Import
3. Triggered the cron run on command line
4. Guess what! that triggered all the exports of my past multiple attempts which had no movements despite several cron runs on command line. I could successfully download the exported file on System >> Export panel

Hope this helps.

[mesnitu]

Still no luck here. I'm on windows 10.
I'm doing a upgrade from 1.9.2 to 2.3.1 , all went more or less ok. I've working on 2.3.1 for about a month. So I saw the update msg, and I thought from 2.3.1 to 3.2 it's a piece of cake.
@agbdxb following your first step I found a error realated to the update:

1 exception(s):
Exception #0 (Magento\Framework\Exception\LocalizedException): Please upgrade your database: Run "bin/magento setup:upgrade" from the Magento root directory.
The following modules are outdated:
Magento_InventoryExportStockApi schema: current version - none, required version - 1.0.0
Magento_InventoryExportStockApi data: current version - none, required version - 1.0.0
Magento_InventoryReservationCli schema: current version - none, required version - 1.0.0
Magento_InventoryReservationCli data: current version - none, required version - 1.0.0
Magento_InventoryExportStock schema: current version - none, required version - 1.0.0
Magento_InventoryExportStock data: current version - none, required version - 1.0.0

So I did the setup:upgrade
But I get no results doing the next steps. ( except the I can actually EXPORT the sample_data file )... funny, right?
So I looked up how to setup a cron schedule on windows, etc ( That on 2.3.1 had no need to do this stuff )
Now I have a cmd popup every 5 minutes, and the cron_schedule table keeps growing. I have 1208 entries and counting.

A lot of missed jobs are created : bulk_cleanup, sales_grid_order_shipment_async_insert, sales_send_order_shipment_emails.
And the table keeps growing.

But i guess the bottom line, is that this was not occurring in 2.3.1 and I don't know why is happening now and how to fix this.

[mesnitu]

Reverting to 2.3.1 there's no such issue.

[NightmareGrl]

Same issue. Any solution yet?

[engcom-Bravo]

Hi again @mesnitu. Here are our steps after vanilla instance installation of 2.3-develop version:

1. Installation + cron:run
   
2. Product created
   
3. Export
   

[mesnitu]

Hi
My solution for now was to revert to 2.3.1 where this issue doesn't occur.
I guess ( really guessing ) that this is related to the cron tasks on windows. If I do that cron:run all I get is that cron_table populate heavily every time a job is fired. It will never end. I'll tested later if I can update to 2.3.2 on a virtual machine or something.

[engcom-Bravo]

@mesnitu, please take a not of this Magento official page, in particular the point about Operating Systems. As You can see Magento supports only Linux. Taking into consideration all aforementioned in this ticket we are supposed to close it. But please feel free to reopen it if You find a way to reproduce it on a vanilla Magento instance running on officially supported server environment. (By the way we also tried to reproduce Your issue on 2.3.2 version, the one You reported about, but trhe result was the same - can not reproduce). Thanks for applying.

[gaurangpadhiyar007]

Reopen this issue.
Not an issue with Linux or operating system.

Reopen issue.
Still present in magento 2.3.2

[gaurangpadhiyar007]

Temporary solution for products export:

public function execute()
{
    if ($this->getRequest()->getPost(ExportModel::FILTER_ELEMENT_GROUP)) {
        try {
            $params = $this->getRequest()->getParams();
            $model = $this->_objectManager->create(\Magento\ImportExport\Model\Export::class);
            $model->setData($this->getRequest()->getParams());
            $this->sessionManager->writeClose();

            return $this->fileFactory->create(
                $model->getFileName(),
                $model->export(),
                \Magento\Framework\App\Filesystem\DirectoryList::VAR_DIR,
                $model->getContentType()
            );

            /** @var ExportInfoFactory $dataObject */
            $dataObject = $this->exportInfoFactory->create(
                $params['file_format'],
                $params['entity'],
                $params['export_filter']
            );

            $this->messagePublisher->publish('import_export.export', $dataObject);
            $this->messageManager->addSuccessMessage(
                __('Message is added to queue, wait to get your file soon')
            );
        } catch (\Exception $e) {
            $this->_objectManager->get(\Psr\Log\LoggerInterface::class)->critical($e);
            $this->messageManager->addError(__('Please correct the data sent value.'));
        }
    } else {
        $this->messageManager->addError(__('Please correct the data sent value.'));
    }
    /** @var \Magento\Backend\Model\View\Result\Redirect $resultRedirect */
    $resultRedirect = $this->resultFactory->create(ResultFactory::TYPE_REDIRECT);
    $resultRedirect->setPath('adminhtml/*/index');
    return $resultRedirect;
}

[jenn0pal]

Hello @mesnitu. Thanks for applying. To get those files now You need Your cron jobs be running. Will You try please to execute php bin/magento cron:run.

I am also experiencing this issue. I also tried manually running the cron task.
Still no job is created

[battamich]

Clean mag 2.3.2 cloud installation (using docker locally) using centos 7
I Installed and run cron jobs, created the product and tried the export but no file has been created for me to download.
I can spot files inside var folder such as 'importexport_{id}' but they are all empty

[hostep]

Can everybody who runs against this, check if the message queue consumer processes are running?
So it works like this:

• the cron searches for running consumer processes
• if it finds a running process, it won't do anything
• if it doesn't find a running process, it should spawn one
• a running consumer process is sitting there and checking every second or so if a new message was added to the queue
• when it finds a new message in the queue, it will start processing it

You can find such processes with a command like:
ps aux | grep [c]onsumer

It should output something like this on a vanilla 2.3.2 installation:

someuser 123  0.0  2.1 530072 175892 ?       S    Aug09   0:33 php /path/to/bin/magento queue:consumers:start product_action_attribute.update --pid-file-path=product_action_attribute.update-some-hostname.pid --max-messages=10000
someuser 124  0.0  2.1 530068 176016 ?       S    Aug09   0:33 php /path/to/bin/magento queue:consumers:start product_action_attribute.website.update --pid-file-path=product_action_attribute.website.update-some-hostname.pid --max-messages=10000
someuser 125  0.0  2.1 530060 175952 ?       S    Aug09   0:33 php /path/to/bin/magento queue:consumers:start exportProcessor --pid-file-path=exportProcessor-some-hostname.pid --max-messages=10000
someuser 126  0.0  2.1 530048 175840 ?       S    Aug09   0:33 php /path/to/bin/magento queue:consumers:start codegeneratorProcessor --pid-file-path=codegeneratorProcessor-some-hostname.pid --max-messages=10000

If you can not find running consumer processes even though the cron is running. Then for some reason those process aren't getting spawned.
I would then start to debug the Magento\MessageQueue\Model\Cron\ConsumersRunner::run method. To see why they aren't getting spawned.

If you think you spotted a bug after debugging, then please let us know!

Alternatively you can also manually execute the bin/magento queue:consumers:start command but first look at the help section and see how this works so you understand the different options you can pass it. Also devdocs has some info on these commands.
This can also be automated using supervisor for example instead of using the cron, here's the documention around that.

[boudga]

I followed HOSTEP's advice and solved my queue problems

[himanshu8dev]

I am facing the same issue can anyone provide the solution for magento2.3.2

[battamich]

Looking at my cron_schedule table I see that the crons are running, even the consumers_runner with success status.

Alternatively you can also manually execute the bin/magento queue:consumers:start command but first look at the help section and see how this works so you understand the different options you can pass it. Also devdocs has some info on these commands.

I tried to manually execute the command as @hostep suggested:

php bin/magento queue:consumers:start exportProcessor

and the files were generated correctly! Thank you very much

[ArthurSCD]

After updating to 2.3.2, I started to experience issues that seem to be related to this as well. What I was doing was setting an attribute for multiple products at once. It would give me a message like so;
http://puu.sh/E5h3e/10b3c9a900.png

It never processed however. I found them sitting like mentioned above:
php bin/magento queue:consumers:list
product_action_attribute.update product_action_attribute.website.update exportProcessor codegeneratorProcessor async.operations.all

[avoelkl]

I just came across this issue.
We updated recently from 2.3.2 to 2.3.3 and the issue exits in Magento Open Source 2.3.3.

Running the command manually works as a temporary fix:
php bin/magento queue:consumers:start exportProcessor

[AndyJAllen]

How does a new version get released and a module is non-functional? If a module is updated shouldn't it be tested?

Something else i noticed; why is there a line in the export table that says "email"? if you try to delete it the page crashes. The following was dropped into system.log:

main.ERROR: Sorry, but the data is invalid or the file is not uploaded. [] []

[hostep]

It would still be great if somebody who runs against this issue can figure out exactly why the consumer processes aren't getting spawned by the cronjob in Magento 2.3.2 or 2.3.3:

I would then start to debug the Magento\MessageQueue\Model\Cron\ConsumersRunner::run method. To see why they aren't getting spawned.

Because out of the box on a vanilla installation it works perfectly fine both in 2.3.2 and 2.3.3, but maybe there are environmental issues which can cause this problem. But that can only be determined if people can put a little bit of time into debugging this 🙂

Thanks!

[ifekaj]

I had the same issue. Product export and mass attribute don't work unless i run the CLI command. i tried the following command ps aux | grep [c]onsumer which didn't return anything.
After a bit of experimenting, i found that changing the following in env.php helped:
'cron_consumers_runner' => array( 'cron_run' => true, 'max_messages' => 20000, 'consumers' => array( 'product_action_attribute.update', 'product_action_attribute.website.update', 'exportProcessor', 'codegeneratorProcessor', 'async.operations.all' ) ),
Not sure if this is a proper way to do it and will it cause more problems.

[hostep]

@ifekaj: that's indeed a solution, but leaving out the entire cron_consumers_runner section should result in the same behavior. See the docs around the defaults.

[AndyJAllen]

@ifekaj: that's indeed a solution, but leaving out the entire cron_consumers_runner section should result in the same behavior. See the docs around the defaults.

This is a very interesting point and can be why some of us are experiencing the issue being discussed here; I had the following in my env.php (which has probably been in there since early 2.2 versions)

'cron_consumers_runner' => [
'cron_run' => false,
'max_messages' => 1000,
'consumers' => [
'async.operations.all'
]

[RalphVB]

Actually, doing what @ifekaj wrote solved the queue problems in all my Magento 2.3.3 projects.
I found that solution in the documentation, here: Queue Solution

Here is how I configured my cron_consumers_runner:
'cron_consumers_runner' => [ 'cron_run' => true, 'max_messages' => 2000, 'consumers' => [ 'product_action_attribute.update', 'product_action_attribute.website.update', 'exportProcessor', 'codegeneratorProcessor' ] ],

[AndyJAllen]

Did you guys happen to get the following dropped into your system.log after modifying your env.php file?

main.INFO: Consumer "async.operations.all" skipped as required connection "amqp" is not configured. Unknown connection name amqp [] []

Sorry to deviate the post.

[RalphVB]

@AndyJAllen Yes, to prevent that message in the log take out from the env. php in the cron_consumers_runner the "async.operations.all". In my projects there has not been any problem taking it out.

[jesperbrejning]

We had the issue with bulk update of products. I can confirm that the solution suggested by @ifekaj is working :-)

[AaronONeill]

We had a similar issue with bulk updating product attributes also, enabling the following worked:

'cron_consumers_runner' => [ 'cron_run' => true, 'max_messages' => 50, 'consumers' => [ 'product_action_attribute.update', 'product_action_attribute.website.update' ] ],

This did however cause another issue where processes are not being killed on the server. This inevitably leads to the whole server crashing. We have disabled the cron_consumers_runner until we figure out how to resolve this but that means the bulk update no longer works.

Is anyone else experiencing any similar issues?

[rf-magma]

same error log after upgrading to 2.3.3 (php7.2), no async.operations.all in env.php (never activated rabbit). Have installed Redis with the upgrade but don't see how it could be the cause. Same for Elasticsearch 6.8.4.
Nothing to do with product export for us.
Sure thing all started with 2.3.2 or 2.3.3 (upgraded closely to solve multiple bugs), htop constantly monitored and "queue:consumers" processes always present as expected.
thanks @hostep for taking the time to advise us. We'll try to dig deeper on this

we've installed RabbitMQ and the system.log error isn't thrown anymore. We suspect a third party module or a magento 2.3.3 core functionality is supposing RabbitMQ installed

[ninex]

For anyone on Magento Cloud with this issue, make sure you have in .magento.env.yaml

stage:
    deploy:
        CRON_CONSUMERS_RUNNER:                                                                                 
            cron_run: true                                                                                       
            max_messages: 20000                                                                                   
            consumers:                                                                                           
                - product_action_attribute.update
                - product_action_attribute.website.update
                - codegeneratorProcessor
                - exportProcessor
                - quoteItemCleaner
                - inventoryQtyCounter
                - async.operations.all

[perfpcs]

Hey all, I also added this to my env.php like this:

'cron_consumers_runner' => [
'cron_run' => false,
'max_messages' => 1000,
'consumers' => [
'async.operations.all'
]
to supposedly fix hundreds of errors in our log like this:
main.INFO: Consumer "async.operations.all" skipped as required connection "amqp" is not configured. Unknown connection name amqp [] []
But we could not generate any coupon codes and so I tried RalphVB's solution:
'cron_consumers_runner' => [ 'cron_run' => true, 'max_messages' => 2000, 'consumers' => [ 'product_action_attribute.update', 'product_action_attribute.website.update', 'exportProcessor', 'codegeneratorProcessor' ] ],
This worked for me in 2.3.2 and we are now able to get coupon codes. Will see if the "async.operations.all" errors reappear. I hope not. Buth thank you RalphVB for solution!

[adamlavery]

we've installed RabbitMQ and the system.log error isn't thrown anymore. We suspect a third party module or a magento 2.3.3 core functionality is supposing RabbitMQ installed

@rf-magma - where did you find the info to correctly setup RabbitMQ? The Magento guide is inaccurate and woefully inadequate! Having finally got a working RabbitMQ (using 3.8.1 for correct distro, not 3.5.6 documented), what next? As the guide says to include the RMQ user, I guess we have to create one, but with what permissions?

It's not clear at all if Magento 2.3 needs RabbitMQ for core functions or if it has just been added for extensions to use! This is becoming all too familiar with M2 releases!

[hostep]

@adamlavery: RabbitMQ is only needed for the Bulk API of Magento (async.operations.all consumer), all the other message queue consumers which are implemented in core Magento use the Mysql database for storing the messages (this is for Magento 2.3.0 until 2.3.3, maybe that will change in upcoming newer versions).
Extensions can use whatever they want, it's defined in the xml files for queues which broker they use.
Hope this helps.

[Bhagyashri-Ranosys]

Temporary solution for products export:

```
public function execute()
{
if ($this->getRequest()->getPost(ExportModel::FILTER_ELEMENT_GROUP)) {
try {
$params = $this->getRequest()->getParams();
$model = $this->_objectManager->create(\Magento\ImportExport\Model\Export::class);
$model->setData($this->getRequest()->getParams());
$this->sessionManager->writeClose();

        return $this->fileFactory->create(
            $model->getFileName(),
            $model->export(),
            \Magento\Framework\App\Filesystem\DirectoryList::VAR_DIR,
            $model->getContentType()
        );

        /** @var ExportInfoFactory $dataObject */
        $dataObject = $this->exportInfoFactory->create(
            $params['file_format'],
            $params['entity'],
            $params['export_filter']
        );

        $this->messagePublisher->publish('import_export.export', $dataObject);
        $this->messageManager->addSuccessMessage(
            __('Message is added to queue, wait to get your file soon')
        );
    } catch (\Exception $e) {
        $this->_objectManager->get(\Psr\Log\LoggerInterface::class)->critical($e);
        $this->messageManager->addError(__('Please correct the data sent value.'));
    }
} else {
    $this->messageManager->addError(__('Please correct the data sent value.'));
}
/** @var \Magento\Backend\Model\View\Result\Redirect $resultRedirect */
$resultRedirect = $this->resultFactory->create(ResultFactory::TYPE_REDIRECT);
$resultRedirect->setPath('adminhtml/*/index');
return $resultRedirect;

}
```Is this worked for you ? and where you added this code

[Bhagyashri-Ranosys]

Hello TawabG,

You can resolve Above issue by overriding Below file, you have to pass
$filename instead of '$Path'.

1. Override
   "Magento\ImportExport\Controller\Adminhtml\Export\File\Download" in custom
   module.
2. Change the code by
   
   
   
   • Before *
     
     public function execute()
     
     {
     
     if (empty($fileName = $this->getRequest()->getParam('filename'))) {
     
     throw new LocalizedException(__('Please provide export file
     
     name'));
     
     }
     
     try {
     
     $path = 'export/' . $fileName;
     
     $directory =
     
     $this->filesystem->getDirectoryRead(DirectoryList::VAR_DIR);
     
     if ($directory->isFile($path)) {
     
     return $this->fileFactory->create(
     
     $path,
     
     $directory->readFile($path),
     
     DirectoryList::VAR_DIR,
     
     'application/octet-stream'
     
     );
     
     }
     
     } catch (LocalizedException | \Exception $exception) {
     
     throw new LocalizedException(__('There are no export file with
     
     such name %1', $fileName));
     
     }
     
     }
     
     After *
     
     public function execute()
     
     {
     
     if (empty($fileName = $this->getRequest()->getParam('filename'))) {
     
     throw new LocalizedException(__('Please provide export file
     
     name'));
     
     }
     
     try {
     
     $path = 'export/' . $fileName;
     
     $directory =
     
     $this->filesystem->getDirectoryRead(DirectoryList::VAR_DIR);
     
     if ($directory->isFile($path)) {
     
     return $this->fileFactory->create(
     
     *$fileName,
     
     $directory->readFile($path),
     
     DirectoryList::VAR_DIR,
     
     'application/octet-stream'
     
     );
     
     }
     
     } catch (LocalizedException | \Exception $exception) {
     
     throw new LocalizedException(__('There are no export file with
     
     such name %1', $fileName));
     
     }
     
     }
   
   

Best Regards,
Bhagyashri Pawar
Software Engineer,

[image: ranosys] http://ranosys.com/
[image: facebook] https://www.facebook.com/ranosys [image: twitter]
https://twitter.com/ranosys [image: linkedin]
https://www.linkedin.com/company/741079?trk=prof-exp-company-name [image:
googleplus] https://plus.google.com/+Ranosys

Head Office:
Oxley Bizhub, #06-48 | 73 Ubi Road 1 | Singapore - 408733
Tel: [phone]

Global Offices:
USA - San Francisco | UK - London | India - Pune, Jaipur, Bikaner

On Thu, Jan 9, 2020 at 4:45 PM TawabG [e-mail] wrote:

Running this command: php bin/magento queue:consumers:start
exportProcessor was a fix to find products, however when actually trying
to download a file I get an error that has to do with:

This site can’t be reached. The webpage at
https://mydomain.com/admin/admin/export_file/download/filename/catalog_product_20200109_101405.csv/key/b72c17d6da3319d91d39190e2445e7c31365a05d7788df6d40c2eabeb67ba5ef/
might be temporarily down or it may have moved permanently to a new web
address.

Can anybody help me with this error?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/magento/magento2/issues/23450?email_source=notifications&email_token=AOD73SEFW2ACEGIMGF6ONZTQ44BNXA5CNFSM4H35E6B2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIP6KEY#issuecomment-572515603,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AOD73SADLHHK4BYUXK4E2XDQ44BNXANCNFSM4H35E6BQ
.

[seattletechworks]

I'm running Magento 2.3.3 (Linux, PHP 7.2) on Nexcess Cloud hosting. I tried exporting products to a csv file and no file is generated. I ran it about 5 times. Then I found an online thread in which someone manually ran cron -- php bin/magento cron:run. After running this, Voila! - all the exports I tried to generate appeared. Subsequent exports are working without issue now. They get dumped to var/export.

Regarding the "email" file in the export table, you can remove it in the var/export directory. It generates an error when trying to remove from the admin console.

How does a new version get released and a module is non-functional? If a module is updated shouldn't it be tested?

Something else i noticed; why is there a line in the export table that says "email"? if you try to delete it the page crashes. The following was dropped into system.log:

main.ERROR: Sorry, but the data is invalid or the file is not uploaded. []

[pramodjadhav1419]

Hello All

I also Tried to export by running php bin/magento cron:run But It won't help, then after some research i got this command :::::: php bin/magento queue:consumers:start exportProcessor :::::
Please run this command and try to export.

Hope this help.

[extremeskillz]

Same issue here with Magento2.3.3 on Ubuntu 16.04lts with exporting anything really.

All other processes for indexing and cache seem to be working fine. Importing products/updates also works as it should.

[TawabG]

After running this command php bin/magento queue:consumers:start exportProcessor I encountered another error.

This site can’t be reached The webpage at https://mydomain.com/admin/admin/export_file/download/filename/catalog_product_20200125.csv/key/ec1fd037eef6ddf02e1159787/ might be temporarily down or it may have moved permanently to a new web address.

I can see a new .csv file created on the server, but for some reason I don't have access to it.
Can anyone help me with this?

[Bhagyashri-Ranosys]

@TawabG,

you have to pass $filename instead of '$Path'. in below file.

Magento\ImportExport\Controller\Adminhtml\Export\File\Download.php

Before

public function execute()
{
if (empty($fileName = $this->getRequest()->getParam('filename'))) {
throw new LocalizedException(__('Please provide export file name'));
}
try {
$path = 'export/' . $fileName;
$directory = $this->filesystem->getDirectoryRead(DirectoryList::VAR_DIR);
if ($directory->isFile($path)) {
return $this->fileFactory->create(
$path,
$directory->readFile($path),
DirectoryList::VAR_DIR,
'application/octet-stream'
);
}
} catch (LocalizedException | \Exception $exception) {
throw new LocalizedException(__('There are no export file with such name %1', $fileName));
}
}
After
public function execute()
{
if (empty($fileName = $this->getRequest()->getParam('filename'))) {
throw new LocalizedException(__('Please provide export file name'));
}
try {
$path = 'export/' . $fileName;
$directory = $this->filesystem->getDirectoryRead(DirectoryList::VAR_DIR);
if ($directory->isFile($path)) {
return $this->fileFactory->create(
$fileName,
$directory->readFile($path),
DirectoryList::VAR_DIR,
'application/octet-stream'
);
}
} catch (LocalizedException | \Exception $exception) {
throw new LocalizedException(__('There are no export file with such name %1', $fileName));
}
}
```

[TawabG]

@TawabG,

you have to pass $filename instead of '$Path'. in below file.

Magento\ImportExport\Controller\Adminhtml\Export\File\Download.php

Before

public function execute()
{
if (empty($fileName = $this->getRequest()->getParam('filename'))) {
throw new LocalizedException(('Please provide export file name')); } try { $path = 'export/' . $fileName; $directory = $this->filesystem->getDirectoryRead(DirectoryList::VAR_DIR); if ($directory->isFile($path)) { return $this->fileFactory->create( $path, $directory->readFile($path), DirectoryList::VAR_DIR, 'application/octet-stream' ); } } catch (LocalizedException | \Exception $exception) { throw new LocalizedException(('There are no export file with such name %1', $fileName));
}
}
After
public function execute()
{
if (empty($fileName = $this->getRequest()->getParam('filename'))) {
throw new LocalizedException(('Please provide export file name')); } try { $path = 'export/' . $fileName; $directory = $this->filesystem->getDirectoryRead(DirectoryList::VAR_DIR); if ($directory->isFile($path)) { return $this->fileFactory->create( $fileName, $directory->readFile($path), DirectoryList::VAR_DIR, 'application/octet-stream' ); } } catch (LocalizedException | \Exception $exception) { throw new LocalizedException(('There are no export file with such name %1', $fileName));
}
}

Thanks for the reply! Your solution partly solved my problem.
Now, I get the notification "Message is added to queue, wait to get your file soon" then when I look at my export folder: a new catalog_product_xxx.CSV is created. However this file is only created when the command "php bin/magento queue:consumers:start exportProcessor" is running. Without this command no export will be done.

In addition, I noticed that the new export files are saved with '664' permissions.

[TawabG]

bin/magento queue:consumers:start

I can't imagine this being the only solution to this problem. This means I have to open CLI and run this command for EVERY export i'm doing!? That's ridiculous.

[seattletechworks]

@TawabG - check your cron processes.

Run "ps -ef | grep cron"

Delete any old processes which may be stuck / still running.

I had similar issue in my dev site. I ran this command -

php bin/magento queue:consumers:list

product_action_attribute.update
product_action_attribute.website.update
exportProcessor
codegeneratorProcessor
async.operations.all

Cron should be kicking off jobs for these consumers. The exportProcessor is the job which handles the export.

You should see similar process in your terminal -

Run "ps -ef | grep consumers"

php bin/magento queue:consumers:start exportProcessor --single-thread --max-messages=10000

[PsmIndia]

There any way to export product csv without run commands ? my magento ver 2.3.4
Running the command manually works as a temporary fix: php bin/magento queue:consumers:start exportProcessor

[lesliemcc]

hi i have no coding experience and i need to export my magento 2.3.2 stores customers over to shopify but when i try the following message appears and nothing happens.

[boudga]

Can you SSH into your server? If so then we can help you out.

On Tue, Jul 21, 2020, 6:39 AM lesliemcc [e-mail] wrote:

hi i have no coding experience and i need to export my magento 2.3.2
stores customers over to shopify but when i try the following message
appears and nothing happens.
[image: lkjhbnm,lkjn]
https://user-images.githubusercontent.com/68594308/88050663-2ad9e780-cb4f-11ea-8e9b-ce5b34c50c69.JPG

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/magento/magento2/issues/23450#issuecomment-661804141,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AADAT2FM5E4TE5RSKOOFIRTR4V47DANCNFSM4H35E6BQ
.

[lesliemcc]

I don’t I’m afraid as an independent team deals with it.

Kind Regards,

Leslie McCulloch,
Miskie London LTD

Phone: [phone]
Email: [email protected]

On 21 Jul 2020, at 14:49, Bruce Gemberling [e-mail] wrote:


Can you SSH into your server? If so then we can help you out.

On Tue, Jul 21, 2020, 6:39 AM lesliemcc [e-mail] wrote:

hi i have no coding experience and i need to export my magento 2.3.2
stores customers over to shopify but when i try the following message
appears and nothing happens.
[image: lkjhbnm,lkjn]
https://user-images.githubusercontent.com/68594308/88050663-2ad9e780-cb4f-11ea-8e9b-ce5b34c50c69.JPG

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/magento/magento2/issues/23450#issuecomment-661804141,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AADAT2FM5E4TE5RSKOOFIRTR4V47DANCNFSM4H35E6BQ
.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.

[adamlavery]

For some reason, Magento Core decided to make exports always asynchronous (run in the background) instead of optionally running big jobs in the background. Which means you have to wait for the job to run and check back on the export page regularly until you see your export file is there. It's a bit rubbish, but that's how it is now! If it never appears, it means your cron is not running (should see a notification about that). That's something you'll need to get your web developer to sort out.

[agbdxb]

Add a cron on the control panel of your webhost and set it to run the below command every 2 min.s (or the longest you are willing to wait). If you are unable to do it yourself, perhaps seek help from your webhost helpdesk.

php bin/magento queue:consumers:start exportProcessor

[HenKun]

Can everybody who runs against this, check if the message queue consumer processes are running?
So it works like this:

* the cron searches for running consumer processes

* if it finds a running process, it won't do anything

* if it doesn't find a running process, it should spawn one

* a running consumer process is sitting there and checking every second or so if a new message was added to the queue

* when it finds a new message in the queue, it will start processing it

You can find such processes with a command like:
ps aux | grep [c]onsumer

It should output something like this on a vanilla 2.3.2 installation:

someuser 123  0.0  2.1 530072 175892 ?       S    Aug09   0:33 php /path/to/bin/magento queue:consumers:start product_action_attribute.update --pid-file-path=product_action_attribute.update-some-hostname.pid --max-messages=10000
someuser 124  0.0  2.1 530068 176016 ?       S    Aug09   0:33 php /path/to/bin/magento queue:consumers:start product_action_attribute.website.update --pid-file-path=product_action_attribute.website.update-some-hostname.pid --max-messages=10000
someuser 125  0.0  2.1 530060 175952 ?       S    Aug09   0:33 php /path/to/bin/magento queue:consumers:start exportProcessor --pid-file-path=exportProcessor-some-hostname.pid --max-messages=10000
someuser 126  0.0  2.1 530048 175840 ?       S    Aug09   0:33 php /path/to/bin/magento queue:consumers:start codegeneratorProcessor --pid-file-path=codegeneratorProcessor-some-hostname.pid --max-messages=10000

If you can not find running consumer processes even though the cron is running. Then for some reason those process aren't getting spawned.
I would then start to debug the Magento\MessageQueue\Model\Cron\ConsumersRunner::run method. To see why they aren't getting spawned.

If you think you spotted a bug after debugging, then please let us know!

Alternatively you can also manually execute the bin/magento queue:consumers:start command but first look at the help section and see how this works so you understand the different options you can pass it. Also devdocs has some info on these commands.
This can also be automated using supervisor for example instead of using the cron, here's the documention around that.

@hostep I debugged in and found the following:

• Cron is running correctly and all settings are fetched correctly, also the empty array [] for cron_consumers_runner does work
• The problem lies in the locking.
• Although no processes for the consumers are spawned (anymore!?), MySQL (in my case MariaDB) still holds locks for all consumers. When the consumers are triggered by cron they are started as single-thread, hence, checking the lock. So the proccesses are not spawned. The locking and unlocking is done in StartConsumerCommand. It might be that there lies the issue, cause unlock is not called, if something goes wrong in $consumer->process($numberOfMessages);. I am not that good in PHP, in C# one needs to put the unlock in a finally statement, so it is safely unlocked, even when process throws. You said "a running consumer process is sitting there and checking every second or so if a new message was added to the queue", which would mean that $consumer->process($numberOfMessages); is not supposed to end? (<-- I get the impression this is true, when calling via cli). Under what circumstances does process return without exception, so that unlock will be called? Alternatively, if $consumer->process($numberOfMessages); would return immediately after spawning the process (leaving the process runnning), the lock would be of no use, so I might not understand the inner workings correctly.
• If you run the consumer in console, it is not single-threaded by default, hence, the lock is not checked. That's why it does work for most of the users, although it does not work via cron. If the lock is held, one can check by using bin/magento queue:consumers:start product_action_attribute.update --single-thread. An error Consumer with the same name is running is returnd if the lock is held.
• Even more unfortunately I can not release the lock manually from database. RELEASE_LOCK(md5(consumerName)); must be called from the thread which triggered the lock. RELEASE_ALL_LOCKS is not yet available in my MariaDB version, and I am not sure if this would remove locks triggered by other threads.

In a nutshell:

• 'cron_consumers_runner' => [ 'cron_run' => true, 'max_messages' => 50, 'consumers' => [ 'product_action_attribute.update', 'product_action_attribute.website.update' ] ], can be completely left out from the env.php, the default values are fetched correctly
• in my case the database held a lock, preventing the consumer proccess from being spawned, when called by cron
• I cannot remove the lock from database
• starting consumers from CLI does work, cause it does not use the --single-thread parameter by default, avoiding the check for the lock.

If it is not fully clear, what I wrote, please ask for clarification :D

[hostep]

@HenKun

I've had some issues in the past as well with locks in mysql getting stuck (not necessarily with queue consumers, it was with cronjobs I memory serves me right, but they both use the same locking mechanism). We've found that switching the lock provider from the database to the filesystem to be a lot more stable.
Here is the documentation to do that: https://devdocs.magento.com/guides/v2.3/install-gde/install/cli/install-cli-subcommands-lock.html#instgde-cli-lockconfig

I hope this can solve your issue!

[HenKun]

Thanks for the link! While this might solve the "stucking" issue for me, since I can interfere with the file system, I think there is a larger underlying problem.

1. The unlock is not guaranteed to be called (as far as I can see). At least it should be in a finally clause.
   1a. If running as single-thread and no max-messages parameter is given, process will never end gracefully (it's in a while(true) loop - ), hence, the lock cannot be freed. (Although I think the consumers are not supposed to be run without this parameter, it is still possible, even from cron if the default settings are changed)
   1b. If running as single-thread and max-messages parameter is given, it is run in a for loop, where exceptions can occur. There is even an exit(0); (although I do not understand when this is called ad-hoc). If an exception occurs, the lock is not freed.
2. The DB locking mechanism is the default, so as far as I can read from multiple GitHub issues and StackOverflow questions, more users seem to have problems with not starting bulk actions, that might be caused by stucked locking. Starting the consumers from CLI seems to solve the issue for many users, but at that moment they don't realize it just solves the issue for a very short period of time. (Like me when I noticed it last week)
3. Is the locking meta data copied, when databases are copied? E.g. when reapplying backup databases or when moving databases from stage to production? That might be another reason for locks being present but no processes being spawned.
4. Does using the file system lock have any performance drawbacks against using the db locking mechanism?
5. Would it be a possibility to not using locks but directly check for a matching running process? Again, I am not that much into PHP and processes, so this is just my naive idea.

[arvind-curotec]

Magento version 2.3.4-p2
All processes running correctly. Still the export not working.

Here are the consumer processes:

php bin/magento queue:consumers:list
product_action_attribute.update
product_action_attribute.website.update
exportProcessor
codegeneratorProcessor
async.operations.all

[HenKun]

Magento version 2.3.4-p2
All processes running correctly. Still the export not working.

Here are the consumer processes:

php bin/magento queue:consumers:list
product_action_attribute.update
product_action_attribute.website.update
exportProcessor
codegeneratorProcessor
async.operations.all

Listing the queue consumers does not mean the processes are running.
Run ps aux to see if they are really spawned.

[arvind-curotec]

@HenKun Please See the output below:

-bash-4.2$ ps aux | grep [c]onsumer
user     11883  0.0  1.0 487880 166556 ?       S    Sep08   1:12 /usr/bin/php httpdocs/bin/magento queue:consumers:start product_action_attribute.update --single-thread --max-messages=10000
user     11885  0.0  1.0 487880 164960 ?       S    Sep08   1:13 /usr/bin/php httpdocs/bin/magento queue:consumers:start product_action_attribute.website.update --single-thread --max-messages=10000
user     11889  0.0  1.0 487884 164348 ?       S    Sep08   1:13 /usr/bin/php httpdocs/bin/magento queue:consumers:start codegeneratorProcessor --single-thread --max-messages=10000
root     14218  1.0  1.0 591576 164296 pts/1   S    09:41   0:00 /usr/bin/php httpdocs/bin/magento queue:consumers:start exportProcessor --single-thread --max-messages=10000

[HenKun]

So obviously they are running. Just wanted to make you aware, that queue:consumers:list is no indicator of running processes.

[perfpcs]

Alot of us are having this bulk update problem and general messaging broker problems with M 2.4.0. Would be nice to get team Magento to fix this bad problem!! https://github.com/magento/magento2/issues/29797

[Cesarorp]

@gaurangpadhiyar007

In what module can we run this?

public function execute()
{
if ($this->getRequest()->getPost(ExportModel::FILTER_ELEMENT_GROUP)) {
try {
$params = $this->getRequest()->getParams();
$model = $this->_objectManager->create(\Magento\ImportExport\Model\Export::class);
$model->setData($this->getRequest()->getParams());
$this->sessionManager->writeClose();

        return $this->fileFactory->create(
            $model->getFileName(),
            $model->export(),
            \Magento\Framework\App\Filesystem\DirectoryList::VAR_DIR,
            $model->getContentType()
        );

        /** @var ExportInfoFactory $dataObject */
        $dataObject = $this->exportInfoFactory->create(
            $params['file_format'],
            $params['entity'],
            $params['export_filter']
        );

        $this->messagePublisher->publish('import_export.export', $dataObject);
        $this->messageManager->addSuccessMessage(
            __('Message is added to queue, wait to get your file soon')
        );
    } catch (\Exception $e) {
        $this->_objectManager->get(\Psr\Log\LoggerInterface::class)->critical($e);
        $this->messageManager->addError(__('Please correct the data sent value.'));
    }
} else {
    $this->messageManager->addError(__('Please correct the data sent value.'));
}
/** @var \Magento\Backend\Model\View\Result\Redirect $resultRedirect */
$resultRedirect = $this->resultFactory->create(ResultFactory::TYPE_REDIRECT);
$resultRedirect->setPath('adminhtml/*/index');
return $resultRedirect;

}

[gaurangpadhiyar007]

@gaurangpadhiyar007

In what module can we run this?

public function execute()
{
if ($this->getRequest()->getPost(ExportModel::FILTER_ELEMENT_GROUP)) {
try {
$params = $this->getRequest()->getParams();
$model = $this->_objectManager->create(\Magento\ImportExport\Model\Export::class);
$model->setData($this->getRequest()->getParams());
$this->sessionManager->writeClose();

        return $this->fileFactory->create(
            $model->getFileName(),
            $model->export(),
            \Magento\Framework\App\Filesystem\DirectoryList::VAR_DIR,
            $model->getContentType()
        );

        /** @var ExportInfoFactory $dataObject */
        $dataObject = $this->exportInfoFactory->create(
            $params['file_format'],
            $params['entity'],
            $params['export_filter']
        );

        $this->messagePublisher->publish('import_export.export', $dataObject);
        $this->messageManager->addSuccessMessage(
            __('Message is added to queue, wait to get your file soon')
        );
    } catch (\Exception $e) {
        $this->_objectManager->get(\Psr\Log\LoggerInterface::class)->critical($e);
        $this->messageManager->addError(__('Please correct the data sent value.'));
    }
} else {
    $this->messageManager->addError(__('Please correct the data sent value.'));
}
/** @var \Magento\Backend\Model\View\Result\Redirect $resultRedirect */
$resultRedirect = $this->resultFactory->create(ResultFactory::TYPE_REDIRECT);
$resultRedirect->setPath('adminhtml/*/index');
return $resultRedirect;

}

This is file.

Magento\ImportExport\Controller\Adminhtml\Export\File\Download

[magecoders]

@hostep As per your suggestion, the consumer processes are running. But both bulk product attribute update and export products operation not working. Any ideas ?

Thanks

[hostep]

@magecoders: the status says 'in progress', so they should be running and your screenshot also shows that those particular consumers are running, so that's good!
But if it takes a really long time and you don't see any progress, then something is wrong somehow. You'll have to start to debug this somewhat I'm afraid. We personally haven't noticed problems like these yet on Magento 2.3.x

[hostep]

@HenKun: you made some good remarks before, could you open a new issue with your findings, because these might be important to get fixed. Thanks!

[magecoders]

@HenKun I agree with your points, I have debug into running consumer process and they stuck due to DB lock in my case.