Laravel-excel: [BUG] Temporary files are creating on one server on queued export

[shineraj94]

Prerequisites

• [x] Have read and understood: https://github.com/Maatwebsite/Laravel-Excel/blob/3.1/.github/SUPPORT.md
• [x] Checked if your Laravel Excel version is still supported: https://docs.laravel-excel.com/3.1/getting-started/support.html#supported-versions
• [x] Checked that your question isn't already asked before.
• [x] Filled in the entire issue template

Versions

• PHP version: 7.2
• Laravel version: 7.22.4
• Package version: 3.1.20

Description

Temporary files are creating on one server on queued export

Additional Information

Hello, I am running my application on laravel vapor (serverless). On queued exporting , the temporary files are not get stored on disk s3. I am stuck on that!

[patrickbrouwers]

Can you show what you are trying, can't help with this little information.

[shineraj94]

@patrickbrouwers
'temporary_files' => [

    /*
    |--------------------------------------------------------------------------
    | Local Temporary Path
    |--------------------------------------------------------------------------
    |
    | When exporting and importing files, we use a temporary file, before
    | storing reading or downloading. Here you can customize that path.
    |
    */
    'local_path' => storage_path('app/public'),

    /*
    |--------------------------------------------------------------------------
    | Remote Temporary Disk
    |--------------------------------------------------------------------------
    |
    | When dealing with a multi server setup with queues in which you
    | cannot rely on having a shared local temporary path, you might
    | want to store the temporary file on a shared disk. During the
    | queue executing, we'll retrieve the temporary file from that
    | location instead. When left to null, it will always use
    | the local path. This setting only has effect when using
    | in conjunction with queued imports and exports.
    |
    */
    'remote_disk' => 's3',
    'remote_prefix' => null,

],

this is my config file , here i am specifying a remote disk for temporary files, but temporary files are not getting stored on that disk

[patrickbrouwers]

Config looks good. I don't know enough about Vapor to give you pointers to debug, but perhaps your config file is cached or something. Maybe people on Laracasts/Stackoverflow will be quicker to help.

[shineraj94]

@patrickbrouwers in my local machine also this issue is coming - temporary files are not getting stored on that disk (s3)

[patrickbrouwers]

Try to go through the code with xdebug/dd() there's probably something wrong in your configuration that I can't see

[patrickbrouwers]

Relevant line: https://github.com/Maatwebsite/Laravel-Excel/blob/3.1/src/ExcelServiceProvider.php#L79

[shineraj94]

@patrickbrouwers I found one case, temp files are writing on the disk, but chunk read happens, it is not taking temp file from s3, any hope ?

[shineraj94]

@patrickbrouwers this is a similar issue like #2655

[patrickbrouwers]

Try setting that force setting to true then?

[shineraj94]

@patrickbrouwers I cannot do it. because my problem is unfortunately vapor doesn't support storing any temp files on the run.
So, maybe, I need support for something called serverless - that is temp files read and write from s3/any disk on chunk reading

[patrickbrouwers]

I believe there are se people using it on Vapor without issues

[shineraj94]

This is the error I am getting on vapor on trying queued export

ErrorException: fopen(/tmp/storage/framework/laravel-excel/laravel-excel-0KCoC8o8oDS1zsMpu927nkeCTzRgCu1E.xlsx): failed to open stream: No such file or directory in /var/task/vendor/phpoffice/phpspreadsheet/src/PhpSpreadsheet/Writer/BaseWriter.php:111
Stack trace:

0 [internal function]: Illuminate\Foundation\Bootstrap\HandleExceptions->handleError(2, 'fopen(/tmp/stor...', '/var/task/vendo...', 111, Array)

1 /var/task/vendor/phpoffice/phpspreadsheet/src/PhpSpreadsheet/Writer/BaseWriter.php(111): fopen('/tmp/storage/fr...', 'wb+')

2 /var/task/vendor/phpoffice/phpspreadsheet/src/PhpSpreadsheet/Writer/Xlsx.php(186): PhpOffice\PhpSpreadsheet\Writer\BaseWriter->openFileHandle('/tmp/storage/fr...')

3 /var/task/vendor/maatwebsite/excel/src/Writer.php(140): PhpOffice\PhpSpreadsheet\Writer\Xlsx->save('/tmp/storage/fr...')

4 /var/task/vendor/maatwebsite/excel/src/Jobs/QueueExport.php(73): Maatwebsite\Excel\Writer->write(Object(App\Exports\TaskExportTemplate), Object(Maatwebsite\Excel\Files\RemoteTemporaryFile), 'Xlsx')

5 [internal function]: Maatwebsite\Excel\Jobs\QueueExport->handle(Object(Maatwebsite\Excel\Writer))

6 /var/task/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(37): call_user_func_array(Array, Array)

7 /var/task/vendor/laravel/framework/src/Illuminate/Container/Util.php(37): Illuminate\Container\BoundMethod::Illuminate\Container{closure}()

8 /var/task/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(95): Illuminate\Container\Util::unwrapIfClosure(Object(Closure))

9 /var/task/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(39): Illuminate\Container\BoundMethod::callBoundMethod(Object(Illuminate\Foundation\Application), Array, Object(Closure))

10 /var/task/vendor/laravel/framework/src/Illuminate/Container/Container.php(596): Illuminate\Container\BoundMethod::call(Object(Illuminate\Foundation\Application), Array, Array, NULL)

11 /var/task/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(94): Illuminate\Container\Container->call(Array)

12 /var/task/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(128): Illuminate\Bus\Dispatcher->Illuminate\Bus{closure}(Object(Maatwebsite\Excel\Jobs\QueueExport))

13 /var/task/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(103): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline{closure}(Object(Maatwebsite\Excel\Jobs\QueueExport))

14 /var/task/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(98): Illuminate\Pipeline\Pipeline->then(Object(Closure))

15 /var/task/vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php(83): Illuminate\Bus\Dispatcher->dispatchNow(Object(Maatwebsite\Excel\Jobs\QueueExport), false)

16 /var/task/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(128): Illuminate\Queue\CallQueuedHandler->Illuminate\Queue{closure}(Object(Maatwebsite\Excel\Jobs\QueueExport))

17 /var/task/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(103): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline{closure}(Object(Maatwebsite\Excel\Jobs\QueueExport))

18 /var/task/vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php(85): Illuminate\Pipeline\Pipeline->then(Object(Closure))

19 /var/task/vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php(59): Illuminate\Queue\CallQueuedHandler->dispatchThroughMiddleware(Object(Laravel\Vapor\Queue\VaporJob), Object(Maatwebsite\Excel\Jobs\QueueExport))

20 /var/task/vendor/laravel/framework/src/Illuminate/Queue/Jobs/Job.php(98): Illuminate\Queue\CallQueuedHandler->call(Object(Laravel\Vapor\Queue\VaporJob), Array)

21 /var/task/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(356): Illuminate\Queue\Jobs\Job->fire()

22 /var/task/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(306): Illuminate\Queue\Worker->process('sqs', Object(Laravel\Vapor\Queue\VaporJob), Object(Illuminate\Queue\WorkerOptions))

23 /var/task/vendor/laravel/vapor-core/src/Queue/VaporWorker.php(31): Illuminate\Queue\Worker->runJob(Object(Laravel\Vapor\Queue\VaporJob), 'sqs', Object(Illuminate\Queue\WorkerOptions))

24 /var/task/vendor/laravel/vapor-core/src/Console/Commands/VaporWorkCommand.php(91): Laravel\Vapor\Queue\VaporWorker->runVaporJob(Object(Laravel\Vapor\Queue\VaporJob), 'sqs', Object(Illuminate\Queue\WorkerOptions))

25 [internal function]: Laravel\Vapor\Console\Commands\VaporWorkCommand->handle()

26 /var/task/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(37): call_user_func_array(Array, Array)

27 /var/task/vendor/laravel/framework/src/Illuminate/Container/Util.php(37): Illuminate\Container\BoundMethod::Illuminate\Container{closure}()

28 /var/task/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(95): Illuminate\Container\Util::unwrapIfClosure(Object(Closure))

29 /var/task/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(39): Illuminate\Container\BoundMethod::callBoundMethod(Object(Illuminate\Foundation\Application), Array, Object(Closure))

30 /var/task/vendor/laravel/framework/src/Illuminate/Container/Container.php(596): Illuminate\Container\BoundMethod::call(Object(Illuminate\Foundation\Application), Array, Array, NULL)

31 /var/task/vendor/laravel/framework/src/Illuminate/Console/Command.php(134): Illuminate\Container\Container->call(Array)

32 /var/task/vendor/symfony/console/Command/Command.php(258): Illuminate\Console\Command->execute(Object(Symfony\Component\Console\Input\StringInput), Object(Illuminate\Console\OutputStyle))

33 /var/task/vendor/laravel/framework/src/Illuminate/Console/Command.php(121): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\StringInput), Object(Illuminate\Console\OutputStyle))

34 /var/task/vendor/symfony/console/Application.php(911): Illuminate\Console\Command->run(Object(Symfony\Component\Console\Input\StringInput), Object(Symfony\Component\Console\Output\BufferedOutput))

35 /var/task/vendor/symfony/console/Application.php(264): Symfony\Component\Console\Application->doRunCommand(Object(Laravel\Vapor\Console\Commands\VaporWorkCommand), Object(Symfony\Component\Console\Input\StringInput), Object(Symfony\Component\Console\Output\BufferedOutput))

36 /var/task/vendor/symfony/console/Application.php(140): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\StringInput), Object(Symfony\Component\Console\Output\BufferedOutput))

37 /var/task/vendor/laravel/framework/src/Illuminate/Console/Application.php(93): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\StringInput), Object(Symfony\Component\Console\Output\BufferedOutput))

38 /var/task/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(129): Illuminate\Console\Application->run(Object(Symfony\Component\Console\Input\StringInput), Object(Symfony\Component\Console\Output\BufferedOutput))

39 /var/task/vendor/laravel/vapor-core/src/Runtime/Handlers/QueueHandler.php(60): Illuminate\Foundation\Console\Kernel->handle(Object(Symfony\Component\Console\Input\StringInput), Object(Symfony\Component\Console\Output\BufferedOutput))

40 /var/task/cliRuntime.php(66): Laravel\Vapor\Runtime\Handlers\QueueHandler->handle(Array)

41 /var/task/vendor/laravel/vapor-core/src/Runtime/LambdaRuntime.php(53): {closure}('afe5516a-587e-5...', Array)

42 /var/task/cliRuntime.php(68): Laravel\Vapor\Runtime\LambdaRuntime->nextInvocation(Object(Closure))

43 /var/task/runtime.php(30): require('/var/task/cliRu...')

44 /opt/bootstrap.php(6): require('/var/task/runti...')

45 {main}

[patrickbrouwers]

There's seems something wrong with your config still. It's not using the correct local path, it's using system's temp folder instead of Laravel's storage folder.

[shineraj94]

@patrickbrouwers Larevel vapor uses s3 as the default storage, but laravel excel still reading from local system/server,

[shineraj94]

@patrickbrouwers Is there a technical limitation preventing being able to read the file directly from s3?

[patrickbrouwers]

Yes it will always read from local after copying from s3. PhpSpreadsheet can only read from local disk

[patrickbrouwers]

With a correct configuration, it should work just fine

[shineraj94]

@patrickbrouwers In vapor, there are no servers...Aws lambda is processing the whole code. So, maybe for queued excel support, we need to go to launch a server and maybe need to run as a micro instance like thing!

[patrickbrouwers]

I know what Vapor and serverless is. I would suggest you try Laracasts or Stackoverflow, I don't think we are getting anywhere if you don't try out my suggestions.

[shineraj94]

I know what Vapor and serverless is. I would suggest you try Laracasts or Stackoverflow, I don't think we are getting anywhere if you don't try out my suggestions.

I tried your suggestions and cross-checked configurations as per documentation.
Temporary files are created on the first stage,
But when we come to the spreadsheet reading case, it throws this error.

Also, another dependency "Laravel snappy" needs this kind of local disk support.

But, this local disk case will not work in my case!

So, when I contacted vapor about this case, they are suggesting to store file on s3 and re-use it. What should i do ?

Hope you understand my situation!

[patrickbrouwers]

Other people have been able to use a local file which syncs with s3. I can't help more than this I'm sorry. Like I've suggested a few times, try a Laravel related forum.

[patrickbrouwers]

Here's an example of someone using queued exports with Vapor: https://github.com/Maatwebsite/Laravel-Excel/pull/2434

[shineraj94]

I will try the above solution

[shineraj94]

Here's an example of someone using queued exports with Vapor: #2434

@patrickbrouwers Extremely thank you for the suggestion! Now its work like a charm!

I just changed

'local_path' => sys_get_temp_dir(),
in excel.php
with remote disk vapor will work like this

Thanks for your patience & suggestions.

[abhijeet9920]

@patrickbrouwers, are these temporary files created by laravel-excel will be auto deleted after particular time? What if I set this value as null in excel.php?