Laravel-backup: All backups gone, and backup:monitor doesn't work now

[peterbrinck]

4 days ago I updated to v6, and everything worked fine, until today.

All of a sudden, all my backups are gone this morning (I have all my scheduled backups jobs running at night), and backup:monitor doesn't work anymore, even after manually running a backup and then backup:monitor, it tells me that there's no backups and it's unhealthy.

backup:list doesn't work correct either. It's showing the wrong disk and no backups as well.
I'm using S3 as disk.

I thought that maybe it was because I forgot to update the backup.php config file, to match the newest v6 file, so I just did that, and it still doesn't work.

But again, it worked without the v6 config file for 4 days, and it still doesn't work with the v6 config.

Any clues to what could cause this?

[sweebee]

maybe similar to mine? #851

[peterbrinck]

I would suspect that they are the same, yeah

Since I only have two backups every morning, they're fresh form today and the day before
And then next day, there's still only two with the same pattern. One from the current and one from the day before

[freekmurze]

We'll try to address this soon. I'd welcome a PR that fixes this.

[peterbrinck]

Good to hear!

I'll dig into the code as well and see if I can find something relevant

[GregorVoelkl]

I ran into the same issue. For me, it turned out, that I did not clear the config cache after updating the backup.php config file.

The backup process is running fine for me with new backup.php changes.

[rubenvanassche]

@peterbrinck I can't reproduce this issue. When using an S3 disk for backups everything seems to work fine(listing/monitoring of backups). Can you please try this in a vanilla Laravel install so we can test this without side effects?

At the moment I'm using following config file for Laravel-backup:
```

return [

'backup' => [

    /*
     * The name of this application. You can use this name to monitor
     * the backups.
     */
    'name' => env('APP_NAME', 'laravel-backup'),

    'source' => [

        'files' => [

            /*
             * The list of directories and files that will be included in the backup.
             */
            'include' => [
                base_path(),
            ],

            /*
             * These directories and files will be excluded from the backup.
             *
             * Directories used by the backup process will automatically be excluded.
             */
            'exclude' => [
                base_path('vendor'),
                base_path('node_modules'),
            ],

            /*
             * Determines if symlinks should be followed.
             */
            'follow_links' => false,
        ],

        /*
         * The names of the connections to the databases that should be backed up
         * MySQL, PostgreSQL, SQLite and Mongo databases are supported.
         *
         * The content of the database dump may be customized for each connection
         * by adding a 'dump' key to the connection settings in config/database.php.
         * E.g.
         * 'mysql' => [
         *       ...
         *      'dump' => [
         *           'excludeTables' => [
         *                'table_to_exclude_from_backup',
         *                'another_table_to_exclude'
         *            ]
         *       ]
         * ],
         *
         * For a complete list of available customization options, see https://github.com/spatie/db-dumper
         */
        'databases' => [
            'mysql',
        ],
    ],

    /*
     * The database dump can be compressed to decrease diskspace usage.
     *
     * Out of the box Laravel-backup supplies
     * Spatie\DbDumper\Compressors\GzipCompressor::class.
     *
     * You can also create custom compressor. More info on that here:
     * https://github.com/spatie/db-dumper#using-compression
     *
     * If you do not want any compressor at all, set it to null.
     */
    'database_dump_compressor' => Spatie\DbDumper\Compressors\GzipCompressor::class,

    'destination' => [

        /*
         * The filename prefix used for the backup zip file.
         */
        'filename_prefix' => 'test-backup',

        /*
         * The disk names on which the backups will be stored.
         */
        'disks' => [
            's3'
        ],
    ],

    /*
     * The directory where the temporary files will be stored.
     */
    'temporary_directory' => storage_path('app/backup-temp'),
],

/*
 * You can get notified when specific events occur. Out of the box you can use 'mail' and 'slack'.
 * For Slack you need to install guzzlehttp/guzzle.
 *
 * You can also use your own notification classes, just make sure the class is named after one of
 * the `Spatie\Backup\Events` classes.
 */
'notifications' => [

    'notifications' => [
        \Spatie\Backup\Notifications\Notifications\BackupHasFailed::class => ['mail'],
        \Spatie\Backup\Notifications\Notifications\UnhealthyBackupWasFound::class => ['mail'],
        \Spatie\Backup\Notifications\Notifications\CleanupHasFailed::class => ['mail'],
        \Spatie\Backup\Notifications\Notifications\BackupWasSuccessful::class => ['mail'],
        \Spatie\Backup\Notifications\Notifications\HealthyBackupWasFound::class => ['mail'],
        \Spatie\Backup\Notifications\Notifications\CleanupWasSuccessful::class => ['mail'],
    ],

    /*
     * Here you can specify the notifiable to which the notifications should be sent. The default
     * notifiable will use the variables specified in this config file.
     */
    'notifiable' => \Spatie\Backup\Notifications\Notifiable::class,

    'mail' => [
        'to' => '[email protected]',
    ],

    'slack' => [
        'webhook_url' => '',

        /*
         * If this is set to null the default channel of the webhook will be used.
         */
        'channel' => null,

        'username' => null,

        'icon' => null,

    ],
],

/*
 * Here you can specify which backups should be monitored.
 * If a backup does not meet the specified requirements the
 * UnHealthyBackupWasFound event will be fired.
 */
'monitor_backups' => [
    [
        'name' => env('APP_NAME', 'laravel-backup'),
        'disks' => ['s3'],
        'health_checks' => [
            \Spatie\Backup\Tasks\Monitor\HealthChecks\MaximumAgeInDays::class => 1,
            \Spatie\Backup\Tasks\Monitor\HealthChecks\MaximumStorageInMegabytes::class => 5000,
        ],
    ],

    /*
    [
        'name' => 'name of the second app',
        'disks' => ['local', 's3'],
        'health_checks' => [
            \Spatie\Backup\Tasks\Monitor\HealthChecks\MaximumAgeInDays::class => 1,
            \Spatie\Backup\Tasks\Monitor\HealthChecks\MaximumStorageInMegabytes::class => 5000,
        ],
    ],
    */
],

'cleanup' => [
    /*
     * The strategy that will be used to cleanup old backups. The default strategy
     * will keep all backups for a certain amount of days. After that period only
     * a daily backup will be kept. After that period only weekly backups will
     * be kept and so on.
     *
     * No matter how you configure it the default strategy will never
     * delete the newest backup.
     */
    'strategy' => \Spatie\Backup\Tasks\Cleanup\Strategies\DefaultStrategy::class,

    'default_strategy' => [

        /*
         * The number of days for which backups must be kept.
         */
        'keep_all_backups_for_days' => 7,

        /*
         * The number of days for which daily backups must be kept.
         */
        'keep_daily_backups_for_days' => 16,

        /*
         * The number of weeks for which one weekly backup must be kept.
         */
        'keep_weekly_backups_for_weeks' => 8,

        /*
         * The number of months for which one monthly backup must be kept.
         */
        'keep_monthly_backups_for_months' => 4,

        /*
         * The number of years for which one yearly backup must be kept.
         */
        'keep_yearly_backups_for_years' => 2,

        /*
         * After cleaning up the backups remove the oldest backup until
         * this amount of megabytes has been reached.
         */
        'delete_oldest_backups_when_using_more_megabytes_than' => 5000,
    ],
],

];

``

[sweebee]

I had the same issue, make sure you copy the new config file. There are some different variables in the new config file.

[rubenvanassche]

I will close this issue for now since there is no response from the poster. Feel free to open it again when the problem occurs.

[ryancwalsh]

@rubenvanassche Maybe this issue needs to be reopened. I upgraded to "^6.1" today, and I was careful to update my config file, and even though it says the following, it deleted all my backups from S3!

'strategy' => \Spatie\Backup\Tasks\Cleanup\Strategies\DefaultStrategy::class,

        'default_strategy' => [

            /*
             * The number of days for which backups must be kept.
             */
            'keep_all_backups_for_days' => 7,

            /*
             * The number of days for which daily backups must be kept.
             */
            'keep_daily_backups_for_days' => 16,

            /*
             * The number of weeks for which one weekly backup must be kept.
             */
            'keep_weekly_backups_for_weeks' => 8,

            /*
             * The number of months for which one monthly backup must be kept.
             */
            'keep_monthly_backups_for_months' => 4,

            /*
             * The number of years for which one yearly backup must be kept.
             */
            'keep_yearly_backups_for_years' => 2,

            /*
             * After cleaning up the backups remove the oldest backup until
             * this amount of megabytes has been reached.
             */
            'delete_oldest_backups_when_using_more_megabytes_than' => 99000,

I'm still trying to figure out how this could have happened. I don't know where I could have made a mistake.

I thought it was a basic upgrade and didn't think I was doing anything risky.

[ryancwalsh]

@rubenvanassche Can you please reopen this issue?

I've made a little bit of progress looking into this. It seems related to https://github.com/spatie/laravel-backup/issues/505

One behavior I noticed is that running php artisan cache:clear before php artisan backup:clean changes what happens within php artisan backup:clean, and I would never have guessed that php artisan backup:clean would be affected by cache.

I don't remember reading any docs saying to always clear cache before running php artisan backup:clean, so there could be a bug here.

For temporary debugging purposes, I've added console logging to this function in DefaultStrategy.php:

    protected function removeBackupsForAllPeriodsExceptOne(Collection $backupsPerPeriod)
    {
        consoleOutput()->info("removeBackupsForAllPeriodsExceptOne");
        $backupsPerPeriod->each(function (Collection $groupedBackupsByDateProperty, string $periodName) {
            consoleOutput()->info("periodName $periodName");
            $groupedBackupsByDateProperty->each(function (Collection $group) {
                $group->shift();

                $group->each(function (Backup $backup) {
                    consoleOutput()->info("delete ". $backup->date()->toDateTimeString());
                    $backup->delete();
                });
            });
        });
    }

Let's say I start with these on S3:

• 2019-03-07-22-30-06.zip
• 2019-03-07-22-35-55.zip
  Then I run php artisan backup:run --only-db, and I now see a third zip in S3:
• 2019-03-07-22-37-17.zip

Now I run php artisan backup:clean.

It says:

periodName daily
periodName weekly
periodName monthly
periodName yearly
Deleted backup `https---m.example.com/2019-03-07-22-35-55.zip`.
Cleanup failed because: File not found at path: https---m.example.com/2019-03-07-22-35-55.zip.

And when I look in S3, all 3 zips are still there. So the "clean" did nothing. (And, why did it have the "File not found" error?)

But then I run php artisan cache:clear and then php artisan backup:clean.

Output:

Cleaning backups of https---m.example.com on disk s3...
removeBackupsForAllPeriodsExceptOne
periodName daily
periodName weekly
periodName monthly
periodName yearly
Deleted backup `https---m.example.com/2019-03-07-22-35-55.zip`.
Deleted backup `https---m.example.com/2019-03-07-22-30-06.zip`.
Cleanup failed because: File not found at path: https---m.example.com/2019-03-07-22-35-55.zip.

Now when I check S3, I see that the first 2 zips are gone, and the only remaining one is 2019-03-07-22-37-17.zip.

So here are the remaining mysteries:

1. Why does removeBackupsForAllPeriodsExceptOne aggressively delete backups that it shouldn't within the "yearly" part of the loop?
2. Why would I get error messages like "Cleanup failed because: File not found at path: https---m.example.com/2019-03-07-22-35-55.zip." from php artisan backup:clean when within that command is what just deleted the file at that path?
3. Why does clearing cache matter?

[rubenvanassche]

@ryancwalsh thank you for the info, I will take a look into this today.

[rubenvanassche]

Hi @ryancwalsh

I've tried your workflow in a vanilla laravel-backup installation(6.1). But can't reproduce the error you are having. Can you please test this workflow in a clean vanilla Laravel application locally? This is the output I get when running your workflow:

 ~/Spatie/laravel-playbox ⮀ ⭠ laravel-backup ● ⮀ a backup:run
Starting backup...
Dumping database playbox...
Determining files to backup...
Zipping 271 files...
Created zip containing 271 files. Size is 509.33 KB
Copying zip to disk named local...
Successfully copied zip to disk named local.
Copying zip to disk named s3...
Successfully copied zip to disk named s3.
Backup completed!
 ~/Spatie/laravel-playbox ⮀ ⭠ laravel-backup ● ⮀ a backup:run
Starting backup...
Dumping database playbox...
Determining files to backup...
Zipping 271 files...
Created zip containing 271 files. Size is 509.72 KB
Copying zip to disk named local...
Successfully copied zip to disk named local.
Copying zip to disk named s3...
Successfully copied zip to disk named s3.
Backup completed!
 ~/Spatie/laravel-playbox ⮀ ⭠ laravel-backup ● ⮀ php artisan backup:run --only-db
Starting backup...
Dumping database playbox...
Determining files to backup...
Zipping 1 files...
Created zip containing 1 files. Size is 1.11 KB
Copying zip to disk named local...
Successfully copied zip to disk named local.
Copying zip to disk named s3...
Successfully copied zip to disk named s3.
Backup completed!
 ~/Spatie/laravel-playbox ⮀ ⭠ laravel-backup ● ⮀ php artisan backup:clean
Starting cleanup...
Cleaning backups of Laravel on disk local...
"removeBackupsForAllPeriodsExceptOne"
"periodName daily"
"periodName weekly"
"periodName monthly"
"periodName yearly"
Used storage after cleanup: 3.95 MB.
Cleaning backups of Laravel on disk s3...
"removeBackupsForAllPeriodsExceptOne"
"periodName daily"
"periodName weekly"
"periodName monthly"
"periodName yearly"
Used storage after cleanup: 1020.16 KB.
Cleanup completed!
 ~/Spatie/laravel-playbox ⮀ ⭠ laravel-backup ● ⮀ php artisan cache:clear
Application cache cleared!
 ~/Spatie/laravel-playbox ⮀ ⭠ laravel-backup ● ⮀ php artisan backup:clean
Starting cleanup...
Cleaning backups of Laravel on disk local...
"removeBackupsForAllPeriodsExceptOne"
"periodName daily"
"periodName weekly"
"periodName monthly"
"periodName yearly"
Used storage after cleanup: 3.95 MB.
Cleaning backups of Laravel on disk s3...
"removeBackupsForAllPeriodsExceptOne"
"periodName daily"
"periodName weekly"
"periodName monthly"
"periodName yearly"
Used storage after cleanup: 1020.16 KB.
Cleanup completed!

[ryancwalsh]

@rubenvanassche I very much appreciate that you looked into this for me. Here is my update (and I expect I'll be able to add more info later).

1. Before writing this earlier comment of mine, I made the mistake of NOT adding enough console logging commands and then accidentally assuming that deletions had been inappropriately happening within the periodName yearly part, when in reality, they could have happened either there or later (such as in removeBackupsOlderThan or removeOldBackupsUntilUsingLessThanMaximumStorage).
2. So then, thinking "With much more logging, I'll be able to narrow down the problem further", I decided to add a whole bunch more console logging to DefaultStrategy.php.
3. Since I prefer editing in Netbeans rather than in Vim, I added more console logging in my local DefaultStrategy.php and then copied and pasted the whole contents of the file into Vim on my production machine (instead of adding console logging manually line by line in Vim on production).
4. After doing that and running my experiments again (expecting to see more specifically where the deletions happened when they shouldn't because of the more thorough logging), I've noticed no problems! (Huh?! 🤷)

This is blowing my mind because I haven't changed the config at all, I'm running the same commands that I always have been, and the only difference between what DefaultStrategy.php used to be and what it is now is that it has lines like this sprinkled throughout it: consoleOutput()->info("removeBackupsForAllPeriodsExceptOne");

These "facts" don't add up, so I'm trying to figure out where I'm wrong.

So far the only hypothesis I have is an extraordinarily unlikely one and unfortunately isn't testable: maybe my deployment process hadn't installed the appropriate new DefaultStrategy.php file during my laravel-backup upgrade, and so when I pasted over it from my local copy, more changed than just console logging. I can't imagine how this would be the case though since all the other files seem appropriately upgraded. I really wish I'd done a "diff" of the production DefaultStrategy.php to my local copy with the extra console logging to first ensure that those were the only edited lines before I pasted over the production copy.

The good news is that I don't seem to have a laravel-backup problem right now. The bad news is that I still don't understand why all my backups got deleted soon after my upgrade to "^6.1", so now I'm super nervous, especially because if I hadn't checked S3, I never would have known they were deleted, because I don't think there is a way for laravel-backup to notify me of what it deletes (maybe I'll extend it to add that feature).

I'll work on a separate project and see if I can reproduce the issue and report back what I've learned.

Thanks for the awesome library and for your help here today.

[rubenvanassche]

Thank you @ryancwalsh for helping to look into this problem. I will close this, for now, let me know if something goes wrong.

On the point of something going wrong during deployment: what usually happens to me is a cache that isn't cleared which causes all sorts of strange problems.

[ryancwalsh]

@rubenvanassche Apparently this is still a problem.

Once again, I upgraded Laravel (and "spatie/laravel-backup"), and once again all my backups from S3 got deleted.

(I feel stupid that I didn't copy the S3 backups somewhere _else_ before my Laravel upgrade since this happened before.)

My diff shows that in recent weeks my composer.json changed from:

"laravel/framework": "5.7.*",
"laravelcollective/html": "^5.4",
"spatie/db-dumper": "2.11.1",
"spatie/laravel-backup": "5.12.0",
},
    "require-dev": {
        "backpack/generators": "^1.2",
        "laravel/dusk": "~4.0",

to

"laravel/framework": "^6.0",
        "laravel/helpers": "^1.1",
        "laravel/slack-notification-channel": "^2.0",
        "laravelcollective/html": "^6.0",
        "spatie/db-dumper": "^2.14",
        "spatie/laravel-backup": "^6.7",
},
    "require-dev": {
        "backpack/generators": "^2.0",
        "laravel/dusk": "^5.5",

(showing only the changes that I think are relevant).

spatie/laravel-backup seems to be working since then.

But it seems super dangerous that all backups get wiped following an upgrade.

And I never saw any errors or received any alerts and only realized the zip files were missing from S3 because I periodically check.

[ryancwalsh]

@rubenvanassche And apparently it wasn't a one-time problem with the upgrade. Since the upgrade, each run (every single day) of php artisan backup:clean has been deleting _all_ of my backups from S3 except the most recent one. It's supposed to _leave_ a lot of backups as defined in defaultStrategy in the config (and as had been working for a long time).

I will post more once I figure out more, and if you have any insight in the meantime, I'm all ears. Thanks.

[rubenvanassche]

@ryancwalsh do you still have the problem? I sadly enough can't reproduce it over here 😞

[ryancwalsh]

@rubenvanassche Thanks for looking into it. Yes, my backups continually get deleted from S3 such that only one remains.

I have "spatie/laravel-backup": "^6.7" installed.

I just did a diff from https://github.com/spatie/laravel-backup/blob/6.7.0/config/backup.php

I'm noticing some differences:

• followLinks follow_links
• gzip_database_dump database_dump_compressor
• temporary_directory
• monitorBackups monitor_backups (and the values inside the array)
• defaultStrategy default_strategy (and the values inside the array)

I now see the note about snake_casing here: https://github.com/spatie/laravel-backup/blob/e3ea9bc9994be5cf8d9e10b202ed838380e0b4e4/CHANGELOG.md#600---2018-12-29

And here https://github.com/spatie/laravel-backup/blob/a9ddd3d79b4e853c79040468403364bd61ba9eb4/UPGRADING.md

I guess I missed those warnings at some point. I wouldn't be surprised if these mistakes are the root cause. I'll post again here once I've tried fixing it all up. Thanks.

[ryancwalsh]

@rubenvanassche I think that was the problem. I see multiple zip files in S3 now, so they haven't been getting deleted every night.

I'd definitely recommend putting a loud, clear, prominent warning at https://github.com/spatie/laravel-backup#changelog saying that changes to the source code will sometimes cause all backups to be deleted if you don't properly update your configs during any upgrades.

Thanks again for this repo and for all the other repos. You all are amazing. :-)

[pinguinoholdings]

Just wanted to leave this here for the next guy to come along. I'm upgrading an old project from 5.12 to 6.11 and had the same issue with the old backups all being deleted.

This isn't a bug, but rather I missed the changes to the config.backup.php file as well, which seems to be the cause.

My first clue something was wrong was running backup:list and getting back "Local" as the disk (actually storing on S3) as well as "No backups present" displayed. When I actually logged into AWS they were all there.

I then ran backup:run, and the manually triggered backup went off smoothly. On the next scheduled (using Forge) update, all backups but the 2 most recent were wiped out. Fortunately this isn't a problem on this project, but I agree the a BIG warning might be useful (I did read through the Changelog prior to deploying and still missed btw.)

To fix, I manually deleted backup.php from the config folder and republished the assets using:

php artisan vendor:publish --provider="Spatie\Backup\BackupServiceProvider"

per the docs.

Using the correct config file has fixed the issue.

Thank you to @ryancwalsh and @rubenvanassche for the back and forth above. Not sure I would have figured this out so quickly. I'll second the kudos for all the Spatie projects. You guys do great work!