Influxdb: Influxd restore command errors when attempting to restore legacy format in influxdb 1.6.0-1

[heckj]

__System info:__ [Include InfluxDB version, operating system name, and other relevant details]

• Ubuntu trusty
• Influxdb 1.6.0-1 package installed from https://repos.influxdata.com/

__Steps to reproduce:__

1. uncompress backup into /tmp/recent (tar xvjf /vagrant/influxdb_backup.recent.tar.bz2)
2. stop any active services (sudo service influxdb stop)
3. restore in metadata (sudo influxd restore -metadir /var/lib/influxdb/meta /tmp/recent)
4. restore the datbase (sudo influxd restore -database mydb -datadir /var/lib/influxdb/data /tmp/recent

__Expected behavior:__

At this point, I expected to see:
2018/07/12 04:18:02 Restoring offline from backup /tmp/recent/mydb.*, which is what is reported and done with InfluxDB 1.5.4.

__Actual behavior:__

However, this final command instead returns an error when attempting to invoke it:
restore: -metadir or -destinationDatabase are required to restore

__Additional info:__ [Include gist of relevant config, logs, etc.]

I through maybe -datadir was renamed and tried to identify that as -destinationDatabase, but that command also returned an error, and influxd restore --help didn't identify these options at all for doing a legacy format restore.

I've worked around this issue currently by installing InfluxDB 1.5.4 directly:

sudo apt-get purge influxdb -y
wget https://dl.influxdata.com/influxdb/releases/influxdb_1.5.4_amd64.deb
sudo dpkg -i influxdb_1.5.4_amd64.deb

which does allow the restore to operate as it did previously
Please note, the quickest way to fix a bug is to open a Pull Request.

[rda0]

I think this bug was introduced in ee90079bb9d705aacf171943bad7514e86b40c6f in PR #9713. The documentation for the legacy restore method is not in line with the documentation located at https://docs.influxdata.com/influxdb/latest/administration/backup_and_restore/:

   -newdb <newdb_name>
       Name of the InfluxDB OSS database into which the archived data will be imported on the target system. Optional. If not
       given, then the value of -db <db_name> is used. The new database name must be unique to the target system.

The statement If not given, then the value of -db <db_name> is used. is not satisfied with the new code, due to the changes made in ee90079bb9d705aacf171943bad7514e86b40c6f.

Please review if this would fix the bug:

diff --git a/cmd/influxd/restore/restore.go b/cmd/influxd/restore/restore.go
index eadd5f0b7..083d32e9f 100644
--- a/cmd/influxd/restore/restore.go
+++ b/cmd/influxd/restore/restore.go
@@ -194,9 +194,13 @@ func (cmd *Command) parseFlags(args []string) error {
                        }
                }
        } else {
+               if cmd.destinationDatabase == "" {
+                       cmd.destinationDatabase = cmd.sourceDatabase
+               }
+
                // validate the arguments
                if cmd.metadir == "" && cmd.destinationDatabase == "" {
-                       return fmt.Errorf("-metadir or -destinationDatabase are required to restore")
+                       return fmt.Errorf("-metadir or -newdb are required to restore")
                }

                if cmd.destinationDatabase != "" && cmd.datadir == "" {
@@ -205,13 +209,13 @@ func (cmd *Command) parseFlags(args []string) error {

                if cmd.shard != 0 {
                        if cmd.destinationDatabase == "" {
-                               return fmt.Errorf("-destinationDatabase is required to restore shard")
+                               return fmt.Errorf("-newdb is required to restore shard")
                        }
                        if cmd.backupRetention == "" {
                                return fmt.Errorf("-retention is required to restore shard")
                        }
                } else if cmd.backupRetention != "" && cmd.destinationDatabase == "" {
-                       return fmt.Errorf("-destinationDatabase is required to restore retention policy")
+                       return fmt.Errorf("-newdb is required to restore retention policy")
                }
        }

Until the bug is fixed, you need to explicitly specify the parameter -newdb and not -destinationDatabase as incorrectly stated by the error message.

Example (InfluxDB >= v1.6.0):

/usr/bin/influxd restore -db mydb -newdb mydb -datadir /var/lib/influxdb/data /path/to/backup

[dadrian]

I can confirm /usr/bin/influxd restore -db mydb -newdb mydb -datadir /var/lib/influxdb/data /path/to/backup is a workaround.

I just got hit by this bug attempting to restore an old version to a new version, so that I can then do an online restore from this new version, to a different new version. Yay computers.

[simplyviks]

System info: [Include InfluxDB version, operating system name, and other relevant details]

• Ubuntu trusty
• Influxdb 1.6.0-1 package installed from https://repos.influxdata.com/

Steps to reproduce:

1. uncompress backup into /tmp/recent (tar xvjf /vagrant/influxdb_backup.recent.tar.bz2)
2. stop any active services (sudo service influxdb stop)
3. restore in metadata (sudo influxd restore -metadir /var/lib/influxdb/meta /tmp/recent)
4. restore the datbase (sudo influxd restore -database mydb -datadir /var/lib/influxdb/data /tmp/recent

Expected behavior:

At this point, I expected to see:
2018/07/12 04:18:02 Restoring offline from backup /tmp/recent/mydb.*, which is what is reported and done with InfluxDB 1.5.4.

Actual behavior:

However, this final command instead returns an error when attempting to invoke it:
restore: -metadir or -destinationDatabase are required to restore

Additional info: [Include gist of relevant config, logs, etc.]

I through maybe -datadir was renamed and tried to identify that as -destinationDatabase, but that command also returned an error, and influxd restore --help didn't identify these options at all for doing a legacy format restore.

I've worked around this issue currently by installing InfluxDB 1.5.4 directly:

sudo apt-get purge influxdb -y
wget https://dl.influxdata.com/influxdb/releases/influxdb_1.5.4_amd64.deb
sudo dpkg -i influxdb_1.5.4_amd64.deb

which does allow the restore to operate as it did previously
_Please note, the quickest way to fix a bug is to open a Pull Request._

yes this worked.Many thanks!

[simplyviks]

I think this bug was introduced in ee90079 in PR #9713. The documentation for the legacy restore method is not in line with the documentation located at https://docs.influxdata.com/influxdb/latest/administration/backup_and_restore/:

   -newdb <newdb_name>
       Name of the InfluxDB OSS database into which the archived data will be imported on the target system. Optional. If not
       given, then the value of -db <db_name> is used. The new database name must be unique to the target system.

The statement If not given, then the value of -db <db_name> is used. is not satisfied with the new code, due to the changes made in ee90079.

Please review if this would fix the bug:

diff --git a/cmd/influxd/restore/restore.go b/cmd/influxd/restore/restore.go
index eadd5f0b7..083d32e9f 100644
--- a/cmd/influxd/restore/restore.go
+++ b/cmd/influxd/restore/restore.go
@@ -194,9 +194,13 @@ func (cmd *Command) parseFlags(args []string) error {
                        }
                }
        } else {
+               if cmd.destinationDatabase == "" {
+                       cmd.destinationDatabase = cmd.sourceDatabase
+               }
+
                // validate the arguments
                if cmd.metadir == "" && cmd.destinationDatabase == "" {
-                       return fmt.Errorf("-metadir or -destinationDatabase are required to restore")
+                       return fmt.Errorf("-metadir or -newdb are required to restore")
                }

                if cmd.destinationDatabase != "" && cmd.datadir == "" {
@@ -205,13 +209,13 @@ func (cmd *Command) parseFlags(args []string) error {

                if cmd.shard != 0 {
                        if cmd.destinationDatabase == "" {
-                               return fmt.Errorf("-destinationDatabase is required to restore shard")
+                               return fmt.Errorf("-newdb is required to restore shard")
                        }
                        if cmd.backupRetention == "" {
                                return fmt.Errorf("-retention is required to restore shard")
                        }
                } else if cmd.backupRetention != "" && cmd.destinationDatabase == "" {
-                       return fmt.Errorf("-destinationDatabase is required to restore retention policy")
+                       return fmt.Errorf("-newdb is required to restore retention policy")
                }
        }

Until the bug is fixed, you need to explicitly specify the parameter -newdb and not -destinationDatabase as incorrectly stated by the error message.

Example (InfluxDB >= v1.6.0):

/usr/bin/influxd restore -db mydb -newdb mydb -datadir /var/lib/influxdb/data /path/to/backup

Hi ,

i am hitting below issue while migrating th db from 0.10.0 to 1.7
please help!.

restore: open /var/lib/influxdb/data/telegraf_jenkins_test/default/2/000000001-000000001.tsm.tmp: no such file or directory
vagrant@host-192-168-0-23:/tmp$

regards,
VIkash

[heckj]

@simplyviks The workaround that I've been using for migrating older data is to install 1.6 somewhere, and do a restore to a 1.6 influx, and then turn around and immediately make a portable backup, which restores beautifully into 1.6 or later.

I found a Vagrant instance with Influx was easiest for my use (smaller amount of data, under 20GB)

With Influx 1.5 the restore commands I used were:

sudo influxd restore -metadir /var/lib/influxdb/meta /vagrant/recent
sudo influxd restore -database sunlink -datadir /var/lib/influxdb/data /vagrant/recent

And with Influx 1.6 installed (on Linux), the restore command changed to:

sudo influxd restore -metadir /var/lib/influxdb/meta -database sunlink -datadir /var/lib/influxdb/data /vagrant/recent

I also did a sudo chown -R influxdb:influxdb /var/lib/influxdb to make sure the files were owned appropriately before restarting the service (with sudo service influx start). I also did a double check of the health of all the files with:

sudo influx_inspect verify -dir /var/lib/influxdb/

Then with the system running, I could make a portable backup:

influxd backup -portable /backups/portable_backup

[simplyviks]

@simplyviks The workaround that I've been using for migrating older data is to install 1.6 somewhere, and do a restore to a 1.6 influx, and then turn around and immediately make a portable backup, which restores beautifully into 1.6 or later.

I found a Vagrant instance with Influx was easiest for my use (smaller amount of data, under 20GB)

With Influx 1.5 the restore commands I used were:

sudo influxd restore -metadir /var/lib/influxdb/meta /vagrant/recent
sudo influxd restore -database sunlink -datadir /var/lib/influxdb/data /vagrant/recent

And with Influx 1.6 installed (on Linux), the restore command changed to:

sudo influxd restore -metadir /var/lib/influxdb/meta -database sunlink -datadir /var/lib/influxdb/data /vagrant/recent

I also did a sudo chown -R influxdb:influxdb /var/lib/influxdb to make sure the files were owned appropriately before restarting the service (with sudo service influx start). I also did a double check of the health of all the files with:

sudo influx_inspect verify -dir /var/lib/influxdb/

Then with the system running, I could make a portable backup:

influxd backup -portable /backups/portable_backup

Thanks for documenting the steps , will be very useful.
i was able to migrate to 1.5.4 , however faced some issues with permissions mainly , after following the steps mentioned by you on the first thread..

[mdbartos]

I am having similar problems with legacy restores to influxdb 1.6 and 1.7. While the workarounds in this thread worked for me, the restore seems to be dropping random shards. This does not occur when restoring to older versions of influxdb (e.g. 1.3).

For instance, the restored backup is missing data from 2018-01-29 to 2018-06-03. When checking the shards using SHOW SHARDS, these shards seem to be accounted for in the metastore:

1210 ARB      autogen          1210        2018-01-29T00:00:00Z 2018-02-05T00:00:00Z 2018-02-05T00:00:00Z
...
1550 ARB      autogen          1550        2018-05-28T00:00:00Z 2018-06-04T00:00:00Z 2018-06-04T00:00:00Z

However, they do not appear in the actual data directory:

$ ls influxdb/data/ARB/autogen/ | grep 1210

These shards are present in the actual backup:

$ ls 2018-08-12T06.30.02.storm-server.ARB | grep 1210
>>> ARB.autogen.01210.00

Any idea what could be causing this problem? I can confirm that this problem does not occur on influxdb 1.3.6

Edit:
This problem also occurs on influxdb 1.5.4.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[rda0]

Fixed in #10206

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed because it has not had recent activity. Please reopen if this issue is still important to you. Thank you for your contributions.