Influxdb: [feature request] influx -import options to specify destination db/rp

[kostasb]

Add destination database and retention policy arguments to influx -import to override the DML options.

Use case: A large database is exported from TSM storage in compressed format. The resulting compressed file includes DDL and DML statements to create and set the target database and retention policy for imported points.

influx_inspect export -database=mydb -datadir=/var/lib/influxdb/data -waldir=/var/lib/influxdb/wal -compress -out=here.gz

# DML
# CONTEXT-DATABASE:mydb
# CONTEXT-RETENTION-POLICY:myrp

If users wants to import the data in a different database or retention policy, they need to unzip and modify the file - which can be infeasible due to the size of the resulting uncompressed file (e.g. 100GB can result in more than 1TB uncompressed).

Proposed feature request:

Include command line arguments like destdb,destrp to override the DML defintions specified in the export file.

influx -import -compressed -path=dumpfile -pps=1000 -destdb=newdb -destrp=newrp

[sachinrase]

+1

also -rentention_factor=n

[kostasb]

@sachinrase in this case the destination database and retention policy can be manually created by the user before the import is started - just so we can keep this request simple. Otherwise a number of parameters would be needed like retention policy duration, shard duration, replication factor.

[pmyjavec]

I just had a talk with @kostasb about this and have plenty of time to get this sorted on Monday, should have a PR sometime sometime in the afternoon, fingers crossed ;)

[sachinrase]

is there any candidate release rpm i can try to test ?

[danielkucera]

I was dealing with the same issue and I cannot afford to upgrade version even if this was implemented. So I figured out a workaround:

influx -import -path <(zcat influxdata-old.gz | sed 's/CONTEXT-RETENTION-POLICY:.*/CONTEXT-RETENTION-POLICY:new-retention-policy/')

Assuming you exported with compress (into influxdata-old.gz), this will decompress the export file, replace retention policy on-the-fly and import the file.

You can also change database name similarly.

[dirkdevriendt]

This would indeed be a useful enhancement.
It took me by surprise actually since the influx -import statement allows for a -database option but it's just not taken into account.

Especially with the crippled backup/restore support in InfluxCloud I'd expect better import/export options to play around with the data...

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed because it has not had recent activity. Please reopen if this issue is still important to you. Thank you for your contributions.

[JogiInnov]

+1
I was exporting a huge database and importing to another influx-server with different database name.
One day timeseries export was ~700GB, then i had to modify the file to update DDL command(i.e CREATE DATABASE XYZ). which take ~3hours+ to modify.