Druid: Deep storage migration from s3 to hdfs

[akashdw]

I'm running hadoop distcp to copy existing segments from s3 to hdfs but druid segments in s3 have : in the paths which is a bad uri for hadoop.
I see https://github.com/druid-io/druid/blob/master/api/src/main/java/io/druid/segment/loading/DataSegmentPusher.java#L96 and https://github.com/druid-io/druid/blob/master/extensions-core/hdfs-storage/src/main/java/io/druid/storage/hdfs/HdfsDataSegmentPusher.java#L221 are different as hdfs one replaces : with _.
Any suggestions how to distcp segments from s3 to hdfs?
@himanshug @gianm @b-slim what do you think of using a common safe path format for s3, hdfs and if possible for other deep storage to avoid similar problems?

[b-slim]

@akashdw I have not done this TBH. Please share with us your tricks once you accomplish this.

[himanshug]

had a quick discussion with @akashdw offline. it appears that Druid is always writing the correct URIs as expected by S3 and HDFS naming conventions.
Problem is Distcp not supporting functionality to rename URIs as it writes to HDFS which doesn't support ':' .

Its a one time activity and my suggestion is to do following.

1. copy relevant segments to another temp location on same S3 region.
2. rename segment files to replace ':' with '_' as S3 can support '_' .
3. copy renamed file from temp location on S3 to HDFS.

then use insert-segment-to-db tool to update the metadata store correctly on the target Druid cluster.

[dragonls]

I faced the same problem while migrating segments from local storage to HDFS storage.
And I did extracly as @himanshug suggested. That works fine.

[stale[bot]]

This issue has been marked as stale due to 280 days of inactivity. It will be closed in 2 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the [email protected] list. Thank you for your contributions.

[stale[bot]]

This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time.