Lizardfs: Request: in-place conversion of local storage to a chunk disk

[biocyberman]

Suppose I have this:

A large data directory of dozens of TB, say 30 TB.
/data
Used space is 27 TB, so I have a bout 3 TB free space left.

This is the current rather safe but slow process:

1. I have to backup /data to another storage, which can well be on another machine over the network.
2. Empty /data and convert it into a chunk disk.
3. Copy backup of /data back to the now lizardfs-backed /data.

Step 1. and 3. are both time consuming, and step 3 is even more time consuming because write speed on /data now suffers the overhead of LizardFS operations.

## The request
It would be faster and more convenient with this process:

1. I make a directory under /data and use it for lizard chunk server without deleting the existing data on /data /data/lizardfs. This can be done automatically.
2. Start chunk server on localhost.
3. Run this pseudocommand line:

migratetochunk --chunk-dir /data/lizardfs \
--chunkserver=localhost \ # 3.1 select the localhost as target to migrate data to.
--tmp-dir=/tmp/lizard # 3.2 this is to deal with extreme cases during migration of files  bigger than current free space on /data. 
--delete-file-after-its-migration \ # 3.3 IMPORTANT, to free up space, for new data chunk to be created
--master=mfsmaster # does this really require for metadata update?

migratetochunk will do some checks of free space, send data to /data/lizardfs and then delete each file after transferring and checking of file integrity by MD5SUM or the like. This program will likely fail if it is migrating a huge file of 4TB (greater than 3TB of my free space), for example. In that case the tmp-dir can be used to store the data chunks of the files, then migratetochunk will delete the huge file on /data and move the data chunks from the tmp-dir into place. Other than that it should work fine without following the current slow process.

If the is currently no similar way, and I want to contribute, how do I get point 3.1 done? (see the pseudo command line).
One can easily see that this feature becomes more and more useful in the migration process of bigger storage, of hundreds terabytes, petabytes, etc.

[onlyjob]

There is no problem to do what you want and no support from LizardFS is needed. Just add data directory in chunkserver's mfshdd.cfg and move data to mount point. If you want data to stay on the same server then you might need to adjust goals (and chunkserver label) to pin data to specific chunkserver. After than normal move operation will do what you want.

[psarna]

I think the request was about migrating without a need to transfer all data over network. It would be theoretically possible to convert files to chunks and then just inform master that they belong to certain files. It is not directly supported and there are pitfalls here (all your data is on precisely one chunkserver for a long while, checksum mismatches etc.). Converting files to chunks is tricky, but not very hard when it comes to regular goals, xors/eresure codes are more complicated.

If you would like to implement it yourself, then one scenario could be to:

1. Create a stub file with given number of chunks.
2. Create chunks of that file manually (slice it to 64MB parts, put checksums here and there, rename properly). There is a lot of details here and it's easy to create invalid chunks, though.
3. Stop a local chunkserver.
4. Inject chunks into a chunkserver by putting created files in its data directory.
5. Restart chunkserver, which should start serving newly created chunks as if they were always there.

This process is pretty advanced and, for now, it is not on a priority list of the development team. Alternatively to doing it yourself, you could try your luck with [email protected].

[4Dolio]

I use rsync all the time for this sort of thing but it would require more free overhead space, which you should have anyway so you can have a minimum goal of 2 after rsync gets the initial transfer done. Depending on the type of data using zfs+compression for the chunk server might gain some space until LFS can do native compression. Anyway, this is the sort of thing I would tend to solve with some crazy shell script.

[biocyberman]

For the time being, a carefully rsync command does the job. However a LizardFS-native program will be better still.
@psarna The step 2 sounds complicated.
And how will the master get updated with metadata?