Lizardfs: Wishlist 5.x - Georeplication & Edge Caching

[Zorlin]

A native georeplication system would be awesome - I'm envisioning being able to select a region for a particular folder (or client) and all data for that folder (or client) being stored in the selected region. I know we have custom goals and topology which sort of provides this, but not quite completely.

All clients would be able to see those files, and could access them as though they were in their region, albeit with slower transfer rates and higher latency.

It would be interesting to cache objects in regions where they are being accessed, too - a small "edge" cache that provides the most frequently requested items.

[mdimura]

This would be THE killer feature for me. Hovewer it seems like there is no need in separate dedicated implementation of "georeplication" or "Edge Caching". Once (minimum/acknowledgment/ACK/GLOBAL_MINIMAL)_goal and "different allocation strategies (performance_prioritised)" functionality is implemented - georeplication and write caching will appear automatically. What will be still missing is read-caching: if the replicas are in Paris and NY, and a client reads it from Sydney (where we also have chunk-servers) - the file will be transfered to the client, but (unfortunately) not replicated in Sydney. Ideally, we would need an option to create "local" replicas on the fly. This might involve replica migration, i.e. when a replica is created (to improve performance) in Sydney, it can be removed in NY, since we have one replica to many. Additionally, one might want to add a maximum_goal, which would set, how many replicas are allowed to stay if their creation is requested upon read.
acknowledgment_goal - how many replicas must be written, that the file-system may acknowledge the file write to the OS (while creating more replicas in background).
main_goal (currently just 'goal') - how many replicas should be written generally (including the replicas written after the file write was acknowledged.
maximum_goal - how many replicas (>=main_goal) can coexists, when additional replicas are created dynamically upon reading from a distant location. If maximum_goal>replicas_present, one old replica should be removed.
This can be generalized to a many-tier storage (e.g. disk These hierarchy level names should probably be user defined as only the level number matters: 0=>disk, 1=>array, 2=>node, 3=>cluster, ...).

[4Dolio]

Recent issues have discussed REDUNDANCY_LEVEL, I'm somewhat behind on most issues... I think this "new to me" feature may be similar to the described acknowledgment_goal. I think REDUNDANCY_LEVEL says that a write fails if not successful to the value of that setting, not sure if it is global or can be a part of a goal definition? I doubt that it results in an ack for the write once reached. It's likely the remaining writes needed to reach the goal, which may be higher, continue to come from the client rather than allowing replication between the chunk servers to finish the remaining writes needed. An early ack from REDUNDANCY_LEVEL would be neat.

The additional of a REDUNDANCY_COMPLETION = { Client, Chunk } would add more tunability for network traffic control and goal fulfillment latency options.

I think the other desire to have more tier control might be extended beyond current topology, goals, and labels by:

• Allowing for multiple labels per chunkserver and perhaps some basic regular expression matching.
• Adding variable proximity to topology values rather than simple binary matches.
  Those are two mechanics are already in place and could maybe be extended without breaking backward compatibility. I am sure that I have mentioned these ideas in the past in other issues (should look those up...)