Label-studio: Add Azure Storage to Cloud Storage connectors.

[kasuteru]

This is pretty straightforward - we have support for s3 and gcs, but Azure Storage is missing from the list.

Describe the solution you'd like
Add a connector for either Azure Blob Storage, Data Lake Storage or both.

I think I might be able to implement this myself, but if someone is already working on this, I don't want to duplicate the work. I am happy to help with testing.

[makseq]

@kasuteru it will be great if you implement it! We are not working on it now.

[kasuteru]

I will start working on this and will send you a PR once I have something.

[makseq]

Awesome! Write here if you need help!

[kasuteru]

I am currently stuck at getting the dev environment set up... I did the following steps (on a Windows machine running anaconda):

• Cloned my forked repo to a local folder
• pip installed the requirements.txt
• Tried to run the app as described in the readme: python label_studio/server.py start labeling_project --init

After some initial hiccups regarding path management in anaconda, I get the app to start correctly in the command line, but the browser simply shows

"Project loading error
The 'label-studio' distribution was not found and is required by the application".

How do I set up the development environment? Are there any additional steps I need to take?

[makseq]

@kasuteru did you install LS package?

python setup.py develop 

[kasuteru]

@kasuteru did you install LS package?

python setup.py develop 

Whoops, that was it. Haven't worked with packages in a while. Thanks!

[kasuteru]

This is more complex than anticipated. https://github.com/heartexlabs/label-studio/blob/master/label_studio/utils/uri_resolver.py also contains hard-coded functions referring to S3 and GCS that I would have to extend. Tbh I think these should first be refactored to reside in the storage section, so that adding a new storage requires only adding a new file there. (See also https://github.com/heartexlabs/label-studio/issues/450).

I will try to get deeper into the code and propose a refactor for this first.

[fromnorthpole]

Hi, @kasuteru ! Please feel free to make a PR, so we can discuss the ways how to do it better. The problem is URI resolver is a somewhat disentangled from storages (practically you don't have to connect storage for resolving URLs - it requires only credentials to be globally accessible). However we've also started to merge these two stories here. So it would be much better to take a look into what you're doing.

[kasuteru]

Hi @fromnorthpole , as mentioned here, I have pushed a first implementation of an ADLS connector here. I will update to this branch when I make progress, it is not finished.

In general, I think it is a good idea to try and disentangle the storage connectors from what is actually accessed (a completion, a json list of tasks, single-file tasks, direct access to files...). It looks like you are going into that direction with your approach. I will keep this in mind, especially when trying to refactor (see Issue #450 )

[kasuteru]

@fromnorthpole, @makseq, I am confused by the naming conventions in the base.py file - since every cloud connector has to implement its own methods, more comments would be highly appreciated. I tried to figure out the naming, here is what I got - please let me know whether this is correct. I would add comments to the file in my PR then.

What I found so far:

url_prefix = Either "" (empty string) for local file system, or the url prefix of the storage including "://", e.g. "s3://"

id = the name of the file, but _without_ any leading file-system-like folder structures.

path = Not the full path, but rather only the address of the bucket. This is confusing for me since in local storage, path is used differently? I would definitely recommend using another naming for this part...

prefix = The "subfolder" on the bucket / storage account. This is not used to define a file prefix (e.g. all files that start with "task_) but instead to point to the correct subfolder, as in aws terminology. Instead, regex() is used to filter files in that subfolder?

key is used as defined by AWS s3. According to base.py/_id_to_key, key consists of prefix+/+id, meaning key is the relative path of the object. This is in slight contradiction to AWS standards, where the prefix actually includes the trailing "/".

key_prefix : Everything but the key, including the url_prefix, the path, and a final /.

full_key : The URL of the file / object, including e.g. "s3://". I would strongly suggest renaming this to url to avoid confusion with key.

[kasuteru]

I pushed an updated version that worked on initial try, but still threw some errors regarding sync between local / cloud after a while. My hunch is that this has to do with how I use path, but not sure. I will look further into this next week.

[makseq]

@kasuteru Hi, have you any news about your PR?

[kasuteru]

Hi,

after some experimentation I noticed that using ADLS directly has too much
latency to fluently display tasks. I have instead written a Redis in-memory
db connector, and an independent backend process to sync to ADLS.

I had to reduce my hours due to Corona-based cuts and haven't managed to
upload it before the holidays. I will get back to it in January.

Thomas

Max [e-mail] schrieb am Mo., 21. Dez. 2020, 13:33:

@kasuteru https://github.com/kasuteru Hi, have you any news about your
PR?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/heartexlabs/label-studio/issues/437#issuecomment-748951102,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AD4KWBAQXY5JY2L64IGR4SDSV46ATANCNFSM4RXKWYBQ
.

[kasuteru]

I opened PR #529 to integrate Redis functionality. In my testing, this was highly performant, even more than the current default, file-based method (since everything is handled in-memory). However, obviously, it requires hosting a redis server in parallel, so it does not intend to replace the default.

I have abandoned my quest to directly write to ADLS storage, since this will in my opinion never be performant enough, so I will close this issue.

In case confusion arises: What I wrote is actually not a cloud connector in any way, it does not use any Azure or other services directly. It simply connects to a Redis database that _may_ be hosted in the cloud.

For anyone stumbling upon this in the future: The way I am doing it is by hosting a Redis server locally on my WebApp instance and syncing the data to ADLS with a custom script. The better way to do it is probably to use Azure's own Redis service, or using the file-based method on the WebApp server if this is compliant with your data protection guidelines.

[keesschollaart81]

@kasuteru You tried with ADLS, what's your view/take on an Azure BLOB storage connector? That should be fairly comparable with GCS/AWS blob storage, right?! I'm thinking about building a PR for Azure BLOB....

[kasuteru]

Blob storage might be faster since ADLS is optimized for big data
workloads, meaning overhead if working with small files. I still think
in-memory caching is the way to go to have a truly performant webapp,
especially since the memory footprint is really small.

I see from the other thread that you are working with images - these I
would store in Blob storage, but I think that is a different connector
again:
https://github.com/heartexlabs/label-studio/blob/4688a36affae6d4dea4514a293b2611e98c8d80c/label_studio/utils/uri_resolver.py

On Tue, Jan 5, 2021 at 5:25 PM Kees Schollaart [e-mail]
wrote:

@kasuteru https://github.com/kasuteru You tried with ADLS, what's your
view/take on an Azure BLOB storage connector? That should be fairly
comparable with GCS/AWS blob storage, right?! I'm thinking about building a
PR for Azure BLOB....

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/heartexlabs/label-studio/issues/437#issuecomment-754742934,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AD4KWBEXSRIIUUH56OW33HDSYM4QBANCNFSM4RXKWYBQ
.