Google-cloud-python: Cloud storage - How to emulate it locally?

[david-gang]

I wanted to know what your recommended way is to emulate google cloud storage locally, if at all.
In stackoverflow there is a question regarding this: https://stackoverflow.com/questions/37542530/is-there-google-cloud-storage-emulator
There are two answers:

• Use the in memory emulator - This is just available with the java library
• Use minio - I am not sure how to specify google cloud to use minio

A third option would to mock classes from google cloud storage but which classes should be mocked?

[chemelnucfin]

Hi David, we are going to use #4840 to keep track of emulator issues as it seems like we would need a broader discussion. Please feel free to follow there.

[selfiebot]

Hi folks,

I've had a similar question from the BBC and looking at the github issue (https://github.com/GoogleCloudPlatform/google-cloud-python/issues/4840) I don't see any updates to this.

Have there been any new discussions around providing an emulator for GCS?

The BBC comment was similar to the stackoverflow one, but here's their detail:

---

We noticed there is an official GCP PubSub emulator, but we couldn’t find an official Cloud Storage emulator. We read this StackOverflow discussion about the topic which advises either the Java in-memory stategy or https://www.minio.io/. Since we’re using Python, it seems the in-memory strategy is not an option, whereas the Python client source code doesn’t seem to implement a similar feature. Since we’re not using AppEngine, it seems the local dev-server isn’t an option as well. What is the recommended way to emulate Cloud Storage locally (or in a container)?

---

Cheers,
Nabeel

On Tuesday, February 20, 2018 at 8:19:04 PM UTC, chemelnucfin wrote:

Hi David, we are going to use #4840 to keep track of emulator issues as it seems like we would need a broader discussion. Please feel free to follow there.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

[selfiebot]

Hi folks,

Just wondering if anyone had any response to this thread? Customer is still
pending an answer...

Cheers,
Nabeel

On Thu, Jul 5, 2018 at 7:17 PM Nabeel Saad [e-mail] wrote:

Hi folks,

I've had a similar question from the BBC and looking at the github issue (
https://github.com/GoogleCloudPlatform/google-cloud-python/issues/4840) I
don't see any updates to this.

Have there been any new discussions around providing an emulator for GCS?

The BBC comment was similar to the stackoverflow one, but here's their
detail:

---

We noticed there is an official GCP PubSub emulator, but we couldn’t find
an official Cloud Storage emulator. We read this StackOverflow discussion
about the topic which advises either the Java in-memory stategy or
https://www.minio.io/. Since we’re using Python, it seems the in-memory
strategy is not an option, whereas the Python client source code doesn’t
seem to implement a similar feature. Since we’re not using AppEngine, it
seems the local dev-server isn’t an option as well. What is the recommended
way to emulate Cloud Storage locally (or in a container)?

---

Cheers,
Nabeel

On Tuesday, February 20, 2018 at 8:19:04 PM UTC, chemelnucfin wrote:

Hi David, we are going to use #4840 to keep track of emulator issues as
it seems like we would need a broader discussion. Please feel free to
follow there.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

--

Nabeel Saad | Solutions Engineer | [email protected] | [phone]

This email may be confidential and privileged. If you received this
communication by mistake, please don't forward it to anyone else, please
erase all copies and attachments, and please let me know that it has gone
to the wrong person.

The above terms reflect a potential business arrangement, are provided
solely as a basis for further discussion, and are not intended to be and do
not constitute a legally binding obligation. No legally binding obligations
will be created, implied, or inferred until an agreement in final form is
executed in writing by all parties involved.

[selfiebot]

+Dave Nettleton [e-mail] is this something you can help with or
point in the right direction?

On Wed, Aug 8, 2018 at 10:29 AM Nabeel Saad [e-mail] wrote:

Hi folks,

Just wondering if anyone had any response to this thread? Customer is
still pending an answer...

Cheers,
Nabeel

On Thu, Jul 5, 2018 at 7:17 PM Nabeel Saad [e-mail] wrote:

Hi folks,

I've had a similar question from the BBC and looking at the github issue (
https://github.com/GoogleCloudPlatform/google-cloud-python/issues/4840)
I don't see any updates to this.

Have there been any new discussions around providing an emulator for GCS?

The BBC comment was similar to the stackoverflow one, but here's their
detail:

---

We noticed there is an official GCP PubSub emulator, but we couldn’t find
an official Cloud Storage emulator. We read this StackOverflow discussion
about the topic which advises either the Java in-memory stategy or
https://www.minio.io/. Since we’re using Python, it seems the in-memory
strategy is not an option, whereas the Python client source code doesn’t
seem to implement a similar feature. Since we’re not using AppEngine, it
seems the local dev-server isn’t an option as well. What is the recommended
way to emulate Cloud Storage locally (or in a container)?

---

Cheers,
Nabeel

On Tuesday, February 20, 2018 at 8:19:04 PM UTC, chemelnucfin wrote:

Hi David, we are going to use #4840 to keep track of emulator issues as
it seems like we would need a broader discussion. Please feel free to
follow there.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

--

Nabeel Saad | Solutions Engineer | [email protected] | +44 (0)7
525611473

This email may be confidential and privileged. If you received this
communication by mistake, please don't forward it to anyone else, please
erase all copies and attachments, and please let me know that it has gone
to the wrong person.

The above terms reflect a potential business arrangement, are provided
solely as a basis for further discussion, and are not intended to be and do
not constitute a legally binding obligation. No legally binding obligations
will be created, implied, or inferred until an agreement in final form is
executed in writing by all parties involved.

>

Nabeel Saad | Solutions Engineer | [email protected] | [phone]

This email may be confidential and privileged. If you received this
communication by mistake, please don't forward it to anyone else, please
erase all copies and attachments, and please let me know that it has gone
to the wrong person.

The above terms reflect a potential business arrangement, are provided
solely as a basis for further discussion, and are not intended to be and do
not constitute a legally binding obligation. No legally binding obligations
will be created, implied, or inferred until an agreement in final form is
executed in writing by all parties involved.

[selfiebot]

+Vaibhav who's owns GCS programmability
Thanks
Dave

On Wed, Aug 8, 2018 at 2:34 AM Nabeel Saad [e-mail] wrote:

+Dave Nettleton [e-mail] is this something you can help with
or point in the right direction?

On Wed, Aug 8, 2018 at 10:29 AM Nabeel Saad [e-mail] wrote:

Hi folks,

Just wondering if anyone had any response to this thread? Customer is
still pending an answer...

Cheers,
Nabeel

On Thu, Jul 5, 2018 at 7:17 PM Nabeel Saad [e-mail] wrote:

Hi folks,

I've had a similar question from the BBC and looking at the github issue
(https://github.com/GoogleCloudPlatform/google-cloud-python/issues/4840)
I don't see any updates to this.

Have there been any new discussions around providing an emulator for GCS?

The BBC comment was similar to the stackoverflow one, but here's their
detail:

---

We noticed there is an official GCP PubSub emulator, but we couldn’t
find an official Cloud Storage emulator. We read this StackOverflow
discussion about the topic which advises either the Java in-memory stategy
or https://www.minio.io/. Since we’re using Python, it seems the
in-memory strategy is not an option, whereas the Python client source code
doesn’t seem to implement a similar feature. Since we’re not using
AppEngine, it seems the local dev-server isn’t an option as well. What is
the recommended way to emulate Cloud Storage locally (or in a container)?

---

Cheers,
Nabeel

On Tuesday, February 20, 2018 at 8:19:04 PM UTC, chemelnucfin wrote:

Hi David, we are going to use #4840 to keep track of emulator issues
as it seems like we would need a broader discussion. Please feel free to
follow there.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

--

Nabeel Saad | Solutions Engineer | [email protected] | +44 (0)7
525611473

This email may be confidential and privileged. If you received this
communication by mistake, please don't forward it to anyone else, please
erase all copies and attachments, and please let me know that it has gone
to the wrong person.

The above terms reflect a potential business arrangement, are provided
solely as a basis for further discussion, and are not intended to be and do
not constitute a legally binding obligation. No legally binding obligations
will be created, implied, or inferred until an agreement in final form is
executed in writing by all parties involved.

>

Nabeel Saad | Solutions Engineer | [email protected] | +44 (0)7
525611473

This email may be confidential and privileged. If you received this
communication by mistake, please don't forward it to anyone else, please
erase all copies and attachments, and please let me know that it has gone
to the wrong person.

The above terms reflect a potential business arrangement, are provided
solely as a basis for further discussion, and are not intended to be and do
not constitute a legally binding obligation. No legally binding obligations
will be created, implied, or inferred until an agreement in final form is
executed in writing by all parties involved.

[selfiebot]

Hi Vaibhav,

Any response on status of GCS emulator solutions? More detail in the below:

I've had a similar question from the BBC and looking at the github issue (
https://github.com/GoogleCloudPlatform/google-cloud-python/issues/4840) I
don't see any updates to this.

Have there been any new discussions around providing an emulator for GCS?

The BBC comment was similar to the stackoverflow one, but here's their
detail:

---

We noticed there is an official GCP PubSub emulator, but we couldn’t find
an official Cloud Storage emulator. We read this StackOverflow discussion
about the topic which advises either the Java in-memory stategy or
https://www.minio.io/. Since we’re using Python, it seems the in-memory
strategy is not an option, whereas the Python client source code doesn’t
seem to implement a similar feature. Since we’re not using AppEngine, it
seems the local dev-server isn’t an option as well. What is the recommended
way to emulate Cloud Storage locally (or in a container)?

---

Cheers,
Nabeel

On Wed, Aug 8, 2018 at 10:41 AM Dave Nettleton [e-mail] wrote:

+Vaibhav who's owns GCS programmability
Thanks
Dave

On Wed, Aug 8, 2018 at 2:34 AM Nabeel Saad [e-mail] wrote:

+Dave Nettleton [e-mail] is this something you can help with
or point in the right direction?

On Wed, Aug 8, 2018 at 10:29 AM Nabeel Saad [e-mail] wrote:

Hi folks,

Just wondering if anyone had any response to this thread? Customer is
still pending an answer...

Cheers,
Nabeel

On Thu, Jul 5, 2018 at 7:17 PM Nabeel Saad [e-mail] wrote:

Hi folks,

I've had a similar question from the BBC and looking at the github
issue (
https://github.com/GoogleCloudPlatform/google-cloud-python/issues/4840)
I don't see any updates to this.

Have there been any new discussions around providing an emulator for
GCS?

The BBC comment was similar to the stackoverflow one, but here's their
detail:

---

We noticed there is an official GCP PubSub emulator, but we couldn’t
find an official Cloud Storage emulator. We read this StackOverflow
discussion about the topic which advises either the Java in-memory stategy
or https://www.minio.io/. Since we’re using Python, it seems the
in-memory strategy is not an option, whereas the Python client source code
doesn’t seem to implement a similar feature. Since we’re not using
AppEngine, it seems the local dev-server isn’t an option as well. What is
the recommended way to emulate Cloud Storage locally (or in a container)?

---

Cheers,
Nabeel

On Tuesday, February 20, 2018 at 8:19:04 PM UTC, chemelnucfin wrote:

Hi David, we are going to use #4840 to keep track of emulator issues
as it seems like we would need a broader discussion. Please feel free to
follow there.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

--

Nabeel Saad | Solutions Engineer | [email protected] | +44 (0)7
525611473

This email may be confidential and privileged. If you received this
communication by mistake, please don't forward it to anyone else, please
erase all copies and attachments, and please let me know that it has gone
to the wrong person.

The above terms reflect a potential business arrangement, are provided
solely as a basis for further discussion, and are not intended to be and do
not constitute a legally binding obligation. No legally binding obligations
will be created, implied, or inferred until an agreement in final form is
executed in writing by all parties involved.

>

Nabeel Saad | Solutions Engineer | [email protected] | +44 (0)7
525611473

This email may be confidential and privileged. If you received this
communication by mistake, please don't forward it to anyone else, please
erase all copies and attachments, and please let me know that it has gone
to the wrong person.

The above terms reflect a potential business arrangement, are provided
solely as a basis for further discussion, and are not intended to be and do
not constitute a legally binding obligation. No legally binding obligations
will be created, implied, or inferred until an agreement in final form is
executed in writing by all parties involved.

>

Nabeel Saad | Solutions Engineer | [email protected] | [phone]

This email may be confidential and privileged. If you received this
communication by mistake, please don't forward it to anyone else, please
erase all copies and attachments, and please let me know that it has gone
to the wrong person.

The above terms reflect a potential business arrangement, are provided
solely as a basis for further discussion, and are not intended to be and do
not constitute a legally binding obligation. No legally binding obligations
will be created, implied, or inferred until an agreement in final form is
executed in writing by all parties involved.

[borovik96]

You can try this one https://github.com/fsouza/fake-gcs-server

[eweitz]

Maintainers, how suitable is googleapis/python-storage/tests/perf/benchwrapper.py for locally emulating GCS? Its README seems to suggest that is possible at some level.

Having a relatively Python-native way to emulate Google Cloud Storage would be quite valuable.

[joejean]

For those of you still looking for a solution for this. I just tried https://gitlab.com/potato-oss/google-cloud/gcloud-storage-emulator/ and it seems to be working fine for me :).

[oittaa]

For those of you still looking for a solution for this. I just tried https://gitlab.com/potato-oss/google-cloud/gcloud-storage-emulator/ and it seems to be working fine for me :).

Thanks! Unfortunately it's a dead project or something. I tried to submit pull requests, but never got any comments or anything back. So I forked it and now the version in my repository has these additional features:

• PATCH requests
• object metadata
• checksums
• correct datetime formats

https://github.com/oittaa/gcp-storage-emulator and pre-built Docker images are located at https://hub.docker.com/r/oittaa/gcp-storage-emulator