Spoke: RFC: port Spoke to Google Cloud Platform ( "GCP" )

[joemcl]

Proposal:

Port Spoke to run on Google Cloud Platform ( GCP )

Motivation:

• Add GCP as an additional option to AWS and Heroku
• Many orgs use G Suite - Gmail, Google Docs etc - and pay for cloud storage for those; potential to save money by running Spoke on GCP and storing data cheaper on that than Heroku or AWS

Detailed Design:

TBD :)

Additional Context:

[joemcl]

@deasterdaywfp @lperson @jbradach ^^

[jbradach]

Nice! I'll try to contribute to this over the over the holidays. I've been doing some of this on my own but wasn't having a lot of luck.

[ibrand]

@jbradach did you end up getting anywhere with this work? Do you have any updates? Thanks for trying this out!

[bchrobot]

We have had success running on GCP's Kubernetes Engine offering. We are using our own database, but I imagine that using GCP's Cloud SQL offerings would be straightforward.

Support for uploading exports to GCS was added in politics-rewired/Spoke#251

[ibrand]

Marked this with help wanted ! First step here would be to document on the issue what changes would need to be made to allow a smooth GCP deploy (if any), if we need code changes creating issues for those and linking here would be awesome.

Then adding a doc to the docs/ folder explaining the deploy process would be awesome!

[navinsivakumar]

I've been playing around with deploying to Cloud Run and documenting as I go. Right now I'm still in the early stages, although I do have a basic deployment up and running. I can report back with more details once I have a better sense of the overall work involved. It does look like a few code changes will be necessary, at least if you want to connect to a Cloud SQL instance.

[deasterdaywfp]

Following this and very interested. As we are currently using Heroku and
seeing its limitations, and also not super excited about supporting AWS
necessarily.

*Due to the covid-19 pandemic and related school closures, my working hours
are limited by childcare needs. I am available daily between the hours 1PM

• 5:30PM ET & 9PM - 11PM Monday - Sunday. Please allow extra time for
  replies to your email. If you are requesting support with peer-to-peer
  texting, please fill out the form here http://bit.ly/T4Ractivate and a
  team of highly-skilled volunteers will also be able to assist you. If you
  have a query about an already running campaign, please consider asking for
  support in the WFP4theMany community Slack http://wfp4themany.slack.com.*

Dylan Easterday (he/him)

Volunteer and Distributed Organizing Manager

[email protected] | [phone]

Join the Party! Text WFP to 30403

Volunteer with WFP! http://www.mobilize.us/workingfamilies

On Tue, Dec 15, 2020 at 9:54 AM navinsivakumar [e-mail]
wrote:

I've been playing around with deploying to Cloud Run and documenting as I
go
https://github.com/navinsivakumar/Spoke/blob/gcp/docs/HOWTO_DEPLOYING_GOOGLE_CLOUD_RUN.md.
Right now I'm still in the early stages, although I do have a basic
deployment up and running. I can report back with more details once I have
a better sense of the overall work involved. It does look like a few code
changes will be necessary, at least if you want to connect to a Cloud SQL
instance.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/MoveOnOrg/Spoke/issues/1274#issuecomment-745344244,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AIQWPSSNXZC3WRX2BZ2ECGTSU52BDANCNFSM4JLJDDWQ
.

[navinsivakumar]

So I played around and got what I would consider a basic, functional Spoke environment running on GCP (Cloud Run + Cloud SQL Postgres + Memorystore for Redis, running on a custom domain, with working Auth0 and Twilio integrations). There's very little code that needs to change for this -- it's mostly documentation about how to configure things. The only code changes I needed were:

• add support for connecting to Cloud SQL via a Unix socket in knex-connect.js
• set the npm option script-prepend-node-path to ensure we always use the version of node that comes with npm

I haven't yet looked into GCS export as an alternative to S3 (nor the implementation @bchrobot mentioned earlier). I imagine S3 export will still work as currently documented but haven't tried it yet.

If that sounds like an adequate basic deployment, I can go ahead and work on the changes. Even if there are other rough edges to come, my feeling is that it might help to get the basic setup and documentation out to encourage other people to try it out and see how their configurations work.

[joemcl]

@navinsivakumar cool! I'll try setting this up tomorrow to test it out.

[deasterdaywfp]

This sounds really great and I'll see if we can test it out at WFP likely
in the new year

On Fri, Dec 18, 2020, 1:11 AM Joe McLaughlin [e-mail]
wrote:

@navinsivakumar https://github.com/navinsivakumar cool! I'll try
setting this up tomorrow to test it out.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/MoveOnOrg/Spoke/issues/1274#issuecomment-747890783,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AIQWPSWZDBUVVB53Y2X43QDSVLW7XANCNFSM4JLJDDWQ
.

[navinsivakumar]

FYI I also started writing a Terraform configuration for my deployment at navinsivakumar/spoke-on-gcp.

[navinsivakumar]

I added a quick but working implementation of (and documentation for) GCS export. It's all available in the gcp branch of navinsivakumar/Spoke.

I think the configuration aspect of this will need a bit of care so that it can extend reasonably well if we start adding other cloud backends, each with AWS and GCP (and maybe Azure or some other platform) variants. Like I suggested in #1907 😃 . My high level proposal for the configuration strategy right now is something like:

• A set of "credentials" properties for each cloud platform:
  
  
  
  • An analog of AWS_ACCESS_AVAILABLE indicating if we are running on the platform (so have service identity available to authenticate automatically)
  
  
  • properties to pass in credentials manually if we're not running on the platform (analogs of AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY)
  
  
• And then for each resource we need from the platform, we'd have:
  
  
  
  • properties identifying the resource we need (e.g. AWS_S3_BUCKET_NAME GCP_STORAGE_BUCKET_NAME, etc.)
  
  
  • a selector property to indicate which backend we want to use (analogous to EXPORT_DRIVER in politics-rewired/Spoke#251)
  
  

[navinsivakumar]

I'm also curious if it might be possible to configure Google Docs import using service identity when running on GCP (rather than passing in a manual credential for a separate service account).

[joemcl]

Cool, @lperson thoughts?

[navinsivakumar]

Docs import using service identity looks pretty straightforward -- I added a quick implementation and documentation notes to my draft branch.