Google-cloud-ruby: ActiveRecord adapter for Datastore?

[sam0x17]

Has anyone hacked together an ActiveRecord adapter for Datastore? If not, is there an ORM designed to work with Datastore? I'm pretty sure I could hack something together myself by overriding some active record methods, but I was wondering if anyone else has done the same or there is a standard way of working with Datastore using rails-esque models?

[jgeewax]

That's a great thought -- thanks for bringing it up. It'd be a bit tricky, but not impossible. I suspect the primary challenges will pop up due to many Datastore queries requiring indexes to work and the consistency model depending on how you structure your data (strongly consistent within a single entity group).

To be more specific:

Simple-ish queries should work just fine:

SELECT * FROM Employees WHERE nickname = 'Jimmy'`

Fancier stuff requires that you add indexes could be frustrating:

SELECT * FROM Employees WHERE salary > 50000 ORDER BY nickname ASC, salary DESC

Hierarchical keys might be weird to both create entities and query over your data:

SELECT * FROM Employees WHERE __key__ HAS ANCESTOR Key('Company', 1234) AND
  nickname = 'Jimmy'

I'm not an expert on ActiveRecord, but I think the attempts at doing this in Django world technically worked, but were a bit frustrating. Unfortunately the way you manage data in something like Datastore (or Mongo, etc) isn't quite the same as a SQL database, but most ORMs are designed with SQL in mind.

[sam0x17]

Oh, I'm prepared for the limitations. Actually, I recall seeing an issue on one of the Mongo repositories where they were talking about adapting the mongo gem to connect to Datastore, maybe this would be the easiest route? Otherwise, what I was picturing was just adding support for getting and setting attributes, .find, .update, .update_all, .save, .create and then only allow passing through Datastore SQL for .where. For me the main advantages of using an ORM are from attribute handling. Also, having models for datastore entities would make it much easier to automatically bubble up attributes to "parent" records using active record hooks.

[quartzmo]

@samkelly I asked @blowmage this question once offline (I should have opened an issue here; I'm glad you did), and I believe his answer was that this is a great idea, but that it should be done in a separate project, both because of the considerable scope of the work, and because the level of abstraction is different. Do I remember this correctly @blowmage?

[blowmage]

I worked on an ActiveModel layer that would bridge the gap between gcloud's Datastore implementation and what most ORMs provide. I'll dig it out and see what I can publish soon.

[bmclean]

I agree with @jgeewax, our initial approach was to try and adapt ActiveRecord but we use mostly ancestor queries and it didn't work very well. However, what is working very well for us is using a layer over ActiveModel. We created a lib similar to the work Google did here (doc, repo) but added support for ancestor queries. We created instance methods such as save, update, destroy and class methods such as all, find_in_batches and find.

[sam0x17]

Yeah, I'd probably take the route of writing an abstraction layer _over_ ActiveRecord rather than merely writing an adapter.

[bmclean]

@blowmage we can provide what we have written so far as well. Let me know if interested.

[blowmage]

@bmclean that would be great! Send to [email protected] if you don't want to make something public.

[bmclean]

@blowmage we can make it public. I will clean it up this morning and publish it.

[sam0x17]

@bmclean that would be great would love to look at that code

[quartzmo]

@bmclean That is so awesome!!

[bmclean]

We created a simple Rails app showing how we are using Gcloud::Datastore with ActiveModel.
The Gcloud::Datastore::Dataset is implemented in config/initializers/cloud_datastore.rb.
The local environment variables are implemented in config/initializers/development.rb.
The ActiveModel wrapper is implemented as a model concern, located in app/models/concerns.
The database.yml contains the dataset_ids.
The local datastore credentials monkey patch is in lib/local_datastore_no_auth.rb.

Let me know if you have any questions!

[quartzmo]

@bmclean That's really cool. I'll try it out soon.

[sam0x17]

@bmclean would you be OK with me using this as the basis for a gem that does this + with some additional functionality? I think I can get it to do some more magic involving indexes and a few other things.

[bmclean]

Sure, feel free to fork it. We released it with the MIT license so just follow that.

[sam0x17]

will do -- thanks!

[quartzmo]

It seems everyone agrees that this won't happen in gcloud-ruby (at least not for now), so closing. But please continue to post progress and links on this issue!

Again:

Agrimatics/active-model-cloud-datastore - ActiveModel Google Cloud Datastore example Rails app

[bmclean]

We have released Agrimatics/activemodel-datastore as a gem.

[quartzmo]

Thanks for the update @bmclean!