Keystone-classic: Sub-collections

[azproduction]

Hello!

I have complex model with nested collections. I'm trying to define such model

{
    title: { type: String, required: true },
    images: {
        type: [{
            name: String,
            image: Types.Url,
            thumb: Types.Url,
            description: String
        }]
    }
}

And I get an error Error: Fields must be specified with a type function @ keystone/lib/list.js:298:9.

Is it possible to define that collection in keystone? (it is possible in Mongoose)

In admin UI this field should look like list of items with Add, Edit and "Trash" buttons. Add and Edit buttons should open modal dialog.

I am new to keystone, but I want to contribute to this project. And it will be great if you guide me a bit.

[JedWatson]

Hello!

Complex models with nested collections (or, for that matter, using mongoose's Mixed Schema Type) are currently (a) possible, and (b) not supported in the Admin UI yet.

The mongoose schema is available for you to modify to your heart's content, so you could still achieve the schema you want like this (assuming your schema snippet is for a gallery):

var Gallery = new keystone.List();
Gallery.add({
    title: { type: String, required: true }
});
Gallery.schema.add({
    images: {
        type: [{
            name: String,
            image: String,
            thumb: String,
            description: String
        }]
    }
});
Gallery.register();

The downside is, obviously, the images field will be completely ignored by the Admin UI, and you won't get those Add, Edit and Trash buttons :)

I'd really like to add this capability in the future (or perhaps support proper nested Lists, in the same way mongoose supports nested Schemas), but haven't had a chance to actually look into doing it yet.

In theory, it actually shouldn't be too hard to do, although when implementing it I'd want to put some work into the UI and make it _really good_. I'm not sure about using modal dialogs (it might be better UX to just render the forms inline) and there are some cases where it would need to be more intelligent (e.g. how should the UI adapt when you have 100+ items in the array? etc.)

Have mentioned this a few times now in different issues, but moving to a client-side rendering system for the Admin UI would make things like this easier to implement. At the moment the client-side of the Admin UI is very basic, so that might be something to do before adding this functionality.

That said if you're up for contributing support for this as a feature, I'd be happy to work with you on it!

[JedWatson]

@floatdrop that's not what's causing the error in this case, Keystone maps the basic Javascript data types to field types.

For reference, the defaults are:

• String = Types.Text
• Number = Types.Number
• Boolean = Types.Boolean
• Date = Types.Datetime

This was done in part to make schema transition from Mongoose to Keystone easier, and is throughout the demo / sample code. Might be a good idea to encourage people to be explicit instead...

[jergason]

Any pointers on getting started in creating an admin UI for nested objects? Where should I begin my explorations in the code?

[JedWatson]

Good question, @jergason. You got me thinking.

I suspect it'll be a new Field Type, that creates its own Schema. Probably best we work out the API on how to define nested fields first, then it might be clearer.

As a start, it could go like this (to use the Gallery example above):

Gallery.add({
    title: { type: Types.Text, required: true },
    images: { type: Types.Nested, fields: {
        name: String,
        image: Types.CloudinaryImage,
        description: Types.Markdown
    }}
});

The Nested field type would create and manage its own Mongoose Schema object, and its addToSchema method would look something like this:

Nested.prototype.addToSchema = function() {
    this.list.schema.path(this.path, [this.schema]);
}

It would also have its own fields array (like the List class does).

You could extend the nested Schema with virtuals / methods / etc like this (before calling List.register())

Gallery.fields.images.schema.methods.something = function() {}

You could also add more fields to its schema after it was added, like this:

Gallery.fields.images.add({
    copyright: String
});

The next step would be to work out how we would work with nested schemas in the UpdateHandler, and finally a somewhat fancy form field template for it.

Let me know what you think. If you like, I'll create a branch where we can work on it together.

[jergason]

That sounds excellent. Next week I might have some time to work on this full-time, since we are looking at using Keystone as an admin interface for some of our mongo data.—
Sent from Mailbox for iPhone

On Wed, Feb 12, 2014 at 5:00 AM, Jed Watson [email protected]
wrote:

Good question, @jergason. You got me thinking.
I suspect it'll be a new Field Type, that creates its own Schema. Probably best we work out the API on how to define nested fields first, then it might be clearer.
As a start, it could go like this (to use the Gallery example above):

Gallery.add({
    title: { type: Types.Text, required: true },
    images: { type: Types.Nested, fields: {
        name: String,
        image: Types.CloudinaryImage,
        description: Types.Markdown
    }}
});

The Nested field type would create and manage its own Mongoose Schema object, and its addToSchema method would look something like this:

Nested.prototype.addToSchema = function() {
  this.list.schema.path(this.path, [this.schema]);
}

It would also have its own fields array (like the List class does).
You could extend the nested Schema with virtuals / methods / etc like this (before calling List.register())

Gallery.fields.images.schema.methods.something = function() {}

You could also add more fields to its schema after it was added, like this:

Gallery.fields.images.add({
    copyright: String
});

The next step would be to work out how we would work with nested schemas in the UpdateHandler, and finally a somewhat fancy form field template for it.

Let me know what you think. If you like, I'll create a branch where we can work on it together.

Reply to this email directly or view it on GitHub:
https://github.com/JedWatson/keystone/issues/153#issuecomment-34862240

[JedWatson]

@jergason how's it looking for working on this this week? if it's looking good, I'll do some prep work to get it ready :)

[jergason]

I can work on this from tomorrow through the end of the week. I'm in the
US, so that is about 18 hours from now.

I'll try to read through the code a bit to prep for it.

On Sun, Feb 16, 2014 at 11:35 PM, Jed Watson [email protected]:

@jergason https://github.com/jergason how's it looking for working on
this this week? if it's looking good, I'll do some prep work to get it
ready :)

Reply to this email directly or view it on GitHubhttps://github.com/JedWatson/keystone/issues/153#issuecomment-35231919
.

[mdciotti]

@jergason any updates on this feature?

[subhog]

Question: I want to have the following schema:

Blog.add({
  admins: [{
    user: {type: Types.Relationship, ref: 'User', index: true},
    role: {type: Types.Select, options: 'editor, moderator},
  }],
});

When I try this directly, I got the same error: Error: Fields must be specified with a type function. However, when I do this via Mongoose schema with Blog.schema.add, I got:

TypeError: Undefined type at `user`
Did you try nesting Schemas? You can only nest using refs or arrays.

How can I achieve the schema I need?

[JedWatson]

@subhog You can't use Keystone field types in nested schemas until this feature is ready. At the moment, the best way to achieve what you're after (if you want the admins for each blog manageable in Keystone) is to create a new list called BlogAdmins and link it to both Blog and User models.

The management workflow isn't going to be ideal though, we should really get around to getting this feature done.

@jergason are you still interested in working on this?

[subhog]

I've started the implementation.

[subhog]

For anyone interested.

I've made a version that accepts the subcolleciton schema and seem to work fine for basic needs. Which is actually surprising, as I needed to pull one dirty trick that doesn't seem like a proper way to solve this problem.

That fork is available here and you can use it in apps by changing dependency path in package.json to "keystone": "subhog/keystone#array-field". Collaboration is welcome.

Api

field: [{
  subfield: {...},
  ...
}],

Or:

field: { type: Array, schema: {
  subfield: {...},
  ...
}},

Or:

field: { type: Types.Array, schema: {
  subfield: {...},
  ...
}},

Notes

• I need to verify whether saving actually works.
• To loop through the data you must use model.field.toObject() instead of just model.field.
• UI is very basic and read-only, just to display something and not crash the app.
• There might be problems if array schema has a nested object.
• I didn't even touch the problem of integration with app updates.

[salzhrani]

Maybe submit a PR and @JedWatson could critique it and and possibly merge it

[subhog]

PR sent. @JedWatson, suggestions?

[JedWatson]

@subhog sorry for the delay, it's a pretty big PR. I'm reviewing it and will get back to you with feedback soon.

[jefersonsv]

Hello,

I'm using admin ui of keystone in commercial application. therefore, this PR is very important, I'm watching to know when this will be resolved.! Thanks for all.

[mavropalias]

I've stumbled upon the same issue, as I need nested collections in my project. I'll keep hitting F5 on this page :)

[mattdesl]

:+1: this would be great!

[zordon]

Hey, I think I'm gonna help, I just made a small dive in the field libs. I have one question, to get this done, basically you need to make a lib and template for that type of field?

[dkhurshudian]

I'm working on this feature, and have done something...

[joelbowen]

Any updates? Or is there any common case solution for something like creating menus as a list?

I'm trying to create a keystone list of menus I can then output for things like primary navigation and footer navigation. My Schema would look something like:

title: { type: String, required: true },
links: {
    type: [{
        label: String,
        href: Types.Url,
        key: String
    }]
}

Just a thought, in general this functionality would have huge benefits (if still editable in the admin UI).

[kapandaw]

@JedWatson Do you have any progress in 'sub-collections' question?
Thanks.

[BerkeleyTrue]

I don't think there has been an progress on this as the admin is being completely rewritten using React and Keystone itself using Express 4. This is slotted to finish early January. Don't think anyone is going to be able to get around to it till then. If someone would like to try to tackle this and submit a PR using the new admin, that would be great!

[kapandaw]

Yes:) it would be great!

[metasansana]

I created a Table field type. Currently it just shows a table in the admin ui. The plan is to add editing
and eventually support for other fields that map to javascript primitives.

You can try it out by cloning my fork of keystone or merge #821 in a topic branch.

Feedback welcome! :)

[wangpingsx]

Really expect to have this feature!!!!!

And what i want is a Dynamic array field, which like tags for a blog post.

I have a temporary solution, I would like to share:

Design.add( {
    design: { type: String , initial: true, required: true, index: true },
    priceGroup: { type: String },
    price: { type: String},
    image: { type: Types.CloudinaryImage },
    },
    "Colours", { 
        colours:{
            1: { 
                type: String,collapse:true
            },
            2: { 
                type: String,collapse:true
            },
            3: { 
                type: String,collapse:true
            },
            4: { 
                type: String,collapse:true
            },
            5: { 
                type: String,collapse:true
            },
            6: { 
                type: String,collapse:true
            },
            7: { 
                type: String,collapse:true
            },
            8: { 
                type: String,collapse:true
            },
            9: { 
                type: String,collapse:true
            },
            10: { 
                type: String,collapse:true
            },
            11: { 
                type: String,collapse:true
            },
            12: { 
                type: String,collapse:true
            },
            13: { 
                type: String,collapse:true
            },
            14: { 
                type: String,collapse:true
            },
            15: { 
                type: String,collapse:true
            },
            16: { 
                type: String,collapse:true
            }
        }
    }   
);

I know I am terrible smart!!!!! :-)

All colours fields are collapsed when you open the create view of admin UI.

if you need only one colour open the first colour field.

All looks great!!, But it will always have 16 sub items in mongodb like this:

{
    "_id" : ObjectId("54ad54c2825783d46abe749b"),
    "design" : "aaa",
    "__v" : 0,
    "colours" : {
        "1" : "red",
        "10" : "",
        "11" : "",
        "12" : "",
        "13" : "",
        "14" : "",
        "15" : "",
        "16" : "",
        "2" : "blue",
        "3" : "",
        "4" : "",
        "5" : "",
        "6" : "",
        "7" : "",
        "8" : "",
        "9" : ""
    },
    "price" : "xxxxxx",
    "priceGroup" : "ddd"
}

[wangpingsx]

Type.TextArray could be a solution for string list only.

but if there is a Type.AnyObjectArray will be great!

[isjackwild]

@JedWatson Is there any ETA on when this feature may be implemented? I'm exploring Keystone for a new project I'm about to start on (due to launch in around 4 months), and I would LOVE to use Keystone for this, I think it seems really promising. More so than other solutions I've explored.

However, the ability to have nested arrays is a pretty make-or-break feature in this case, as we need it to be nice and easy to add new repeat sections and media in a post. At least in time for when we launch.

[mandb]

This should be required Type.nested looks like the right way to go about it. @JedWatson any way we can contribute to getting this added into main branch other than with PR? Thanks.

[morenoh149]

@mandb what's wrong with a PR?

[mandb]

I'm asking if there is a way to contribute to a feature other than by providing code.

[morenoh149]

@mandb in-depth discussion and coordination can be done on the slack. I'll send you an invite. pm or email me your email.

[mandb]

I would be interested thanks!
On Jan 16, 2015 6:06 PM, "Harry Moreno" [email protected] wrote:

@mandb https://github.com/mandb in-depth discussion and coordination
can be done on the slack. If you're interested in an invite let me know

—
Reply to this email directly or view it on GitHub
https://github.com/keystonejs/keystone/issues/153#issuecomment-70337282.

[metasansana]

@mandb feedback on #821 will also be useful. If all the other fields ui's are embeddable it should be possible to manage them in table form.

[wangpingsx]

Hi guys,

I have solved this issue

This blog explains how to use it:
http://baiduhix.blogspot.co.uk/2015/01/a-new-type-of-keystonejs-objectarray.html

This blog explains the source code of my solution:
http://baiduhix.blogspot.co.uk/2015/01/a-new-type-of-keystonejs-objectarray_26.html

@JedWatson let me know if you have any issue with that.

Thanks,
Peter

[mandb]

@wangpingsx - very nice plugin, will look into it but this seems to be the best approach so far.

[mandb]

@wangpingsx, does it only support string-objects or can you use more complex object types in the sub object definitions?

[wangpingsx]

@mandb only string for sub-object' fields at that moment. you can make it to support number by this:

E.g. i can make the "ext" field as number by this:

xxxxx.add(
{

    //***** Note, this is nessary !!!  Start *****//
    'contacts': { type: ObjectArray, initial: true,
        subObjectDefine: { 
            role: { type: String}, //only string now
            ext:{ type: String}
        }
    }
    //***** Note, this is nessary !!!  End *****//
}

);

//** Note, this is also nessary !!! Start **//
var mongoose = keystone.mongoose;
var schemaPaths = new mongoose.Schema({
role: String,
ext:Number
});

// schema.add(this._path.addTo({}, [schemaPaths]));

storeObject.schema.add({
contacts: { type: [schemaPaths] }
});

But it still display as a string input on admin ui, which means you only lose your number format ability.

If the user type "abc" in this field, he/she will get error for that.

[medv]

Just sitting here hoping this gets some progress, so I can advise my client for a rough ETA (so far I have said that Keystone gets a lot of love, but it could still be another year). It's a key feature for this project and most others that I can imagine. Looking forward to using Keystone for more contract work. Thank you : )

[jkumara]

Really anticipating this feature too! So far I've been able to work around this, but this would definitely be a huge improvement in admin functionality. Thanks for all your great work guys!

[svenrobin]

same here, this is essential for me to even be able to use keystone

[mandb]

Do we want to sponsor a feature then? How can we go about that.

[medv]

I think we might have to wait since there is the whole refactor to react thing going on - plus talk of making the admin panel a single-page-app (also required). This is what has been stopping me from getting in there myself, as any code that isn't in collaboration with owner would quickly be outdated. I have pushed on with using Keystone for a 2nd contract, and working around this issue. It's not the greatest for some pages unfortunately, but fantastic otherwise.

[mandb]

I believe it was on hold until the reactjs update, since a kit will be
changing and any pr's before it will likely conflict. Is this still the
case?
On Apr 25, 2015 6:20 PM, "Harry Moreno" [email protected] wrote:

@svenrobin https://github.com/svenrobin wanna join the slack and help
build it?

—
Reply to this email directly or view it on GitHub
https://github.com/keystonejs/keystone/issues/153#issuecomment-96288236.

[Bockit]

I'm trying to use the mongoose mixed schema type and I'm running into some trouble.

I tried to follow the steps above with the mixed schema type but my changes aren't saving (I am using markModified())

User.schema.add({
    testResults: {
       type: {}
    }
})

Then in a schema method:

User.schema.methods.updateTestProgress = (moduleId, recordId, passed) ->
    @testResults = {foo: 'bar'}
    @markModified('testResults')

Calling user.save() after user.updateTestProgress(...) doesn't save the testResults data to the document.

Does anyone know what I'm doing incorrectly? Thanks for your time.

[jack-guy]

I also could really use this feature!

[JedWatson]

Yes - we will get to it. There are still bigger concerns we're working through that will make this an easier thing to add in the future.

[micahbule]

Any updates on this feature? I'm willing to help out in any way I can.

[mandb]

Mongo 3.2 has the new $lookup aggregate which allows you to effectively do
a left outer join - this would be a good candidate.
On Nov 5, 2015 9:38 PM, "Micah Andrew F. Bule" [email protected]
wrote:

Any updates on this feature? I'm willing to help out in any way I can.

—
Reply to this email directly or view it on GitHub
https://github.com/keystonejs/keystone/issues/153#issuecomment-154266031
.

[morenoh149]

@mandb :+1: I saw that as well. Haven't used it yet though.

[mandb]

Doesn't work on sharded collections and can only work with exact matching
fields, but it's a good step and can give exactly this functionality.
On Nov 8, 2015 7:59 PM, "Harry Moreno" [email protected] wrote:

@mandb https://github.com/mandb [image: :+1:] I saw that as well.
Haven't used it yet though.

—
Reply to this email directly or view it on GitHub
https://github.com/keystonejs/keystone/issues/153#issuecomment-154894754
.

[CubicflowDan]

+10 This is the only thing preventing me from using keystone in a number of production websites/apps.

[simonwidjaja]

Any news on this?

[jogonru]

+1

[wmertens]

@Bockit Did you get it to save?
@JedWatson without saving there is no workaround for this :cry:.

If this gets implemented, does that mean that the current conversion of nested objects into flat fields will become objects again? If so, that's a breaking change of course, if not, how to do it?

Note that with React, making nested editors for nested objects is easy, simply recurse editors in the rendering function.

[sachss]

I'd suggest anyone interested in this see and support #2265.

[Bockit]

@wmertens I did, the code is at home, I'll try and dig it up. IIRC I was misunderstanding something about how the feature worked.

[VinayaSathyanarayana]

Any updates on this? Will this be part of V0.4?

[jjhesk]

if i need to use cloudinary in the api just to insert the info as the object. How can i use the cloudinary schema object as the new insert object? any sample code?

[jbpin]

I admit it will be really pleasant to have an ObjectArray type instead of using relations. +1

[wmertens]

Arrays of Objects are being worked on, they are already working in fact :)

See https://github.com/keystonejs/keystone/pull/3300 and https://github.com/keystonejs/keystone/pull/3282 for progress.

I hope that we can replace all the Array types with them (in a backwards compatible way). Then we can implement reordering too.

[simonwidjaja]

+1 I would be interested in an update as well...

[ghuroo]

Any chance of seeing this in a 4.0.6 beta?
Or can we implement sub-collections trusting that this feature will be out someday soon?

[Adrian-Kardasz]

Yeah this is such a critical thing for me so that the web app admins can edit data easily (they're not to savy)

[mrth2]

yah, this should be done for the extendable admin 👍
any update @JedWatson ?

[semeano]

This would be so awesome!

[vlatri]

Hey, @JedWatson, are there any updates on this one?

[semeano]

I've ended up creating a new type called List that allows that. You can check my fork of keystone.

images: { type: Types.List, required: true, initial: true, default: [],
    fields: {
        title: { type: Types.Text, required: false, initial: true },
        alt: { type: Types.Text, required: false, initial: true },
        url: { type: Types.Text, required: true, initial: true }
    }
}

[dejavu1987]

+1

[ohadperry]

+10

[leouz]

+1

[gautamsi]

unfortunately nested lists are dropped from keystone 4, see #3282, Keystone 5 promises to pick it up. see https://github.com/keystonejs/keystone-5/issues/195

closing this for tidiness.