Eleventy: Can't iterate over a global data subfolder

[octoxalis]

My global data are contained in subfolders of the main data folder.
For instance: data/subdata.
My Nunjucks for loop is:

{% for sub in subdata %}
.....
{% endfor %}

However I never get anything: no looping!
But if I put:
subdata.entry.property or subdata['entry'].property
I get that property. I mean, if I know the name of the data subfolder, I can retrieve anything declared in that data file.
Does it mean that I can't iterate over the data subfolders?

[jevets]

You should be able to iterate over them, but it looks like you're trying to access them in your template as if the files lived at _data/subdata.json.

A file at _data/subdata/items.json would be accessible as subdata.items.

Is the data iterable as arrays? Not sure which template language you're using, but you may be trying to iterate over objects when the language expects an array.

[octoxalis]

Thanks a lot for answering.
But I've not been cute enough to explain what I'm trying. (BTW, I'm using Nunjucks templates and Javascript files for my data).

For some sections (collection pages) of my site, I use JS files (that could be generated by a back-end DB if necessary). Each section has its JS files and I know how to access each one when I know the section item (JS file) requested.

But if I want to iterate over all files (i.e. exported Object/Array modules) within a section, the Nunjucks for loop gives nothing.
I've carefully read the docs and tried every possible way, but still no success. According to the Javascript data files page:

_data/section/sub_1.js
_data/section/sub_2.js

are accessed as

section.sub_1.property...
section.sub_2.property...

and I can iterate on section.sub_1 or section.sub_2 to access every property.

But I cant iterate on section to get an iterable sub_1, sub_2... Object/Array:

{% for subsection in section %}
   subsection.property...  // i.e. section.sub_1.property
{% endfor %}

However, subsection yields nothing!

[kleinfreund]

Right. section is not a data file; hence, it can not be used as data in Eleventy. You could create a _data/section.js file which in turn returns the data structure you want to iterate over.

[octoxalis]

@kleinfreund: Your answer confirms what I found.

My workaround is a small JS module I've written to iterate over the files in the data folders I want to use: a bit more complicated but it works.
That's the magic of JS data files: really a great enhancement compared to JSON data files, because I can leverage my data files with a bunch of logic if I need. Can't do that with front matter.

[jevets]

Yup, section is nothing but a key.

Your workaround is the right choice; build up the structure yourself, then iterate over your own structure.

In cases like these, I've found myself storing the data outside of _data, then just using a file like _data/sections.js to build up the data structure that I need in my templates. This approach also provides a spot to do any necessary transformations to the data, since even though they may be static files now, I may get the data from an API in the future that returns a different structure. Great place to handle those transformations.

[andreapernici]

I think this is maybe related to overrhiding something in an md.

If I use data: [{ etc etc }] the data is populated if I do data: subfolder.data with the same content of the inline array of objects I don't get any data.

How the js should be created in order to make the sufolder object accessible?

[jevets]

You'd just need to create a _data/subfolder.js file. In this file, you'd bring in whatever data you want, then export it as an array (assuming you want an array).

// _data/users.js

const user1 = { name: 'Jane', id: 1 }
const user2 = { name: 'John', id: 2 }

module.exports = () => {
  return [
    user1,
    user2
  ]
}

{% for user in users %}
  <p>{{ user.name }} has an ID of {{ user.id }}</p>
{% endfor %}

If you really need each user as its own data file, too:

// _data/users/user1.json
{
  "id": 1,
  "name": "Jane"
}

// _data/users/user2.json
{
  "id": 2,
  "name": "John"
}

// _data/users.js
const user1 = require('./users/user1.json')
const user2 = require('./users/user2.json')

module.exports = () => {
  return [
    user1,
    user2 
  ]
}

Bottom line is: if you need _data.subdata to be iterable, it has to be an actual file that provides an array. 11ty does not automatically build up users as an array just because you've got them in a sub folder. 11ty simply allows you group certain data files under subfolders for your own organization purposes.

[andreapernici]

I think I'm missing something here but for example considering the json file to be already an array of object.

_data/2013_feedbacks.js

const fb = require('./2013/feedbacks.json');
module.exports = function(fb) {
  return fb;
};

then applying the following in an .md file

layout: layouts/base.njk
enableFasciaFeedback: true
feedbacks: {{ 2013_feedbacks }}
---

I expect feedback to be populated with the content of _data/2013/feedbacks.json but I don't get any value.

[octoxalis]

@jevets: Woo! The Array solution is pretty expansive because I can have many data files to declare and the Array has to be updated each time I add a new file to be included in the Array.
My solution, which leverage the Node file system functions is:

• make a JS module exporting a function to walk thru the subfolder I want to look at;
• require the module and invoque the function in the data files where I need it.

That module can reside anywhere in the source folder.
For instance, I put it (and any other modules I want to create to operate on data files in a tools folder at the root of my source folder.

Using JS data files shines, compared to using JSON files or Yaml/Toml etc.. Not only your code is lighter (every JS programmer hates the JSON property quotes, without optional comments!) but you can introduce a bunch of JS processing in your data files, using the full power of a programming language. JSON data files are pure declaration files, JS data files are pure programming files.
If you need a dynamic building of your site content, JS data files change the deal.

[octoxalis]

@andreapernici: AFAIK you can't use JS data inside your front matter (even if you use JS front matter). You can just invoque JS library functions I think.
Use a JS data file to do what you want and access that data file in your templates (well, you can in Nunjucks templates, not sure for other template engines).

IMO, it's a better practice to use front-matter only to declare page constants and use JS data files to process the data as you see fit.

[andreapernici]

Yes I noitced that putting those variables in the NJK works, but in my use case it could be useful to overrhide those value based on a JSON file downloaded from external APIs.

Obviously creating the js object file is not as straight forward as curling the API response.

I'll end up probably create a sort of dynamic templates that uses a var as the folder selector id is not possible to dynamically overrhide values.

[zachleat]

Whoa, this thread kinda went off the rails and there is a lot of misinformation in here—sorry, everyone. Wish I would’ve stepped in sooner.

The original problem @octoxalis had was a simple misunderstanding of how Nunjucks loops work.

Consider the following project structure:

Here’s what {{ subdata | dump }} looks like:

It’s true that this outputs nothing:

{% for key in subdata %}
{{ key }}
{% endfor %}

However that’s only because you’re looping over an object literal, not an array. Absolutely you can iterate over subdata!

This is what you want:

{% for key, val in subdata %}
{{ key }}
{% endfor %}

Output:

Does that help everyone?

[zachleat]

(More info at https://mozilla.github.io/nunjucks/templating.html#for)

[jevets]

@zachleat

Could've sworn I tested that folder isn't accessible automatically given a data file like _data/folder/items.json

Was actually thinking about suggesting this as a feature, but it's already available! Thanks for the clarification.

The key, value tripped me up at first but it does indeed work.

@octoxalis @kleinfreund my statement above that section is just a key isn't quite accurate... FYI

[zachleat]

No worries @jevets—just a simple mistake 👍 I appreciate the help you’ve been providing on the tracker!!

[octoxalis]

It's perfect. Sometimes the most evident things are out of scope!

[zachleat]

Thanks all! Closing

[plainspace]

Whoa, this thread kinda went off the rails and there is a lot of misinformation in here—sorry, everyone. Wish I would’ve stepped in sooner.

The original problem @octoxalis had was a simple misunderstanding of how Nunjucks loops work.

Consider the following project structure:

Here’s what {{ subdata | dump }} looks like:

It’s true that this outputs nothing:

{% for key in subdata %}
{{ key }}
{% endfor %}

However that’s only because you’re looping over an object literal, not an array. Absolutely you can iterate over subdata!

This is what you want:

{% for key, val in subdata %}
{{ key }}
{% endfor %}

Output:

Does that help everyone?

The key, value piece is tripping _me_ up. I have multiple files in _data/team with a name like bob.json and structured like so:

{
  "name": "Bob",
  "github": "bobsgithub",
  "twitter": "bobstwitter",
  "linkedin": "https://www.linkedin.com/in/bob/",
  "blog": "https://medium.com/@bob"
}

I can {{ team | dump }} and see all of the json output.

But I can't figure out how to use the data in a template like so: {{ name }}, {{ blog }}, etc.

Any pointers?

[pdehaan]

@plainspace Assuming a data file of ./_data/team/bob.json, I believe you'd use something like the following:

<p>Hello, my name is {{ team.bob.name }} and you can tweet me at https://twitter.com/{{ team.bob.twitter }}.</p>

---

<p>Hello, my name is {{ team.bob.name }} and you can tweet me at https://twitter.com/{{ team.bob.twitter }}.</p>
{# Output:
  <p>Hello, my name is Bob and you can tweet me at https://twitter.com/bobstwitter.</p>
#}


{# loop over the _data/team/bob.json object... #}
{%- for key, value in team.bob %}
KEY={{ key }}, VALUE={{ value }}<br/>
{%- endfor %}
{# Output:
  KEY=name, VALUE=Bob<br/>
  KEY=github, VALUE=bobsgithub<br/>
  KEY=twitter, VALUE=bobstwitter<br/>
  KEY=linkedin, VALUE=https://www.linkedin.com/in/bob/<br/>
  KEY=blog, VALUE=https://medium.com/@bob<br/>
#}

{# debug the _data/team/members.json array... #}
<p>Members: {{ team.members | dump(2) | safe }}</p>
{# Output:
  <p>Members: [
    "bob",
    "david"
  ]</p>
#}

{# loop over the array of names in _data/team/members.json and
   fetch their respective _data/team/{{ name }}.json file... #}
<ul>
{%- for member_name in team.members %}
  {%- set member = team[member_name] %}
  <li>{{ member.name }} &mdash; @{{ member.twitter }}</li>
{%- endfor %}
</ul>
{# Output:
  <ul>
    <li>Bob &mdash; @bobstwitter</li>
    <li>David &mdash; @davidstwitter</li>
  </ul>
#}

[plainspace]

Thanks! In the last example, what does the members.json file need to look like? Is there a way to do that without the members.json file and instead just pull date from the respective _data/team/{{ name }}.json files?

[plainspace]

Also, in the first example the output is:

KEY=bob, VALUE=[object Object]

[plainspace]

🐥I figured it out. Thanks!

[pdehaan]

In the last example, what does the members.json file need to look like?

[
    "bob",
    "david"
  ]

[plainspace]

Thanks! Not sure if this is the right place to ask but let's see:

I'd like to sort the data now. Some of the _data/team/{{ name }}.json files have a weight attribute in them.

I'm expecting that the following would move the members with a weight to the top of the list and order them ascending and then order those without a weight alphabetically. I'm assuming that isn't working because the order in members.json is defining the order.

{%- for member_name in team.members|sort(attribute='weight') %}

[pdehaan]

@plainspace You could move the weights into the members.json file. I can't see how you could create a custom collection from files in the _data/ directory.

[
  {"name": "bob", "weight": 20},
  {"name": "david", "weight": 10}
]

Then our nunjucks template can sort by weight:

<ul>
{%- for member in team.members | sort(false, false, "weight") %}
  <!-- {{ member | dump | safe }} -->
  {%- set weight = member.weight %}
  {%- set member = team[member.name] %}
  <li>{{ member.name }} &mdash; @{{ member.twitter }} &mdash; {{ weight }}</li>
{%- endfor %}
</ul>

OUTPUT:

<ul>
  <!-- {"name":"david","weight":10} -->
  <li>David &mdash; @davidstwitter &mdash; 10</li>
  <!-- {"name":"bob","weight":20} -->
  <li>Bob &mdash; @bobstwitter &mdash; 20</li>
</ul>

You could also move all the members into a single data file to make it easier to sort. But it really depends on how you're using the data.