Hugo: regression: Fail to unmarshal JSON arrays

[JoshStrobl]

As of Hugo 0.35, I am no longer able to parse a valid JSON file for use in .Site.Data.var. Running --debug reports:

WARN 2018/01/31 17:26:12 Failed to read data from mirrors.json/mirrors.json: json: cannot unmarshal array into Go value of type map[string]interface {}                                                
WARN 2018/01/31 17:26:12 Failed to read data from sws.json/sws.json: json: cannot unmarshal array into Go value of type map[string]interface {}

mirrors.json is saved in data/, with the contents of:

[
    {
        "Latitude": 52.3702,
        "Location": "Amsterdam",
        "Longitude": 4.8952,
        "Type": "official",
        "Url": "https://stroblindustries.com/isos/"
    },
    {
        "Latitude": 33.753746,
        "Location": "Atlanta",
        "Longitude": -84.386330,
        "Type": "official",
        "Url": "http://us.justin.id.au/iso/"
    },
    {
        "Latitude": 52.5200,
        "Location": "Germany",
        "Longitude": 13.4050,
        "Type": "community",
        "Url": "http://solus.veatnet.de/iso/"
    },
    {
        "Latitude": 51.507351,
        "Location": "London",
        "Longitude": -0.127758,
        "Type": "official",
        "Url": "https://soluslond1iso.stroblindustries.com/"
    },
    {
        "Latitude": 40.357298,
        "Location": "New Jersey",
        "Longitude": -74.667223,
        "Type": "community",
        "Url": "http://mirror.math.princeton.edu/pub/solus-iso/"
    },
    {
        "Latitude": 43.086002,
        "Location": "Rochester",
        "Longitude": -77.674538,
        "Type": "community",
        "Url": "https://mirrors.rit.edu/solus/images/"
    },
    {
        "Latitude": 37.7749,
        "Location": "San Francisco",
        "Longitude": -122.4194,
        "Type": "official",
        "Url": "https://solussf1iso.stroblindustries.com/"
    }
]

sws.json is saved in data/, with the contents of:

[
    {
        "episode": "10",
        "description": "The team talks about work done since the snapshot 2017.04.18.0, ferryd, cuppa, plans for the new site, and more.",
        "date": "Fri, 14 Apr 2017 01:33:12 +0300",
        "length": [
            "21474938",
            "43046701"
        ],
        "twis": "/2017/04/24/this-week-in-solus-install-43"
    }
]

The failing to parse this JSON results in errors like:

ERROR 2018/01/31 17:26:12 error processing shortcode "theme/shortcodes/sundays-with-solus.html" for page "blog/2017-04-24-this-week-in-solus-install-43.md": template: theme/shortcodes/sundays-with-so
lus.html:6:10: executing "theme/shortcodes/sundays-with-solus.html" at <where $site.Data.sws...>: error calling where: can't iterate over <nil>

With the respective line that it's erroring having: {{ range where $site.Data.sws "episode" "==" $episode }}

---

I can confirm that it works as intended with Hugo 0.34, I released a blog not just yesterday, were both the Sundays with Solus shortcode, as well as Download mirrors dropdown work as intended. See https://solus-project.com/blog/ and https://solus-project.com/download/

[JoshStrobl]

I can confirm that it's not a permissions issue for reading (no permissions were changed between blogging yesterday and building the site today). Here is a summarized output (removing irrelevant dir content) of ls -al data:

drwxrwxrwx  9 joshua joshua 4096 Dec  5 04:35 .
drwxrwxrwx 12 joshua joshua 4096 Jan 26 15:38 ..
-rw-rw-r--  1 joshua joshua 1085 Sep 21 12:09 mirrors.json
-rw-rw-r--  1 joshua joshua  308 May 14  2017 sws.json

[moorereason]

Likely related to the change from interface{} to map[string]interface{} in hugolib.Page.update() from https://github.com/gohugoio/hugo/commit/91bb774ae4e129f7ed0624754b31479c960ef774. May have to roll that back. Should have had tests for this use case.

/cc @vassudanagunta

[vassudanagunta]

on it

[JoshStrobl]

Thanks @vassudanagunta :smile:

[vassudanagunta]

OK, looking into this, while it is a regression, it worked before but partly out of luck, since parts of the code assume data files along with frontmatter are maps at the top level. Hence JSON arrays fail if you have two of them at the same level (#4366). A single JSON array file works, but not YAML (#3890, also forum discussion), which I confirmed in test. Other unmarshaling issues that may be connected/resolved together: #4138, #2441.

If I can't come up with a broader solution soon, I'll role back just enough of 91bb774 to fix your case.

[bep]

@vassudanagunta any updates on this?

[vassudanagunta]

In progress. Mostly have it. I'll PR a proposed fix within a few hours when I get some free time.

[vassudanagunta]

@bep The main difficulty is that the various parsers (YAML, JSON, TOML) all behave differently. Getting close. What's the deadline? It would be nice to fix this rather than roll back (which trades one bug for another). If I get this right, I'll have squashed a number of bugs.