Hugo: Add template function for making general HTTP requests

[atishay]

A lot of Jamstack services like Formspree require the Authorization header for authentication. Without being able to supply headers, we cannot access the API provided by these services from within Hugo. getJSON and getCSV are very limited in scope and do not support any means for authentication or passing data. We cannot handle non 200 series status codes in Hugo templates as well.

NOTE: With transform.Unmarshal hugo already supports parsing textual responses.

[moorereason]

I actually started working on such a function a few weeks ago, but it's not finished. Below is what I was wanting to implement. Comments welcome.

Function signature

getHTTP [OPTIONS] URL

Optional Parameters Dictionary

• method = string (GET, POST, etc)
• headers = dict (map[string]string)
• body = string
• no-cache = bool (bypass Hugo's caching mechanism; not sure this is useful) ❓
• retries = int
• timeout = string (parsed by time.Duration)
• tls-insecure-skip-verify = bool

Returns

A HTTP Response object (a simplified view of http.Response) containing:

• Ok (bool; if StatusCode >= 200 && StatusCode <= 299)
• StatusCode (int)
• Headers (map[string]string)
• Body (string or keep it as []byte?) ❓

Example Usage

{{ $opts := dict "method" "POST" "body" $body "headers" (dict "Authorization" "blah") }}
{{ $resp := $url | getHTTP $opts }}
{{ if eq resp.StatusCode 200 }}
  {{ $json := transform.Unmarshal resp.Body }}
{{ end }}

Cc: @bep

Edits ℹ️

• 2020-09-16: Updated to put URL as the last parameter as suggested by Bjørn Erik
• 2020-09-17: Updated Parameters to include retries, timeout, and tls-insecure-skip-verify. Updated Response object to include a Ok property.

[bep]

We have discussed this, and I'm convinced that the method signature should be something like:

{{ $resource := $url | resources.GetRemote $opts }}

But I would I would love it if we could find a great Go library with a struct config that we could fairly easily adapt).

[moorereason]

Updated my "proposal" comment above to move the URL parameter to the end of the function signature.

[atishay]

@moorereason Hugo defines maxAge for controlling its caching. Instead of no-cache, maxAgewhich can override the default provided in the config on a per request basis may make more sense.

[bep]

To repeat myself, we're not adding some low-level HTTP API to Hugo, so this:

{{ if eq resp.StatusCode 200 }}
  {{ $json := transform.Unmarshal resp.Body }}
{{ end }}

Is not something I want to teach the average Hugo user on the forum.

[atishay]

How about we just have {{ $json := transform.Unmarshal resp.Body }}. It can throw and error and fail the build if the StatusCode is not in 200/300 series. (I am assuming redirects will be automatically followed). We can have resp.RawBody which will never fail. Sometimes response body gives useful detail about the error on the server.

That way users that understand status codes get status codes and access to the raw body of the response. Users that don't can just use resp.Body. It is useful to get response headers, especially for cases like 201s where the created content id is present in headers in many cases.

[dabrain34]

Hello,

I'm interested by this feature as I'm looking for a way to generate a small preview of a static website (blog post). This API would be great.

But before to have something landed is there another way to get the header from a random website and use it ? a JS snippet somewhere ?

[davidsneighbour]

@dabrain34 why use a static website generator to do what a headless browser testing tool already can? https://headlesstesting.com/support/start/screenshots.html

[dabrain34]

because I wanted to generate a mosaic from posts containing only a link where I could retrieve these info (title, description, thumbnail).