Amphtml: FR: Support Algolia results

[Haroenv]

Intro

Algolia is a hosted search as a service. Users can upload their data, so that it gets indexed and replicated over several machines, in several locations even if a DSN is used.

The most used way of bringing Algolia search, is by doing it in the frontend by using one of our InstantSearch libraries. These exist for vanilla, React and Vue currently. AMP could become one of those list, if Widgets can be made available for this platform.

Proposed API

The <amp-algolia> could be loosely based on the <amp-list> component, since it's a more complicated version of storing something dynamically in a list. This components provides props for

1. account information
   
   
   
   • app-id
   
   
   • api-key
   
   
   • index-name
   
   
2. API parameters
   
   
   
   • query
   
   
   • page
   
   
   • ... see the documentation
   
   

The index-name and all API parameters should be changeable with AMP-state, since it's a convenient way of storing the information externally of this one component.

<amp-state id="search">
  <script type="application/json">
    {
      "query": "",
      "page": 0
    }
  </script>
</amp-state>

<input on="input:AMP.setState({query: event.value})" />

<amp-search-algolia
  app-id="latency"
  api-key="3d9875e51fbd20c7754e65422f7ce5e1"
  index-name="bestbuy"
  [query]="query"
  [page]="page">
  <!-- arbitrary props are passed through to the fetch -->

  <!-- the template for a single hit -->
  <template type="mustache">
    <article>
      <h1>{{name}}</h1>
      <p>{{_highlightResult.description.value}}</p>
      <amp-img 
        src="{{image.src}}"
        alt="{{image.alt}}"
        height="{{image.height}}"
        width="{{image.width}}">
      </amp-img>
    </article>
  </template>

</amp-search-algolia>

<button on="tap:AMP.setState({page: search.page + 1})">next page</button>

Widgets

What would be nice is having several components like in our InstantSearch libraries for things like disjunctive faceting etcetera. This should be out of scope for this proof of concept, but might be something that makes searching easier in the future. The only question then is how to bind these components. A feasible solution would be to add events to those as well, and linking it up to the global state. That's quite manual, but probably the most "magic" we can get with AMP.

Current issues

see also #10523

url is dynamic

Algolia has a four-way redundant URL scheme, based on the appId:

• appid-dsn.algolia.net
• appid-1.algolianet.com
• appid-2.algolianet.com
• appid-3.algolianet.com

This means that adding a simple url to amp-list will not be sufficient, without forking it and taking care of the url to send a request to. This isn't really a problem, but more a reason to have an own component over which more custom behaviour can be had.

This means that an <amp-algolia> will be stateful and will retry failed requests with the next URL and its timeout. Preferrably there would be some control over the several timeouts that can happen, but it's not in the Fetch spec, and FetchController isn't supported anywhere yet.

Algolia can’t return AMP CORS

Adding the CORS requirements to an Algolia response will be problematic, there are three main ways to solve this:

1. a proxy
2. a setting in Algolia that will forward the correct headers
3. send raw requests

The first might be the simplest at first sight, but it has significant impact on performance. If there's just one host, that means that all AMP traffic will go through the same machine, instead of being distributed otherwise. This also means that the advantage of DSN will be impossible to achieve, without having an as-big infrastructure, just for AMP.

The second option might be more complicated, but it seems to be the only way that this can be solved in vanilla AMP, since sending these requests is required.

The last option will probably be impossible within AMP, since this is your project, and there are valid reasons to expect having extra headers to be sent.

tl;dr Algolia is a Search as a Service, it provides a JSON endpoint with hits to loop over, currently quite complicated to add the necessary headers but might be possible if interest is shown.

[dvoytenko]

@Haroenv Thanks for the ITI! This sounds good. I have a couple questions:

1. What's you current typical rendering strategy: (a) Do you likewise return JSON and let the caller render it? (b) Do you predefine some rendering rules? (c) Do you prefer to do rendering in your widgets with a couple customizable classes?
2. I couldn't quite understand AMP CORS section. Do you not support CORS itself? Do you simply not need credentialed requests?

/cc @ericlindley-g @cramforce

[cramforce]

Have the same questions as Dima, but overall looks good. FMI: Is there a category for this type of service? We generally prefer names that say what they do like amp-search-algolia

[Haroenv]

Well the "blocker" now is mostly that our API can not:

1. Return arbitrary headers
2. Work with unknown parameters

These two issues make it so that the regular AMP CORS flow is not yet possible.

If there's a way to avoid doing the AMP CORS requests, but instead regular Simple CORS requests (x-www-form), which is what we do in the client to avoid the OPTIONS preflight request, that would solve the problem

Requests are secured with an API key header, which could be added based on the prop.

We have two main ways qua rendering, the first is by using the Algolia api client. As a return you'll get a JSON object with hits, facets (not really relevant for a PoC), and some other information. We also offer InstantSearch (vanilla, React and Vue), which make that simpler, but likewise don't make any assumptions on the rendering of hits.

InstantSearch actually also uses Mustache for the rendering of hit templates, so that's very similar already.

The only rendering requirement we have is that our logo is displayed for users of the free plan.

Renaming it to amp-search-Algolia seems fine to me. Amp-instant-search is also possible but it might become a branding problem at that point.

[Haroenv]

The technical term for our category would be Search as a Service

[dvoytenko]

@Haroenv The shape of the network request is generally up to you. AMP CORS is currently only required for credentialed requests (credentials: include). Same goes to the request body - it's definitely fine to use x-www-form-urlencoded and we indeed recommend everyone to avoid preflight CORS if possible.

InstantSearch actually also uses Mustache for the rendering of hit templates, so that's very similar already.

This sounds good. I'd recommend, though, to provide a default template if possible and makes sense.

[cramforce]

AMP CORS is only required if CORS is essential to the security model. Might be fine without. In a world where serving from cdn.ampproject.org VS domain.cdn.ampproject.org is enabled it is much less needed.

[Haroenv]

Argh, I'm going to have to close this one. I'm sorry but we only support JSON requests, x-www-form-urlencoded won't be supported by our engine any time soon.

I'm wondering however if I could depend on part of our Algolia engine (namely Requester and search), this whole headache could be avoided, and amp-search-algolia could focus on its actual use. If we do that, our whole retry strategy wouldn't need to be done again.

V4 of our JS client, which allows this individual importing of methods as described, is not ready yet, that will still take some time, but I'll keep this use case in mind when doing the implementation.

I don't know think a default template would be possible, records are shemaless, so the best we could do is just a list of the attributes. I'm not even sure you can iterate over the object values properly in Mustache. amp-list doesn't have a default rendering either as well, right?

Thanks for your answers already 😄

[dvoytenko]

@Haroenv the shape of the request is generally up to you - you can use application/json requests if you'd like. Per CORS rules this would trigger preflight, but it's not a big issue.

I don't know think a default template would be possible
Sure, if your responses do not have a schema - then default template is not possible, so it's fine.

I'm wondering however if I could depend on part of our Algolia engine (namely Requester and search)

Do you mean you'd want to compile this code in? If you do so, we request the relevant code to be placed in the third_party, so it's not much help. And we do not allow 3p JS references in the 1p AMP JS code. Though maybe I misunderstood?

[Haroenv]

Indeed, that’s what I meant. This means that technically if I wanted to depend on our own Request strategy, it would be possible, but mean that people import it from /3p?

I have a feeling that going that direction will be __significantly__ simpler for us, as it wouldn’t require an exception of AMP in the engine, as the requests done by our Requester are doing Simple Requests where possible (avoiding the OPTIONS request).

The 3p directory seems to be mostly for embeds, but maybe that’s just because no service like Algolia has been part of AMP before.

[dvoytenko]

@Haroenv I'm not 100% sure I understand. Preflight request can be avoided with ease and as an extension you'd have a wide range of tools. Compiling 3p code in is a bit more complicated, but possible as long as it's in the third_party folder. We run all 3p code (and of course 1p code as well) via security reviews. As part of AMP we make a promise that the code has been security reviewed and to make this promise we have to be able to control what's compiled/deployed in 1p.

But maybe you could point me to the code that you'd like to use so that I can understand this better? Is it open-sourced?

[Haroenv]

algoliasearch-client-javascript, yep, it’s open source.

This is the older v3 version (12kb gzipped), but v4 will be a lot more flexible and allow for importing just the “search request” part.

[dvoytenko]

@Haroenv The current policy is to include a binary (minified or maximized - doesn't matter) in the third_party/ along with readme with links, licenses, etc. You can place it in the common third_party/ or inside your extension/third_party/. We can then ask for a security review.

[ampprojectbot]

This issue doesn't have a category which makes it harder for us to keep track of it. @dvoytenko Please add an appropriate category.

[ampprojectbot]

This issue doesn't have a category which makes it harder for us to keep track of it. @dvoytenko Please add an appropriate category.

[Haroenv]

This should probably be Category: User Input @dvoytenko

[dvoytenko]

Also added Dynamic Content.

[ampprojectbot]

This issue hasn't been updated in awhile. @dvoytenko Do you have any updates?

[Haroenv]

No updates so far. We haven’t heard too many requests for it on our part, so priorities were focused elsewhere. Might pick this up again later though

[kmarshall008]

As AMP adoption increases beyond Publishers with a huge push in Ecommerce, along with other sectors deploying apps with AMP pages for performance, searchability and user experience benefits, it's increasingly important for us to make the switch. Soon we're going to have to choose "AMP or Algolia" and unfortunately we'd have no choice but to select AMP. What would it take for us to be able to use Algolia within an AMP page? They are saying now that it may be possible for them to allow the injection of JS into an AMP page. Perhaps you could reach out to their team to collaborate as a beta library worth supporting. The AMP Project team is easy to get in touch with and they are very passionate about creating awesome, high performant user experiences. AMP + Algolia is a no brainer marketing win.

[dvoytenko]

@ericlindley-g could you pls reach out to the vendor?

[ericlindley-g]

Sure — I'll take a look at some of the options to get in touch. It's a promising sign that they have a GitHub repo

[Haroenv]

I’m working at Algolia FWIW, would love to get some more info from someone on the team, was a bit confused by earlier responses

[ericlindley-g]

Sorry about that — I should have read the thread more before jumping in. @Haroenv and @dvoytenko , does it make sense for you to sync in a higher bandwidth channel like Slack or video chat? (e.g. design review)

[mrjoro]

Is this I2I still active?

[Haroenv]

I have not actively been working on this because of lack of demand from the community to support AMP directly. However if this is in demand from Google we could reconsider I think.

[mrjoro]

Thanks for the update @Haroenv; that sounds good. I'll leave it to @aghassemi and the @ampproject/wg-ui to help figure out whether the demand is there.

For now I'll leave this as a Feature Request but remove the Intent to Implement label.

[aghassemi]

Please feel free to reopen the issue when/if you plan to start the implementation.

[alankent]

I should have searched here first, but I had a chat with some Algolia folks this morning looking to see if amp-autocomplete could be useful with Algolia. Short answer, not at present because the API assumption of the frontend and backend are different. (AMP assumes a particular response type, Algolia does not have an API returning the JSON AMP expects. Might be some HTTP header issues as well.)

I have been floating with a few people the idea of having a "protocol adapter" for want of better name. A JavaScript (or similar) function that runs in a worker thread that I can write in a separate file (sort of like amp-script). It intercepts the AMP outbound request, has the full power of JavaScript to encode the request as required for the real endpoint (base64, XML, JSON, etc), and massages the response into the JSON structure AMP wants. It can also cache. It allows AMP to cope with existing backend APIs rather than expecting the APIs to cope with it. (A service worker is close, but they are not guaranteed to be loaded. This JavaScript hook function would be guaranteed to load and called for all outbound src= requests, for amp-autocomplete, amp-list, etc.)

[Haroenv]

Great idea @alankent, this would prevent us from needing to add more amp-specific logic

[sebastianbenz]

Update: you can now use amp-script to integrate Algolia search.

Here is a demo: https://amp-script-demos.glitch.me/algolia.html

[Haroenv]

Thanks @sebastianbenz, that's great news!

[Haroenv]

Where's the source of that demo? I'd love to see what it looks like to use this @sebastianbenz

[sebastianbenz]

HTML source: https://glitch.com/edit/#!/amp-script-demos?path=public/algolia.html
JS source: https://glitch.com/edit/#!/amp-script-demos?path=public/algolia.js