Elasticpress: Autosuggest - how does it work?

[scooterlord]

Describe your question
I am trying to make autosuggest work, but have no idea where to start since it is not documented anywhere... I tried to unluckily connect bits and pieces I found on the web, with no luck.

I am using an elasticsearch server hosted by elastic.co.

My questions as follows:

• Do I need to create my own REST endpoint to connect to autosuggest?
• After this endpoint is created, is it necessary also to define a callback function?
• Do I have to connect this endpoint to the hosted elasticsearch solution somehow?
• Can the autosuggest be customized to also show custom content, eg images?
• Adding the ep_autosuggest class on an input currently completely hides the input - using the inspector a 'display:none' property is applied via css. Is this normal?

Thank you everyone for your time, looking forward to your replies and help! Please be thorough in your replies!

[scooterlord]

Hello everyone, I made some progress, but still haven't been able to find a solution.

I'm sorry if below I am _stating the obvious_ but for me it wasn't _that_ obvious. After a lot of trial and error I realized and mixing bits and snippets of code, I realized that, in fact, you _need_ to create the REST endpoint and using a callback function return the results.

Here is my progress so far, I created the endpoint:

 add_action( 'rest_api_init', function ( $data ) {
     register_rest_route( 'elasticpress', '/autosuggest/', [
         'methods' => \WP_REST_Server::CREATABLE, // Changing method to 'POST' allows to hit from browser
     'callback' => 'ep_autosuggest'
     ] );
 } );

I then created a function based on the only snippet I could find online:

use ElasticPress\Elasticsearch as Elasticsearch;

 function ep_autosuggest( WP_REST_Request $data ) {
     // Elasticsearch PHP Client
     $client = ClientBuilder::create()->build();
     $params = [
         'index' => 'localhostmn-post-1',
         'type' => 'post',
         'body' => $data->get_body()
     ];
     $response = $client->search( $params );
     return $response;
}

This gives me a '500' error in the browser when I use the search field -the problem is the $client = ClientBuilder::create()->build(); line... the ClientBuilder doesn't exist, even if I add the `use Elasticsearch\ClientBuilder;'

Emptying the function, and returning $data->get_body() eturns '200' to the browser with the query.

So, the only missing link is how to query the server properly.

Any pointers are more than welcome. Spend a full day on this one... at least I am a lot further than 3 days ago where I had no clue!

[scooterlord]

Anyone? Any help? Any pointers?

I am surprised how people figured this out without documentation!

[n9yty]

I haven't gotten it to work either, but I'm not sure if you need to create the REST endpoint so much as you need a redirect in your web server to forward that endpoint over to the elasticsearch endpoint. That is what the sample nginx config I found showed, but I will confess that I haven't managed to get that working.

It would be nice to have a working config documented.

[n9yty]

I also see ticket #1504 here which seems to suggest the same. I had tried, but the apache2 instance installed on the test site I was using did not have proxy installed. After installing proxy, one line was enough in my htaccess file:

RewriteRule ^your_endpoint$ http://127.0.0.1:9200/_search [P]

I also found this page with more info if you wanted to have basic auth set up on your Elasticsearch endpoint, but I don't know how to tell ElasticPress to use that.

And while it may not be the best security (still investigating, testing, looking) you could preceded that RewriteRule with a RewriteCond to only proxy requests that were referred by your main server.

RewriteCond "%{HTTP_REFERER}" "server.domain.here" [NC]

I haven't tested extensively, but it allows autosuggest to work and if I hit the endpoint directly from outside I get a 404 error as the request is not proxied. NOTE: Referer is easily spoofed, so this really adds nothing in the way of security.

[scooterlord]

Thanks for your reply! But I have a problem with the breaking callback function. If you use the rewrite what’s the purpose of creating an endpoint anyway?

Your reply confused me more than actually helping me!

[n9yty]

My point was that you don’t need to create a WP REST endpoint. You are doing a lot of work for nothing. The ElasticSearch service already has a running endpoint, you just have to proxy the requests to it.

[scooterlord]

This makes more sense, but still, how do you retrieve results back to WP?

[n9yty]

It is all handled automatically. The ElasticPress plugin puts a listener on the search field (or however they are doing it) and as you type characters it is making a call to the endpoint specified in the settings, which is then proxied to the ElasticSearch _search endpoint, and it takes those results and updates the dropdown it creates with the results of the search results. Basically, once I put the proxy in place and turned the autosuggest on in the settings, making sure the endpoint matched with my proxy, it just started working with no other work to be done in my theme or custom code bits on WordPress.

[scooterlord]

This is very helpful! I will look into it and report back as soon as I have results!

So, no wp endpoint, I will just define it via htacces rewrite and see what happens

[n9yty]

Yes, that is all I configured. The only place the endpoint/URL was set up was in the htaccess rewrite rule, WP didn't have to know about it at all.

I hope it goes well for you. You have already put in a lot of work, which unfortunately wasn't needed. Just like you, I found very little clear instruction on how this needed to be done. I wish they could document this better.

[scooterlord]

I've been trying with trial and error but still no luck... I asked for help from my hosting provider, they might be able to help..

[n9yty]

You might need to verify that your Apache has mod_proxy installed. Mine did not, but it did not complain about that configuration, it just silently did not work. If you have the JavaScript console open when trying the search field, you should see errors in there for every character you type. I was seeing 404 errors or others, depending on what I had tried. That would be my next thing to look at because it caused it to fail silently. Once I had the proxy and proxy_http modules installed everything was fine.

[scooterlord]

Thank you very much for your support so far! However, I don't see any errors either in the console or the network XHR.

However, hitting https://mydomain.com/es-search now gives a 500 error. That's a good thing, right?

[n9yty]

Just to be sure, in the configuration page for ElasticPress, you put in the FULL URL for the endpoint, right?

So you would put in:

https://mydomain.com/es-search

Then the htaccess rewrite would be:

RewriteRule ^es-search$ http://127.0.0.1:9200/_search [P]

Assuming your ElasticSearch is running on the localhost on port 9200.

Then, after you verify it is working, you might put this in above it to block others on the Internet from accessing that endpoint:

RewriteCond "%{HTTP_REFERER}" "mydomain.com" [NC]

You should be able to view your web server error log files to see if there is more information about the 500 errors.

[scooterlord]

I am using Elastic Cloud service, not a local environment... and for the endpoint I just added /es-search. I am going to test with the full endpoint, after I type this.

Since I am using the elastic cloud service, I followed the other link you posted and essentially here is what I put in my .htaccess. For some reason it was not working at all inside the <ifModule mod_rewrite.c> so I put the following code outside of it - and I replaced XXXXXXX with the base64 as mentioned in the other page.

SetEnvIf Request_URI ^/es-search$/* ADD_ELASTICSEARCH_BASIC_AUTH
RequestHeader set Authorization "Basic XXXXXXXXXXXXXXXXXXX" env=ADD_ELASTICSEARCH_BASIC_AUTH
RewriteRule ^es-search$ https://elasticclouddomain:9243/_search [P]

[brandwaffle]

@n9yty @scooterlord running your own endpoint is certainly possible, and the approach you are taking will definitely enable autosuggest. The reason we don't provide a tutorial on this is that the resulting autosuggest endpoint becomes fully available to the world, and therefore your data can be easily modified/deleted by someone with knowledge of Elasticsearch and JavaScript. As you noted, you can put some sort of access credentials in front of this, but unless you inject those only conditionally via the proxy script using a very complex ruleset to handle things like 'private' vs 'public' posts, the proxy essentially acts as a way to bypass any security and allow unfettered access to your Elasticsearch instance.

As you pointed out, if you run your own infrastructure we also can't enforce any sort of security, even a basic username/password (which still allows for request manipulation and possible content deletion/defacing). For these reasons (differing requirements, approaches, configs, and the lack of built-in security in Elasticsearch), we aren't able to provide a recommended approach, since we don't recommend doing things this way. That said, in the spirit of GPL we make it fully possible if you have a solid understanding of your web server's configuration options, Elasticsearch security, and feel confident making security decisions for these applications in your production environment.

[n9yty]

@brandwaffle So instead of the request to just document how this works, it would be nice to have it documented how to do this in a secure fashion, at least the basics of how it could be secured. It doesn't have to come from the maintainers, but if anyone knows how it could be done, please share. :)

Maybe a reference here to some closed issues:

1633 #1027 #956 #249

[scooterlord]

...looks like my server doesn't allow redirection through apache because it doesn't support mod_proxy... so I need to figure out another solution.

So, from what I realize from the conversations above I just to create the wordpress REST API and connect it through the admin. What about the callback action though?

@brandwaffle Will this be a more secure option compared to Apache redirects?

Edit: So, I finally managed to make it work _locally_ though. Have no idea how to set this up on my live environment due to host restrictions, at least now I realize what's going on.

As been advised by @n9yty above, here are the steps I did:

• Installed mod_proxy, mod_proxy_http and mod_proxy_http2
• Added the full endpoint url in the Endpoint URL in elasticpress settings like so: http://localhost/mysite/es-search
• Added the following line in .htaccess that points to my local Elasticsearch server: RewriteRule ^es-search$ http://localhost:9200/_search [P]
• Success!

Well, now I have no idea how to make this on my live env, so I guess I am going to either manually setup an apache server or investigate the other solution with the wordpress REST endpoints. I will report back with my findings.

[brandwaffle]

@scooterlord it will definitely be more secure, as you can enforce more granular checks via the REST API. The tradeoff of course is that you go from a < 100ms direct call to ES to probably a few hundred ms to talk through WP to ES.

@n9yty not sure what exactly happened to the thread--I have an email reply I was trying to address for you about security implications, but it seems to be gone from here. To address those questions, ElasticPress.io doesn't put credentials into the client JS, it simply pushes Autosuggest requests into a separate endpoint that only allows post types and statuses based on the latest _authenticated_ request the service has received. This ensures that someone can't modify the request body itself to spy on your content. For example, if you have EP's WooCommerce Feature active alongside Protected Content to enable EP to search your customer orders, an unscrupulous visitor could simply change the 'product' post type to 'order' in the JSON body and start grabbing all your customer addresses, etc. A news site could have their unpublished content scraped, etc.

We don't really document the HTTP auth in-URL approach, because the format of that is pretty widely known regardless of whether you're using ES or any other service that supports htpasswd auth: https://username:[email protected] in the URL field will use those credentials. For your proxy URL, you would set that separately in the appropriate field. You can also set this in wp-config.php which is documented here http://10up.github.io/ElasticPress/tutorial-security.html

Of course, I'm sure you see where that starts to create new security concerns for us--if that field is accidentally copy/pasted into the Autosuggest field, the user will inadvertently expose their ES credentials in JS, and now literally anything can be done on that server (unless an IP block is added, etc).

It's definitely possible to achieve _some_ level of security with a custom setup, and depending on what data you decide to store, this might be perfectly fine for your use-case. That said, we perform the request body-level checks on every incoming request on ElasticPress.io and have spent significant effort to provide both full security and direct connection to the service through custom server-side authorization logic, simply because there isn't a great off-the-shelf way to ensure that certain data isn't exposed because it's part of the request body, not the request method or any other server-level conditionals.

If you choose to index private content, I would use @scooterlord 's approach of a REST API endpoint and use that to ensure the incoming request isn't snooping, and if you don't index private content, I'd go with a quicker Apache proxy to hide your auth credentials and limit access to all paths except _search and maybe even block certain request types (DELETE, PUT).

[n9yty]

@brandwaffle That is a wonderful summary. I wish you could put that in the "docs". :). Very helpful! Thank you for taking the time to spell this out, hopefully it will avoid someone from a data disaster if they see it spelled out this clearly.

I wonder if something like this would be helpful to protect the autosuggest... You could still have protected data available but for the autosuggest endpoint you could use a unique Apache proxy that would go through a filter, which is what I think you have already suggested.

This is old, but an idea, and maybe it might still work, I haven't tried yet:
https://github.com/lukas-vlcek/node.es

That code could be modified to look at the incoming _search request and, as you say, make sure that only non-sensitive data is being queried and maybe put other restrictions in place. Probably faster an a WP REST endpoint and that overhead.

UPDATE: Another good point about not using a simple proxy: https://search-guard.com/elasticsearch-proxy-security/

So much to learn. That Search Guard product looks interesting, and they have a community edition to at least do some testing with, I think I will start there.

UPDATE2: So, Search Guard may be using code inappropriately borrowed. ;( But here is a blog article about using the free ElasticSearch security features: https://www.elastic.co/blog/getting-started-with-elasticsearch-security

[scooterlord]

@brandwaffle thank you for the thorough explanations - obviously there is also a ton of new things to learn especially if it is out of your field.

Regarding the endpoint I still haven’t managed to make it work. I created the REST endpoint and I can hit it. I am having problems with creating the callback function. Is there any up to date snippet for the callback function? I tried every single snippet I could find online but no luck... any pointers would be greatly appreciated and I am certain it will help others as well.

[brandwaffle]

@scooterlord Unfortunately I don't have any examples. There might be others going the self-hosted ES route who have tried this approach, but since ElasticPress.io was built to handle this specifically, I'm not personally aware of anyone using any REST API-based workarounds.

I appreciate both you and @n9yty engaging in this dialog with me so candidly and courteously. In that spirit, I'd like to just ask: is there a reason why ElasticPress.io is not an option you're considering? We really have spent a lot of time and effort to solve all these problems and provide a seamless experience, and as you've pointed out, this isn't exactly a simple problem to solve, so we think ElasticPress.io can be a great value once you factor that in.