Parse-server: Full text search support

[ranhsd]

Hi, I wanted to know if parse-server support full text search. The full text search is available since MongoDB version 2.6 and it is possible to create text index (currently only one) which provides efficient search capabilities on text fields.

From the code it looks like that this capability has not been implemented ...

In case we need to add this capability i think we should perform the following tasks:

1. Write the server side code (probably modify the MongoDB storage adapter)
2. Update Parse rest api endpoints
3. Update all client SDK's (at least iOS/Android/Javascript)

Thanks.

[flovilmart]

Can you point out the documentation for full text search? If it requires a special operator?

[ranhsd]

Hi @flovilmart . Yes it actually required 2 things and there is also one limitation

1. Create text index on one or more columns (multiple column is the compound index)
2. Use the following command the run the search:

db.{collection_name}.find({$text: {$search: "{search string}"}}

Full text search is supported since v2.6 of Mongo and currently you can create only one text index per collection.

There is a great user guide + tutorial which explain what it is and to use it:

https://code.tutsplus.com/tutorials/full-text-search-in-mongodb--cms-24835

[flovilmart]

Ok that's something worth considering!

[ranhsd]

Yes it is! until now the solution was to create the full text search indexes in elastic search via mongoostatic.
The idea is to replicate all the text indexes column data into elastic search and when the client sending a search query the library execute the query in front of elastic instead of MongoDB. The big advantage of this approach is that elastic search engine is much more powerful than MongoDB native search capabilities and also you can create multiple full text search indexes with this approach. Unfortunately we cannot integrate this solution because parse-server not use Mongoose behind the scenes so we must use the native MongoDB capabilities

BTW! how much effort is to use Mongoose for executing the queries to the DB instead of the native MongoDB driver? (I am not talking about replacing the whole solution but maybe use it as optional dependency)

[hhanesand]

Looks like it has already implemented this at https://github.com/seriph/parse-server

@seriph Could you throw together a pull request? 😄

[seriph]

Sure, I put this together right around the time that this thread was started. Code works great but I consider it beta. Note that it requires changes to both the server and the API. I'll submit a pull soon.

Until then, if you have urgent need in your projects, feel free to use the npm packages I submitted. Just drop and replace with parse and parse -server.

https://www.npmjs.com/package/parse-server-text
https://www.npmjs.com/package/parse-text

I've only updated the node/js API so far. Would love to see someone update for the other platforms.

[flovilmart]

@seriph nice, we'll take care of the SDK's once the feature lands in parse-server (no need to rush those). @rogerhu and I will take care of the android and iOS SDK's.

[ranhsd]

We need also to update the docs about this new capability. Thanks a lot !

[SebC99]

Any news on this? It would be awesome! :)

[dplewis]

@flovilmart I can do a PR for this as Postgres also supports full text.

1. Should the tests go in ParseQuery.spec?
2. What would we call this query.fullTextSearch()?

[flovilmart]

1. YES! Amazing
2. Ok! That's fine for me

For the REST format, let's stick with something close to the mongo query format (as the GeoPoints do)

[dplewis]

Should I include all endpoints? Or do 1 at a time

{
  $text:
    {
      $search: <string>,
      $language: <string>,
      $caseSensitive: <boolean>,
      $diacriticSensitive: <boolean>
    }
}

[flovilmart]

The REST format ultimately should support all payloads, you can put validation on all at first, and write failing tests that are xit'ed this way we know this support is ongoing if you prefer spliting the work in multiple PR

[seriph]

How does postgres apply?

On Jun 2, 2017 6:20 PM, "Florent Vilmart" [e-mail] wrote:

The REST format ultimately should support all payloads, you can put
validation on all at first, and write failing tests that are xit'ed this
way we know this support is ongoing if you prefer spliting the work in
multiple PR

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/parse-community/parse-server/issues/3747#issuecomment-305936421,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABUl40u1IR54geM0mn9h2Fv5syJNRwv5ks5sAKbkgaJpZM4NF72e
.

[dplewis]

Postgres is another NOSQL database Parse supports like Mongo.
We try to have as much functionality supported by both databases.

https://www.postgresql.org/docs/9.5/static/textsearch.html

[seriph]

Pull request submitted @flovilmart @hhanesand @SebC99

[seriph]

3905

[nuborian]

Anything new to this topic? :)

[SebC99]

It's available on parse-server since 1.5.0 but only accessible via REST API
The php sdk PR is on hold as they want to include a default index creation with it (I don't know why it has to be done at the same time...) and so are the other sdk. I'm working on the JS one, and will open a PR soon for this.

[nuborian]

Thanks for the Update! @SebC99 Are there any docs about this? Couldn't find it on the REST Api Guide

[SebC99]

There's a PR from @dplewis in the docs repo

[DIMEGLIOM]

Are there any instructions out there regarding using this feature w/in iOS? This feature looks very promising and I was hoping to use it in a project I am currently working on.

Thanks in advance to anyone who can get back to me.

[rihadavid]

@dplewis I am thinking about implementing support for full-text search over multiple fields. Would you be able to help me with this? I made it work on my local MongoDB, so I know it is possible. But I am not familiar with Postgres, REST, how to run tests and don't have much experience with contributing :)

What was needed to make it work on MongoDB:

1. create index in mongo shell - example: db.getCollection("_User").createIndex( { name: "text", username: "text", about: "text"} )
2. add '$text' to allowConstraints in ClassesRouter.js
   allow providing null as the key
3. somehow bypass error 85 in createTextIndexesIfNeeded in MonoStorageAdapter.js (you wrote that function, so maybe you could help with correct solution? Maybe just do not auto-create index if doing query over multiple fields?)
4. define the $text query without the key, like query._extraOptions['$text'] = { '$search': options }; in javascript (cloudcode)
   call the query with null key query._addCondition(null, '$text', { '$search': options }); in javascript (cloudcode)
5. got the results! it works!

What do you think? Could we make it, please? :)

[dplewis]

@rihadavid We came up with a proposal for handling full-text over multiple fields. Adding indexes via Schema API

https://github.com/parse-community/parse-server/issues/4212#issuecomment-333328197

I should have some time next week to work on it. If you want to open a PR feel free.

@flovilmart We can provide an immediate solution to this problem or should we wait until the Schema API gets indexes? What do you think?

[rihadavid]

@dplewis Actually, I had a bug in my test from previous comment and it is much easier than I thought. Just providing null as the key, allowing the null as the key and bypassing the error 85 makes the query search over the multiple fields.

Like this:
where={"null":{"$text":{"$search":{"$term":"some words"}}}}

This really seems easy, maybe I will try the PR. Do you think it's OK to call it this way, with null? I am surprised it works, but it does!

[dplewis]

@rihadavid There are reasons why automatic index creation and catching error 85 are in place. Please see this comment here. https://github.com/parse-community/parse-server/issues/4212#issuecomment-332580563

Removing error 85 will allow for multiple indexes but we will like to keep it in place for now.

[rihadavid]

Ok, sure :) It seems too complicated, I am not a big friend with indexes. So I will rather let you work on it and keep my experiments in my local environment until you guys make it a stable feature :) Thank you for implementing this!

[dplewis]

@rihadavid Contributions are always welcomed. Trust me I didn't know about full-text, indexes, postgres, rest when I did this.

[grosscorporation]

so will the following work? I tried it aint, or do I have to recreate my Schema

where : { 'null' : { '$text' : { '$search' : { '$term' : val } } } }

[dplewis]

@GoGross The REST docs has what you need. You don't need to recreate your schema
http://docs.parseplatform.org/rest/guide/#queries-on-string-values

where : { '$text' : { '$search' : { '$term' : val } } }

[grosscorporation]

Do I have to update my schema? this fails

[dplewis]

Please update to the latest version of parse server.

https://github.com/parse-community/parse-server/releases

[MHX792]

any way to use fulltext search within the iOS SDK?

[rihadavid]

PR for iOS SDK already exists here: https://github.com/parse-community/Parse-SDK-iOS-OSX/pull/1196

[montymxb]

Going to go ahead and close this out now that the feature is merged in. Feel free to keep discussing however, if anything significant comes up feel free to open up another issue or let us know to pop this one back open, whichever works.

Also relevant PRs look to be open on the sdks. From here I'll see if we can't get those in place now 👍 .

• iOS
• JS
• PHP(merged)

For Android it's issue #751. If someone has a moment we would appreciate a PR over there as well!

[rihadavid]

If anyone is reading this still being interested in the full-text search over multiple fields, I just figured out that it works out of the box - if you had manually created the multi-field text index in the db.

All you need to do is to provide any of the fields defined in the text index (so that the parse-server is not trying to create the index for you). It will search over all of those fields defined in the text index, not just the one you provide 👍

var query = new Parse.Query(BarbecueSauce);
query.fullText('name', 'bbq'); //provide any of the fields in the index, it will search over all of them

Tested on Parse Server 2.8.4 with Mongo.
Maybe this could also be mentioned in the docs, but it would probably require some more testing.

[dplewis]

@rihadavid Would you like to submit a PR? Passing in an array to create a compound index automatically.

[ranhsd]

Hi @rihadavid Yes I already request it as a feature few days ago here: #5152

@dplewis I don't think that we should replace the current function that we have and use array instead of string because It can break current implementations. We can either create additional function something like: fullTextCompound or check inside the current function if the input is an array or string and create the indexes accordingly.

[rihadavid]

@dplewis Nope, sorry, but I am currently buried deeply in heaps of C# code that should have been finished months ago 😭
@ranhsd what a coincidence! :)