Haystack: Introduce QueryClassifier

[tholor]

Is your feature request related to a problem? Please describe.
With the new flexible Pipelines introduced in https://github.com/deepset-ai/haystack/pull/596, we can build way more flexlible and complex search routes.
One common challenge that we saw in deployments: We need to distinguish between real questions and keyword queries that come in. We only want to route questions to the Reader branch in order to maximize the accuracy of results and minimize computation efforts/costs.

Describe the solution you'd like

New class QueryClassifier that takes a query as input and determines if it is a question or a keyword query.
We could start with a very basic version (maybe even rule-based) here and later extend it to use a classification model.
The run method would need to return query, "output_1" for a question and query, "output_2" for a keyword query in order to allow branching in the DAG.

Describe alternatives you've considered
Later it might also make sense to distinguish into more types (e.g. full sentence but not a question)

Additional context
We could use it like this in a pipeline

[stefanondisponibile]

I like the idea @tholor . I'd be willing to help.

[tholor]

Great! Do you want to create a first draft PR @stefanondisponibile ?
Happy to guide you then and give early feedback.

The core will be of course the classification of the string into "query" or "keyword". Either via a small rule set or alternatively via small, light-weight ML model.

The requirements for running this class as a node in the Pipeline are then relatively straight forward:

• run() method
• class attribute outgoing_edges = 2
• returning a tuple (query, "output_1") or (query, "output_2")

Just let me know if you need any help to get started!

[stefanondisponibile]

Thanks for the hint @tholor , I guess I'll do a little analysis and come back with some ideas, or directly a draft PR.

[Threepointone4]

@tholor I have implemented a simple method based on my understanding, let me know if this is proper or not. If yes I will go ahead and create a branch and push the changes.

question_check can be an ML or rule-based function which checks query is question or keyword. Right now I have done this using some rules.

Thanks

[tholor]

Hey @Threepointone4, this looks good! You also need a class attribute to define the number of the two different outgoing edges:

class QueryClassifier():
    outgoing_edges = 2
def ...

The interesting part will be question_check(), I guess.
Would be great if you can create an early draft PR. Maybe you can then collaborate with @stefanondisponibile on finalizing it together. I am sure merging your two approaches will yield something great!

I will of course give feedback and support you along the way.

[Threepointone4]

@tholor Yes class attribute is there but it's not showing in the screenshot. I am testing with some flows and will create a draft PR soon and will share ASAP.

[stefanondisponibile]

Good job @Threepointone4 . Have you implemented the _classifier_ itself yet. I think that should be also something that any user can pass or tune on their own. The node receiving an input and going either to one output node or another should be independent from the specific need of a query classifier (I may want to either go to different nodes depending on some other logic). What you think @tholor ?

[lalitpagaria]

Sorry for asking question bit late.

I am just curious about use of classification model for query identification.
Isn't bit heavy to determine whether query it keyword or question by model?
I might not getting bigger picture.

For simple purpose I rather go by how elastic search solve it. We can have simple DSL or borrow concept from Lucene's DSL as follows -

• For keywords: term: <query>
• For questions: query: <query>
• For NERs: ner: <query>
• For fuzzy queries: fuzzy: <query>
  etc

Also supporting different type of filters - range, bool, in, not_in, greater, lower, matched, not_matched etc
Ultimately DSL parser will decide which part of query will go where and then it will join results from these parts.

Again apologies if my understanding is wrong here.

[tholor]

The use case is straight forward: Imagine you have a search bar somewhere on a website. Naturally, some people will do keyword queries (e.g. "address deepset"), others will ask natural questions ("What is the address of deepset?").
In the case of keyword queries we just want to call the retriever (as the reader would be a waste of resources), in the case of natural questions we want to go the full mile and include the reader / generator.
With the above DSL, you would need two different search bars or something like a checkbox in the UI to determine what type of query to trigger. That is not feasible from a UX perspective in most cases. That's why we want a very simple, light-weight query classifier (no BERT).

Besides that use case, we have plans to determine in more detail what type of question it is (e.g. one directed to text documents or tables or SQL DB) and route it accordingly in our DAG.

Does this make sense for you?

[lalitpagaria]

With the above DSL, you would need two different search bars or something like a checkbox in the UI to determine what type of query to trigger. That is not feasible from a UX perspective in most cases.

I was suggesting to use single search bar but with keywords support. Like searching in Kibana.

Naturally, some people will do keyword queries (e.g. "address deepset"), others will ask natural questions ("What is the address of deepset?").

How about user want to use reader / generator but typing queries in non natural question ways?

Yes I understand the need of QueryClassifier but it should have capability to override default nature, ie user able to call reader/generator with non-natural queries and call retriever only for natural question type queries.

[tholor]

I was suggesting to use single search bar but with keywords support. Like searching in Kibana.

How would you separate a keyword query from a natural language one then?

How about user want to use reader / generator but typing queries in non natural question ways?

I think we have a misunderstanding here. The QueryClassifier will just be an optional node that you can put in your pipeline DAG. Every user is of course still free (and encouraged!) to build their own custom pipeline. The standard ExtractiveQAPipeline won't include this node and I personally believe that we will see many variants of QueryClassifier nodes and people will start to implement their own depending on their "categories of queries".

[lalitpagaria]

How would you separate a keyword query from a natural language one then?

I can have reserve keywords (keyword, question, generate, summarise, classifier, etc) which then understand by QueryParser. So UI will pass search string to parser and it then have (filter support as well like Kibana). By default parser assume all queries as keywords queries.

• "keyword: " -> Parser will treat it as keyword query
• "question: " -> Parser will treat it as natural question query
• "keyword: AND summarise: all" -> Parser will run summarisation after keyword query
• "keyword: AND summarise: each" -> Parser will run summarisation for each docs retrieved by keyword query separately
• "question: AND summarise: all" -> Parser will run summarisation after keyword query
• "question: AND summarise: each" -> Parser will run summarisation for each docs retrieved by keyword query separately
• "keyword: AND generate: all" -> Parser will run generator after keyword query
• "keyword: AND generate: each" -> Parser will run generator for each docs retrieved by keyword query separately
• "question: AND generate: all" -> Parser will run generator after question query
• "question: AND generate: each" -> Parser will run generator for each docs retrieved by question query separately
• "keyword: AND generate: all AND query_docs:10 AND generate_docs:2" -> Parser will run summarisation after keyword query retrieve up to 10 docs and generator will generate 2 docs.
• "classifier: " -> Parser will check with classifier to determine type of query and then execute it

Ultimately it will create one DSL (Lucene can be tweaked for this), which QueryParser will understand and parse it and then execute it. Treat Generator, Summariser, Reader, Classifier etc as different optional components of the Haystack and on production user/company can add/remove based on need via new Pipeline APIs (Similar to elasticsearch plugins/add-ons) at runtime.

Anyway I understand it might not be aligned with current goals. But yes I do understand the need of QueryClassifier.

[stefanondisponibile]

hello @lalitpagaria and thank for your support on the issue. I don't completely understand the use case of your last comment, but I think what you're trying to say is what you actually said: "_I was suggesting to use single search bar but with keywords support. Like searching in Kibana._" / "_How about user want to use reader / generator but typing queries in non natural question ways?_". This last one is quite interesting by the way, and an interesting feature for the classifier.

However, to understand why we feel the need of a QueryClassifier imagine the following situation:

Our index may look something like this:

[
  {
    "id": 1,
    "text": "Haystack is an end-to-end framework for Question Answering & Neural search that enables you to do a lot of fancy things."
  },
  {
    "id": 2,
    "text": "Haystack is cool."
  }
]

Now, let's discuss the problem before rushing to the obvious solution. And of course, let's assume we _don't_ have a QueryClassifier yet. Let's start by pretending we don't even have _Haystack_ at all, just a .json in your filesystem.

Some user reaches our search bar and types in the following query:

What can I do with Haystack?

Alright, we have tranformers or some model to extract the answer from the query. So we load it and run the query against each of the contexts in our index:

[
{
    "id": 1,
    "context": "Haystack is an end-to-end framework for Question Answering & Neural search that enables you to do a lot of fancy things.",
    "question": "What can I do with Haystack?" ,
    "answer": "do a lot of fancy things",
    "accuracy": 0.548
  },
{
    "id": 2,
    "context": "Haystack is cool.",
    "question": "What can I do with Haystack?" ,
    "answer": "cool",
    "accuracy": 0.326
  }
]

So we're happy, we keep all the records with a satisfying accuracy (let's assume 50% here), order them accordingly, and give back the best one to the user:

What can I do with Haystack? => do a lot of fancy things

---

However, the user types in another query:

Haystack

Alright, let's run this weird query against our docs:

[
{
    "id": 1,
    "context": "Haystack is an end-to-end framework for Question Answering & Neural search that enables you to do a lot of fancy things.",
    "question": "Haystack" ,
    "answer": "Question Answering & Neural search",
    "accuracy": 0.326
  },
{
    "id": 2,
    "context": "Haystack is cool.",
    "question": "What can I do with Haystack?" ,
    "answer": "cool",
    "accuracy": 0.319
  }
]

Well, 🤔 I don't know, what should I do with this? Having to judge, I feel that the document with id: 2 would be more relevant here, because it's shorter and the "Haystack" token has a major impact on it (i.e., _to me_ the second document has a bigger _score_).

Haystack => ????

---

So, we have 2 problems at least:

1. We can't scale: we get slower and slower each time we index a new document, because we have to pass all the documents to our model (i.e. my _Reader_).
2. The user is not requesting something in a semantic fashion, so we're spending useless computation (and time) trying to figure out how to extract an _answer_ from our _context_, using something that's not even a _question_. Odd, isn't it? 🤯

Elasticsearch to the rescue. 🦸

We're having a coffee with our colleague, complaining about our scaling problem, and he comes up with a nice solution.

Why not putting some search filter on top of you're current flow, so each time you will pass just a few docs to your Reader?

Elasticsearch sounds like the perfect solution for that! We throw away our json-database and index everything inside Elasticsearch. This let us also separate two concerns that were living in our model before: _searching_ vs _extracting_.

So the new plan is using a _Retriever_ to gather some documents related the user query (and rely just on that for our search) and then passing those results to a _Reader_ that will extract the proper answer from that subset of our index.

Cool! We solved problem 1, and our structure is more robust and lean now!
Yet problem 2 is still quite an issue 🤔 I mean, yeah, we can do something at the end of our flow, like having a look at our results, and if all the Reader score are bad maybe just returning the document having the best Elasticsearch score, but again...odd 🤷‍♂️

So we take another coffee with our colleague, and he goes like:

Well, in those cases, just skip the Reader.

So that's not a bad idea, pretty obvious, we could actually "skip the Reader", but our problem is identifying "_those cases_".

So @tholor opens his browser, goes to Github and tells:

"Hey, we need a QueryClassifier to identify those cases in which the search style could (optionally) be influenced by the fact that the user is (or is not) expressing semantically. This may influence the search Pipeline in many ways, surely by performing dynamically the answer extraction done by the Reader."

Some issues around this feature:

1. The classifier _itself_ is the core of this issue. (let's forget about Pipelines for a second)
2. The classifier must be fast and not computationally intensive. (that's probably the main problem to face)
3. The classifier should be multilingual.

I've been trying to create a dataset taking the "Quora Question Pairs" and using Spacy to craft a "keyword" version of each question. To give you an example from it:

What is Morse code? 1
Morse code  0
What is the best backend for my app?    1
best backend app    0

Not bad, as a start, but I still haven't come to any _lean_ solution. Moreover, unless having some tricks applied to the classifier, this wouldn't be multilingual.

Back to Pipelines: Binary/Dynamic Node.

Whatever you may call it, even if we _had_ a classifier, there's another feature we would need: a Pipeline node to handle the result of the c̶l̶a̶s̶s̶i̶f̶i̶e̶r̶ callable and use it to point the Pipeline in the right direction of the graph. I think we could already submit a PR for this, @Threepointone4 kinda proposed it already, so along its idea, as a sketch:

class BinaryNode:
    def __init__(self, node_a: str, node_b: str, evaluate: Callable):
        self.node_a = node_a
        self.node_b = node_b
        self.evaluate = evaluate
        self.outgoing_edges = 2
        ...

    def run(self, **kwargs):
        a = self.evaluate(**kwargs) # evaluate function returns a Bool
        return kwargs.get("query"), self.node_a if a else self.node_b

This opens some ideas about having the "evaluate" function itself defining the next node id, but this approach would have some drawbacks to consider. However, thing is the evaluating function (or classifier) is injected, and that's it. If you want to skip the reader or whatever else based on the time of the day, from the starwars api or whatever function you desire, the user can handle that himself. That could also work as a middle-ground solution until a proper QueryClassifier is implemented. When it is, it will just inherit from the BinaryNode, as a specialized version of it.

---

What do you think?

[lalitpagaria]

@stefanondisponibile thank you detailed analysis. I agree with you. 👍

You have brought many interesting points.

[tholor]

@stefanondisponibile Awesome analysis / explanation! Very nice!

A few comments:

The classifier itself is the core of this issue. (let's forget about Pipelines for a second)
The classifier must be fast and not computationally intensive. (that's probably the main problem to face)
The classifier should be multilingual.

I totally agree with all of these points :+1:

I've been trying to create a dataset taking the "Quora Question Pairs" and using Spacy to craft a "keyword" version

I like the direction of this as it could be easily extended to other QA datasets out there. Another potential source for your training data might be information retrieval datasets containing (mostly) keyword queries.

Whatever you may call it, even if we had a classifier, there's another feature we would need: a Pipeline node to handle the result of the c̶l̶a̶s̶s̶i̶f̶i̶e̶r̶ callable and use it to point the Pipeline in the right direction of the graph. I think we could already submit a PR for this, [...]

Maybe I am missing something here, but we already have such functionality in Haystack. @tanaysoni and I discussed the initial design already quite intensively as they are a lot of options to implement such a "branching" in a DAG.
What we settled on, for now, is somehow similar to what you sketched above - maybe a bit simpler though.
This node would already work as of today in the Pipeline class:

    class QueryClassifier():
        outgoing_edges = 2

        def run(self, **kwargs):
            if "?" in kwargs["query"]:
                return (kwargs, "output_1")

            else:
                return (kwargs, "output_2")

    pipe = Pipeline()
    pipe.add_node(component=QueryClassifier(), name="QueryClassifier", inputs=["Query"])
    pipe.add_node(component=es_retriever, name="ESRetriever", inputs=["QueryClassifier.output_1"])
    pipe.add_node(component=dpr_retriever, name="DPRRetriever", inputs=["QueryClassifier.output_2"])
    pipe.add_node(component=JoinDocuments(join_mode="concatenate"), name="JoinResults",
                  inputs=["ESRetriever", "DPRRetriever"])
    pipe.add_node(component=reader, name="QAReader", inputs=["JoinResults"])
    res = p.run(question="What did Einstein work on?", top_k_retriever=1)
    print(res)

The key is really the classifier that would replace the naive if-condition from above.

[stefanondisponibile]

Thank you @tholor, and thanks for the catch on those datasets, _do you have any particular link or resource I could check_?

---

About your doubt, I'm probably overengineering then with that _BinaryNode_ thing. I'll maybe make a PR to discuss the issue directly there without polluting this argument :)

Hopefully I'll make a PR for the RequestClassifier too, as soon as I can find a solution that's lean enough.

[tholor]

do you have any particular link or resource I could check?

I did not have anything particular in mind, but maybe one of those is helpful:

• https://trec.nist.gov/data.html (collection of datasets)
• https://trec.nist.gov/data/million.query09.html
• https://www.kaggle.com/c/yandex-personalized-web-search-challenge/data
• https://webscope.sandbox.yahoo.com/catalog.php?datatype=l&guccounter=1

In any case, I think it would be great of getting a "first version" out there even if the accuracy is not perfect in the beginning. We can always iterate on the datasets (and potentially invest with some labeling power from our side). The design of the implementation and choice of model is currently more important, I believe. I don't know if you had already any thoughts here, but besides a fast inference speed it would be also great to not introduce crazy new dependencies in Haystack. The usual suspects that I see: lightweight NN (PyTorch) or some tree based method (sklearn). My gut feeling would be that a very simple model can already give us a pretty decent solution if it picks up on some standard words/patterns that indicate a question (What, where, How much, Is, Are ...).

[Threepointone4]

Hi Sorry for late response.
I have done simple solution for this as first pass. I have done simple comparison for this if the word starts with wh or certain words its question.

The question_check in second pass can be extended to a model. I am training a simple model will share soon.

@tholor what do you think ?

[tholor]

@Threepointone4 Yes, such a rule-based system could be a first, simple starting point. I can also see that we have multiple classes in the end. Maybe one "RuleQueryClassifier" and one "ModelQueryClassifier" - so people can choose.

Regarding your implementation: you are currently only returning a string in run(). You would need to make this a tuple of (query, out_put)