Azure-docs: QnA Maker Train API behavior is undocumented

[CeciAc]

@hansmbakker commented on Mon Feb 11 2019

QnA Maker apparently has a Train API, but it is not documented at https://westus.dev.cognitive.microsoft.com/docs/services/5a93fcf85b4ccd136866eb37/operations/5ac266295b4ccd1554da75ff.

It is not documented what the expected response should be.

The link to the Train API on https://docs.microsoft.com/en-us/azure/cognitive-services/qnamaker/how-to/improve-knowledge-base is a sample, not documentation.

Furthermore, in the Google search cache I found that the Train process is run only after 50 feedback entries are received but this is listed nowhere on https://docs.microsoft.com/en-us/azure/cognitive-services/qnamaker/how-to/improve-knowledge-base.

---

Documentdetails

⚠ Dit gedeelte niet bewerken. Het is vereist om problemen te koppelen tussen docs.microsoft.com en GitHub.

• ID: e302f635-c2b9-e505-7b9e-d947dab75514
• Version Independent ID: 2c1e8216-c9d5-68bd-cbca-6c80dcb2f2b3
• Content: Verbetering van knowledge base - QnA Maker - Azure Cognitive Services
• Content Source: articles/cognitive-services/QnAMaker/How-To/improve-knowledge-base.md
• Service: cognitive-services
• GitHub Login: @diberry
• Microsoft Alias: diberry

---

@hansmbakker commented on Mon Feb 11 2019

Screenshot from Google:

I see that this links to an old page (which redirects to qnamaker.ai now) but it seems that feedback items still are not picked up immediately.

Also I do not see any change in the qnamaker portal or in the confidence scores of the question I'm giving feedback on using POST requests to the Train API.

---

@hansmbakker commented on Mon Feb 11 2019

Also it is not clear if calls to the Train will result in a suggestion being created in the QnA Maker portal that still needs to be accepted by a moderator, or that the model will automatically be retrained and published.

[CeciAc]

Hi team,
I'm moving this here since the user is reporting missing information in the documentation

best regards

[YutongTie-MSFT]

@CeciAc Thanks for the feedback! I have assigned the issue to the content author to investigate further and update the document as appropriate.

@diberry Hi Dina, could you please help with this document issue? Thank you.

[diberry]

in-progress

[diberry]

@tulasim88 @YutongTie-MSFT @CeciAc

The conceptual documentation doesn't document APIs. That should be referred to the service team to update the swagger (or whatever their method is).

For the issue about active learning needing manual steps, I'll get that fixed.

As for the information about how many requests have to happen before suggestions appear in the portal, I need guidance on what the doc should say from the service team.

[sshradha]

Hi @CeciAc
I am also interested in QnA Maker's Train API feature, in fact I would like to implement it in our Production Bot's QnA Maker KB.
But there are few queries which are not clear to me related to Train API:

1. Is there any Train API here --> https://westus.dev.cognitive.microsoft.com/docs/services/5a93fcf85b4ccd136866eb37/operations/5ac266295b4ccd1554da75ff . I cant find it.

2. I am interested in understanding the implementation of below Train API in our bot’s nodejs code. If below shows the request, what is the expected response?
   

3. What is userId : 1 ?

4. “Question-text” --> I believe is the user query/utterance entered by user for qnaId : 1, is this understanding correct?
5. Is this implementation available here https://github.com/Microsoft/BotBuilder-Samples/tree/master/experimental/javascript_nodejs/qnamaker-activelearning-bot , again I could not find it.

[iMicknl]

Besides that the documentation and API reference isn't complete yet, I am also hitting the issue where suggestions aren't showing up in the qnamaker.ai portal. The API response is a status 204, but nothing shows up in the portal.

[hansmbakker]

@iMicknl for me they usually do show up after some time, but it is unclear whether they take a certain time to appear (is there some background job that runs periodically? or does it just take some time to process?), or whether the same train call needs to be made several times (as the google cache result suggests).

[iMicknl]

@hansmbakker you are right, after +1 hour it did show up in my QnAMaker. Documentation on this field would be really helpful.

I believe the article you mentioned from Google Cache is for the old QnAMaker version (3), where active learning wasn't built with suggestions, but directly after x people gave the same feedback.

[YutongTie-MSFT]

@iMicknl @hansmbakker Hi Mick and Hans, thanks for your feedback. I highly recommend you to create a new ticket for this issue. It would help a lot! Thanks in advance ^^

[hansmbakker]

@YutongTie-MSFT excuse me, why is this current issue not good enough? Your own colleague @CeciAc moved this issue here in the first place. Part of my question is about whether it still takes 50 Train calls for a suggestion to show up, and in a more general sense, how the suggestion behavior works.

I believe the suggestion behavior is related to the Active Learning behavior and Train behavior and I feel it should be documented in the same effort.

[iMicknl]

@YutongTie-MSFT please refer to the right place to create a ticket. Would it be in BotFramework-Services, since QnAMaker doesn't have a repo to track issues.

[YutongTie-MSFT]

@hansmbakker Hi Hans, sorry if I make you confused. ^^ What I want to say is if you have suggestions other than this post, please open a new ticket. This ensures your suggestions will be covered. Of course, if you now feel that this post has covered all your ideas, it is perfect.

[hansmbakker]

If this issue is enough to document the following:

• the Active Learning behavior in general
• the Train REST call
  
  
  
  • what parameters are there?
  
  
  • what are the inner workings?
  
  
• the suggestion behavior
  
  
  
  • on what conditions does a suggestion show up?
  
  
  • after what time does a suggestion show up?
  
  
  • are suggestions auto-generated or do they only come from Train calls?
  
  
  • are they also generated during normal use from GenerateAnswer queries?
  
  

then it covers the various questions that are asked. I believe all of this is related, so it can be documented in a related way, too.

[sshradha]

@iMicknl @hansmbakker Hi Mick and Hans, in my QnA Maker suggestions did show up 4-5 days after enabling Active learning i.e. on last friday 22nd feb.
7 suggestions were shown and all were kind of garbage. I had rejected all. Now suggestions are not appearing since last friday. No idea when these will reappear again. Our bot is being used continuously since then. So QnA Maker API is also being hit continuously.

[tulasim88]

adding @diberry and @prakul1991

thanks for the feedback @hansmbakker @sshradha - we are working on improving the documentation or active learning so that the logic for active learning, suggestions and train API are together and more transparently explained.

@hansmbakker - to answer you specific question- it no longer takes 50 train API calls to make suggestions show up.

@sshradha - we are continually improving the suggestion generation process. if suggestions are not showing up- this is probably because the questions users are asking are not generating top N answers that have similar confidence scores. Just calling QNA Maker API is not reason for the suggestions to show up.

Hope this helps and let us know if you have other questions.

[YutongTie-MSFT]

@CeciAc
We will now proceed to close this thread. If there are further questions regarding this matter, please respond here and @YutongTie-MSFT and we will gladly continue the discussion.