Datasets: Add SQA

[NielsRogge]

Adding a Dataset

• Name: SQA (Sequential Question Answering) by Microsoft.
• Description: The SQA dataset was created to explore the task of answering sequences of inter-related questions on HTML tables. It has 6,066 sequences with 17,553 questions in total.
• Paper: https://www.microsoft.com/en-us/research/publication/search-based-neural-structured-learning-sequential-question-answering/
• Data: https://www.microsoft.com/en-us/download/details.aspx?id=54253
• Motivation: currently, the Tapas algorithm by Google AI is being added to the Transformers library (see https://github.com/huggingface/transformers/pull/8113). It would be great to use that model in combination with this dataset, on which it achieves SOTA results (average question accuracy of 0.71).

Note 1: this dataset actually consists of 2 types of files:
1) TSV files, containing the questions, answer coordinates and answer texts (for training, dev and test)
2) a folder of csv files, which contain the actual tabular data

Note 2: if you download the dataset straight from the download link above, then you will see that the answer_coordinates and answer_text columns are string lists of string tuples and strings respectively, which is not ideal. It would be better to make them true Python lists of tuples and strings respectively (using ast.literal_eval), before uploading them to the HuggingFace hub.

Adding this would be great! Then we could possibly also add WTQ (WikiTable Questions) and TabFact (Tabular Fact Checking) on which TAPAS also achieves state-of-the-art results. Note that the TAPAS algorithm requires these datasets to first be converted into the SQA format.

Instructions to add a new dataset can be found here.

[thomwolf]

I’ll take this one to test the workflow for the sprint next week cc @yjernite @lhoestq

[NielsRogge]

@thomwolf here's a slightly adapted version of the code from the official Tapas repository that is used to turn the answer_coordinates and answer_texts columns into true Python lists of tuples/strings:

import pandas as pd
import ast

data = pd.read_csv("/content/sqa_data/random-split-1-dev.tsv", sep='\t')

def _parse_answer_coordinates(answer_coordinate_str):
  """Parses the answer_coordinates of a question.
  Args:
    answer_coordinate_str: A string representation of a Python list of tuple
      strings.
      For example: "['(1, 4)','(1, 3)', ...]"
  """

  try:
    answer_coordinates = []
    # make a list of strings
    coords = ast.literal_eval(answer_coordinate_str)
    # parse each string as a tuple
    for row_index, column_index in sorted(
        ast.literal_eval(coord) for coord in coords):
      answer_coordinates.append((row_index, column_index))
  except SyntaxError:
    raise ValueError('Unable to evaluate %s' % answer_coordinate_str)

  return answer_coordinates


def _parse_answer_text(answer_text):
  """Populates the answer_texts field of `answer` by parsing `answer_text`.
  Args:
    answer_text: A string representation of a Python list of strings.
      For example: "[u'test', u'hello', ...]"
  """
  try:
    answer = []
    for value in ast.literal_eval(answer_text):
      answer.append(value)
  except SyntaxError:
    raise ValueError('Unable to evaluate %s' % answer_text)

  return answer

data['answer_coordinates'] = data['answer_coordinates'].apply(lambda coords_str: _parse_answer_coordinates(coords_str))
data['answer_text'] = data['answer_text'].apply(lambda txt: _parse_answer_text(txt))

Here I'm using Pandas to read in one of the TSV files (the dev set).

[lhoestq]

Closing since SQA was added in #1566