Haystack: Error while fine-tuning Q/A model

[bipinkc19]

Question
While using the prescribed annotator and trying to fine-tune the model I keep on getting this error.
This is the annotator I used from haystack
https://annotate.deepset.ai/

While following this tutorial

https://colab.research.google.com/github/deepset-ai/haystack/blob/master/tutorials/Tutorial2_Finetune_a_model_on_your_data.ipynb

In this line
reader.train(data_dir="sample_data", train_filename="answers.json", use_gpu=True, n_epochs=10, save_dir="my_model", num_processes=4, dev_split=0.01)

Additional context
Here is the json file I wan't to fine tune on just for the proof of concept. It is very small size file with only a few questions.

{
  "data": [
    {
      "paragraphs": [
        {
          "qas": [
            {
              "question": "What is the range of investment they are looking to invest?",
              "id": 14292,
              "answers": [
                {
                  "text": "Our direct investments range from €250k to €15m ",
                  "answer_start": 6662,
                  "answer_category": null
                }
              ],
              "is_impossible": false
            },
            {
              "question": "What are the investor’s target sectors?",
              "id": 14293,
              "answers": [
                {
                  "text": null,
                  "answer_start": null,
                  "answer_category": "NOT_GIVEN"
                }
              ],
              "is_impossible": false
            },
            {
              "question": "Do they invest directly, with partners or other means?",
              "id": 14294,
              "answers": [
                {
                  "text": "We invest in businesses directly, but we also co-invest alongside a selection of hand-picked funds who share our sector focus",
                  "answer_start": 6160,
                  "answer_category": null
                }
              ],
              "is_impossible": false
            },
            {
              "question": "Are the investors majority or minority investors or both?",
              "id": 14295,
              "answers": [
                {
                  "text": "We’re generally an active minority shareholder with a seat on the board, but we’re open to majority stakes ",
                  "answer_start": 6487,
                  "answer_category": null
                }
              ],
              "is_impossible": false
            },
            {
              "question": "What's the name of the organization?",
              "id": 13981,
              "answers": [
                {
                  "text": "UNILEVER VENTURES ",
                  "answer_start": 0,
                  "answer_category": null
                }
              ],
              "is_impossible": false
            }
          ],
          "context": "UNILEVER VENTURES \r\nExplore
THE BEST OF BOTH WORLDS
We have the global reach of Unilever
THE BEST OF BOTH WORLDS
We have the agility of a venture capital team
THE BEST OF BOTH WORLDS
We scale businesses and grow strong brands
WE ARE UNILEVER VENTURES
AGILITY WITH SCALE
Unilever Ventures is the venture capital and private equity arm of Unilever. We invest in young, promising companies, accelerating growth by providing access to Unilever’s global ecosystem, assets and expertise. We’re looking for tomorrow’s world-beaters in Personal Care and Digital. ACCESS TO UNILEVER
We provide access to a company with unrivalled global reach. On any given day, two billion people use Unilever products to look good, feel good and get more out of life. With more than 400 brands focused on health and wellbeing, no company touches so many people’s lives in so many different ways.
Unilever Ventures works closely with The Unilever Foundry to provide technology innovators with access to paid pilot projects, mentorship and growth financing. \r\nUNILEVER VENTURES AT A GLANCE Founded in 2002
€450 million under management
Over 40 direct investments \r\nHundreds more via equity partners
Operations in Europe, North America and Asia
3rd fund launched in 2014 focused on Personal Care and Digital UNILEVER GROUP AT A GLANCE
More than 174,000 employees
Turnover of €53.3 billion in 2015 with products sold in over 190 countries Uniquely positioned in emerging markets (57% of the business)
Annual Brand and Marketing investment budget of more than €8bn
#1 Corporate Sustainability Leader for 5th year in a row
Around €1 billion invested every year in R&D
Item 0
Item 1
Item 2
Item 3 \r\nItem 4
Item 5
Item 6
Item 7
\"It's time for the brand to go to the next level. Unilever is a great company with great principles; it has a profound understanding of brands and global reach; it makes the perfect partner to help REN fulfil its future potential worldwide\" \r\nANTONY BUCK, CO-FOUNDER, REN SKINCARE
\"As one of the largest marketing technology organizations in the world, it has been amazing to work with Unilever building a multi-year and increasingly strategic and broad relationship. Unilever Ventures has been instrumental in helping share our vision with the key stakeholders in Unilever.\"
JAMES GROSS, PRESIDENT & CO-FOUNDER, PERCOLATE
\"Unilever Ventures combine the strategic and financial benefits of a solid venture capital firm with the unique ability to leverage Unilever, helping us to tune our proposition so that we could appeal to large consumer goods companies.\"
JOHN KEARON, BRAINJUICER, CEO & FOUNDER
\"UV’s operating model allows small companies to combine their entrepreneurial spirit with strong financial discipline.\"
BRENDAN HARRIS, CHAIRMAN & CEO, FROOSH AB
\"Unilever Ventures has been an important partner in helping us grow and nurture our vision through capital infusion and access to a global network of experts and resources\"
NAVEEN BHAVANISHANKAR, FOUNDER, B.LAB
\"Unilever Ventures has been critical to the difficult decisions a rapidly growing company faces.\" BRYAN BRISTER, CEO, VOLTEA
\"Unilever Ventures has been an ideal partner for us at this critical time of growth for our brand. They believe in our mission and they are committed to helping us reach our potential, while maintaining our rigorous safety standards. Their desire to share their extensive industry knowledge has already proved invaluable.\"
HILLARY PETERSON, FOUNDER & PRESIDENT, TRUE BOTANICALS
\"Unilever has been a fantastic investor for us. They have been incredibly supportive, providing insights and feedback where we need them, but giving us the space to stay focused on our mission. It's a great relationship.\"
CHRISTINA MACE-TURNER, CO-FOUNDER & CEO, TRUE BOTANICALS
WHAT WE DO
EXPERTISE THAT COUNTS
Voted most influential Corporate Venture fund in the consumer sector, Unilever Ventures brings together an experienced and flexible team of investment and operational experts with a proven track record of creating market-leading consumer and tech companies. We play to our strengths and value people for theirs.
WE CAN HELP...
DEVELOP A BRAND
We fundamentally understand brands. We know how to build, nurture and protect them. \r\nBUILD COMPETITIVE EDGE
As part of a global CPG ecosystem, we can fine-tune a business’s strengths. TURBO-CHARGE A BUSINESS
We have access to substantial in-house Unilever resources: a coach, a local expert, even a transformational client. Unilever is the 2nd largest advertiser in the world.
STRENGTHEN THE FINANCIAL BASE
As a committed partner; we are able to provide critical follow-on financing to take a business to the next level.
THROUGH UNRIVALLED CAPABILITIES IN:
CONSUMER INSIGHT
Deeper consumer and market understanding through Unilever’s global market intelligence. MARKETING
Improved brand awareness through insight-driven branding, communication and media planning.
SUPPLY CHAIN
Cost savings and best practice across procurement, logistics and manufacturing.
R&D
Strengthened scientific credibility through new technology, formulation capabilities, clinical studies and IP assessment.
LEGAL
Brand protection, and regulatory and legal expertise for international rollout.
DISTRIBUTION
Geographic and channel expansion from introductions in traditional retail, foodservice and drug/pharmacy.
HOW WE WORK
INVESTING WITH A LONG TERM VIEW
We take a long-term view on companies’ performance. We encourage a long-term perspective and motivate key managers by providing them with significant incentives.
INVESTING WITH INTEGRITY
As the venture capital and private equity arm of Unilever, our corporate reputation is of the utmost importance to us. We approach everything we do with integrity.
INVESTING WITH A SHARED GOAL
We work closely with a company to establish a strategic and operating plan to maximise long-term equity value. We always look for ways to leverage the scale of Unilever. We only succeed when you do.
INVESTING WITH FLEXIBILITY
Whilst we are tightly focused on key sectors we understand that different companies have different needs, so we are flexible in the structure, size and stage of our investments.
DIRECT OR PARTNER-LED
We invest in businesses directly, but we also co-invest alongside a selection of hand-picked funds who share our sector focus.
EARLY OR LATER \r\nWe invest in early and later stage businesses ranging from new growth capital to pre-exit funding. We mainly look for a proven model – seed investments are rare.
MAJORITY
OR MINORITY \r\nWe’re generally an active minority shareholder with a seat on the board, but we’re open to majority stakes – on our own or in collaboration with partner funds.
LARGE OR SMALL
Our direct investments range from €250k to €15m with a sweet spot of €8m to €10m. Significantly larger investments can be considered for highly strategic opportunities. GLOBAL OR LOCAL \r\nWe primarily invest in Europe, North America and Asia. However, we will continue to invest elsewhere if the opportunity is right. \r\nWHAT WE'RE LOOKING FOR
PASSION, INNOVATION, DISRUPTION
Great companies led by outstanding, passionate people – unless we’re confident that a team can deliver, we won’t invest.
Genuine innovation – brands that create defendable, sustainable, profitable positions. Differentiated businesses with clear opportunities for global expansion that can disrupt or compete with market leaders in our target sectors.
Companies that could benefit from Unilever’s global reach, market and consumer knowledge and its capabilities.
PERSONAL CARE
We’re looking for an authentic brand story which fuses beauty with superior technology and draws upon advances in science or the riches of nature to provide a great consumer experience.
We are focused on premium opportunities in cosmeceuticals/nutraceuticals, professional skincare, natural skincare and haircare, beauty devices, hair loss and scalp health, male grooming and digitally enabled beauty businesses. We also look for ground-breaking technologies that could play a role in new innovations in Unilever personal care brands. DIGITAL
Digital technologies are transforming nearly every aspect of the consumer journey. Unilever Ventures is seeking investments in dynamic, digitally-powered marketing businesses which can allow companies like Unilever to deliver a better return on their marketing investments and engage more deeply with consumers - at-shelf, on-line, and on-mobile.
Key focus areas include content creation, data analytics, ecommerce, Internet of Things, mobile marketing, social media, and video in both developed and emerging markets.
PORTFOLIO
Allpersonal caredigitalrefreshmentsustainable livingother
ALL \r\nPERSONAL CARE DIGITAL REFRESHMENT SUSTAINABLE LIVING OTHER \r\nIndirectExited Personalised gut health \r\nThe leading provider of eCommerce intelligence. DIRECT?EXITED? \r\nDelivers groceries at your doorstep, every morning, free of charge Automating Mom & Pop stores in Emerging Markets
Happy skin needs happy bacteria – Skincare for the microbiome \r\nCruelty-free, long-lasting cosmetics to sweeten your life
SaaS creative management platform for digital advertising
Changing the way women choose, use, buy and apply their make-up \r\nHealthy cooking made easy: Organic and non-GMO ingredients & delicious recipes delivered weekly \r\nNutraceutical Supplement for Hair Loss and Regrowth Results without toxins \r\nThe grocery delivery service that delivers in as little as an hour! \r\nLimitless is an innovative Crowd ServiceTM platform that is transforming customer service forever by connecting customers with skilled people who can help them on-demand. \r\nAugmented reality mobile game, gamifying consumer and brand engagement \r\nBeauty for Humanity – pure, rare, and impact-sourced luxury skincare \r\nThe Consumer Connection Video Platform \r\nLeading provider of home water filtration systems for health conscious consumers. DIRECT? \r\nPioneers and develops better delivery forms for proteins & biological therapies. \r\nAffinity purification camelid antibody products. Acquired by Life Technologies in 2013. EXITED? \r\nUK provider of beauty on-demand services \r\nAdvanced Location data technology \r\nA specialist music agency servicing brands and their partner agencies. \r\nA marketing and brand consultancy with market research solutions from behavioural science. EXITED? \r\nInnovative technology group developing world class oxidation catalysts that make processes faster, cleaner and safer. \r\nNovel green chemistry in sustainable sources of biofuels/chemicals. \r\nIPL technology for at-home beauty devices such as permanent hair removal. EXITED? \r\nThe leading premium smoothie brand in the Nordics. EXITED? \r\nA leading player in the rapidly emerging UK ‘recipe box’ delivery market \r\nA regional ice cream & dairy brand in India sold via retail and premium scooping parlours. DIRECT? \r\nHigh tech beauty devices, topicals and intelligent textiles for use at home. Personalised Precision Skincare from France. \r\nMasstige skin care and cosmetics brand using natural arctic ingredients. DIRECT? \r\nBeauty and health product company in China focusing on nutri-beauty. DIRECT? \r\nNative in-video advertising \r\nLeader in institutional water treatment in South India. DIRECT? \r\nTurns consumer generated images into valuable brand assets. EXITED? \r\nGlobal leader in liquid repellant nano-technology that makes electronic gadgets waterproof. \r\nThe System of Record for Marketing, a complete web and mobile software platform to manage marketing in one place. \r\nSpecialty oral and transdermal pharma company. Acquired by Orexo AB in 2008. EXITED? \r\nIron based catalyst to replace cobalt in paint drying. Sold to OM Group in 2011. EXITED? \r\nA social platform that incentivises people to take environmental actions. DIRECT? \r\nDelivering healthy meals and nutritional programs to schools through retail partners. DIRECT? \r\nInnovative water technology with solutions for water purification and softening. \r\nClean Skincare focused on Performance, Purity and Pleasure. \r\nAward winning, premium hair care in a timeless design for professionals and consumers alike \r\nA premium frozen yogurt brand sold in UK supermarkets and own stores. EXITED? \r\nA chain of branded day spas operating in UK, Canada and Spain. EXITED? \r\nAntibody platform for functional food products targeting gastrointestinal infections.
An award winning technology company that wants to change the way the world uses water. \r\nUK’s first natural vitamin water. Acquired by PepsiCo in 2008 (now SoBe V-Water). EXITED? \r\nDevelops tools for water utilities to help customers save water and money. DIRECT? \r\nAn online platform
for recipe searches, nutrition guidance, food shopping, and cooking. \r\nCASE STUDIES
REN SKINCARE
Acquired by Unilever Ventures in April 2015 as part of its strategy to invest in and scale high potential, innovative Personal Care brands.
PERCOLATE
We invested in Percolate in 2013 alongside best-in-class investors with deep expertise in the enterprise SaaS space to help corporations create effective, efficient, and consistent marketing. BRAINJUICER
We invested in BrainJuicer when they had only one founder employee. Our partnership helped the company scale to over 140 employees operating in 11 countries before we exited the business in 2011.
FROOSH
Having acquired a majority stake in froosh, we have re-launched this fruit smoothie brand and business to drive it to number one in the market.
ILUMINAGE BEAUTY
We invested $25m to grow Iluminage Beauty, expanding distribution, developing award-winning new products and refreshing this home beauty device brand.
BRANDTONE
We co-founded mobile marketing company, Brandtone, introducing them to blue chip companies and helping them scale internationally.
INVESTMENT TEAM
Our core team of investment experts are based in London,
Mumbai and New York.
Olivier Garel
Head of Ventures
John Coombs
Managing Director
Andrew Lane
Director
Jan Harley
Director \r\nRachel Harris
Director
Anna Ohlsson-Baskerville
Director
Pawan Chaturvedi
Investment Director, Asia
Stephen Willson
Investment Director
Ian Lane
Investment Director
Amelia Townsend
Investment Associate
Paul Hart
Finance Controller - Direct
Jay Miranda
Financial Controller - Funds
Amanda Clarke
Tax Manager
Lana Base
PA/Office Manager
ADVISERS AND PORTFOLIO COMPANY NON EXEC DIRECTORS
We have a global network of world-class sector and functional experts inside and outside Unilever.
Allpersonal caredigitalother
ALL
PERSONAL CARE
DIGITAL
OTHER
Dr Shimon Eckhouse
NXD, Iluminage
Chairman Syneron Medical
Jack Gallagher
NXD, Iluminage
Ex-President Clarisonic
Vasiliki Petrou
Global SVP Prestige, Unilever
Personal Care & Digital Advisory Boards
Pamela Dickson
VP Finance, Prestige, Unilever
Personal Care Advisory Board
David Wheldon
NXD Brandtone
Mark McClennon \r\nVP ETS, IT, Unilever
Digital Advisory Board
Luis Di Como
SVP Global Media, Unilever Digital Advisory Board Keith Higgins \r\nSVP Customer, Global eCom, Unilever Digital Advisory Board
Aline Santos
SVP Marketing, Unilever \r\nDigital Advisory Board Stan Sthanunathan VP CMI, Unilever Digital Advisory Board Roland Sieker \r\nVP Strategy Foods, Unilever
Digital Advisory Board
Dorcas Lau
VP Digital Marketing & eCom, North Asia, Unilever Digital Advisory Board \r\nFrederic Court
General Partner & Founder, Felix Capital
Graham Blashill
NXD, Froosh
Investor
Dr Alan Smith CBE FRS
NXD Arecor
Ex-Chief Scientific Officer Genzyme
Dr Andrew Richards
NXD Arecor
Biotech Angel Investor
Bryan Morton
NXD VH2
David Greensmith
NXD Catexel
EQUITY PARTNERS
Funds that Unilever Ventures works with include Langholm Capital, L Catterton, Felix Capital, Physic Ventures, ClearVue Partners, Blume Ventures, Capvent, Swander Pace and Ambit. \r\nEVENTS
2018 Start Up Grind Europe Conference
DATE:
13 June 2018
LOCATION:
London
NAME OF CONTACT ATTENDING:
Ian Lane & Jan Harley
MORE INFORMATION
Millennial 20-20
DATE:
13th March – 14th March 2018
LOCATION:
London
NAME OF CONTACT ATTENDING:
Anna Ohlsson-Baskerville
MORE INFORMATION
NEWS
30th May 2018
Trinny London opens doors to London pop-up
22nd May 2018
Milkbasket raises $7 million in Series A funding led by Kalaari Capital
16th May 2018
Thryve Secures $1.4 Million Funding to Expand Personalized Probiotics and Individual Microbiome Testing
9th May 2018
P2i ranks second in Intellectual Property League Table 2018
13th March 2018
Ready Meal Provider Gousto Attracts £28.5m In Backing
1
2
3
4
5
6
7
8
9
10 \r\n11
12
13
14
15
16
17
18
19
CONTACT US
To start the journey, get in touch at [email protected] or on [phone]. LONDON \r\nUnilever Ventures 1st Floor
16 Charles II Street London \r\nSW1Y 4QU
UK
Get Directions
MUMBAI
Unilever Ventures India Advisory Private Ltd Unilever House \r\nBD Sawant Marg
Chakala, Andheri East
Mumbai 400099
India
Get Directions
NEW YORK
Unilever Ventures
940 Sylvan Avenue
Englewood Cliffs
New Jersey
NJ07632
USA
Get Directions
© Copyright 2020 Unilever Ventures Limited \r\n",
          "document_id": 22881
        }
      ]
    }
  ]
}

[bipinkc19]

I keep getting the same error using FARM
https://colab.research.google.com/drive/1tqOJyMw3L5I3eXHLO846eq1fA7O9U2s8

I have downloaded the annotated data in SQUAD format.
What is the problem here?

[Timoeller]

Thanks for pointing this out and also giving a minimal example to replicate this.

The error seems similar to the one reported in #268

We will look more closely into it and report back here.

[Timoeller]

Hey we had a chance to look into it. It really helped having a small failing example.

The problem is coming from multiple angles and it will take some time to fix it. We have included a quick hotfix for you in the gist at the bottom:

1. We included the option to mark unanswerable questions in the annotation tool. But those unanswerable questions are exported in a wrong way in the answer.json. (see answers where answer_category is "NOT_GIVEN"). We are currently fixing this in the label tool
2. FARM is not fail save when loading incorrect QA pairs. Data handling is improved at current FARM master (not version 0.4.6 that is used in haystack). We are improving the handling even more as mentioned here https://github.com/deepset-ai/FARM/issues/484

Quick hotfix to correct the answer.json
To have something working fast we updated a script that should correct the QA pairs for you. You can find it here: https://gist.github.com/Timoeller/be6dfd8e34cdcd84fdca4c4aa72f42fc

[bipinkc19]

Thank you so much!

It would have taken me days to understand the current working mechanism of code and find an answer. It works now.

You guys saved me so much time and effort by such quick replies.

I would also like to request feature where we can label multiple spans as answers for a single question in https://annotate.deepset.ai/.

Cheers

[Timoeller]

No problem, always happy to fix bugs.

About the feature request. Totally useful, I guess you were thinking of Natural Questions style annotations? We put it on the feature list. For now you could create the same question twice to mark multiple occurences. But agreed currently it is not the most efficient way to label, or check these multi answer annotations.

[Timoeller]

Issue seems fixed, closing now.