Spacy: vocab["-0.23"].like_num is False

[phdowling]

Pretty straight forward, it seems like negative numbers are currently not flagged as numbers in the Lexeme object.

[sakatipomu]

Yes. I am also facing the same issue but with token objects. Negative numbers are tagged as PUNCT Did you find any work around for this?

text = "My readings are -4.72, 4.72, -3.45 and 4.05"
text_nlp = nlp(text)
for token in text_nlp:
    print token.i,token.text,token.pos_,token.tag_

0 My ADJ PRP$
1 readings NOUN NNS
2 are VERB VBP
3 -4.72 PUNCT :
4 , PUNCT ,
5 4.72 NUM CD
6 , PUNCT ,
7 -3.45 PUNCT :
8 and CCONJ CC
9 4.05 NUM CD

[DuyguA]

I checked is_num feature for you:

https://github.com/explosion/spaCy/blob/master/spacy/lang/en/lex_attrs.py

As far as I see, minus sign in front is not parsed. @ines what do you say?

[phdowling]

FYI, a leading "+" also is not parsed. Maybe both should just be included?

[DuyguA]

Sure, I meant no "sign bit" is included in parsing :wink: We can skip the initial sign character during the parse.

[sakatipomu]

I already tried like_num attribute in token class, but it is not working. Currently I am doing this

is_num=""
try:
    float(token.text)
    is_num = True
except ValueError:
    is_num = False

[ines]

This would be good to handle, yes. I'd propose the following:

def like_num(text):
    if text.startswith('+') or text.startswith('-'):
        text = text[1:]
    # rest of the function

[DuyguA]

Btw @ines , startswith can take a tuple as argument when I first found about it, I instantly fall in love :heart: So this is possible:

def like_num(text):
  if text.startswith(('+', '-')):
    text = text[1:]
 #rest of the func

[ines]

Ahh, nice! That's even better. I'm just writing some tests for this for all languages that implement like_num, so we can test that it works as expected.

[phdowling]

What about possibly also handling "~5", etc? Also, "±1" ?I know we're getting into the grey area here, and this isn't very high priority, but those kind of cases might also be interesting. Of course at some point, it's probably up to the user to just handle this or use a regex for the use case. + and - are definitely a good improvement in the general case though.

[DuyguA]

From my side, why not... additions can go in a similar fashion. Then most probably we'd like to do:

def like_num(text):
  polarity_signs = ('-', '+', '~', '±') #More can come here
  if text.startswith(polarity_signs):
    text = text[1:]
 #rest of the func

[ines]

Sure! Just tested it and it seems to work as expected – it was only a case of adding more characters to the startswith check.

I also noticed that the tokenizer was currently always splitting the + prefix. This was a problem, because it meant that a token +123 could never exist. I changed it to only split if the next character is not 0-9. All existing tests pass, so I hope there aren't any unintended side-effects of this change.

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.