Compromise: big size library version with big list of names, places & English words for use in Offline Apps/extension

[wis]

I would like to use this library in my extension and app, which will be optimized for a good experience after installation and not payload size.

it is not detecting "White House" as a place, and many more cases..
at the end of the day, you need a list of the names and places to work correctly close to 100% of the time.
is there a possibility to have a "Offline version" of this library that can be several Mega bytes in size. not just the 100 Kilo byte library.

[spencermountain]

hey Wesam, that's great. Yeah, this is possible using a custom lexicon. You can make it any size you want, and tag 'white house' as a Place, or whatever. You can see the existing known words here

we're working on a new version that lets you compress your own lexicon, like we do internally. It should be ready in a few weeks

[wis]

Thanks Spencer @spencermountain.
this video is fascinating!
the issue is that according to zipf's law, few words are used so often and so many words are used rarely. and you need to have a list of them all to work correctly 100% of the time, I imagine the perfect complete uncompressed list would be a 1 or few hundred mega bytes in size in memory with JS vm's UTF-16 encoding (16 bits/2 bytes per character).
this is not a problem for my case.
now I gotta make the lists and start testing.

[spencermountain]

:smile: yeah! zipfs was the whole reason i started this project. It's the best. Our internal wordlist is something like 16,000 words, once the conjugations are all blasted-out and everything. It's still way-smaller than jquery, uncompressed.

[spencermountain]

would love some help getting it right at the best trade-offs for filesize. Take a peek around and see if there's anything that can be done better. As I said, work's underway to expose our word compression library in the api, so people can throw fat wordlists into the web-browser without busting anything.
you sound like my type of dude!

[owendall]

:+1: for exposing the word compression library in the API.

[wis]

neato! great compression ratios, and the projects it's based on are very interesting also.
check out SMAZ maybe there's anything to learn from it.
I would be glad to contribute if I can in any way.

[spencermountain]

agh, this library is a thing of beauty. Thanks for sharing it. Just getting back from holidays.

so it looks like it compresses common substrings into integers, using a custom mapping. - This is not my expertise - chars are 16 bits each? but you can get 8 bit integers with that Uint8Array thing?
is there a clever way to serialize it for going over the web? json? .. back to a string?

[wis]

The Uint8Array typed array represents an array of 8-bit unsigned integers (0-255). [0]
Both UTF-8 and UTF-16 are variable length encodings. However, in UTF-8 a character may occupy a minimum of 8 bits, while in UTF-16 character length starts with 16 bits. [1]
UTF-16 uses a minimum of 2 bytes/16 bits. This makes it incompatible with ASCII. [1]
Javascript engines use UCS-2 or UTF-16 internally. [2]

Yeah, these common substrings were generated by a ruby script from English corpuses and the English Wikipedia, according to the original author in a HackerNews thread.
we can make a better list, I don't think ".com" or " compromise only supports English, optimal encoding for size savings is plain old ASCII in a Uint8Array.
serializing compressed text to json defeats the purpose I believe. what do you think about Protobuf?

[0] - Uint8Array - MDN
[1] - Difference between UTF-8 and UTF-16? - StackOverflow
[2] - JavaScript’s internal character encoding: UCS-2 or UTF-16?

[spencermountain]

hey - i've been thinking of another thing I wanted to bounce off you.
A lot of people are using compromise to parse a big document, and then search it for a list of known words - like, here's my huge list of cities, which ones are in this doc..

i wanna build some kind of word index. right now we (i know..) are just looping through each term in the document.

1. the words change (the index becomes dirty)
2. we're not matching sub-string, just full-on str===word
3. it's gotta support multiple-word sequences (like 'toronto raptors')

i was looking at trying to make aho-corisack algorithm per-word, instead of per-character, but couldn't wrap my head around it.
any ideas?

[wis]

you should try SMAZ with your abbrevs list and see the resulting compression ratios,
why not have an application specific English encoding method like ascii but in place of the terminal characters like ╝ ╟ have the abbrevs list (I know they are 2-3 chars long, that's the point).

i'm sure you are stronger than me in algorithms, i'm not tbh, but i'm thinking about it.

[spencermountain]

hey, I looked into SMAZ and found the trie-stuff in efrt was more suited to our purposes. @wesamco it's a little rough still, but you can see how it's possible now to ship a big lexicon with more metadata in v11, as a plugin

lemme know if you see anything weird
cheers