Plots2: Call for related search terms

[shubhscoder]

So recently, we implemented a feature wherein the search for a particular term at publiclab.org/search also returns results for the terms that are related to it. #4823 is the relevant issue and #4848 is the solution for the same.

However for the solution to work we have used a dictionary file here - just a simple text file. This dictionary would use a variety of domain specific words and thus we need everyone's help in setting this dictionary properly.

Following types of words are expected to go into the dictionary :

1. Hyphenated terms when for corresponding non-hyphenated ones. For example, we could have the following pair :

purpleair: purple-air

1. Related terms, we often use short forms or abbreviations for commonly used terms. But the search engine most of the time misses out on this abbreviations. So we should construct a dictionary of such abbreviations along with the actual corresponding word. For example, this could be one of the pairs:

h2s: Hydrogen Sulphide

Thank you so much!

• https://github.com/publiclab/plots2/blob/9d36bd7a1c9a285d48b9025da563037507001217/app/services/search_criteria.rb#L46
• https://github.com/publiclab/plots2/blob/984066aa8825883f28078b22c091897dc62e25c8/lib/text_search.rb
• https://github.com/publiclab/plots2/blob/984066aa8825883f28078b22c091897dc62e25c8/lib/related_and_hyphenated_terms.dict.txt

[shubhscoder]

So @jywarren, do we want people to submit their words by making Pull requests ? Or is there some other way?

[grvsachdeva]

Nice issue @shubhscoder! How about asking contributors to comment the related terms in issue's comments and we can add them weekly/monthly?

[shubhscoder]

Yes we could do that @gauravano. Also I have one more idea, how about posting a new wiki on public labs and ask people to comment the related terms there?

[grvsachdeva]

We can also ask people to edit the wiki?

[shubhscoder]

Yeah that sounds good. We can do that!

[jywarren]

I think possibly @ebarry suggested this?

voc: volatile-organic-compund

Others I can think of are:

discussion-list: mailing-list
mailing-list: discussion-list
listserv: mailing-list
dust: pm
pm: dust
particulates: dust
tds: total-dissolved-solids
datalogger: data-logging

[ebarry]

event: meetup
meetup: event

formaldehyde: HCHO
HCHO: formaldehyde

formaldehyde: CH2O
CH2O: formaldehyde

ag: agriculture
agriculture: ag

agriculture: farming
farming: agriculture

are these pairs not reversable, as in, do i have to write the pairs in the other direction as well?

[shubhscoder]

I think even the reverse is required.

On Thu, Mar 7, 2019, 7:19 PM Liz Barry [e-mail] wrote:

event: meetup
formaldehyde: HCHO
formaldehyde: CH2O
ag: agriculture
agriculture: farming

are these pairs not reversable, as in, do i have to write the pairs in the
other direction as well?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/publiclab/plots2/issues/4928#issuecomment-470531847,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ARofxTHZGgPmmKwj42hO2qgodXTy3oaJks5vURjsgaJpZM4bhaVG
.

[ebarry]

alright, well let's use this issue to grab quick thoughts, and then maybe we set up a spreadsheet somewhere with a formula to generate all the reverse forms of the pairs?

cafo: factory-farm
factory-farm: cafo

cafo: feedlot
feedlot: cafo

confined-animal-feed-operation: cafo
cafo: confined-animal-feed-operation

sewage: wastewater
wastewater: sewage

salinity: conductivity
conductivity: salinity _(do we agree with this?)_

[shubhscoder]

Yeah that sounds good👍

On Thu, Mar 7, 2019, 8:21 PM Liz Barry [e-mail] wrote:

alright, well let's use this issue to grab quick thoughts, and then maybe
we set up a spreadsheet somewhere with a formula to generate all the
reverse forms of the pairs?

cafo: factory-farm
cafo: feedlot
sewage: wastewater
salinity: conductivity

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/publiclab/plots2/issues/4928#issuecomment-470555431,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ARofxfq6j0TRB0sNt3qW2dqAv-YKt2hdks5vUSd8gaJpZM4bhaVG
.

[jywarren]

We may want to do reverse in some cases but not others, so let's keep doing it manually. Some terms might be related to broader terms but we wouldn't want the reverse.

valverde: val-verde

[jywarren]

data-logging: datalogging
data-logging: data logging

[jywarren]

Added all these in a batch: #5503

Noting that we're not sure about how to do multiple word additions; for example:

h2s "Hydrogen Sulphide" <--- liz thinks this is "sulfide"
data-logging "data logging"
cafo "factory farm"

So I'm asking over here: https://github.com/yohasebe/lemmatizer/issues/7

[ebarry]

urban-planning: urban-design
urban-design: urban-planning _(happy to discuss as once upon a time this was my field, but pragmatically think these should be joined for Public Lab)_

soil: dirt
dirt: soil

community-engagement: communityengagement
communityengagement: community-engagement

marine: water
water: marine

[ebarry]

ewb: engineers-without-borders
engineers-without-borders: ewb

pole-aerial-photography: pap
pap: pole-aerial-photography

global-climate-change: climate-change
climate-change: global-climate-change

urbanwater: urbanwaters
urbanwaters: urbanwater

[jywarren]

Noting that as of the latest https://github.com/yohasebe/lemmatizer we can now have spaces in the right-hand terms:

https://github.com/yohasebe/lemmatizer#resolving-abbreviations

Thanks to @yohasebe!!!!! ❤️

[ebarry]

colorimetry: colorimetric
colorimetric: colorimetry

colorimeter: colorimeters
colorimeters: colorimeter

[ebarry]

hot-spot: hot-spots
hot-spots: hot-spot

microscope: microscopes
microscopes: microscope

PS Is there a way to automatically relate a word with the same word that just has an "s" on the end? Asking for a friend!

[ebarry]

plastic: plastics
plastics: plastic

radiation: radioactive
radioactive: radiation

superfund-site: superfund
superfund: superfund-sites

spectra: spectrum
spectrum: spectra

spectra-lines: spectra
spectra: spectra-lines

spectral: ???????

plume: plumes
plumes: plume

[ebarry]

diagram | diagrams | diagramming <-- how to associate a cluster?

noise: sound
sound: noise

[ebarry]

booklet: publication
publication: booklet

[ebarry]

filter: filtering
filtering: filter

filter: filtration
filtration: filter

filtration: filtering
filtering: filtration

[jywarren]

Hi Liz, i believe all of these will be covered by lemmatization, thanks!

On Tue, May 21, 2019 at 12:04 PM Liz Barry [e-mail] wrote:

filter: filtering
filtering: filter

filter: filtration
filtration: filter

filtration: filtering
filtering: filtration

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/publiclab/plots2/issues/4928?email_source=notifications&email_token=AAAF6JYMFEZ2KDA77NZFDDLPWQMO5A5CNFSM4G4FUVDKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODV4MM7Q#issuecomment-494454398,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAF6J4P46JXO77AD3LSGUDPWQMO5ANCNFSM4G4FUVDA
.

[ebarry]

fossil-fuel: oil-and-gas
oil-and-gas: fossil-fuel

[jywarren]

Closing this as we are mainly running on google search now! Thank you!

[ebarry]

eu: europe

[ebarry]

ok great -- i have moved the eu: europe set over to https://github.com/publiclab/plots2/issues/4928 which is where all the related search terms were being aggregated. Since that list is so huge, i suggest we either just keep adding on there, or move to a google spreadsheet.

[jywarren]

And, for purposes of migrating tags, I suggest NOT doing all of these, but
choosing only those which we know there is content for and which we want to
take this big action on. But this is a good source to skim to ID such tags!

On Wed, Nov 20, 2019 at 11:55 AM Liz Barry [e-mail] wrote:

ok great -- i have moved the eu: europe set over to #4928
https://github.com/publiclab/plots2/issues/4928 which is where all the
related search terms were being aggregated. Since that list is so huge, i
suggest we either just keep adding on there, or move to a google
spreadsheet.

—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/publiclab/plots2/issues/4928?email_source=notifications&email_token=AAAF6J2ZGVSZ6AP6MIVKVITQUVTXVA5CNFSM4G4FUVDKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEESWJEI#issuecomment-556098705,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAAF6J4HZOSEPLMDOHY4TD3QUVTXVANCNFSM4G4FUVDA
.