Mycroft-core: Consolidated proposal for adding metadata to dialog files

[ChanceNCounter]

Addresses #1865 and #2293

What:

The proposal would add whitespace-delimited metadata to the ends of dialog lines, separated from the dialog line itself by carat ^:
this is a line of dialog^attribute1=value attribute2=value

Why:

To quote Jarbas in chat,

if i'm not missing anything we are discussing 3 features enabled by this metadata scheme

• gender value for grammatical correctness
• atitude value, weights for this would be global (i really wanna say increase sarcasm by 20%)
• individual line weights (after atitude selection)

How:

MustacheDialogRenderer's existing logic would be extended to parse the file for lines matching the desired attributes. I've looked at a few different ways of doing this, and I think recursive, broader searches, dropping the "lowest value" attribute each time, are sadly the easiest route.

In this case, that would likely mean that Mycroft would first look for a line matching the device's grammatical gender and its attitude, plucking random lines until it runs out. After that, it would look for a line matching the device's grammatical gender and default/no attitude. Failing that, it would revert to the existing logic and pull random lines with no regard for metadata.

I worry about the computational consequences if, and only if, several such attributes are added and dialog files start cropping up with hundreds of lines. I stress tested the existing renderer when recent_phrases was added, and it's lightning for any reasonable values of max_recent_phrases. Still, this system does run the risk of parsing entire dialog files n + 1 times for n attributes, and there's further multiplicative complexity within the loop. Right now, we're talking about kilobytes, so who cares, but... yeah...

---

Regarding "individual line weights," that was in reference to easter eggs that get obnoxious if they fire too often. There'd be a few ways to do it, but I think the most straightforward would be to shove them in a dict that works like recent_phrases, phrases being the keys, and their values representing a countdown until the phrase gets "unlocked." Each time a line is rendered, the values in that dict would decrement, and any keys with the value 0 would be removed from the dict, allowing the parser to choose them for rendering again.

In that case, 'weight' might be the wrong term. The attribute would instead reflect a "cooldown" before the phrase is spoken again. Also, I should point out that, because the dict I describe would be per dialog renderer, the cooldown would not guarantee that any other phrases from the same .dialog file are spoken in between instances of the weighted line. It would only guarantee a minimal number of interactions with Mycroft before the line is spoken a second time.

It also lives in memory, so it remains possible that a user hears your easter egg on their last skill invocation before shutdown, and again on their first invocation after startup.

---

I propose, and will likely prototype during the break, adding the metadata immediately, and simultaneously adding attitude weights to mycroft.conf, along with an intent for tweaking those values verbally. Increase sarcasm by 20%, indeed! That's at least three of us now who are tickled by that reference 👨‍🚀

Lingua Franca has a part to play in handling grammatical gender, so I think that should probably be added to core at the time of the switch. That way, Lingua Franca can report whether the configured language is gendered, what (if any) default gender it uses, and any other information which might be relevant to the problem.

[ChanceNCounter]

ping: @JarbasAl, @KathyReid, @ftyers in case I've missed something or messed it up.

[ChanceNCounter]

I should also acknowledge the obvious alternatives:

1) Read lines into a data structure, representative of the various attributes, then hit that with random.choice(lines where all(attribute in line.attributes for attribute in desired_attributes)). This massively decreases the maximum runtime, but massively increases the minimum runtime.
2) Parse the file, discarding invalid lines as we go, and return the first one that meets all criteria. Same hypothetical minimum runtime - 1 loop over 1 line - but the minimum runtime for certain attributes, in the case of well-organized dialog files, becomes 1 + the number of lines without those attributes that appear earlier in the file. In this case, pulling random entries is likely to be more performant on average. (edit: it also limits the number of unique lines that could satisfy a particular set of criteria; any in excess of max_recent_phrases just wouldn't be reached.)
3) Cache the files in the first place. As of today, this would be fine. As the skill repo grows, it would become ruder and ruder to eat up limited RPi memory. Right now, most of us only have a few dozen dialog files on our computers, but someday (we hope!) we'll all have hundreds or thousands. Or maybe not, if some other form of language processing enters the mix, but I don't wanna write code on the assumption that it can suck because it'll be obsolete. That's how two-digit years became an epoch problem unto themselves ;)

[forslund]

Sounds very interesting. Some thoughts after reading (and since I haven't read the actual discussions yet I might be completely off base)

Did you consider creating a new file format for dialog files using json or yaml to define the dialogs and meta data. The benefit would be that we'd use a standard format and don't need to add additional code for parsing the variables. This would also make in easier to differentiate an old style simple dialog from a new style dialog. And it would be trivial to create a small tool to help write the dialog files allowing to specify the attributes etc.

Can some things be done at load time like dropping non-matching gender lines? This would require triggering reload of dialogs if the gender of the system is changed...

One thing to consider as well is the translation process. How will the scripts generating data for the translate server need to be altered to handle this? Sounds like the metadata need to be kept in the string or the translators need to be made aware of the specifics surrounding the string...

[ChanceNCounter]

Re: creating a new format, the objective was to avoid breaking changes. This proposal was born after Jarbas and I shared with each other colliding proposals to use filenames as metadata.

Re: dropping non-matching lines at load time, I'm not opposed, but we'd need a tmpfile per dialog file, yes?

I didn't realize there was any machine translation taking place, I defer to others.

[forslund]

Well keeping the meta data like that will sort of be a breaking change, old versions of Mycroft-core would interpret the metadata as normal dialog and we'd be forced to make this a major release only thing (It's not too long to 20.02 though so that could possibly be fine). But having a new file format living alongside the old could be better...We'd still support the old format and load .dialog files like we do now, using defaults for the metadata. While the new format (.dyaml or something) is parsed using the newer code.

I don't think we'd need tmp files, just load it into memory like we do now. if memory becomes an issue we could do in memory compression of some sort. (but to be honest this isn't the first place I'd be optimizing to improve the memory consumption of mycroft-core)

There isn't any machine translation, it's just us monkeys on typewriters doing the actual translation but there are a scripts transferring the dialog and vocabs to and from the translation system.

[ChanceNCounter]

Well keeping the meta data like that will sort of be a breaking change, old versions of Mycroft-core would interpret the metadata as normal dialog and we'd be forced to make this a major release only thing (It's not too long to 20.02 though so that could possibly be fine

Quite! I hadn't thought about old versions of Mycroft-core parsing new .dialog files. Indeed, when I was thinking about breaking changes, I was thinking about the new system's compatibility with old dialog files, lacking metadata.

having a new file format living alongside the old could be better...We'd still support the old format and load .dialog files like we do now, using defaults for the metadata. While the new format (.dyaml or something) is parsed using the newer code.

Couldn't hurt performance-wise, but, to quote Jarbas again,

one other thing i like in this approach is that it is stupidly simple for newcomers to take advantage off, one of our objectives is AI for everyone, including non technical folks

imo, it should come down to whatever's easiest for people to get their heads around if their first Mycroft skill is their first real coding experience. json and yaml are great at what they do, but there's a lot of syntax to worry about.

[forslund]

I agree that we need to keep it as simple as possible and keep the original basic format around as well as a gateway drug. A yaml implementation could look like this:

this is a dialog:
  sarcasm: 10
  speaker_gender: male
  listener_gender: female
  extra: douglas adams fan
this is a second dialog:
  sarcasm: 0
  speaker_gender: male

Resulting in this python dict

which wouldn't be much more syntax than the ^ separation with spaces. The issues may be when using colons and quotation marks.

[krisgesling]

To me, the new yaml format seems like the cleanest method for adopting this meta data. I won't rehash what's been said, but also thought that as we've adopted yaml for other aspects of Skills - settings and manifest, it's already a format that authors are likely going to need so this keeps more consistency in that respect.

[JarbasAl]

quotes from a chat with @forslund

---

i'm wondering a bit how to make that the most human friendly possible, i think a list with entries of this sort would be easy to understand
{"utterance": "XXX", "atitude": "sarcastic", "gender": "male", "weight": 0.3}

so on load it would be filtered by gender, falling back to a lang default / male because thats mycrot default voice

EDIT: to clarify, if gender is missing we assume gender==default, there isn't a case where gender is not set, some value for "no gender" can however be specified /EDIT

then filtered by attitude according to @ChanceNCounter 's proof of concept , and finally weight would be taken into account to have some dialog lines more likely to be chosen than others

my only doubt at this point is how to handle previous change to avoid repeating dialogs, i think weight kinda deprecates that functionality

id say to keep it for .dialog files and simply ignore it in the new format

and what extension should we use? i don't like .json with it being in the locale folder it's confusing
.dialog.json maybe?

[JarbasAl]

i personally prefer json over yaml , would be nice to hear the preferences of non-developers

[ftyers]

I oppose yaml files in general. I would also like to repeat my request that Mycroft stick to a validateable, standard list of Feature=Value pairs for linguistic information and not make up ad hoc categories. I prefer @JarbasAI's proposal, but note that using .json is far less human writeable than the proposal that @ChanceNCounter came up with originally. Note that in Universal dependencies features can be subscripted like Gender[speaker]=Masc and Gender[listener]=Fem.

[JarbasAl]

this example sums up the categories we need, everything else being an extra with no clear use case yet
{"utterance": "XXX", "atitude": "sarcastic", "gender": "male", "weight": 0.3}

NOTE: gender == device gender

@ftyers while i like the way you are thinking, it seems to not solve our issue and to impose artificial limitations, i would like for skill authors to be able to add ad hoc categories for personal use if they like

[ftyers]

@JarbasAI i don't see how having consistent Feature=Value pairs for linguistic information is an artificial limitation, it's a coding style and a compatibility issue. I'm not saying that people shouldn't be able to make up their own arbitrary categories and values. But the ones that refer to linguistic information should be consistent and standard.

I would make it:

 {"Utt": "XXX", "Attitude": "Sarcastic", "Gender[device]": "Masc", "Gender[listener]": "Fem", "Weight": 0.3}

I would also be happy with:

 {"Utt": "XXX", "Features": ["Attitude=Sarcastic", "Gender[device]=Masc", "Gender[listener]=Fem"], "Weight": 0.3}

[JarbasAl]

the only linguistic related value here is the gender value, everything else is more related to a personality than language, can you please share some relevant links to these Feature=Value pairs?

[ChanceNCounter]

my only doubt at this point is how to handle previous change to avoid repeating dialogs, i think weight kinda deprecates that functionality

I'm not sure. I don't think we want to weight every phrase, or even most phrases, but it would still be nice to keep simple little skills from repeating dialog over and over. I've already uploaded one skill that consists of nothing but boilerplate, and there's no reason Mycroft needs to prefer "Okay" to "Whatever you say" when you tell him off.

I was thinking about it after I commented earlier, and I thought this might work:

cooldown_phrases = {"easter egg": 1, "another thing": 3}
recent_phrases = [3_ago, 2_ago, 1_ago]
...
def render():
  ...
  locked_phrases = recent_phrases + cooldown_phrases.keys()

and then use locked_phrases instead of recent_phrases in the existing logic.

It's also worth mentioning that weights for individual phrases are surely the least important bit here =P

[ftyers]

@JarbasAI From this list, potentially relevant features are:

• NounClass: In languages with NounClass instead of Gender, e.g. Bantu languages
• Animacy: In languages with a distinction between animate and inanimate, is Mycroft animate or inanimate?
• Polite: In languages with Politeness distinctions, which one should be used?
• Clusivity: In languages with inclusive/exclusive plurals, should inclusive or exclusive be used?
• Number: Are we talking about addressing one person or many?
• Evident: What kind of information are we transmitting? First hand or non-first hand? -- or other?

My point is that, this is a fairly easy thing to use, doesn't require more programming, is extensible and is inclusive and thoughtful in terms of language diversity.

[ChanceNCounter]

Also, if we're moving to markup, I wanna +1 yaml. I prefer JSON myself, but non- (read: new) developers will be much more comfortable with whitespace-delimited data, and yaml dictionaries are 100% whitespace.

Of course, if they're learning to develop skills, they'll need to learn all kinds of syntax that's much more complicated than JSON, but it's another prerequisite to getting their feet wet. I'd love it if Mycroft skills were a budding developer's first foray. Right now, you can develop a skill that doesn't need settings, but it's hard to develop a skill that doesn't need dialog.

[ftyers]

@ChanceNCounter some languages use whitespace to separate syllables instead of words, e.g. Vietnamese. Would it then be necessary to come up with hacks to represent Vietnamese? How might yaml work for Vietnamese?

[JarbasAl]

@JarbasAI From this list, potentially relevant features are:

* NounClass: In languages with NounClass instead of Gender, e.g. Bantu languages

* Animacy: In languages with a distinction between animate and inanimate, is Mycroft animate or inanimate?

* Polite: In languages with Politeness distinctions, which one should be used?

* Clusivity: In languages with inclusive/exclusive plurals, should inclusive or exclusive be used?

* Number: Are we talking about addressing one person or many?

* Evident: What kind of information are we transmitting? First hand or non-first hand? -- or other?

My point is that, this is a fairly easy thing to use, doesn't require more programming, is extensible and is inclusive and thoughtful in terms of language diversity.

this makes things clearer, it does complicate a bit the dialog selection, i guess we need to decide which values we want to account for in core, and which values should be only for developers to consume, while all those are very good to have, taking them all into account for selection is unpractical, if only because we don't have a mechanism yet to know how to use those

the politeness thing is also very relevant for portuguese

[ChanceNCounter]

@ChanceNCounter some languages use whitespace to separate syllables instead of words, e.g. Vietnamese. Would it then be necessary to come up with hacks to represent Vietnamese? How might yaml work for Vietnamese?

For these purposes, only leading whitespace matters, and line breaks.

[ftyers]

@ChanceNCounter so whitespace does not separate values? like in @forslund's example?

[ftyers]

this makes things clearer, it does complicate a bit the dialog selection, i guess we need to decide which values we want to account for in core, and which values should be only for developers to consume, while all those are very good to have, taking them all into account for selection is unpractical, if only because we don't have a mechanism yet to know how to use those

the politeness thing is also very relevant for portuguese

Couldn't something like a maximum intersection between the Feature=Value pairs set in the device and the Feature=Value pairs in the dialogue be used to select between competing utterances? --- in addition to the weight that you mention.

[ChanceNCounter]

@ChanceNCounter so whitespace does not separate values? like in @forslund's example?

I believe that's one value, {"extra": "douglas adams fan"}

[adocampo]

I prefer YAML above JSON because I use it a lot with ansible, docker-compose files, and the so, but I must admit YAML is quite tricky at the beginning and more error prone.

YAML have a ton of very cool features like anchors, serialization, _comments_, recursive structures, or even embed a JSON inside a YAML...

[JarbasAl]

id rather have json fail loudly than yaml errors slipping silently

[JarbasAl]

@ftyers am i understanding correctly these features are at the word level, i'm not sure if they extrapolate well for a full sentence, it's also important to be sure we are not mislabeling things if they don't fit within that spec

EDIT: again, this metadata is mostly related to a personality, not exactly linguistic features

[JarbasAl]

one more thing i like about json is the direct correspondence to a python dict, that way files could easily be loaded for custom handling of dialog without surprises (not using self.speak_dialog)

in a follow up PR we can think about also allowing devs to influence dialog selection trough some extra arguments in speak_dialog

(sorry for triple posting)

[ftyers]

@JarbasAI they can apply at the level of speech act too, which is what the dialogue files represent no ?

[JarbasAl]

related PR that i closed https://github.com/MycroftAI/mycroft-core/pull/1874

this discussion replaces the approach in that PR, just commenting to keep a link between PRs here