Libelektra: Elektra-web 1.3

[markus2330]

improvements

:+1: new instances design
:+1: "play" button to directly configure instances is great
:+1: enums work really great

errors

• [x] there should be a error message when connecting to a not-running elektrad (and not only when trying to navigate)
• [x] view&dismiss opens a pop up on the very top, although the button is in the very bottom?
• [x] errors with binary keys e.g. when navigating into "user/tests" and "user/kdb-tests/umlauts", from "import" in blobs.libelektra.org
• [x] when clicking on an namespace with not-working (binary) keys, I get the error: "Error: only absolute urls are supported" (see #1858)

bugs

• [x] duplicating keys buggy:
  
  
  
  1. create test
  
  
  2. duplicate test -> testCopy
  
  
  3. duplicate test -> error, rename to testCopy2
  
  
  4. you suddenly get testCopy2Copy
  
  
• [x] or when duplicating a key, and giving it another name, also a Copy gets added

accessibility

• [x] tab not usable in tree view?
• [x] Elektra-web selecting text is not possible on firefox bug #1851

searching

• [x] when creating a key in a not-yet-opened namespace, the key is not found using the search, even if "reloading" was pressed

types

• [x] when changing type of key, the message does not clearly say if data will now be lost or not.
• [x] when changing from any -> boolean, the value is discarded invalidly: it is neither 0 or 1 but empty string.
• [x] also regex should make sure that key is according the regex
• [x] validation should only allow characters that are useful/possible in current type/regex

installation

• [x] It should be kdb run-webd and not kdb run-web?
• [x] cleanup all npm files for make uninstall, see old issue #1244
• [x] Please support KDB env var
• [x] please check for Elektra version and warn users if Elektra is too old

interface

• [x] flickering when moving over keys, very bad when too many keys are shown :cry:
• [x] visibility: >= placed wrongly. It should be in instances, not in metadata. So a key is e.g. important, and instance visibility ">= important" shows the key, ">= critical" would not show the key.
• [x] keys which are too deep in hierarchy get white and flicker
• [x] instances do not remember how the tree was open before
• [x] placement of elements in tree view could be optimized (see images/bugs/placement.png in blob.libelektra.org)

use cases

• [x] undo/redo
• [x] copy&paste does not work with chromium (nor Firefox): when copying a key (green + is shown) it is actually moved. Pressing ctrl does not help.
• [x] not possible to copy/move into a key which does not have subkeys

earlier discussions see: #1857

metadata

• [x] readonly -> restrict/write
• [x] implement other restrict/*

KeySet features

• [x] null keys do not look different to empty keys
• [x] improve usability for arrays (indexing of #0, #1, ...)

testing

• [x] test all features on Firefox ESR

[markus2330]

@omnidan It looks much better now! There are, however, still several bugs -- some related to usability and validation.

[omnidan]

It should be kdb run-webd and not kdb run-web?

I am using the tool name here, which is web. My idea was that later on elektrad might be replaced by a separate project, which would only leave kdb run-web for this project.

improve usability for arrays (indexing of #0, #1, ...)

What do you mean by indexing of #0, #1, ...?

null keys do not look different to empty keys

How should they look like? I could set the grey hint text to (null), what do you think?

cleanup all npm files for make uninstall, see old issue #1244

I checked this and it seems like the whole tool_data and tool_exec folders get completely removed on make uninstall (that includes the npm files). However, the kdb command does not uninstall cleanly on my system. It might have to do with the homebrew version and the compiled version conflicting in some ways, though (I did uninstall the homebrew version first).

[omnidan]

@markus2330 is there a difference when getting null keys with kdb get? It seems to produce the same output as empty keys:

> kdb set user/null
Create a new key user/null with null value
> kdb get user/null

> kdb set user/empty ""
Create a new key user/empty with string ""
> kdb get user/empty

[markus2330]

I am using the tool name here, which is web. My idea was that later on elektrad might be replaced by a separate project, which would only leave kdb run-web for this project.

Okay.

What do you mean by indexing of #0, #1, ...?

Sometimes configuration settings do not have a (useful) name, they simply enumerate things. In such situations Elektra has the array convention (see src/libs/elektra/keyname.c or https://www.libelektra.org/ftp/elektra/publications/raab2017context.pdf page 96).

So when pressing "+" on a key which is an array (only has #... as subkeys), you do not need to ask for a name (increment the number and prefix with one more _ if the number of digits increased) and can visualize the #0 more nicely.

See also qt-gui, which implements this feature. I would, however, prefer not to get an extra button. Simply suggest that the user can pick #0 as name if an array is wanted. And then the + button automatically increments the numbers when adding new array members. So this should not clutter the interface at all but help people creating an array a lot.

How should they look like? I could set the grey hint text to (null), what do you think?

Yes, making it gray is an excellent idea.

However, the kdb command does not uninstall cleanly on my system. It might have to do with the homebrew version and the compiled version conflicting in some ways, though (I did uninstall the homebrew version first).

Could you please create a separate issue for this problem?

For me there are also left-overs, I added them to #1627. They seem to be unrelated to Web-UI, though.

is there a difference when getting null keys with kdb get? It seems to produce the same output as empty keys:

I added 38b8a5e8cdca75b7268196ad3fc8ea6fcbe2d183 so that you can check the differences but the output is not suitable for the Web-UI.

To make it easier for the Web-UI, simply check for the metadata "binary". If it is make the value gray and read-only. (Without having metadata restrict/write, when clicking on the value, simply say that web-UI currently does not support changing binary keys.) Null keys are binary keys, so you automatically handle them without adding extra support.

I added "support for changing binary keys" to #1849.

[markus2330]

Btw. but removing binary keys and making binary keys non-binary (by unchecking "binary" in their Metadata dialog; the value can be discarded with a warning then) should work.

I updated the text in the list above.

[omnidan]

@markus2330 I am now requiring elektra 0.8.22 or higher (does the 0.8.22 release already contain the fix or should we bump the version to 0.8.23 on master and require that?)

[markus2330]

Yes, please require 0.8.23. I pumped the version number in c01c7d02e2c6b573ba76ac516c45eb8a86abdf06. (Please rebase if you already have an branch.)

Please also write release notes because the web-ui will be one of the highlights of the 0.8.23 release. Ideally you guide people through a use case how to use the web-ui. (e.g. editing /etc/hosts)

[omnidan]

errors with binary keys e.g. when navigating into "user/tests" and "user/kdb-tests/umlauts", from "import" in blobs.libelektra.org

when clicking on an namespace with not-working (binary) keys, I get the error: "Error: only absolute urls are supported" (see #1858)

@markus2330 I have no idea what to do about these issues... There are two issues here:

1. sometimes when executing the kdb command binary keys are not parsed properly
2. some keys are invalid with the yajl plugin: Yajl parser error: lexical error: invalid character inside string.

Both problems would be fixed with proper JavaScript bindings, but we are still quite far from that. I don't know any workarounds either. Maybe we could find a workaround for the first issue, but then we would still have to adjust the yajl plugin (and probably even the yajl library itself) to support these keys.

[markus2330]

Yes, you have somehow lost if you try to export it with JSON.

I think it would also be okay if you do not display the value of a binary key. Important is that the behavior is defined (nothing weird happens). Binary keys are always present, e.g. system/elektra/modules/dump/exports/get. We do not want to surprise users if they navigate there.

Is it possible for you to avoid getting the keys? (If you see that they have meta data binary, do not try to get them?) That would be the easiest solution.

If this does not work, we could make the JSON plugin to base64 encode binary values (like done by @sanssecours in YAML). Unfortunately the kdb export currently does not respect the contract of the plugin... (similar to problem #1178)

[omnidan]

@markus2330 the problem is that the key name already has problematic values in it. So unfortunately it won't make a difference if I get the value or not. I think this is something that will have to be fixed in the YAJL plugin.

BTW: I tried navigating to system/elektra/modules/dump/exports/get and it seems like the YAJL plugin already properly escapes binary values, so it works fine. Only binary key names (like user/tests in the real-world config) are problematic.

[markus2330]

I see. I was confused because you referred to binary keys but this problem has nothing to do with binary keys. In key names null bytes are prohibited anyway. But all other characters are allowed (1-255).

How do you encode (escape) key names via the API? Where exactly are the problems and which characters are problematic?

[omnidan]

@markus2330 The first issue has to do with key names that cause problems when executed in the kdb command:

Command failed: "kdb" lsmeta -0 "user/tests/+�+M  $0�1\"s�=

�sd�y2 =  \"�[sec�iy1 ct{on`3"
/bin/sh: -c: line 2: unexpected EOF while looking for matching ``'
/bin/sh: -c: line 3: syntax error: unexpected end of file

Or:

Command failed: "kdb" lsmeta -0 "user/tests/$M  $0�1\"s�=

�3"
Did not find key

The second issue has to do with key names that yajl cannot parse (also some keys in user/tests):

Yajl parser error: lexical error: invalid character inside string.

Are you sure the user/tests keyset was exported properly? It looks quite broken in general:

[markus2330]

problems when executed in the kdb command
The second issue has to do with key names that yajl cannot parse

I mean that you document exactly which characters work and which not (and which ones fail at which layer). There are only 255 characters, so it is not so difficult to find out.

And you also did not describe how your (http/rest) API works with unusual characters.

These are important implementation challenges. If you can prove that your parts work and only the CLI/JSON plugin has problems we might be able to live with some limitations (to be fixed with the elektrad reimplementation).

Without any thoughts about the encoding (which is needed if you use REST and JSON) it is only luck if you see in KDB and WebUI the same characters. I added the task "describe encoding" to "API blueprint". (Btw. your API differs here completely with the snippet API. For the snippets, only scope and slug are encoded within the URL and not complete Elektra paths.)

Are you sure the user/tests keyset was exported properly?

In the dump files the content is written down literally, I do not see how it could be wrongly exported. In the INI files there might be problems because INI might also interpret some characters. Thus I added both dump and INI files. (It would be a bug though, please report if you find some differences in the INI and dump files.)

[markus2330]

Btw. one way to demonstrate that the web-ui works properly would be to mock elektrad to returns a tree with all characters.

Btw2. how are newlines rendered in the web-ui? It might be not a good idea to actually have a new line within the key name or value.

Btw3. Some kind of automatic testing of elektrad and the web-ui would be very much appreciated. (At least some curl tests against elektrad.)

[omnidan]

@markus2330 I debugged the issue and the problem seems to be the way newlines get passed on to the kdb command. I am trying to figure out a way to escape bash command arguments with newlines, but I haven't found a solution yet.

The API is treating newlines fine, as newlines in values are no problem. But newlines in key names cause the kdb command to contain newlines, which fails.

[omnidan]

Even when I replace the newlines with escaped newline characters, bash still fails executing the kdb command...

Command failed: \"kdb\" lsmeta -0 \"user/tests/+�+M  $0�1\\\"s�=\\n\\n�sd�y2 =  \\\"�[sec�iy1 ct{on`3\"
/bin/sh: -c: line 0: unexpected EOF while looking for matching ``'
/bin/sh: -c: line 1: syntax error: unexpected end of file

[omnidan]

Alright, I figured it out - newline characters (\n) and backticks (`) needed to be escaped.

The real-world config loads fine without errors now.

I added the task "describe encoding" to "API blueprint".

The encoding for key names is simply URL encoding, which is a standard. The encoding for other data is JSON. Both formats should support all characters that elektra does (and even all utf-8 characters). The problem was not in the encoding of the API, it was the way it got executed in the shell (which was interpreting newlines and backtick characters).

[markus2330]

Looks fine here, I added new issues to #1892