Bitbar: Configuration for bitbar and plugins

[manojlds]

Read config from something like ~/.bitbarrc. This should also enable plugins to hook into it.

For example, for the jenkins plugin, within ~/.bitbarrc:

[jenkins]
user="user"
password="password"

If not for this approach, some config management will be of use.

[matryer]

How about environment variables?

On 5 Jan 2016, at 19:46, Manoj [email protected] wrote:

Read config from something like ~/.bitbarrc. This should also enable plugins to hook into it.

For example, for the jenkins plugin:

[jenkins]
user="user"
password="password"

If not for this approach, some config management will be of use.

—
Reply to this email directly or view it on GitHub https://github.com/matryer/bitbar/issues/88.

[BrendanThompson]

+1 to a configuration file. Would prefer that to env variables. As can keep conf in dotfiles repo.

[manojlds]

Yeah, prefer config file based approach. @matryer can you add labels to the issues, so that we can triage must have features, so that people know what is needed? And, thanks for the awesome tool :)

[jcaille]

+1 for a configuration file as well !

[matryer]

Guys, an environment file could be a config file like below. What do you think?

plugin.conf.sh

export SETTING1="val"
export SETTING2="val"
export SETTING3="val"

Then in the plugin:

source /path/to/plugin.conf.sh

[jcaille]

This would not really work for plugins that are not shell scripts (for example if you have a #!/usr/bin/python shebang at the top), and would require plugins to manually subscribe to the configuration.

However, sourcing the NSTask in the ExecutablePlugin with this configuration (by setting its environment property) might work, and plugins should be able to access the environment regardless of the language they're using.

With that in mind, i would prefer a conf file that is not a script (just a json file or other file format would be nice). That should make it easier to parse when launching a new task.

[manojlds]

That approach is not suitable for non-shell scripts right. Something that a plugin can feed in to the scripts, say as arguments, would be great. And what BitBar feeds to a particular plugin should be based on some convention for how the configuration is saved.

[manojlds]

@jcaille - +1 to most of what you said. The approach could be similar to what git hooks receive from git. Any kind of script can run them. Many of the inputs are parameters or sent via STDIN.

[keithamus]

How about having a ~/.bitbarrc combined with environment variables?

[myplugin.sh]
token=foobar
[otherplugin.rb]
password=hunter2

myplugin.sh:

#!/usr/bin/env sh
echo $token #foobar

otherplugin.rb:

#!/usr/bin/env ruby
ENV["password"] #hunter2

[jcaille]

I've actually gone ahead and implemented a ~/.bitbarrc for configuration used in some of the plugins we're using at my company.
Right now, Bitbar itself does not interact with the config file, and plugins that need it just read directly from this file. This allows me to centrally update the plugins without end-users having to apply their patch.

I'd really like to try and implement this feature in the next few days / weeks, as it seems to be a prerequisite for more advanced features (plugin management, automatic plugins update, ...).

I like @keithamus idea that Bitbar should handle the configuration file, and route relevant parameters to each script through the environment when running them. I just see a few wrinkles that we might need to work out.

---

One of the problem that I ran into (and that we might have if we offer a standardised solution) is providing default values and discoverability for those parameters. What I mean by this is without reading the script, there is absolutely no way to discover what parameters are available, and what values are accepted or provided by default. I think there are multiple ways to address this

• Documentation

We could handle parameters discoverability through documentation. Plugins developper should provide documentation about what parameters are accepted, ideally in their plugin file or in some other documentation. This makes it easy to implement for us, and puts most of the work on the shoulder of plugins developers.

• Automatic parameter discovery

Another (more convoluted) solution that comes to my mind is to have a configuration setup similar to the one used by Sublime Text, with two files. I realise this is very overkill, but I like the flexibility it offers (and think it would work well with a robust plugin management solution).

~/.bitbarrc_defaults contains the keys, default values and documentation of parameters for each installed plugins. Ideally, installing or running a plugin should add all the parameters used by the plugin to this configuration file. Unfortunately, i don't see a simple and clean way to do this at the moment.
~/.bitbarrc contains parameters that are overloaded by the user.

When launching a script, we successively parse the files and populate the environnement with the relevant parameters.

Off the top of my head, other problems might be :

• Namespacing : plugin A should not receive plugin B's parameters. How do we provide namespaces (plugin file name ? What about symlinks with different names ? What if the plugins filename changes ?)
• Data types : Could we provide advanced data types through this config ? I'm thinking lists or dictionary of values (e.g. : Here's the list of Jenkins jobs you should check) ? What syntax could we use ?

What do you think ? Do you have ideas for handling those issues ? Do you see other problems that might arise ?

If nobody has objections, i'll start by implementing the parsing and routing of parameters contains in a ~/.bitbarrc file, which seems like a common feature required by most solutions.

[keithamus]

One of the problem ... is providing default values...

I think defaults can be done easily enough in userland, even in bash can do defaults.

Automatic parameter discovery

What about passing an argument to a plugin on first run? e.g. when BitBar first detects my plugin (myscript.sh) it calls myscript.sh setup. Every scripting langauge has a way to get argv. We can use this to then echo different output - a way for the plugin to set it self up:

#!/usr/bin/env sh
if [ "$1" = "setup" ]
then
  echo "option TOKEN"
  exit 0
fi
...

#!/usr/bin/env python
import sys
if sys.argv[1] == "setup"
    print "option TOKEN"
    sys.exit(0)
...

Namespacing

If we had an ini like I described:

[myplugin.sh]
token=foobar
[otherplugin.rb]
password=hunter2

myplugin.sh would only have token set as an env var, but not password. Similarly, otherplugin.rb only gets password but not token. This keeps things nicely sandboxed.

Data types

Seems like, especially for a first pass, keeping things to Strings will be fine. Need an advanced config? Use serialised strings - like JSON.

[jcaille]

+1 to what you said. I like the setup parameter given to the file. It keeps it nice and tidy, should default pretty well (just an additional script execution if argv is not used), and might even do a few other things in the future.

A few questions to clear up my understanding :

• In the setup phase, could we output the default lines in the ~/.bitbarrc verbatim ? Makes it easy to write them to the file and prevent an additional parsing phase.
• Upon launching or resetting bitbar, we could do the following steps :

   * List plugins in the directory
   * Parse existing config file
   * For every plugins where no configuration is present
       * Run `plugin setup` 
       * If output start with keyword -[configuration]- 
           * The plugin has a setup phase
           * Write output to the config file
   * For every plugins where a configuration has been found
       * Run `plugin setup`
       * Compare config output to existing parameter
       * If a new parameter key is found, add it to the config file with the script-provided default
    * Start updating plugins as usual

This should keep the config file up to date with new and updated plugins. Downside is startup could be much longer, needing to run plugin setup for every plugin (and possibly defaulting to running the plugin if no setup is provided)

• Namespacing or sandboxing is a must-have imo. Do we agree that the namespace should be constructed this way ?

    name.refresh_period.extenstion -> [name.extension]
    jenkins.10m.rb -> jenkins.rb
    timeup.sh -> timeup.sh

This should allows the user to update the refresh period without breaking parameter integration.

If there's no objections, i'll start implementing this over the weekend

[keithamus]

Personally I agree with everything you've said. But maybe wait for @matryer's approval before working on a PR?

[jcaille]

Good point

[keithamus]

@matryer Do you think this is a good idea to implement?

[mcchrish]

Inline with configuration file discussion, is it ok to implement XDG base directories?
http://standards.freedesktop.org/basedir-spec/basedir-spec-0.6.html
So instead of ~/.bitbarrc, the config file will be ~/.config/bitbar/config

[astrostl]

Major +1 to configuration files. With variables-in-scripts, it makes staying in sync with master a nightmare. Without, end users are just a git pull away from a clean update.

[keithamus]

I concur with @mcchrish; XDG base directories would be a good idea

[ryansydnor]

:+1: to a config file. I've played around with a "global" ~/.bitbarrc and it's working well for needs. However, I see the need for namespacing on a larger scale.

Globals with the ability for a namespace to override seem like a potentially useful feature to reduce configuration sprawl.

I'd also consider potentially offloading the burden of config parsing to the scripts. This will mean your proposed "config discovery" functionality is unnecessary. Build a configuration parser (can we just use JSON or YAML for the config file?) and pass relevant values (globals and namespaced) to the script as JSON. Allow the script to parse the values and throw intelligent errors if a value they expect is absent.

[matryer]

I'm happy to do this - @keithamus do you want to suggest a final design from all the ideas posted here?

[keithamus]

Sure. Here's something close to a final design; I think @manojlds and @jcaille should critique this before we go head with anything though.

---

• Upon startup or refresh, BitBar reads $XDG_CONFIG_HOME/bitbar.conf
  
  
  
  • if $XDG_CONFIG_HOME is not present, then it defaults to $HOME/.config
  
  
  • if $XDG_CONFIG_HOME is not present, bitbar should create the folder
  
  
  • if $XDG_CONFIG_HOME/bitbar.conf is not present, bitbar should create the file
  
  
  • $XDG_CONFIG_HOME/bitbar.conf is an ini formatted file.
  
  
  • For every script inside of the bitbar enabled plugins folder
  
  
  • Find the section in the ini corresponds to the plugin name - for example Dev/jenkins.30s.sh would find a section name of [jenkins].
  
  
  • If that section does not exist, call the plugin with the given argument setup (for example Dev/jenkins.30s.sh setup)
  
  
  • Attempt to parse the output of setup - the output should be an enumeration of keys, with the syntax option <keyname> <defaultvalue>.
    
    
    
    
    
    • If the output of the setup command does not follow this format, disregard its output
    
    
    
    • If the output of the setup command does follow the output, add a section name to the ini file, representing the name of the script, and a set of key/value pairs matching the option declarations from the setup script
    
    
    
  
  
• When a script is executed, it is passed a set of environment variables
  
  
  
  • For every top level key/value in the ini file, the key becomes the name of an environment variable and the value becomes the value of that environment variable
  
  
  • If the ini file contains a section block (e.g. [foo]) and that section block corresponds to the script name (e.g. [jekins] for Dev/jenkins.30s.sh) - then every key in that section becomes the name of the environment variable, and every value becomes the value
  
  

---

To provide an example user journey:

1. I set up bitbar for the first time. ~/.config/bitbar.conf is created, it is an empty file.
2. I put a helloworld.30s.sh script into my bitbar enabled plugins folder. It contains:

bash #!/bin/bash if [ "$1" = "setup" ]; then echo "option PHRASE hello world" exit 0 fi echo $PHRASE

1. I click refresh on BitBar. It runs helloworld.30s.sh setup and determines that it requires the PHRASE option, set to hello world.
2. My ~/.config/bitbar.conf file now contains the following:

ini [helloworld] PHRASE = "hello world"

1. BitBar executes the script for the first time, it passes the environment variable PHRASE, set to "hello world", making the output of my helloworld.30s.sh script hello world
2. I edit my ~/.config/bitbar.conf file, it now contains the following:

ini [helloworld] PHRASE = "goodbye world"

1. I hit refresh on bitbar. My script now outputs goodbye world
2. I edit my ~/.config/bitbar.conf file, it now contains the following:

ini ; these are all global variables that are added to every script PHRASE = "goodbye world"

1. I click refresh on bitbar, my script still outputs goodbye world
2. I edit the file one more time:

``` ini
; these are all global variables that are added to every script
PHRASE = "goodbye world"
; these are specific settings for each plugin
[helloworld]
PHRASE = "this one"
```

1. Now when I click refresh on my bitbar, the script outputs this one.

[ryansydnor]

@keithamus thanks for writing that up!

A few questions for you:

1. How do you handle error cases? The couple of examples I can think of:
   
   
   
   1. Manually editing the bitbar.conf file and removing values
   
   
   2. Plugin updating the vars it uses - bitbar not recognizing it needs to run setup again because [section] already exists
   
   
2. Are you considering differentiating between "required" and "optional" variables? In this model, it looks like everything is "required" (with a sensible default).
3. Are you rolling your own configuration format? If so, why? One of my favorite aspects of bitbar is the ease of use - adding complexity via a bitbar specific config syntax could be daunting to certain users. Why isn't a YAML dictionary easier from both the implementation and UX perspectives?

[keithamus]

Manually editing the bitbar.conf file and removing values

I think this can be solved in userland. If a value is removed from bitbar.conf, the next time a refresh (or initialisation) happens, the value wont be given to the script. If a script depends on a value, it should either provide a sensible default or fail with useful messages. To wit:

#!/bin/bash
if [ "$1" = "setup" ];
then
  echo "option PHRASE hello world"
  exit 0
fi
if [ "$PHRASE" = "" ];
then
  echo "PHRASE must be set"
  exit 1
fi
echo $PHRASE

Plugin updating the vars it uses - bitbar not recognizing it needs to run setup again because [section] already exists

Same as above - if scripts add or remove values, they can simply check for existence of the new variables and warn the user.

Are you considering differentiating between "required" and "optional" variables? In this model, it looks like everything is "required" (with a sensible default).

Again, this is a userland thing. If a value is optional, plugins should provide sensible defaults. If it is required and the existing value doesn't satisfy, the plugin should exit with an explanation.

Are you rolling your own configuration format? If so, why? One of my favorite aspects of bitbar is the ease of use - adding complexity via a bitbar specific config syntax could be daunting to certain users. Why isn't a YAML dictionary easier from both the implementation and UX perspectives?

No, this is not a custom format. This is the INI standard. YAML, JSON, PLIST or other configuration formats could be used - but INI works well because it fits all of the requirements and nothing more. We need a key value store with namespacing, and comments would be nice. Ini format is key value with namespacing (sections) and comments. JSON is less readable than ini, and offers extras that we don't need (like deep nesting, arrays, numbers, etc). YAML is arguably more readable but offers _loads_ more stuff that we don't need.

[ryansydnor]

Thanks for getting back so quickly!

Proposal sounds good to me. Looking forward to code reviewing the PR :)

[vogonistic]

I like this, especially having configuration passed in through environment, but I would vote for 2 changes.

1. Allow the script to optionally provide a comment to be associated with the option, in case there are requirements any one editing options needs to know about.
2. Instead of $XDG_CONFIG_HOME (or in addition to) read the configuration path out of NSUserDefaults, which is significantly easier to set programmatically.

Regarding the syntax from plugin.sh setup, any particular reason it can't just output raw .ini?

[tresni]

We already have plugin metadata. Why not just extended that? Then plugins don't have to implement a special $1 case.

# <bitbar.option>USERNAME</bitbar.option>
# <bitbar.option>PASSWORD</bitbar.option>
# <bitbar.option>I_LIKE_PIZZA</bitbar.option>

If you want to allow specifying "type" you can either do <bitbar.option.string> or (more xml proper) <bitbar.option type="string">.

[keithamus]

@tresni right now, to my knowledge, the metadata is only used on the getbitbar website. We could repurpose it for option declaration, I suppose - personally I'm not the biggest fan of this idea though. I like @vogonistic's idea of plugin.sh setup outputting raw ini format though.

[chrismetcalf]

I just want to throw in my vote for something as simple as a $BITBAR_PLUGINS_DIR/.env or $HOME/.bitbarrc file that contains environment variables that get sourced before each script is run. Doesn't help the GUI-dependent folks, but it's super simple for the rest of us (and could be checked into private homedir repos for those of us who do that).

[astrostl]

+1 for simple. Perfect may be getting in the way of good here.

[chrismetcalf]

There's nothing saying that a GUI couldn't manage $HOME/.bitbarrc either, but I'd like something simple for config vars sooner rather than later. I accidentally checked a GitHub token in and GitHub discovered it in one of their automatic crawls looking for dumbasses like me.

[chrismetcalf]

I just filed a pull request that should be at least a start on this one.

It just reads config from a config.json file like this:

{
    "GITHUB_TOKEN" : "1234abcd"
}

Those become environment variables you can access from within scripts, like this:

github_api_key = os.getenv( 'GITHUB_TOKEN', 'TOKENGOESHERE' )

I know almost nothing about ObjectiveC, so I kind of threw it together. It doesn't auto-update the environment when you refresh your plugins, so you need to quit and restart BitBar to update your variables. But it works for my needs, and it's simple, which is what I was looking for.

[chrismetcalf]

I found a creative workaround that works for stock BitBar.

If you follow the instructions in this StackOverflow question and create an environment.plist file, you can set login-global environment variables you pull into your BItBar scripts. Just set it up, and it works pretty well for me.

[astrostl]

Also using that now. Works!

[svetlyak40wt]

Guys, you a wasting time. It is easy to set environment variables in the way, such BitBar plugins will see them. Just do:

launchctl setenv TEST_BITBAR blahminor

And restart BitBar.

[GlenCoakley]

Since this is still open I'd like to add my vote for XDG support. I also have another suggestion regarding sensitive data.

Like other people, I have accidentally committed a Github access token to a repository. In my case it was because I am not only using an XDG setup but, I also keep my ~/.config in a github repository so that I can keep it in sync across the different machines that I use. (I keep my sensitive tokens in some files under ~/.ssh./ Adding support for XDG would include using its XDG_CONFIG_DIRS which is just what I need as long as Bitbar doesn't stop looking when it finds one configuration file. Rather, I like to see it look through all of the XDG_CONFIG_DIRS and read every matching configuration file it finds applying each of them, such that later definitions overwrite earlier ones. This would allow keeping secure data such as Github tokens and sensitive data such as passwords in a location that is not (backed up/stored as) part of common application configuration. (This is not defined by the XDG spec. but, I am trying to have it added.)

My setup is, of course, a bit of a pain because many applications do not support XDG or otherwise segment their machine specific configuration (window geometry, cache files) from the machine agnostic configuration (app. behavior). I handle this differently for different apps. when moving between machines. For some apps. I don't commit their configuration, for others I merge the diffs, etc.