Kibana: Directory structure with the new platform

[kimjoar]

With the addition of the new platform, we need to adjust our directory structure to support both legacy and new code simultaneously. The directory structure should as clearly as possible separate out concerns and distinguish between new and old.

Latest proposal: https://github.com/elastic/kibana/issues/12466#issuecomment-386809436

Original description

We need to decide on a directory structure for the new platform.

@epixa created a suggestion in https://gist.github.com/epixa/5af7474d3205f9558113bb6b30248050

Discussions

• 8 Nov 2017
  
  
  
  • talked about philosophy of dir structure
  
  
  • decided we should have tenets
  
  
  • notes on tenets: https://gist.github.com/archanid/8f9075816be79a72af4817bc59b72cdf
    
    

[spalger]

I feel that we don't need to separate dev and tools. I'd even wonder if test should be a part of dev...

[epixa]

The dev directory in my example is a place for input/output artifacts from development, essentially. Unless we wanted to pre-seed the development setup, the directory wouldn't even need to be checked in. When you start up a dev run of Kibana, it'd write things like the pid, data directory, logs, etc. to this directory. On the other hand, tools and tests are permanent, checked-in files. The former could be things like a backporting tool or a version bumping tool, whereas the latter would be for functional test files.

[spalger]

Ah, I thought the dev directory was kind of a "dev tool source" directory, makes sense

[rhoboat]

Added to the issue description to include notes from today's discussion

[epixa]

@kjbekkelund You're getting close to merging the initial platform work into master. To unblock the directory structure work, it would be helpful if you could outline specific concerns you have for the directory structure related to the new platform itself. Like where to put core plugins, how we separate server and client code in a plugin, etc.

[epixa]

I've taken a more pragmatic swing at this. We've learned a lot about how we plan to roll out the new platform since this issue was opened, and I think a complete overhaul of our entire directory structure is not necessary right now. Instead, I focused on how the legacy code and new platform code would live by side by.

I moved the new platform code into src, and then I rearranged some directories to more clearly separate out legacy functionality from the new. All existing src directories are represented here, though I've tweaked some names across the board.

  build
    babel-register
    optimize
  core (new platform core)
    cli
    public
    server
      config
      logging
      legacy_compat
    test_helpers
  dev
    build
    docs
    ci_setup
    eslint
    ...
  plugins (new platform plugins)
    timepicker
      public
      server
    ...
  legacy
    cli
    cli_keystore
    cli_plugin
    deprecation
    core_plugins
      console
      dev_mode
      elasticsearch
      ...
    plugin_discovery
    server
    ui
    utils
  test
    es_archiver
    fixtures
    functional_test_runner
    utils
x-pack
  dev
    build_chromium
    gulp_helpers
    dev-tools
  docs
  legacy
    common
    plugins
      apm
      cloud
      console_extensions
      ...
    server
    webpackShims
  plugins (new platform plugins)
    license
      public
      server
    security
      public
      server
    ...
  scripts
  test

One key thing I want to point out here is that a plugin is either all legacy or all new platform, at least from a technical standpoint. So for legacy plugins that are doing a gradual migration over to the new platform, they are technically migrating to a new plugin. Given recent conversations I've had with @spalger about how we might treat legacy plugin code in the new platform (at least in the UI), I think this makes a lot of sense. It also gives us a clear indication of what is and is not new platform stuff, and we can choose to enforce different linting rules and such between old and new.

• Edit 8 May 2018: browser -> public
• Edit 8 May 2018: x-pack directories
• Edit 22 May 2018: xpack_core -> license

[azasypkin]

Looks good to me, I only have mixed feelings about browser, why not just good old client (not ui since technically client code in plugin or core may not provide any user interface)?

[epixa]

@azasypkin @spalger brought up the browser name as well, and ultimately we decided to use public for now so we don't have to overhaul a ton of glob references in our build system for client-side code. Down the line, we can always rename to client or browser if we think it's important, but I suspect we won't care much.

[epixa]

I updated the example in https://github.com/elastic/kibana/issues/12466#issuecomment-386809436 to use public instead of browser and to include the proposed x-pack directories.

[epixa]

I took assignment of this issue since it's really just about driving to consensus rather than being coupled to the server implementation. It is possible (if not likely) that @azasypkin will need to deal with decisions we make here before anyone else, but it's also possible that @spalger will need to deal with this first if he ends up pushing forward the client-side stuff quicker than we can resolve the server-side typescript changes.

[rhoboat]

I see a discrepancy in not having a src directory within x-pack/. x-pack is shown as a sibling of src, and it contains some source directories (by the standard we use today--i.e. not tasks, not scripts, etc) as well as dev tooling, scripts, and test related code. In this model, what other kinds of things are siblings of src/ besides x-pack? Knowing that will help figure out what src/ really means in this proposal.

[epixa]

The only tests/scripts/dev tooling in this proposal are things that are already under src or x-pack. I'm not proposing [here] that we move anything into or out of them.

[snide]

@epixa I assume the "kibana/ui_framework" directory is getting moved into legacy as well? I didn't see it specifically listed here.

[epixa]

@snide I was only dealing with moving directories around inside src and x-pack, since those are the only places that impact the new platform. We certainly could move other directories around as well if we think it makes sense. Technically speaking, there's nothing about ui_framework that makes it incompatible with the new platform, though we have no intention of using it there since clearly its days of limited.

[cjcenizal]

@epixa Could you explain the reasoning behind organizing plugins by plugin first and public/server second [1], instead of by public/server first and plugin second [2]?

[1]
  plugins
    timepicker
      public
      server

[2]
  plugins
    public
      timepicker
    server
      timepicker

[uboness]

@epixa Could you explain the reasoning behind organizing plugins by plugin first and public/server second [1], instead of by public/server first and plugin second [2]?

can't speak on @epixa's behalf, but [1] totally makes sense to me (and [2] totally doesn't in the context of plugins). Group by functionality regardless of implementation details. If someone want to understand what's going on in plugin X, they just look at the plugin directory without the noise of all other plugins. It's as close to isolation as you can get... Also some plugins have server side other don't, some have client side others don't. mixing all server side or client side codebases of all plugins in one place creates noise and makes it hard to rationalise of what actually makes a single plugin.

+++ on [1]

[epixa]

There are a couple reasons why a plugin-first directory structure (your example 1) is ideal for Kibana, ranging from technical to philosophical.

From a technical perspective, our goal is for plugins to be independent systems, which means they are isolated from the world around them, have their own dependencies, etc. This is only possible for node modules with a shared top level directory for each plugin.

From a philosophical perspective, Kibana is essentially just a lightweight core with a bunch of plugins composed together. Each plugin is a building block, and as a whole it includes server, public, and any common code. Each plugin can be added or removed by either uninstalling it (third party plugins) or disabling it (core plugins), which removes all aspects of that plugin from the experience in Kibana, and since a goal of the new platform effort is to have our architecture expressed through our directory structure, it makes sense for all of this combined functionality to be represented as a single directory.

Also, well before we had any notion of rearchitecting Kibana, it has been a goal of ours to have third party plugins look and behave like core plugins. Each third party plugin is encapsulated in a top level directory, so our plugins should be as well.

[cjcenizal]

Thanks for sharing your perspective @uboness and thanks for the explanation @epixa.

[cjcenizal]

@epixa Can we close this issue? It seems like @joshdover and the rest of the people working on NP migration have established directory structure patterns at this point.