Graylog2-server: Support for multiple retention strategies

[joschi]

Enable Graylog to store different types of log messages for different periods of time, based on streams.

https://graylog.ideas.aha.io/ideas/GL2E-I-356

Steps

• [x] Create explicit default stream #2881
  
  
  
  • [ ] ~Add migration job adding the default stream as start page for users who don't have a start page set and have the required permissions for the default stream~
  
  
• [ ] ~Remove default output handling from OutputBufferProcessor~
• [x] Create "write targets" (combination of index prefix, rotation strategy, and retention strategy)
• [x] Allow to map streams onto write targets
• [x] Add migration job which creates write target from current rotation/retention config + index prefix
• [x] Make sure that Searches/IndexHelper classes uses correct index prefixes
• [x] Add Write Target management functionality to web interface (refactor indices page)
• [x] Add Stream → Write Target mapping functionality to web interface
• [x] Check if we need an UI element to mark one stream as a default stream. Might be needed if a user deletes the default stream. Otherwise there is no way back. If we create the possibility to mark a stream as the default stream, we have to ensure that only one stream can be a default stream.

Open TODOs

• [x] Document stream REST API changes regarding the new mandatory index_set_id field - breaking change (in UPGRADING.rst) (@bernd #3341)
• [ ] ~Document indices resource API changes (in UPGRADING.rst)~ (not needed, no breaking changes)
• [x] Create unique index on index_prefix field to avoid "problems" (@bernd #3193)
• [x] Check if index template handling still works with multiple index sets (one template per index set?) (@bernd)
• [x] Make index prefix matching more strict. What happens if index-set-1 uses prefix graylog and index-set-2 uses prefix graylog_yolo? This is also a problem in index templates as they are using graylog_* to match indices! (@bernd #3193)
  
  
  
  • Idea: Not allowing _ character in new index set prefixes should be enough
  
  
• [x] Rotation/retention strategies need to load the index set config from the database instead of using the old cluster config data
• [x] Check validation for IndexSetSummary and IndexSetConfig - currently it's possible to create an empty index_prefix field even though it's annotated with @NotBlank (#3188)
• [x] Make sure index template and index is created when a new index set is created (@bernd)
  
  
  
  • Will be created by the index rotation job that runs every 10 seconds
  
  
• [x] Check all "TODO 2.2" comments in graylog server and plugins and fix them (#3211, #3285 ongoing)
• [x] Make it possible to mark an index set as "default" to unbreak content pack import - only one index set can be marked as default (@bernd #3209)
• [x] Show the index set a stream is assigned to on the stream overview page (@edmundoa)
• [x] Adjust index set deletion REST API to forbid deletion if the index set is assigned to a stream (partly #3207)
• [x] Check IndexSet and IndexSetRegistry interfaces for naming. Some method names might be misleading or weird (@bernd #3342)
• [x] creation_date for IndexSetConfig is a string in MongoDB due to date serialization problems. Because of that, sorting is wrong. (#3198)
• [x] Check if the remove matches from default stream flag in streams works properly. (@bernd)

Nice to have

• [ ] ~Validate index_prefix field while editing field in UI during index set creation~ (not for 2.2)
• [x] Display index count/message count on Index Sets overview page. (@bernd #3322)

[bernd]

Finally closing this.