Kibana: Sharing saved-objects in multiple spaces

[jinmu03]

Requirements

Currently, Kibana only supports the ability to copy a saved-object to multiple-spaces. When copying a saved-object, all copies are separate saved-objects and the modification to any copy is not reflected in the other copies. This behavior is ideal in some situations, and problematic for others. This proposal is focused on adding the ability to share saved-objects in multiple spaces, so changes are automatically reflected in all spaces in which the saved-object has been shared.

The ability to share saved-objects in multiple spaces will alleviate two primary drawbacks to only supporting copies:
1) Difficult to keep copies in sync
2) Performance and storage impact of requiring a copy

When a user copies a saved-object to multiple spaces, the updates to the saved-objects are isolated from each other. This is problematic when the user wishes to update all copies. The user is currently required to manually update all copies utilizing the “Copy to Space” functionality that was introduced in 7.4, or cobble together some other solution. This is a common complaint for index-patterns. When an index-pattern’s mappings are refreshed or a scripted-field is added, if the intent is for this change to be made in all copies it’s a manual and quite painful process.

Additionally, some usages of saved-objects have a significant impact on the performance and storage requirements of both Elasticsearch and Kibana. In these situations, when a saved-object is copied the copy itself will consume as many resources as the original saved-object. An example of this is ML. When ML starts using Kibana’s saved-objects to provide more granular access to their jobs and other entities, a copy is a potentially dangerous operation. An Elasticsearch ML job consumes a potentially large amount of cluster resources to run, and as such a job should be copied with care.

When a saved-object is shared to a space, the saved-object will be modifiable in any of the spaces in which it has been shared. Since the saved-object can be modified in any space its been shared, this introduces some complexities for saved-objects which have references to other saved-objects. Before a saved-object can be shared to a space, all direct and transitive references to saved-objects must be shared. For example, dashboards have references to visualizations, which have a reference to an index-pattern. When a dashboard is shared to a space, all referenced visualizations and in-turn index-patterns must be shared before the dashboard is shared.

Additionally, when a saved-object is either created or updated and references new saved-objects, the references will be checked to ensure they exist in at least all spaces which the saved-object exists in. This will prevent saved-objects from being updated or created with broken references.

In the following situation, Bob only has access to the marketing space and creates a dashboard and two visualizations.

Alice has access to both the Marketing and Sales spaces, and shares the dashboard and visualizations to the Sales space.

Both Alice and Bob are able to edit Dashboard A; however, both are constrained to only adding visualizations which exist in both the Marketing and Sales space. This limits Bob in the types of edits he can make, but it doesn’t completely remove his ability to make edits.

Further elaborating upon the previous situation, if Alice was to unshare Visualization A from the Marketing space, we could potentially prevent her operation from occurring because it would make Dashboard A have a broken reference from the Marketing space. However, we already have a similar situation when we allow a Visualization to be deleted when it is referenced by a Dashboard within the same space.

This proposal relies on high-level referential integrity checking by default on create/update logic to ensure that references aren’t being broken. Programatically, developers will be able to circumvent this protection; however, it will be enabled by default. Broken references are generally beneficial to avoid, but in the case where they are broken it’s not a cataclysmic event.

From an authorization perspective, not many changes are required. When a saved-object is shared to a new space, the user will need to be authorized to write the saved-object in both the source and target spaces. However, when a user edits a saved-object, they will only need to be authorized to write the saved-object in their current space. For example, in the following situation Bob is only authorized to edit dashboards and visualizations in the Marketing space.

Bob will therefore be able to edit the dashboards and visualizations which are shared in the Marketing and Sales spaces. However, once object-level security is implemented the user will be able to specify a per-object ACL to make the shared saved-object read-only.

Implementation Phases

The ability to share a saved-objects in multiple spaces will be implemented in three primary phases. Not all saved-objects of the specific "category" will automatically be able to be shared, and this will require effort from application and solution teams before a saved-object can be shared. For example, with Phase 1 not all new saved-object types will automatically be "shareable".

Phase 1: New saved-object types without references

Release: 7.8 #54043

This will allow ML to begin taking advantage of the ability to share saved-objects in multiple spaces. ML jobs haven't transitioned to take advantage of saved-objects, and this will allow us to reduce the complexity that migrations introduce

Phase 1.5: Management

Release: 7.10 #58139

This will support import/export for "shareable" types (saved object types marked as multiNamespace) without references. It will also support existing operations in the Saved Objects Management Screen (server side and client side) and add a UI for sharing a saved object to other space(s).

Ancillary task: Sharing to all spaces

Release: 7.10 #69808

As originally envisioned, sharing saved objects would only support _explicitly-named spaces_. However, recent use cases have surfaced that would benefit from the ability to share to _all current and future spaces_. This will support such use cases.

Phase 2: Existing saved-object types without references.

Release: 7.11 (estimated) #54837

This will allow index-patterns to be shared in multiple-spaces. It requires that migrations and aliases are implemented, but doesn't require that the referential integrity checking be implemented.

Phase 3: All saved-object types with references

Release: 7.12 (estimated) #67380

Everything can now begin taking advantage of being shared in multiple spaces. As previously stated, given the level of effort that it will take to customize the various applications to inform the user that the saved-object is shared in multiple spaces and allow direct and transitive references to be shared, this will be taken advantage of incrementally by the application and solution teams.

Technical Solution

When implementing Spaces, we prefixed a saved object’s Elasticsearch document ID with the namespace and added a singular namespace field. Prefixing the Elasticsearch document ID with the namespace allowed us to copy saved-objects from one Space to another without having to worry about conflicts, as the Elasticsearch document IDs are unique. Since the end-goal is to have saved-objects belong to multiple namespaces , we can no longer prefix the document ID with the namespace. The following technical solution expands upon the method in which we’re proposing to remove namespace from the Elasticsearch document ID, change the namespace field to namespaces and maintain backwards compatibility with existing Kibana URLs.

Brief Summary

This is the current document structure:

{
  _id: NAMESPACE:TYPE:ID;
  type: string;
  namespace: string;
}

We’d like to remove the NAMESPACE, from the _id and change namespace to namespaces. This would leave us with this new document structure:

{
  _id: TYPE:ID;
  type: string;
  namespaces: string[];
}

The issue is that we’re likely going to have quite a few _id collisions when doing so. Kibana currently supports the ability to have multiple saved-objects with the same ID in different namespaces. For example:

{
  _id: ‘marketing:index-pattern:1’;
  type: ‘index-pattern’;
  namespace: ‘marketing’;
}

And

{
  _id: sales:index-pattern:1’;
  type: ‘index-pattern’;
  namespace: sales;
}

These two documents will exist if we copy an index-pattern from the sales space to the marketing space, or vice-versa. If we remove the NAMESPACE from the _id for both of these documents, we’d end up with two documents with an _id of index-pattern:1.When we detect this collision during a migration, we have the option of regenerating an ID. However, we’ve been primarily concerned that doing so is going to break existing URLs in Kibana.

To prevent us from breaking URLs in Kibana and address another pain-point which users have experienced, we’d like to introduce the ability for end-users to manage saved-object aliases. Saved-object aliases are used when translating from an identifier in an application’s front-end route to the referenced saved-object. Prior to this change, front-end routes would include the actual saved-object ID, for example: http://localhost:5601/app/kibana#/dashboard/edf84fe0-e1a0-11e7-b6d5-4dc382ef7f5b. However, with the introduction of saved-object aliases, we’d allow the user to create a custom alias which references the actual saved-object ID. This would allow a front-end route like http://localhost:5601/app/kibana#/dashboard/foo to be used instead of the aforementioned route which includes the explicit saved-object ID.

Now that we have saved-object aliases, when we detect a collision during the migration, we regenerate a new ID and create an alias document, similar to the following:

{
  _id: alias:sales:index-pattern:1’;
  type: ‘alias’;
  namespaces: [‘sales’];
  ‘alias’: {
    type: ‘index-pattern’,
    id: ‘2’
  }
}

We also need to keep track of the old and new IDs. For the sake of the migration itself, an in-memory map is sufficient; however, to better support imports when an ID had to be regenerated, we’ll be persisting these in a regenerated-id saved-object.

This leaves us with the following documents to model the previous situation where we had an index-pattern with ID of 1 in both the sales and marketing namespaces.

{
  _id: ‘index-pattern:1’,
  type: ‘index-pattern’,
  namespaces: [‘marketing’]
}

{
  _id: ‘index-pattern:2’,
  type: ‘index-pattern’,
  namespaces: [‘sales’]
}

{
  _id: ‘alias:sales:index-pattern:1’,
  type: ‘alias’,
  namespaces: [‘sales’].
  ‘alias’: {
    type: ‘index-pattern’,
    id: ‘2’
  }
}

{
  _id: ‘regenerated-id:1’,
  type: ‘regenerated-id’,
  namespaces: [‘sales’],
  ‘regenerated-id’: {
    ‘old’: 1,
    ‘new’: 2
  }
}

Then, we no longer rely directly upon SavedObjectsClient::get when retrieving the saved object based on the identifier in the URL. Instead, we add a new SavedObjectsClient::resolve, which will return a saved-object using the type and identifier. The resolve method will look for an alias which matches the type and identifier in addition to a normal saved-object which matches the type and identifier. If we find only an alias, we return the referenced saved-object. If we only find an actual saved-object, we return the actual saved-object. If we find both, we throw an error and require the user to remedy this using Saved Object management. This way, we don’t alter the behavior of SavedObjectClient::get, and it always returns saved-objects based on their ID.

Migration Pseudocode

The following assumes we’ve topologically sorted all saved objects by saved-object types:

const savedObjectIds = new Set();
const inMemoryRegeneratedIds = new Map();

for (const savedObject of topologicallySortedSavedObjects) {
  if (savedObjectIds.has(`${savedObject.type}:${savedObject.id}`)) {
      const id = new Guid();

      shortUrls.create({
          url: savedObject.id,
          namespace: savedObject.namespace,
          type: savedObject.type,
          id: id
      });

      inMemoryRegeneratedIds.add(`${savedObject.type}:${savedObject.id}`, id);
      persistentRegeneratedIds.create({ // these are used later during import
          type: savedObject.type,
          old: savedObject.id,
          new: id,
      });

      savedObject.id = id;
  }

  for (const reference of savedObject.references) {
      if (inMemoryRegeneratedIds.has(`${reference.type}:${reference.id}`) {
          reference.id = inMemoryRegeneratedIds.get(`${reference.type}:${reference.id}`);
      }
  }

  savedObjectIds.add(savedObject.id);
}

SavedObjectsClient::resolve Pseudocode

const { type, identifier } = request.params;
const namespace = getNamespace(request.url);

const actualSavedObject = await savedObjectsClient.get(type, identifier);
const alias = await savedObjectsClient.get(alias, `${namespace}:${type}:${identifier}`);

if (!actualSavedObject && !alias) {
 throw new NotFoundError();
}

if (actualSavedObject && alias) {
 throw new GeneralError(`You've shadowed a saved-object, you must delete the alias.`);
}

if (actualSavedObject) {
 return actualSavedObject;
}

if (alias) {
 return await savedObjectsClient.get(customUrl.attributes.type, customUrl.attributes.id);
}

SavedObjectsRepository Operations

The following logic changes will be implemented within the SavedObjectsRepository, which exists within OSS Kibana. Currently, the SavedObjectsRepository ensures that saved-objects in different namespaces are isolated from each other and no conflicts exist. This is by virtue of the namespace being part of the Elasticsearch document ID. Additionally, the SavedObjectsRepository ensures that saved-objects which are being retrieved using the get, bulkGet and find methods all respect the namespace that is specified, and saved-objects in other namespaces are not retrieved.

With the changes being proposed, we can no longer ensure that saved-objects in different namespaces are always isolated from each other and no conflicts exist. Instead, we will be modifying the logic within the SavedObjectsRepository to reflect the new reality that conflicts may exist between saved-objects in namespaces.

Create and BulkCreate

• Perform a pre-create check to ensure all referenced saved-objects exist within the current namespace
• If the id isn’t specified
  
  
  
  • Create a new ID and a new saved-object is created
  
  
• If the id is specified and overwrite is false,
  
  
  
  • Try to insert as-is. If there’s a conflict, throw the conflict error
  
  
• If the id is specified and overwrite is true
  
  
  
  • Perform a pre-create check to ensure that we aren’t overwriting a saved-object which exists in a different namespace.
  
  
  • Execute a GET or _bulk_get to retrieve any existing saved-objects which already exist
  
  
  • If the saved-objects already exist, ensure the current namespace is already in the saved-object’s namespaces.
    
    
    
    
    
    • If it is, execute an update excluding the namespaces field, preserving the existing namespaces
    
    
    
    • If it is not, return a 409 Conflict
    
    
    
  
  
  • If the saved-object doesn’t already exist, execute a create
  
  

Update

• Perform a pre-update check to ensure that we’re updating a saved-object which exists in the current namespace and all newly referenced saved-objects exist within the current namespace.
  
  
  
  • GET the existing saved-object
  
  
  • If the current namespace isn’t in the namespaces, we throw a 409 Conflict error
  
  
  • For any newly referenced saved-objects perform a _bulk_get. If their namespaces don't include the updated saved-objects namespaces, throw an error.
  
  
  • Otherwise, proceed with the update
  
  

Get and BulkGet

• Perform the normal get or bulkGet
• Filter out any responses which don’t include the current namespace

Find

• Change the query generated from namespace to namespaces

Delete

• Perform a pre-delete check to ensure that the saved-object we’re deleting exists in the current namespace
  
  
  
  • GET the existing saved-object
  
  
  • If the saved-object exists in the current namespace, delete the saved-object
  
  
  • Otherwise, throw a 404 Not Found
  
  

DeleteByNamespace

_Note: This is the internal method which is used to delete a space and all of the saved-objects in that space. It likely should be renamed based on the changes in logic I'm proposing._

• Perform a _delete_by_query for all saved-objects which only exist in the current namespace
• Perform a _update_by_query to remove the current namespace from all saved-objects which exist in multiple namespaces

Other Saved Object Operations

Export

• Nothing should need to be changed ehre

Import

_Note: The following is a gross over-simplification and needs further thought and investigation_

• When validating references
  
  
  
  • If a reference doesn’t exist, consult the regenerated-id saved-objects and use the new ID
  
  
• When creating the objects
  
  
  
  • If bulkCreate returns a conflict, there are now multiple remediations
  
  
  • Overwrite
    
    
    
    
    
    • Only possible when the user has overwrite false
    
    
    
  
  
  • Generate new ID
    
    
    
    
    
    • When regenerating the ID, we’d also update all references to use the new ID
    
    
    
    • Create a new regenerated-id saved-object
    
    
    
  
  
  • Use previously generated new ID
    
    
    
    
    
    • Only possible when there’s a relevant regenerated-id saved-object. There could be multiple, if we let there be...
    
    
    
    • When choosing this, we’d also have to update all references to use the new ID
    
    
    
  
  

[elasticmachine]

Pinging @elastic/kibana-security

[Mekk]

I raised #35380 for scripted fields in particular. Those are sometimes uneasy to write, require patching, and having to update them in 17 spaces made by various people is just painful.

Some loose ideas:

• what about _inheriting_ new space from another one (so everything created on the „base” space is shared everywhere, but not the other way around – this would make it possible to maintain „exemplary/initial/official” spaces with curated scripted fields, patterns, common queries etc from which people would derive their private/team subspaces)

• what about explicit „import from another space” (or mayhaps „link to object from another space”) - on individual objects level, for example import just those two sripted fields defs

• what about „global” scripted fields available to all spaces based on index(es) those global fields target?

[Matheu-Chambers]

HI, Just thought I'd get in on this too... I've not much to add from what has already been said, really. My situation is that I'm creating dashboards for different users, but that use the same index-pattern, so having that available across spaces would be ideal...

thanks

[Mekk]

I abandoned using scripted fields (in part because of this problem), but I miss some way of „inheriting exemplary space” more and more – I maintain curated space with generally useful queries, visualizations, and dashboards, I encourage people to create their own spaces so they can freely create and save their own objects, but once they do it, they loose all the base (and even if they export/import them, they no longer get them updated after base changes, and no longer see new items). In the end people mostly switch between spaces, but this is confusing.

So, I think it would be best to „inherit”, so queries, dashboards, visualizations, visualizations and index patters can be seen and used in the child space.

Of course changes made in the child space should not impact the base (probably it would suffice to reject an attempt to save updated query or dashboard if it is imported, save as will work…, or mayhaps in such a case new object should be created in child space shielding the original)

[kobelb]

For those following along, the description has been updated with the requirements and general technical approach

[geheim01]

Is it working? I'm pretty interesed in having a visualization in two spaces, synced at the same time. So pretty much what you wrote. But this thread reads more like an idea of how to do it than a proper implementation. Is there any best practice how to do it?

Many thanks

[arisonl]

@ModernMozart Thanks for your interest. This feature is at an early stage of development and will be available in future releases. Would be great to know more about your use case, if you could share.

[XavierRamosORGADATA]

Yes, having dashboards and visualizations synced across different spaces would be awesome!

[Mekk]

My use case, just in case ;-)

1. I, powerful and kibana-skillful Mekk, maintain curated "Main" space. There I have 10 crucial index patterns, 24 useful queries, 11 dashboards, some timelions, a few painless fields, etc. I also have some useful settings like a bit different list of time ranges for Discover, much bigger row width and more compact datetime format. From time to time (usually a few times a month) I add sth new, or tweak some element.

2. New users come regularly. When some Paul or Jenny arrives and asks how to start with Kibana, I want him or her to have own space (after all they are going to start saving own queries fairly quickly, and attempts to share single space ended up with horrible mess of 250 chaotically named queries nobody could use anymore). But I also want him or her to see my queries and dashboards, as they are handy. I am also tired with showing how to clickety-create 10 index patterns.
   So I dream about being able to say „click Create Space, choose inherit from Main, and you will be done, then start from trying queries you will find in Discover/Open”.

3. I'd also like them to see my modifications. New query for new kind of data. New index pattern. New column in old query. Updated dashboard. I dream about them seeing those without the need to perform any action. There are many of them, those users. They miss mails, they misclick, they don't care. And they need not even know that I fixed dashoard definition because it had wrong formula. And I really don't want to check whether they see sth wrong because they still have unpatched one.

4. Well, as Paula started to write interesting dashboards used by testers, it would be nice to let her curate Tests space. My queries are helpful for them too, so it would be nice if Tests inherited from Main, while Joe, Philip, Marie (and Paula too) could create personal spaces inheriting from Tests (so they could see both items created in Main, and in Tests). Best if those who already created space could patch it to change inheritance without loosing own saved items. Or maybe add new inheritance without removing previous one. I am not quite sure whether it is better to have inheritance chain, or multi-ihneritance. Both

     Main <---- Tests <---- Philip

     Main<----Philip
                  |
                 \/
                Tests

would probably do.

1. Getting changes upstream (like getting to Main nice dashboard Adam wrote in his own space) is less critical, here export/import may be sufficient (not that I'd complain if there were some glue to click it directly and detect and resolve possible conflicts). But it is more important to have this dashboard automatically visible by everybody who inherited from Main once it arrives.

[kobelb]

Hey @Mekk, these use-cases that you've provided will be possible once we add the ability to share saved-objects in multiple space in addition to object-level security. Object-level security will allow you to on a per saved-object level specify which users should be able to read, write or share the saved-object. This will allow you to either create an individual space for each of these users, share your dashboards, visualization and index-patterns to those per-user spaces in a read-only manner.

[Mekk]

From my point of view object-level security is not needed, and may in fact defeat the use case. Sharing and importing whole space would be by far sufficient – I wish to share whole space, all in this space, and nothing less. The only security needed is that few people can edit this Main space, while everybody can import/inherit from that space (use those objects), but without the right to modify.
I'd prefer to avoid the need to explicitly grant anything to anybody on object level (I sense very irritating hundred clicks whenever either new user, or new item appears). And I'm 100% sure I wouldn't like people to cherry-pick items (or need to select them in any way). Whole point of my use-case is that users can use everything what was prepared in curated Main space without any effort, including seeing changes.

(I suppose object-level security is more intended towards people sharing single space. IMHO this is worse than having separate spaces as such space ends up polluted and being difficult to use.)

[kobelb]

@Mekk do you currently model your use-case by granting these users read-only access to the curated "Main" space and then all access to their individual spaces? This would allow them to copy the dashboards, visualization, etc. from the Main space into their own individual spaces if they ever want to start making modifications, or else they can just defer to the saved-objects in the Main space.

[rozling]

It strikes me that the inheritance model @Mekk describes above sounds like a basic git workflow... maybe there is no need to reinvent the wheel here?

If you are looking for use cases, currently we have the following setup:

• Using Elasticsearch for logs
• Single cluster, divided into spaces representing our CI / Release / Production environments
• In order to grant users access per space, indices are prefixed ci-_project_, rel-_project_, prod-_project_
• Roles are then created to grant access to these index patterns
• A few savvy users generally create dashboards in Prod and want to replicate them to other spaces.

This is kind of a problem at the moment, as e.g. to create a TSVB visualization they need to specify an index pattern. The index pattern then has to be manually changed for each visualization that gets copied to another space.

When _importing_ saved objects there's a mechanism for changing the associated index pattern, however this doesn't seem to be an option when copying saved objects between spaces (it would be excellent if we could have this!).

In fact, if you copy a dashboard it also copies the related index pattern, which then has to be manually deleted.

I would have expected that a possible workaround would be to point visualizations to custom index IDs, instead of the exact index pattern name, however e.g. in my tests the TSVB doesn't seem to understand custom index pattern IDs at all.

But yeah, unless I'm really approaching things the wrong way, I don't think our issues would be solved by object-level security either, they're more to do with the sharing / copying mechanism itself.

[Mekk]

I am currently encouraging people to export/import, but it doesn't really work (this is troublesome and people don't do it). More often they switch spaces depending on the task at hand, but this is also confusing. So I am simply not happy with the current state of things with respect to sharing.

Regarding previous comment: I don't really like CI/Release/Prod metaphor. For me it is mostly „there is a bunch of things everybody needs + then they have their own specific problems and tasks".

[rudolf]

Why do we need to remove the namespace from the id, what if we just updated the format from ${namespace}:${type}:${id} to ${type}:${namespace+id}? Since ID's can be any string, it would not guarantee that there are no conflicts but it would reduce the probability of a conflict to the same as that of a new random UUID.

With such low probabilities, I think we can treat this as a failed migration which requires manual intervention from the user. In the future, the migration system can just mark the document causing an ID conflict as "invalid" which means it will be unavailable until fixed, but won't stop Kibana from being available.

I'm probably missing something 😅 but the only downside I can think of is that we have ugly ID's like 'security-spacec7437e02-5529-405b-8a34-5e0fd0d1ece2' which might seem like this document is part of the security space, but in-fact it might have been moved to any number of spaces.

[legrego]

what if we just updated the format from ${namespace}:${type}:${id} to ${type}:${namespace+id}

While we commonly auto-generate IDs using a UUID, the API supports end-users creating their own saved object ids, so we could unfortunately end up with a collision fairly easily:

visualization:my-marketing-viz could mean any of the following:
1) a visualization in the default space with user-supplied id my-marketing-viz
2) a visualization in the my- space with user-supplied id marketing-viz
3) a visualization in the my-marketing space with a user-supplied id -viz
4) a visualization in the my-marketing- space with a user-supplied id viz
...etc.

[jportner]

Why do we need to remove the namespace from the id, what if we just updated the format from ${namespace}:${type}:${id} to ${type}:${namespace+id}?

I struggled with this too as I started to work on implementing Phase 1. To add to Larry's comment:

Just to clarify, this migration will only take place when an existing saved object type is changing to support multiple namespaces (so it can be shared between spaces). When that happens, we can no longer leave the namespace in the ID. This is because that saved object may exist in multiple spaces, and the namespace isn't known to consumers, it's injected by the SpacesSavedObjectsClient.

So if we had an object with an ID that is currently space1:dashboard:foo, and we change the ID to dashboard:space1foo, then the user decides to share the saved object to space2, how will the SavedObjectsRepository know what ID to look for when the client wants to retrieve a saved object of type dashboard and ID foo? The only way I see of us supporting that is creating a separate alias of dashboard:<namespace>foo for each space that this saved object is shared to. Then we're in a similar situation, just with more aliases floating around, and the additional potential for collisions / nondeterministic behavior that Larry outlined.

I'm not in love with the whole approach of removing the namespace from the ID and using aliases to preserve URLs for existing saved objects, but it seems to be the solution with the fewest downsides and the least long-term potential for undesired behavior.

[pmuellr]

Question about the use of namespace in Technical Solution above. I may have missed this in the description / discussion above ...

Currently namespace is a field in SO, but is only set when the namespace isn't the "default" namespace. So the search for SOs in the default space has to use a must_not / exists clause:

https://github.com/elastic/kibana/blob/ea7c78d10eaf2f80265eea0a564eda0607c58147/src/core/server/saved_objects/service/lib/search_dsl/query_params.test.ts#L66-L84

So seems like there is at least a problem with the structure that uses an array of namespaces, at least if the mapping for namespaces is keyword and we treat it as an array, since we won't be able to represent the default space.

{
  _id: TYPE:ID;
  type: string;
  namespaces: string[];  // what do we store for the default space?
}

There's also the name spaceId (used as parameter name, variable name, etc), whose definition (I believe) is essentially

namespace ?? 'default'

I'm wondering if we're going to have to use the "spaceId" value instead of "namespace" value in various places.

And then what that means if you have spaces disabled :-)

For reference, as we're implementing the Alerting "event log", we need to store references to saved objects (current design) in event documents, so my current thinking is that I'll go ahead and make that shape an array instead of single string, and probably name it space_id and use THAT value rather than namespace, to be a little clearer.

[jportner]

So seems like there is at least a problem with the structure that uses an array of namespaces, at least if the mapping for namespaces is keyword and we treat it as an array, since we won't be able to represent the default space.

In Phase 1 of the implementation (#54605), the namespaces attribute isn't comparable to the namespace attribute in that sense. It actually treats the Default namespace as a special case and maps that back to its spaceId.

E.g., a saved object in the Default space and the Foo space will have the following attribute:
namespaces: ['default', 'foo']

If Spaces are disabled, under the hood everything operates in the Default space -- so newly-created shareable saved objects will have an attribute of namespaces: ['default'].

As far as references are concerned, I would continue with your old approach. If you use the SavedObjectsRepository to get an object, you still specify the namespace, not the spaceId.

[rudolf]

I should have been more clear in what problem I see this solving. It wasn't intended as a way to remove aliases, but rather a way to deterministically generate id's that would guarantee no conflicts could arise. If id's are deterministically generated this becomes a standard saved object migration, we don't need to topologically sort saved objects according to references, we can just load documents one at a time and update their id's and their reference id's and write them back.

This has a further benefit in that we could remove the saved object migration lock, if migrations are deterministic we could have multiple Kibana nodes participate without any synchronisation or shared state.

@kobelb and @jportner suggested constructing these ID's using uuid.v5 which is a nice way to to remove the semantic meaning in the namespace from the id to avoid any user confusion it might cause.

[rudolf]

The downside to having a truly deterministic migration that doesn't require a lock is that it would require regenerating ALL id's not just the conflicting ones.

If we have a lock we can still avoid topologically sorting all objects, but we would have to load all objects into memory to detect possible collisions before we start the migration. Once we've done this there isn't such a big benefit to not trying to topologically sort them.

[streamich]

When that happens, we can no longer leave the namespace in the ID. This is because that saved object may exist in multiple spaces, and the namespace isn't known to consumers, it's injected by the SpacesSavedObjectsClient.

@jportner Is it true that the only reason we want to change the IDs is because how SpacesSavedObjectsClient works? Or are there other reasons?

If the above is the only reason, I think we should seriously think about what we can do to keep the IDs as they are and from now on treat them as opaque strings.

Instead of changing the IDs and implementing all the changes necessary for that throughout Kibana (e.g https://github.com/elastic/kibana/issues/59598, https://github.com/elastic/kibana/issues/59960) change how SpacesSavedObjectsClient works and do a one time migration to facilitate that.

[rudolf]

The downside to having a truly deterministic migration that doesn't require a lock is that it would require regenerating ALL id's not just the conflicting ones.

This was in the context of evaluating if we could stop creating a new index for each saved object migration and re-use the existing index instead. If, however, we keep a copy of the outdated index it means a new node can always compute the list of conflicting id's. This means even if another node is halfway through it's migration a new node would still be able to determine the conflicting id's and only deterministically regenerate those.