Xarray: Removing dimensions from Dataset objects

[brynpickering]

test_dataset = xr.Dataset(dict(
    empty_array=xr.DataArray([], dims='a'), 
    populated_array=xr.DataArray([1], {'b':['1']}, 'b')
))

I have a dataset that is produced programatically and can at times end up producing an empty DataArray. This isn't an issue, per-se, because I can remove those empty DataArrays later. However, I cannot find any way in which to remove the unused, empty dimension! I've tried deleting, dropping, resetting indeces, etc., and have had no luck in purging this empty dimension. It causes issues down the line as existence of entries in the dimensions list triggers certain events.

Is there a way to remove a dimension (and possibly then all data variables which depend on it)?

[max-sixty]

I think SO is the best place for user Qs, so the answers can be searchable for future generations.

To respond immediately though, have you tried .squeeze? Or, to confirm, you actually want to remove a rather than b here?

In [1]: import xarray as xr

In [2]: test_dataset = xr.Dataset(dict(
   ...:     empty_array=xr.DataArray([], dims='a'),
   ...:     populated_array=xr.DataArray([1], {'b':['1']}, 'b')
   ...: ))

In [3]: test_dataset
Out[3]:
<xarray.Dataset>
Dimensions:          (a: 0, b: 1)
Coordinates:
  * b                (b) <U1 '1'
Dimensions without coordinates: a
Data variables:
    empty_array      (a) float64
    populated_array  (b) int64 1

In [4]: test_dataset.squeeze()
Out[4]:
<xarray.Dataset>
Dimensions:          (a: 0)
Coordinates:
    b                <U1 '1'
Dimensions without coordinates: a
Data variables:
    empty_array      (a) float64
    populated_array  int64 1

[brynpickering]

I don't think it's actually possible to purge a, hence why I started an issue rather than SO Q. As you can see, squeeze() removes b from the list of dimensions, but not a (as a has a length 0, not 1).

[max-sixty]

Hmmm, this is harder than I originally expected. I imagine someone will comment with an easy solution, otherwise I'll have another look

[shoyer]

If you're OK creating a new Dataset, it works to remove any variables using a dimension, e.g.,

In [25]: test_dataset = xr.Dataset(dict(
    ...:     empty_array=xr.DataArray([], dims='a'),
    ...:     populated_array=xr.DataArray([1], {'b':['1']}, 'b')
    ...: ))
    ...:

In [26]: test_dataset
Out[26]:
<xarray.Dataset>
Dimensions:          (a: 0, b: 1)
Coordinates:
  * b                (b) <U1 '1'
Dimensions without coordinates: a
Data variables:
    empty_array      (a) float64
    populated_array  (b) int64 1

In [27]: test_dataset.drop('empty_array')
Out[27]:
<xarray.Dataset>
Dimensions:          (b: 1)
Coordinates:
  * b                (b) <U1 '1'
Data variables:
    populated_array  (b) int64 1

You're right that this doesn't work to remove dimensions from existing datasets (e.g., with del). But this isn't specific to dimensions of length 0 -- you can't delete any dimensions. At best, you can delete a coordinate variable corresponding to a dimension:

In [46]: test_dataset
Out[46]:
<xarray.Dataset>
Dimensions:          (a: 0, b: 1)
Coordinates:
  * b                (b) <U1 '1'
Dimensions without coordinates: a
Data variables:
    empty_array      (a) float64
    populated_array  (b) int64 1

In [47]: del test_dataset['b']

In [48]: test_dataset
Out[48]:
<xarray.Dataset>
Dimensions:          (a: 0, b: 1)
Dimensions without coordinates: a, b
Data variables:
    empty_array      (a) float64
    populated_array  (b) int64 1

In [49]: del test_dataset['populated_array']

In [50]: test_dataset
Out[50]:
<xarray.Dataset>
Dimensions:      (a: 0, b: 1)
Dimensions without coordinates: a, b
Data variables:
    empty_array  (a) float64

Is there a way to remove a dimension (and possibly then all data variables which depend on it)?

This used to be possible in the xarray data model prior to v0.9.0. del should to delete a dimension corresponding to a coordinate variable and all other associated variables.

When we made coordinates optional, I updated del to only delete data/coordinate variables. This made sense, but meant there is now no way to get rid of dimensions.

I'd like to suggest two possible fixes:

1. Update del dataset[key] to remove all "orphaned" dimensions that don't correspond to a dimension on any variable. This would enforce the invariant "all dimensions on a dataset correspond to a dimension on one of its variables." If there are other places where we violate this invariant those should be fixed, too.
2. Update Dataset.dims to allow deleting elements. Deleting a dimension on a dataset means all associated variables are also deleted.

[brynpickering]

The drop technique seems reasonable, if a bit long-winded for the programmatic case (loop over all dimensions, find any that are empty -> loop over all variables, drop any that contain those empty dimensions).

As an addition, if the empty dimension also has an associated empty coordinate then it requires an extra step to get rid of it:

In [21]: test_dataset = xr.Dataset(dict(
    ...:     empty_array=xr.DataArray([], dims='a', coords={'a':[]}), 
    ...:     populated_array=xr.DataArray([1], {'b':['1']}, 'b')
    ...: ))

In [22]: test_dataset
Out[22]: 
<xarray.Dataset>
Dimensions:          (a: 0, b: 1)
Coordinates:
  * a                (a) float64 
  * b                (b) <U1 '1'
Data variables:
    empty_array      (a) float64 
    populated_array  (b) int32 1

In [23]: test_dataset.drop('empty_array')
Out[23]: 
<xarray.Dataset>
Dimensions:          (a: 0, b: 1)
Coordinates:
  * a                (a) float64 
  * b                (b) <U1 '1'
Data variables:
    populated_array  (b) int32 1

In [24]: del test_dataset['a']

In [25]: test_dataset.drop('empty_array')
Out[25]: 
<xarray.Dataset>
Dimensions:          (b: 1)
Coordinates:
  * b                (b) <U1 '1'
Data variables:
    populated_array  (b) int32 1

Fixes seem reasonable, based on how we use xarray over at https://github.com/calliope-project/calliope/. The second one also provides more scope to remove subsets of data (all corresponding dims, coords, vars) if the dimension becomes superfluous for any reason, whether or not the dimension is empty.

[shoyer]

The second one also provides more scope to remove subsets of data (all corresponding dims, coords, vars) if the dimension becomes superfluous for any reason, whether or not the dimension is empty.

Yes, this was a useful feature that we lost.

Note that in general we try to encourage using methods to create new Datasets rather than modifying existing ones inplace. So it might also make sense to add a drop_dims() method that return a Dataset with the given dimensions removed.

[FaustinCarter]

So does this mean that the following line in the docs is now false: "If a dimension name is given as an argument to drop, it also drops all variables that use that dimension"

This is at http://xarray.pydata.org/en/stable/data-structures.html#dataarray.

It does not seem to work as advertised. Before drop:

<xarray.Dataset>
Dimensions:      (Frequency: 2000, Index: 1, Power: 1, Temperature: 1)
Coordinates:
  * Power        (Power) float64 -117.0
  * Temperature  (Temperature) float64 1.16
  * Index        (Index) int64 0
  * Frequency    (Frequency) float64 3.865e+09 3.865e+09 3.865e+09 3.865e+09 ...
Data variables:
    I            (Power, Temperature, Index, Frequency) float64 ...
    Q            (Power, Temperature, Index, Frequency) float64 ...
    Time         (Power, Temperature, Index) datetime64[ns] ...

Then I call ds.drop('Frequency'), expecting that this will kill off the Frequency dimension, coordinate, and the I and Q variables, since they depend on Frequency, but what I get is:

<xarray.Dataset>
Dimensions:      (Frequency: 2000, Index: 1, Power: 1, Temperature: 1)
Coordinates:
  * Power        (Power) float64 -117.0
  * Temperature  (Temperature) float64 1.16
  * Index        (Index) int64 0
Dimensions without coordinates: Frequency
Data variables:
    I            (Power, Temperature, Index, Frequency) float64 ...
    Q            (Power, Temperature, Index, Frequency) float64 ...
    Time         (Power, Temperature, Index) datetime64[ns] ...

If I then try dropping Frequency again, it complains that there are no variables named 'Frequency'. So probably this issue should include an update to the documentation. Or maybe that should be a new issue.

[shoyer]

So does this mean that the following line in the docs is now false: "If a dimension name is given as an argument to drop, it also drops all variables that use that dimension"

Oops -- yes, that line in the docs / example is broken!

[kmsquire]

I was looking for a way to drop dimensions, similar to the OP, and found this issue. I created an implementation of Dataset.drop_dims in #2767 which should work.