Sktime: Function to convert long format to nested pandas DataFrame

[mloning]

Describe the solution you'd like

• The inverse function of from_nested_to_long()
• Takes in a pd.DataFrame in long format with columns for the instance, time point, variable and value and returns a nested pd.DataFrame.

[fkiraly]

I assume it should create NAs where different series have different indices, and, say, the series in column A does not have an entry at the index of column B?

[hamzahiqb]

Hi,

I tried to write this function but ran into an issue. Here's the replication code:

import numpy as np
import pandas as pd
from sktime.utils.data_container import from_nested_to_long
from sktime.utils._testing import generate_df_from_array, _assert_almost_equal

# Create a nested array
n_obs_X = 20
n_cols_X = 3
X = generate_df_from_array(np.random.normal(size=n_obs_X), n_rows=10,n_cols=n_cols_X)

# Turn it into long format
Y = from_nested_to_long(X)

The issue that I face is that the index dtype does not match for X and Y. That is:

X.iloc[0,0].index.dtype => dtype('int64')

Y.time_index.dtype => dtype('O')

I use the following script to convert Y back to nested format:

def squish(y):
    x=pd.pivot(y, index="time_index", columns="column", values="value").sort_index()
    x.index.name = None
    d = pd.Series({idx: x.loc[:,idx] for idx in x.columns})
    return d

X_renest = Y.groupby(["index"]).apply(lambda x: squish(x))

X_renest.iloc[0,0].index.dtype
# dtype('O')

Not sure if this dtype change is expected out of from_nested_to_long.

Additional side effects I noticed:
type(Y.time_index.iloc[0]) => str
Y.index.is_unique => False

The time index in X is a RangeIndex dtype. When we convert to long format, we lose the dtype information from X. Since RangeIndex is an object dtype, the conversion to long gets the correct time values but converts it into a string object. We can turn the string object to an int, but not sure how one could ensure exact replication from long to nested

Since we start with 3 nested columns in X, Y get 3 duplicate index values (20 time points * 10 rows).

[mloning]

Sorry, I don't fully understand what you mean.

• I think the index dtype shouldn't change when going from one format to another. That's probably something we should fix in from_nested_to_long();
• For the inverse from_long_to_nested(), we also don't want to change the index type and the index of the returned nested data frame are the unique values of the index of the long data frame.

Does that help?

[hamzahiqb]

@mloning got it. I can open an issue for the dtype change in from_nested_to_long().

[Rohith295]

take

[mloning]

Best to discuss between the two of you @hiqbal2 and @Rohith295 who wants to tackle this! There's also the related #311 issue!

[hamzahiqb]

take

[TonyBagnall]

hi, is this enhancing what is already there? https://github.com/alan-turing-institute/sktime/blob/master/sktime/utils/load_data.py def from_long_to_nested(long_dataframe):

[mloning]

Good catch @TonyBagnall!

@hiqbal2 check out the existing function which I forgot about. I'd still appreciate a PR to ensure that index types are preserved through the conversion!