Sktime: Design/implementation of time series data container (pandas vs xpandas ... or sth else?)

[fkiraly]

As @sajaysurya pointed out, we may not at all need xpandas for the core use cases.

This is a high-priority issue to be decided by w/c Feb 4 meeting, since it the data container is a central design decision. This thread is to collect pro/con until the decision is made - issue is complete once we decide to remain with, or leave xpandas as the data container solution for the API and implement the alternative (note: in the case of leave the issue is only complete once the alternative is implemented in the existing code).

[fkiraly]

One aspect to consider is type-awareness of xpandas. Later on, we may like to make a query through high-level interface and/or registry such as "please find a transformer/estimator combination suitable for this task/data", or "please transform all series columns to primitive columns using the standard". Vanilla pandas cannot distinguish string columns from series columns via the dtype argument, so helper functionality would have to be written.

[fkiraly]

Here's vitaly's response:

"Hm, our main purpose was providing researchers who work with different type of data a unified interface and data container. So the one can store images/series/... and anything else in homogeneous style inside a "data frame". It makes sense to use use the exact dtype, not abstract object because it reduces the potential error user might make. For example, user can randomly put a series inside an image column -> transformer will fail on that cell -> it's hard to find that kind of errors.
I think it sounds legit, what do you think?"

Sounds similar to my point above?

[TonyBagnall]

So this is my 2p's worth. Bare in mind I am not a python or R programmer of any experience.

1. I believe xpandas extends pandas. Therefore, until we run into a scenario where xpandas functionality is required, algorithm development should I feel continue with pandas. Simplest solution being the best and all. There is nothing lost, and if we want to enforce xpanda input it will be very easy to update it later.
2. python is a weakly typed language, and has no equivalent to generics in java or templates in c++ to restrict typing afaik. You are trying to enforce the characteristic loved in typed languages on a language that actively discourages type safety of any sensible kind (from a Java perspective). I think this may be confusing for people who like weakly typed languages (I'm using empathy here, I have no idea how such people think, except that they like ducks).
3. My preference would be to enforce type safety through data loading routines, e.g. load timeSeries or somesuch would throw exceptions if images or ducks were attempted to be loaded via a method meant for loading time series
4. I think classifiers/regressors should check whether they have the capabilities to handle the type of data given to them, throwing exceptions if they cannot.

to summarise, my preference would be to proceed with pandas until we have the vanilla/univariate TSC algorithms at a decent stage, then reassess. This will give us a suite of classifiers that are completely scikit-learn compatible, which we can then enhance with specialisations able to handle special cases such as multivariate/unequal length. Similarly with forecasting and regressors. From the algorithm development side, it is not that big a deal switching, and delaying the implementation side of things will hold up the project.

[fkiraly]

@TonyBagnall , makes sense - I'll have some further thoughts but I might start to agree.

[fkiraly]

Suggestion for discussion in Feb 5 meeting: pandas seems to have all features necessary for low-level modelling and most of high-level modelling. Hence make pandas standard, but define the features it needs to have. xpandas should be compatible via duck-typing, and we may require it later for high-level model or registry when we make selection-by-type.

[fkiraly]

(that's the decision made)

@mloning, @sajaysurya, please confirm once code is migrated to close this issue.

[sajaysurya]

The migration is complete, and currently we don't have any explicit dependency on xpandas.

[sajaysurya]

Dask (http://docs.dask.org/en/latest/) can also be considered as an alternative (explained in #22)

[mloning]

Reasons for extending pandas:

• typing columns using scientific types (scitypes),
• separation of meta-data for high-level interface
• single input checks (n_cols, missing values, unequal length series, etc.)
• slicing/selecting in nested dataframe, possibly with time as third dimension
• linking data to file via secure hash values

[TonyBagnall]

So I'm leaning towards extending pandas when we have done the minibake off, primarily for storing metadata such as contains missing, unequal length etc. Otherwise we rely on the classifier to check this, which seems a little wasteful, brittle and poorly encapsulated, these are properties of the data not the classifier. I dont think we necessarily need to use XPandas to do this, we could just write a stripped down extension ourselves, but obvs things will be different in forecasting world. Lets revisit after march meeting.

[fkiraly]

Here's some related discussion on the MLJ project:
https://github.com/alan-turing-institute/MLJ.jl/issues/86
python is not a strongly typed language, but even in Julia we ended up introducing a second type of type - the scientific type. Which is the kind of metadata that, now that I think about it, would be nice to have kept track of by a data container and task class system.

[matteogales]

Came across this as a way of providing custom properties on a Pandas DataFrame, might be a nice alternative to subclassing....

http://pandas.pydata.org/pandas-docs/stable/development/extending.html#registering-custom-accessors

[mloning]

Related comment from @big-o in #200

Data containers

I realise this is a hot topic that's already being looked at, but I'll add my thoughts here as I believe it affects the forecasters. @mloning recently wrote a branch that changed forecasters to work with a normal Series instead of the "nested dataframes" used elsewhere. In my opinion this is a huge improvement. The nested frames were fiddly and unnecessarily complex, especially for new users. Unfortunately it doesn't escape the fact that you still need to convert things to/from nested frames if you want your forecaster to interact with any other parts of the library. I hit on this issue when writing the code for #198 - I had to deseasonalise/reseasonalise my series within the forecaster. To do that I have to perform some messy transforms on the data containers. Fortunately this is hidden from the users, but what if the user wants to build a pipeline like [Transform, Forecast]? I think they will have the same issue.

One idea I discussed with @mloning was to move to using xarray instead, which supports N-D dataframes and therefore covers off multiple time series. The issues with xarray, as I understand, are:

It won't support multiple unequal-length time series.
It won't support multiple series with unaligned time indices.
We can address both issues with padding, but what value do we use to pad with? If we use the obvious choice, NaN, then we lose the ability to distinguish between missing data in the inputs and our padding. This to me isn't a problem with the data container; it's a lower-level problem caused by the fact that numpy defines two kinds of missing data, NA (Unknown Yet Existing) and IGNORE (Doesn't Exist), yet it provides no way of dealing with both types at once.

I initially thought we could create a new type of padding object that behaves in a similar way to NaN but is distinguishable from it, or use sparse matrices that can contain NaN values, but I can't think of a good way to implement either of these approaches. Instead, I thought we could let the user provide a padding mask instead. This mask could be optional, that way the use case of having aligned, equal length time series can be achieved with a simple xarray (or even a flat dataframe for univariate series). For unaligned/unequal series, a helper function could be provided that converts a list of pandas.Series into an (xarray, padmask) tuple. The padmask can then be used internally to distinguish the NA NaNs (missing) from the IGNORE NaNs (padding). Hopefully this won't make it too difficult to implement algorithms, but I'll leave that for others with more knowledge to comment on.

In the case of forecasters and transformers, this would mean having an new optional padmask parameter in fit(). I don't think it would be needed in predict as the padmask should be the same for transformers (we don't transform non-existent data) and irrelevant for forecasters.

I appreciate that this would be a huge change, but I believe it would pay off in the long term to move away from the nested format if possible and now is the time to make sweeping changes while the project is still young.

[fkiraly]

Regarding the general question, based in what we've learnt since the early days, I think the following things are important:
(i) inspectability, especially regarding (implicit) type, content, format; e.g., to allow type checks or content dependent control statements - one of the reasons we were considering xpandas as a custom solution
(ii) efficiency and access layer - repeated and possibly unnecessary data re-formatting can be a bottleneck, as composite implementations of highly composite learner such as RISE/BOSS have shown
(iii) light-weight and intuitive user interface, optimally pandas data frame like

My problem with your suggested padding solution, @big-o, and/or the mask is that it does not seem to be inspectable and at least in parts rely on "implicit interface conventions" that are not encapsulated in the container object itself. Which of course is also true for the status quo (nested pandas), if you are strict.

Even if we move to internal xarray with masking convention, I fear changing the representation wouldn't solve the key issue (ii) if used in vanilla form, and might not be too different in terms of (iii) because you have to keep track of the mask suddenly.

So far, the only solution to that would indeed be a dedicated data container class which can contain non-primitive data types, and has the option to quickly present the data in different formats when queried via an access interface.
Once this exists, it is easy to not only store time series, but also other hierarchical (non-primitive) data types such as segmentations, shapes, arrays-within-arrays, etc.

Though it sounds like a major effort - and a resource question - to do this properly.
If anyone wants to make this, we might probably be early adopters...

[fkiraly]

Regarding the specific issues arising from the de/reseasonalizer:
I may be able to guess, but would you mind stating explicitly, for information, what exactly the data format transformations are that you need to make under the hood?

In addition: why would passing to xarray plus masking convention solve this specific need to reformat?

[mloning]

I've tried to pull together the key points from our discussions in our wiki. Feel free to make changes, I've added your points @fkiraly.

[matteogales]

While I can understand the logic described above for proceeding with the current design until later down the road, I do agree with @big-o in that the way in which the time-series data is currently inserted as entries into a DataFrame does feel slightly unintuitive and feels like a misuse of the class. However, in agreement with @fkiraly, while I can see how it does alleviate one problem, using padding and/or masks seems to introduce some complexity/obfuscation that would potentially reduce the benefits of using well known/loved Pandas data structures.

Like @fkiraly, I also think that using a custom data structure may be the way to go, both in terms of providing a more intuitively accessible entry point to the underlying time-series data, but also to act as a central point for providing routines for serialisation and automated data-conversion etc. I have seen an analogous situation in a couple of banks that I worked at where we wanted to ship around several related but distinct structured financial objects (groups of foreign exchange rates, interest rate term structures, deals…) as a cohesive unit to analytics routines and for that container to take care of serialisation etc. We ended up with what is essentially a Dict on steroids, where the keys were strings that if they met a certain naming conventions (e.g. “SWAP.3M.USD.LCH” for a USD swap curve with 3-month tenor cleared at LCH) allowed the analytics routines to easily and automatically detect the presence of objects in the container.

Part of me likes the above approach (possibly using some combination of attribute names and Pandas frequencies as the keys) as it is simple and not a huge amount of work to get something to at least start using as a tangible target for further design discussions. Possibly adding a few simplistic forecast/classification routines that make use of it, so as not to disturb existing code/development and in case even after several iterations of any design it proves not to be a winner, would also seem useful.

Now that I actually have some free time again (initial start-up life killed any capacity that I had to much else this last 4 months), I'm happy to have a look at this, if people think that this could be useful?

[fkiraly]

While I can understand the logic described above for proceeding with the current design until later down the road, I do agree with @big-o in that the way in which the time-series data is currently inserted as entries into a DataFrame does feel slightly unintuitive and feels like a misuse of the class.

Yes, I actually agree with that aspect of the discussion, too.
I didn't like the misuse of the poor pandas data frames from the start, but there are not many low-resource alternatives.

A compromise could be a data frame like facade class similar to Tables.jl in Julia?

I'm happy to have a look at this, if people think that this could be useful?

Absolutely! Would be nice if we could meet for a discussion of alternatives and plan.

PS: we built xpandas
https://github.com/alan-turing-institute/xpandas
with sktime as an eventual use case in mind, but that got binned in the early stages of sktime.
Perhaps it's worth looking at it, even if only to learn from failure.

[matteogales]

@fkiraly - thanks for the xpandas and Tables.jl references; I'll take a look to see what features look like they could be useful. There are also a bunch of packages for storing/representing time-series (using NoSQL databases and compressed files) that have their roots in financial data (more focus on large amounts, due to application domain) that may be of use, so will also sift through those:

https://github.com/man-group/arctic
https://github.com/ranaroussi/pystore

Let me first have a look at the above and any others that Google sends my way, and have an initial go (in my own Fork/branch) at building a quick and dirty prototype (just to clarify some of my own thoughts), and I'll mail out to the group and head into the ATI to discuss.

One thing is certain though, as we are storing a group of Pandas objects, the name of the class is going to be Cupboard... http://www.visiontimes.com/2014/11/07/what-do-you-call-a-group-of-baby-pandas-its-not-what-you-think-video.html

[big-o]

Just to expand on my earlier comments - I was trying to suggest what I considered the best compromise under the assumption that no one wanted to invest the time and ongoing effort in building and maintaining a new data container - especially if it needed to be as feature-rich as pandas. I can see from this thread that I was mistaken and there is indeed an appetite for that - if someone can build a suitable data container then I think it's hard to argue that having to couple two independent objects, which my padmask idea would involve, would be better.

If you do end up creating a new data structure, would it be as an independent project to sktime?

Since there's appetite for work on a larger scale, I'll throw my other idea out there that I was too scared to suggest the first time around... instead of building a new data container, would it be possible to extend the existing data containers by adding support to numpy for dealing with the two distinct types of missing data (NA - aka "missing but exists", and IGNORE - aka padding) at once? _Maybe_ you could do this by adding a new type of missing value (pad) which always gets ignored/dropped from computations. If you had this then I wonder if that would be enough to make existing containers (potentially including sparse ones) suitable for sktime use cases? I appreciate this would have to be done very carefully since numpy is used so widely and could therefore take a while. We would also need to get buy-in from a lot of other people. But I just wanted to throw the idea out there for consideration.

[mloning]

modin may also be an option, as they parallelise many of the pandas operations under the hood.

[fkiraly]

@matteogales, @big-o, thanks for your thoughts!
Both, I'd suggest we have a brief chat about requirements for sktime before doing any substantial implementation. For example, @mloning and @jasonlines have insight in whether a proposed container would remedy the data reformatting bottleneck in RISE/BOSS composites.

@matteogales, arctic and pystore look relevant from an architectural standpoint, but perhaps not exactly what we need.

@big-o, regarding extending data containers for sktime and more general use cases: it might make a lot of sense to do this as a standalone project.
However, I would doubt whether your suggestion introducing NA vs padding fields would truly solve the issue, because I think we do not only need a "jagged array" data structure, but the ability to represent nested jagged data frames, with inspectable structure and possibly named index lists (for variables etc) or index typing (for variable types).
I.e., there should be a distinction whether something is "just" a jagged array, and some additional pandas style inspectable container annotation on whether and which indices are interpreted as samples, time, etc.
From a development perspective, taking pandas as basis rather than numpy array seems more promising to me.

Though of course even efficient jagged multidimensional arrays would already be very useful. Does this exist in python (at good quality)?
For jagged arrays, I'm not sure whether the "mask" idea, or trying to build something from scratch would be the more efficient route.

[fkiraly]

One thing is certain though, as we are storing a group of Pandas objects, the name of the class is going to be Cupboard...

@matteogales , I guess we're lucky that the pandas container is called pandas and not crows then. Or grackles.

[fkiraly]

@mloning , re modin - that's almost the kind of architecture I had in mind.
Nice that it already exists - the question is then how to integrate a custom interface layer module (and/or whether it really is just integrating a custom interface layer module).

[mloning]

Issue to combine awkward-array with xarray https://github.com/pydata/xarray/issues/4285

[vnmabus]

I am not a sktime user, but I stumbled into this issue and I though that maybe I could bring my two cents. I am currently the main developer of scikit-fda, a Python package for performing Functional Data Analysis (FDA). As the objects of study of FDA are functions, with are a generalization of time series or images, maybe you can take a look at our types for inspiration (or to see if they suit you). Currently we provide a type FDataGrid for functions of arbitrary domain and codomain dimension evaluated at a grid or points (not necessarily equispaced, but the same for all the functions of the set) and a type FDataBasis for functions represented as a basis expansion (such as Fourier series).

We also have some utilities that can work with our types, and we try to make our functionality compatible with other packages in the Python scientific stack, such as Pandas or scikit-learn. Our types inherit from Pandas ExtensionArray, so they can be used as columns in a DataFrame.

Even if you decide to use a different approach, we would love to have feedback from sktime users, and we would want to be as compatible with your chosen solution as possible, in order to facilitate the combination of sktime and scikit-fda tools.

[mloning]

Hi @vnmabus, thanks for reaching out!

We have played around with extension arrays but they didn't address the main problem were facing, which is handling unequal length panel or multivariate data (or more generally time-heterogenous data where instances and/or variables do not share time indices). My currently favoured solution is awkward-array, but we haven't started integrating it into sktime.

Where do you see integration points of sktime and scikit-fda, would love to develop an interface to make it easy to use both packages in the same workflow! Perhaps we should open a separate issue for that.

Also feel free to open a PR to add scikit-fda to our list of related software here!