Sktime: Design and implementation of time series annotation framework

[mloning]

For a preliminary design discussion, see our wiki entry.

[ianboru]

Hey this whole project looks amazing. Is there any sense of when time series annotation will be usable?

[mloning]

Hi @ianboru and thanks! It's hard to say, we have written some initial design ideas here but haven't started implementing anything yet. Any help would be very welcome! What kind of problems are you working on?

[ianboru]

I am trying to do some interpretation and classification of electrical signals in the body! Time series annotation or clustering seem to be right in line with the ability to do so. I can potentially find a couple hours in the week to help. What areas do you need most help with? I look at the design page so it looks like you guys are still choosing algorithms and figuring out how to port them, is that a fair assessment?

[mloning]

Yes, algorithm choice is part of it. If you have a good overview, that would be helpful already. But there's also the API design problem: it's not clear yet what a good API for time series annotation would look like. Clarifying that in the enhancement proposal seems like the first step. Implementing a few simple algorithms would then be the next one.

There's is also ongoing work for creating a clustering API (see #501).

Your help would be very welcome! I can invite you to our slack space if you want to send me your email address (e.g. via a direct message on Gitter).

[ianboru]

"If you have a good overview, that would be helpful already" - Do you mean an overview of my project/use case? [email protected] for the slack invite.

[mloning]

Overview of common algorithms for time series annotation. But note that there are different variants of the time series annotation task. I think it's best to continue working on the enhancement proposal: https://github.com/sktime/enhancement-proposals/blob/master/steps/06_time_series_annotation/step.md

[ianboru]

Just reviewed that. I guess I wonder what your process is for defining an MVP. Do you have guidelines to help bound the research phase? I'm happy to chat in slack to get a better idea of an atomic task that I can start and finish in a reasonable amount of time.

[fkiraly]

"MVP" depends - are you considering to invest time helping out, @ianboru?
The process is investing some work in planning (on the core dev side) if there is interest from the community, then proposing smaller chunks as issues or work items, working closely with members of the community.

What the design is still missing is explicit specification of input/output types for the annotators, at least for a "core" set of annotation tasks.

Btw, looking quickly, you don´t seem to have much coding (or python) on your Github account? Are you proposing to contribute, or just wondering in general how we operate?

[ianboru]

@fkiraly I am considering investing time. I reached out to Markus on slack but I'm happy to continue the conversation anywhere. I guess most of my repos are private but I regularly work in python. react, react native, swift and android java. I have used machine learning in various capacities in my biophysics/bioinformatics phd, data scientis/data science director roles and my personal projects.

My primary goal earnest is to find something that works for my personal electrophysiology project, but if I can contribute along the way then that's great.

[fkiraly]

ah, let's continue on slack then. Wasn't aware of that conversation, sorry.

[fkiraly]

Btw, some interesting items about annotation, @ianboru:

• the design in the STEP is not fully complete - we should work out input/output signatures. Some of this is in flux due to the re-work of the forecasting design, but should be hammered out.
• suppose we identify a core set of annotation tasks, e.g., segmentation and change point detection. What would be a "good first prototype?", e.g., in terms of algorithms to implement and initial interface?

[mloning]

Hi @ianboru and @fkiraly, sorry for the confusion, I prefer design discussions on GitHub (either here or via a PR on https://github.com/sktime/enhancement-proposals) so that others can get involved and use slack only for quick questions (debugging, minor clarifications, etc).

[fkiraly]

I prefer design discussions on GitHub

@mloning, so do I, but I thought there were already some minor org or onboarding discussions ongoing on slack. Perhaps I misunderstood.

Generally, it is good practice to keep any technical and high-level org things open.

@ianboru, slightly confused now - were you referring to an existing conversation on slack?

[ianboru]

@mloning @fkiraly In reference to the MVP, I usually try to think of what minimal piece of work would create value for the user. For myself as a user, I have two major use cases of varying complexity and value.

1) The simplest and lowest value is to input 1 time series with no labels, and output an indication for every time point whether the point is in a region of "change". I have data about when I might expect the changes to happen but the unsupervised result is more interesting to me. As I understand it, ruptures should do this to some degree but I haven't gotten it to work fully as expected yet.

2) The next level of complexity and value is to input 1 time series with no labels, and output a label for each region of "change" for an unknown number of regions and labels.

So for me an MVP is sufficient if I can do 1 in a reliable fashion that let's me visualize the original data and the labels.

Now presumably the goal of sktime is not to simply wrap another api, but for example if there are any limitations in ruptures that can be overcome either in terms of accuracy, or in terms of not needing to predefine the number of breakpoints for example, then fixing those problems and wrapping it in sktime could still be valuable.

So ultimately, I think each use case can have its own MVP so it can be shipped and tested ASAP instead of waiting for all use cases to be solved first. Currently I don't know what you all see as the limitations in the "related" software, which would help understand what problems are left to solve.

RE: design process/workflow

Personally I find discussions that can have many questions, recapping of history, onboarding and many off-the-cuff ideas that likely won't make it into any final designs, to be had as close to real-time as possible. Then the summaries and relevant points of those discussions can be documented somewhere like git. The limitation to this format is that we have to post and wait, sometimes just to clarify confusion caused by short statements and answers.

@fkiraly to clarify about my mention of slack, I sent @mloning a message to start explaining how I might help but he re-directed me here so no substantial conversation occurred.

[mloning]

@ianboru yes, that makes a lot of sense!

A large part of sktime is to clarify and distinguish different learning problems in the time series domain. If other packages implement specific algorithms, we do wrap them so that it becomes easier for users to combine them in the same workflow without having write glue code. If other important algorithms are missing, we want to implement them.

Regarding use case 1, I'd start by taking a look at https://adtk.readthedocs.io/en/stable/. statsmodels also has some change-point detection methods. The question then is how can we provide a unified interface to these methods?

Since there's currently no-one else working on it, you're more or less free to proceed how you prefer. Working on the enhancement proposal and/or opening a PR early will make it easier for us to give feedback. I'm happy to jump on a phone call and discuss some concrete proposals.

[fkiraly]

@mloning, @ianboru, agreed to what you are saying. The first step is to hammer out a full design for a circumscribed set of use cases.

@ianboru, you describe two use cases, in my opinion these are identical with:

• unsupervised change-point detection - which, as you say, ruptures already does reasonably well. Might be worthwhile discussing and understanding what you think it doesn't do well or why it doesn't work. @markus, I think change-point detection is different from (but closely related to) anomaly detection which is the problem that ADTK solves.
• unsupervised segmentation - you are asking for segments, optionally also with inferring number and/or consistence choice of segment labels

Some important points to think about i.m.o.:

• in the change-point detection case, it is worth thinking about how different label types are represented using a similar interface, and how the closely related tasks/problems are classified, e.g., change-point vs anomaly detection
• segmentation seems not to be supported by any toolbox I am aware of, so thinking about a segmentation interface and input/output format might be key
• I would also consider it important to design the interface keeping compatibility with the the respective supervised variant in mind.

[fkiraly]

Now presumably the goal of sktime is not to simply wrap another api, but for example if there are any limitations in ruptures that can be overcome either in terms of accuracy, or in terms of not needing to predefine the number of breakpoints for example, then fixing those problems and wrapping it in sktime could still be valuable.

"number of breakpoints" would be a hyper-parameter of a specific method in a sklearn-like API.

[ianboru]

Ok thanks @mloning @fkiraly I will digest all of this and see what I can get to this week. I think I can at least put out some ideas for the interfaces, dig into ruptures a bit to see if it works for me or if it falls short, and see if I can identify the steps to MVP for the simplest use case + implementation combination.

[ianboru]

@fkiraly @mloning

Ruptures Overview:

After reading most of the ruptures related paper, looking more closely at the docs and then using ruptures again, I must say it is quite well done and comprehensive. Apparently it does include algorithms with AND without a known number of changes i.e. semi-supervised and unsupervised. Moreover it can handle multiple time series at once. It also has parameters like min_size and jump to help speed up searches based on the scales at which changes are expected to occur. Originally I didn't see those parameters so it ran quite slowly on my data.

As far as the API, it follows the standard .fit .predict pattern. It also comes with its own display method that hooks returns a matplotlib figure. Although there might be some slight personal preferences I have about how the documentation could be better designed, overall the api is well thought out and works well for its use cases.

In terms of integrating ruptures into sktime, one clear options is to just attach it as is. But if you ultimately want to include other library that have a higher level categorization such as supervised and unsupervised than it might be better to add different algorithms in ruptures to different parts of the sktime api. Currently the first level of the ruptures api is separated by algorith e.g. Pelt, Dynamic Programming, window sliding segmentation rather than specifically by the type of problem being solved or some other conceptual distinction.

What's missing:

After this deep dive, I see ruptures clearly covers my first use case. There is a substantial "limitations" section in the ruptures related paper that described a few classes of algorithms that are left out of ruptures, so that is one place to find other methods to gather for sktime. For my second use case I wondered if I could use the breakpoints given by ruptures to feed into some classification method like a random forest or neural net. I then wondered what I could find about just using neural net for the sequence annotation/classification problem.

My next step for my personal project is to investigate that avenue which includes these two interesting resources:
https://www.kaggle.com/meaninglesslives/simple-neural-net-for-time-series-classification
https://towardsdatascience.com/how-to-use-convolutional-neural-networks-for-time-series-classification-56b1b0a07a57

For now I have limited time for this project in general so I will focus on collecting and implementing methods to solve my own use cases. If you'd like I can report back and maybe further down the road if there is still work needed to integrate some of these methods then I can help.

Thanks,
Ian