Sktime: Folder structure

[mloning]

When we want to add regressors, we have to adopt the folder structure so that we can factor out parts that are used by both regressors and classifiers. We're currently doing this in sktime-dl, see #22, however for sktime, it's a bit more complicated because we have additional methods for composition and model selection that is specific to tasks.

Here's my proposal:

├── forecasting
│   ├── base.py
│   ├── composition.py
│   ├── exponential_smoothing.py
│   ├── model_selection.py
│   └── ...
├── series_as_features  # any better name? supervised?
│   ├── base.py  # can be used for common base classes for classifiers and regressor
│   ├── classification
│   │   ├── dictionary_based.py
│   │   ├── frequency_based.py
│   │   └── ...
│   ├── clustering
│   │   └── kmeans.py
│   │   └── ...
│   ├── composition.py
│   ├── model_selection.py
│   └── regression
│       ├── dictionary_based.py
│       ├── frequency_based.py
│       └── ...
├── transformer
└── ...

Any thoughts?

[TonyBagnall]

I dont want to put every classifier of each type in one file, sorry, it will just become hugely bloated and I will never be able to find anything. It is also a matter of ownership. If a new algorithm comes along (we have a new interval classifier), then we will have joint ownership of single files, which is confusing. I want to retain the sub directories for classifiers. Series as features doesnt work either, since distance functions and distance based algorithms need to go somewhere. Our four categories are forecasting, prediction, segmentation and event modelling. But then forecasting is prediction. The problem is, what then to call classification, regression and clustering. Apart from machine learning, I dont know, sorry! Can we not just get rid of series_as_features and have forecasting/classification/regression/clustering/transformer ... or should be transformation if we are not using nouns. I personally prefer classifiers/clusterers/regressors/transformers, but then I am object oriented :)

[TonyBagnall]

this is how tsml currently stands. Not ideal actually, dont like elastic_distance_measures as a package, but for info

[fkiraly]

yes, agreed with @TonyBagnall - the draft governance process (ownership, contribution, and maintenance) wouldn't work properly if classifiers are merged in a large file. I propos sub-folders instead, similar to the current structure.

The top-level structure makes sense.

[fkiraly]

Actually, that´s pretty similar to tsml.

[mloning]

I'm not proposing to put every classifier in a single file! We would still have the categories that we're working with now and happy to have more sub-folders if that's preferred.

The problem is

• where do we put common functionality for the series-as-feature (for lack of a better name) setting (e.g. Tuning, composition, reduction) which is not applicable to forecasting or annotation?
• where do we put common base classes shared by both classifiers and regressors?

[fkiraly]

I'm not proposing to put every classifier in a single file! We would still have the categories that we're working with now and happy to have more sub-folders if that's preferred.

Yes, but what I believe what Tony is saying is that the categories-per-file are still too coarse. People may want to own folders rather than just files. @TonyBagnall ?

where do we put common functionality for the series-as-feature (for lack of a better name) setting (e.g. Tuning, composition, reduction) which is not applicable to forecasting or annotation?

I would put it in the folder of the "resultant" scitype, in a separate folder and/or file.
That should be one of the existing categories?

where do we put common base classes shared by both classifiers and regressors?

"base" or "framework" folder?

[TonyBagnall]

yes I want categories per directory, as we have now. I will then personally keep our classifiers in their own file, its just tidier imo although I realise it may not be pythonesque. However, it doesnt have to be a rule of course, in fact we have some variants that would go well in the same file.

[mloning]

Sorry about the confusion, I was more concerned with the top-level structure, I'm happy to keep folders for categories and files for algorithms.

So, how about this:

    ├── forecasting
    │   ├── base.py
    │   ├── composition.py
    │   ├── exponential_smoothing.py
    │   └── model_selection.py
    ├── series_as_features  # any better name?
    │   ├── base.py  # for base classes shared by classifiers and regressors
    │   ├── classification
    │   │   ├── dictionary_based
    |   |   ├── __init__.py  # algorithms can be exposed to users on this level by loading them into the __init__.py file
    │   │   │   └── _boss.py  # individual files for algorithms
    │   │   ├── frequency_based
    │   │   │   └── _rise.py
    │   │   └── shapelet_based
    │   │       └── _shapelet.py
    │   ├── clustering
    │   │   └── kmeans.py
    │   ├── composition.py
    │   ├── model_selection.py
    │   └── regression  
    │       ├── __init__.py
    │       ├── dictionary_based
    │       │   └── _boss.py
    │       ├── frequency_based
    │       │   └── _rise.py
    │       └── shapelet_based
    │           └── _shapelet.py
    └── transformer

[James-Large]

Copied over from https://github.com/sktime/sktime-dl/pull/22#issuecomment-584070979. Counter argument is still e.g. composition methods that apply to classification/regression, but not to e.g. segmentation (that is, to more than one task but not all).

Scatter brain/rambling/not fully formed thoughts ahead, sorry

Ultimately there's going to be redundancy somewhere in the structure most likely, question is what redundancy do you want to accept.

In what ways can these things reasonably be grouped?

1. Input/Output
   1a. series -> series transformers
   1b. series -> vector transformers
   1c. series -> double/vector predictors
   1d. model -> model meta processes (composition, tuning, etc)
   1e. a -> b general/other utilities
   1f. etc.

1.2 Output only?

1. High level data tasks
   2a. prediction (classification, regression, forecasting, etc.), aka predictors, estimators, models etc.
   2b. composition
   2c. selection
   2d. tuning
   2e. representation/transformation
   2f. evaluation
   2g. visualisation?
   etcetc.

2. Prediction tasks (basically the members of 2a)
   3a. classification
   3b. regression
   etc.

3. Conceptual algorithm type
   3a. shapelet vs interval vs dictionary etc
   3b. classical vs modern?

Others?

4 is certainly the weakest (or maybe most niche?) grouping IMO, but parties want it in. I'd argue for 2 being the strongest. There's the question of whether prediction (or whatever name it could be) should be an encapsulating folder, or instead if classification,regression... should be their own top level folders, but yeah you need somewhere to put base.py for estimators so it makes sense for the prediction containing folder to be there really.

Then when it comes to which redundancy to accept in relation to e.g. specific evaluation methods as per example above, you probably repeat the classification,regression... categories in each of composition, selection etc. as needed

sktime/
    prediction/
        base.py
        classification/
             base.py
             dictionary_based/
                  boss.py
             ...
        regression/
            base.py
            dictionary_based/
                ...
            ...
         ...
    evaluation/
         base.py 
         ...
         classification/ 
              classification_evaluation_methods.py  # only if needed beyond base/shared methods
         forecasting/
              sliding_window_splitter.py
              ...         
         ...
    composition/
         base.py 
         ...
         classification/ 
              classification_composition_methods.py  # only if needed beyond base/shared methods
         ...

If you argue for 3 being the strongest grouping over 2, you flip the priority of classification,regression etc and put that on top level, while duplicating prediction,evaluation,composition directories

In terms of usage, the imports for some experiment you want to run would then be something like

from sktime.prediction.forecasting import FancyForecaster
from sktime.evaluation import SomeGenericEvalMetric
from sktime.evaluation.forecasting import SlidingWindowSplitter 
...

and so on. From the point of view of a user, the redundancy might not even be that unattractive really, makes it obvious what methods can be shared between prediction tasks and what are specific.

So long as you ignore any code that might be useful for more than one but not all prediction tasks...

[mloning]

I want to sort this out for the next release and I'll start working on this in the next few days.

As @James-Large pointed out, we basically have two sensible choices here (unless someone has another idea): either taking purpose/functionality as the top organising principle or the learning task. I'd appreciate a quick vote on them.

Option 1: by purpose

sktime/
    algorithms/  
        base.py
        classification/
             base.py
             dictionary_based/
             ...
        regression/
            base.py
            dictionary_based/
            ...
        forecasting/
            base.py
            arima/
            ...
         ...
    model_selection/
        base.py 
        ...
        series_as_features/  # classificaton/rergession/clustering
            GridSearchCV.py
        forecasting/
            sliding_window_splitter.py
              ...         
         ...
     composition/
         base.py 
         ...
         series_as_features/ 
             pipeline.py 
         ...
         forecasting/
             pipeline.py
         ...

Option 2: by learning tasks

sktime/
    classification/  
        base.py
         dictionary_based/
         ...
    regression/
        base.py
        dictionary_based/
        ...
    series_as_features/
        algorithms/  # base algorithms shared by classifiers/regressors
            dictionary_based/
            ...
        composition/
            pipeline.py
            ...
        model_selection/
            GridSearchCV.py
            ...
    forecasting/
        base.py
        arima.py
        ...
        composition/
            pipeline.py
              ...
        model_selection/
            sliding_window_splitter.py
            ... 

[fkiraly]

2

[fkiraly]

also, random paradigm shifting idea: design a file system in which it is possible to have the order some sub-folders not matter, e.g., model_selection/forecasting and forecasting/model_selection access the same folder

[fkiraly]

maybe also the order of letters, e.g., acefginorst/cdeeeillmnoost_

[James-Large]

Getting back into all this, I think 2 as well. When using, you'd more likely want many things for one task, instead of doing many tasks that suit this purpose.

That'd be nice Franz until you hit the algorithmic/logarithmic dirs :stuck_out_tongue:

[fkiraly]

We could rename sktime to kismet (=the fate that cannot be escaped)

[mloning]

Closed by #246