Gluon-ts: UnboundLocalError: local variable 'lv' referenced before assignment

[kaijennissen]

Description

I cannot include a validation dataset during the training of a multivariate model. When training without validation dataset everything works.

To Reproduce

(Please provide minimal example of code snippet that reproduces the error. For existing examples, please provide link.)

from gluonts.dataset.common import TrainDatasets
from gluonts.dataset.multivariate_grouper import MultivariateGrouper
from gluonts.dataset.repository.datasets import get_dataset
from gluonts.model.gpvar import GPVAREstimator
from gluonts.mx.trainer import Trainer

NUM_OF_SERIES = 8


def load_multivariate_dataset(dataset_name: str):
    ds = get_dataset(dataset_name)
    grouper_train = MultivariateGrouper(max_target_dim=NUM_OF_SERIES)
    grouper_test = MultivariateGrouper(max_target_dim=NUM_OF_SERIES)
    return TrainDatasets(
        metadata=ds.metadata,
        train=grouper_train(ds.train),
        test=grouper_test(ds.test),
    )


dataset = load_multivariate_dataset(
    dataset_name="exchange_rate"
)
metadata = dataset.metadata

estimator = GPVAREstimator(
    prediction_length=metadata.prediction_length,
    target_dim=NUM_OF_SERIES,
    freq=metadata.freq,
    trainer=Trainer(
        epochs=50,
        batch_size=4,
        num_batches_per_epoch=10,
        patience=5,
    )
)

predictor = estimator.train(
    training_data=dataset.train, validation_data=dataset.test)

Error message or code output

UnboundLocalError: local variable 'lv' referenced before assignment

Environment

• Operating system: Mac OSX 10.15.5
• Python version: 3.7.6
• GluonTS version: 0.5.0
• MXNet version: 1.6.0

[lostella]

The problem occurs also on master. The root cause seems appears to be the way ValidationDataLoader extracts batches of data: the data transformation pipeline gets applied with is_train=True (see here) so that the "future" target is included in the data, and the loss associated with it (say, negative log-likelihood) can be computed; however, this also has the consequence that the instance splitter selects a random number of time windows from the validation dataset.

Now, because of the usage of MultivariateGrouper, the validation dataset consists of a single 8-dimensional series, out of which 0 or more "validation" windows gets sampled by the instance sampler. When 0 are sampled, no validation loss is computed and lv never gets assigned.

In fact, the following minimal, univariate example shows the number of elements produced by the ValidationDataLoader to be sometimes 0, sometimes 1.

import mxnet as mx

from gluonts.dataset.common import ListDataset
from gluonts.dataset.loader import ValidationDataLoader
from gluonts.model.deepar import DeepAREstimator

dataset = ListDataset(
    [{"start": "2020-01-01", "target": list(range(50))}],
    freq="1D"
)

estimator = DeepAREstimator(
    prediction_length=10,
    freq="1D",
)

data_loader = ValidationDataLoader(
    dataset=dataset,
    transform=estimator.create_transformation(),
    batch_size=4,
    ctx=mx.context.cpu(),
)

for _ in range(20):
    print(len(list(data_loader)))

This problem will occur any time the ValidationDataLoader is constructed out of a "singleton" dataset (but may happen with some other small number of time series in the dataset).

The solution to this is to rethink how the ValidationDataLoader should behave and how it is defined: not an easy one, but thank you kindly for submitting this @kaijennissen, this motivates us in improving this part.

[PascalIversen]

@lostella For now, why can't we just do:

import mxnet as mx
from gluonts.dataset.common import ListDataset, Dataset
from gluonts.dataset.loader import TrainDataLoader
from gluonts.model.deepar import DeepAREstimator
from gluonts.transform import Transformation
from functools import partial
from gluonts.mx.batchify import batchify

from typing import Callable, Optional

class ValidationDataLoader(TrainDataLoader):
    def __init__(self,
                dataset: Dataset,
                transform: Transformation,
                batch_size: int,
                stack_fn: Callable,
                n_eval_steps: Optional[int]=1,
                num_workers: Optional[int]=None,
                max_queue_size: Optional[int]=None,
                num_prefetch: Optional[int] = None,
                shuffle_buffer_length: Optional[int]=None
            ) -> None:
        super().__init__(dataset=dataset,
                        transform=transform,
                        batch_size=batch_size,
                        stack_fn=stack_fn,
                        num_batches_per_epoch=n_eval_steps)


dataset = ListDataset(
    [{"start": "2020-01-01", "target": list(range(1000))}],
    freq="1D"
)

estimator = DeepAREstimator(
    prediction_length=10,
    freq="1D",
)


data_loader = ValidationDataLoader(
    dataset=dataset,
    transform=estimator.create_transformation(),
    batch_size=4,
    stack_fn=partial(batchify, ctx=mx.context.cpu()),
    n_eval_steps=1
)

for _ in range(10):
    print(len(list(data_loader))) # 1 1 1 1 1 1 1 1 1 1 

This would also make @kaijennissen's code run and I can't think of any disadvantage of this approach compared to how the validation loader is currently defined.

This does not take away from the need to rethink validation, since the validation mechanism applied by using

predictor = estimator.train(
    training_data=dataset.train, validation_data=dataset.test)

is not in line with the users' expectation.

[kaijennissen]

@PascalIversen Could you share your code for the batchify function?

[PascalIversen]

@kaijennissen Sure, I edited my last comment to include the missing imports. (This will work on the current state of the master branch)
However, currently, the ValidationData loader naively samples from the whole validation dataset. Since the dataset.train and dataset.test overlap except for a prediction_length part at the end using validation_data=dataset.test will produce an optimistic prediction loss estimate during validation. Instead you can cut a validation_dataset from the train data so that validation and train data overlap only for a context_length part.

[kaijennissen]

Thanks. Replacing the ValidationDataLoader inside the GluonEstimator with the one you proposed seems to fix the issue.

[linehammer]

The Unboundlocalerror: local variable referenced before assignment is raised when you try to use a variable before it has been assigned in the local context. Python doesn't have variable declarations , so it has to figure out the scope of variables itself. It does so by a simple rule: If there is an assignment to a variable inside a function, that variable is considered local . To solve this problem, you can explicitly say it's a global by putting global declaration in you function. The global statement does not have to be at the beginning of the function definition, but that is where it is usually placed. Wherever it is placed, the global declaration makes a variable to global variable everywhere in the function.

[DayanSiddiquiNXD]

@kaijennissen Sure, I edited my last comment to include the missing imports. (This will work on the current state of the master branch)
However, currently, the ValidationData loader naively samples from the whole validation dataset. Since the dataset.train and dataset.test overlap except for a prediction_length part at the end using validation_data=dataset.test will produce an optimistic prediction loss estimate during validation. Instead you can cut a validation_dataset from the train data so that validation and train data overlap only for a context_length part.

I dont quite follow, I see the problem that the validation set will validate on the data seen in the train set and so will improve even as the model starts overfitting, but how would we alter the data loader or our validation data to fix that?

[PascalIversen]

@DayanSiddiquiNXD, this would be a way to cut a validation dataset which does not overlap with the train dataset:

from gluonts.dataset.repository.datasets import get_dataset
from gluonts.dataset.split import OffsetSplitter
dataset = get_dataset("m4_hourly")
train_length = len(next(iter(dataset.train))["target"])
print(train_length) # 700

def vertical_split(dataset, offset_from_end):
    """
    Split a dataset time-wise in a train and validation dataset.
    """
    dataset_length = len(next(iter(dataset))["target"])

    split_offset = dataset_length - offset_from_end

    splitter = OffsetSplitter(
        prediction_length=offset_from_end,
        split_offset=split_offset,
        max_history=offset_from_end)

    (_, dataset_train), (_, dataset_validation) = splitter.split(dataset)
    return dataset_train, dataset_validation

dataset_train, dataset_validation = vertical_split(dataset.train, 40)

len(dataset_validation[0]["target"]) # 40

len(dataset_train[0]["target"]) #660

Note, that m4_hourly is a square dataset (all time series have the same length). If you do not have a square dataset, you can use the DateSplitter instead.

In case you have multiple time series, another option you have is splitting the dataset horizontally, meaning that you reserve some time series for validation.
You could do it like this, assuming your time series are well shuffled:

def horizontal_split(dataset, item_split_ratio):
    """
    Split a dataset item-wise.
    """
    item_ids = [entry["item_id"] for entry in list(dataset)]

    n_train_items = int(len(set(item_ids))* item_split_ratio)

    dataset_in_sample = [
        ts for ts in dataset if int(ts["item_id"]) < n_train_items
    ]  # assuming items are zero indexed

    dataset_out_of_sample = [
        ts for ts in dataset if int(ts["item_id"]) >= n_train_items
    ]

    return dataset_in_sample, dataset_out_of_sample

[DayanSiddiquiNXD]

@PascalIversen this is very helpful. what about the context length overlap in the case of a vertical split? you dont have to go through the trouble to coding it up but i'm assuming we would need to make the test and val datasets overlap for the context length so the first val prediction is immediately after the last train value. given that i didnt define a context length and the estimator would have used the default, how would i get the context length variable to be able to create this split?

[PascalIversen]

You can a get the context_length with estimator.context_length. For many estimators it defaults to context_length = prediction_length.
Overlapping datasets you could produce like this:

splitter = OffsetSplitter(
        prediction_length=offset_from_end,
        split_offset=split_offset,
        max_history=offset_from_end + context_length)

However, I would advise against an overlap since many estimators will calculate the loss also on the context_length during training and validation.

I need to clarify that for the prediction_length argument of the OffsetSplitter, you should not put the prediction_length of the model, but the amount of timepoints you want to use for validation (In the OffsetSplitter this argument is called prediction_length, since it is most often used to create test datasets, but to avoid confusion, a better name would maybe be future_length).

So to make my example complete, this would maybe be a reasonable way to do validation:

from gluonts.dataset.repository.datasets import get_dataset
from gluonts.dataset.split import OffsetSplitter
from gluonts.model.deepar import DeepAREstimator

dataset = get_dataset("m4_hourly")
train_length = len(next(iter(dataset.train))["target"])
print(train_length) # 700
prediction_length=10
context_length=10
estimator = DeepAREstimator(prediction_length=prediction_length, context_length=context_length, freq="H")


def vertical_split(dataset, offset_from_end):
    """
    Split a dataset time-wise in a train and validation dataset.
    """
    dataset_length = len(next(iter(dataset))["target"])

    split_offset = dataset_length - offset_from_end

    splitter = OffsetSplitter(
        prediction_length=offset_from_end,
        split_offset=split_offset,
        max_history=offset_from_end)

    (_, dataset_train), (_, dataset_validation) = splitter.split(dataset)
    return dataset_train, dataset_validation

dataset_train, dataset_validation = vertical_split(dataset.train, 2*(context_length+prediction_length))
estimator.train(training_data=dataset_train, validation_data=dataset_validation)

We are aware that implementing this is currently not super intuitive, so we are hoping to improve validation soon.

[DayanSiddiquiNXD]

ok. I might as well ask about my actual end use case with this. will this allow me to use validation loss for early stopping while training my deepAR model (or at least see roughly how many epochs it takes to minimize val loss)? because i'm using this on realtively simple dummy data (a sin wave with some noise) and the val loss just keeps jumping around, not following the expected val loss pattern shown below

[kaijennissen]

@DayanSiddiquiNXD #555

[DayanSiddiquiNXD]

@DayanSiddiquiNXD #555

ty ty

[bingblackbean]

Does this issue have any further update? For me, I managed to set random seed of mx and numpy to "Pass" this issue, anyway this is not a good solution.

[lostella]

@kaijennissen @bingblackbean @DayanSiddiquiNXD @mariosantosprivate I’ve been working on #1256 which, among other things, gets rid of this problem. There, estimators are configured by default in such a way that the validation data loader will (deterministically) yield, from a given dataset, the “trailing” instance from each time series.

That is to say, validation batches will have the prediction interval at the end of the time series. This way the issue in the OP here is avoided (no random sampling of instances, but one instance per time series instead) and one can easily split a dataset in the training/validation parts: just remove (at least) prediction_length time steps from each time series, to obtain the training dataset. This is exactly what is done in #1158, for example.

What I think is nice about this, is that this is entirely controlled by the “instance sampling” mechanism, so that one can even decide to sample validation instances differently if they want, or we could even avoid requiring a separate dataset object for validation, and instead do the split automatically.

[lostella]

fixed in #1256 and #1284