Mne-python: support user-defined event fields in mne.read_epochs_eeglab()

[nbara]

Hi all,

EEGLAB supports adding custom field names in the event structure, on top of the default ones (type, latency, duration, urevent, epoch) :
https://sccn.ucsd.edu/wiki/Chapter_03:_Event_Processing#Event_fields

For instance I use a behaviour field to store the subjects responses on each epoch.

It would be very useful if one could somehow access those custom fields in MNE (maybe the option already exists but I haven't found it).

Best wishes,
Nicolas

[agramfort]

if you only have a single value per epoch you can use the second column of
the events array

epochs.events

will keep track of the info.

HTH

[nbara]

Hi @agramfort, thanks for the fast response!

Actually I use more than one custom field in my event structure in EEGLAB. Here's a MATLAB screenshot:

Also, in python the second column of my epochs.events array is only 0s (after mne.read_epochs_eeglab()):

[jona-sassenhagen]

You can set the middle value to whatever you want.

One option if you have more than 1 value to store is to set the middle column to an index, and then retrieve from a pandas dataframe using that index.

E.g.

covariates = df[epochs.events[:, 1]]

For legacy reasons, the events array will not be extented beyond a 3-column integer array.

[nbara]

Hi @jona-sassenhagen, I'm not sure I understand what you mean.

I would like to import my custom event fields from my .set file, how do I do that?

For instance, in the screenshot above, my EEGLAB/Matlab's event structure has fields named value, conf_task.

How do I access them in python/mne? This info seems to be lost using epochs = mne.read_epochs_eeglab('eeglabfile.set')

[larsoner]

@jona-sassenhagen it sounds like we might need a specialized read_events_eeglab or something to give full access to the information, since we currently simplify it?

[nbara]

@Eric89GXL Could it possibly stored in epochs.drop_event or something similar?

[larsoner]

It could be exposed via a property of the EpochsEEGLAB class, sure. But whatever it is, ideally it would contain all of the same data that EEGLAB exposes, and be complete. Maybe a numpy structured array?

[mmagnuski]

I have used something like this in the past to get info from the set file to a pandas dataframe:

import pandas as pd
from scipy.io import loadmat

def read_set_events(filename, ignore_fields=None):
    '''Open set file, read events and turn them into a dataframe
    Parameters
    ----------
    filename: str
        Name of the set file to read (absolute or relative path)
    ignore_fields: list of str | None
        Event fields to ignore
    Returns
    -------
    df: pandas.DatFrame
        Events read into a dataframe
    '''
    EEG = loadmat(filename, uint16_codec='latin1',
                  struct_as_record=False, squeeze_me=True)['EEG']
    flds = [f for f in dir(EEG.event[0]) if not f.startswith('_')]
    events = EEG.event
    df_dict = dict()
    for f in flds:
        df_dict[f] = [ev.__getattribute__(f) for ev in events]
    df = pd.DataFrame(df_dict)

    # reorder columns:
    take_fields = ['epoch', 'type']
    ignore_fields = list() if ignore_fields is None else ignore_fields
    take_fields.extend([col for col in df.columns if not
                     (col in take_fields or col in ignore_fields)])
    return df.loc[:, take_fields]

it worked in my usecases - it goes through all EEG.event elements and extracts relevant fields so that every row in returned dataframe correspons to one EEG.event.
You might not need the uint16_codec argument when reading your set files though.
(and may have been more elegant with struct_as_record=True... :) )

[larsoner]

We don't want a dependency on pandas. Could we get away with a simple structured array, e.g. what loadmat returns in the first line (but with struct_as_record=True)?

[mmagnuski]

@Eric89GXL Yes - I was not proposing to put something like this in mne :) only trying help @nbara. Structured array should work well.

[larsoner]

@mmagnuski any preference on what such a property would be called? EpochsEEGLAB.original_events or something?

[mmagnuski]

It would be eeglabish to call it urevents (but misleading and ugly :smile:). original_events are ok for me.

[larsoner]

It would be eeglabish to call it urevents (but misleading and ugly ). original_events are ok for me.

Yikes. What is that short for? user_events? If so then we can just use that name.

[nbara]

i think 'ur' stands for 'original' (in which case it has German origins https://en.wiktionary.org/wiki/ur-#Etymology) but I can't be sure...

[mmagnuski]

@nbara - that is correct :) But I was joking, eeglab was not really designed to have a clean API (it is mostly GUI-centered) so its internals can be a bit messy. In eeglab urevents are the original, untouched events - which means that if you remove some epochs or time segments your events change, but urevents stay the same (so you can for example later compare events and urevents and infer which epochs were removed - as this information is not saved in eeglab files etc.).
original_events is fine, really.

[jasmainak]

Etymology lessons on MNE :) @Eric89GXL have we considered putting some of these useful scripts (like the one from @mmagnuski ) in one place? It's probably nice to point users towards them when an issue crops up without necessarily having to make a PR for everything. I know that such scripts also exist for mangling data from FieldTrip into MNE ...

[mmagnuski]

@jasmainak - this is a good point, maybe mne-sandbox? + something like FAQ with links/examples (in the long run)?

[larsoner]

We could add links to gists in the FAQ

[jasmainak]

Actually, I just found this page on the wiki which appears to be extremely underutilized.

[mmagnuski]

@jasmainak my guess is that most users go to online docs and rarely visit mne github wiki.

[jona-sassenhagen]

We could have the EEGLAB reader warn if events are dropped and point to _read_eeglab_events for reading in the raw event structure.

[larsoner]

Let's not point users to a private functions, it encourages bad behavior. If there is some functionality there that we need, we should make a public API.

[jona-sassenhagen]

Edit: this is actually the wrong issue I think.