Plasmapy: New filament statistics analysis module?

[TomNicholas]

Hi,

Firstly, what you're doing here is great, this kind of integrated open-source tooling is exactly what the plasma community should be doing!

I am planning to write a repo of tools specifically for analysing the statistics of Scrape-Off-Layer filaments, in collaboration with a couple of others (at York and Tromso). It would include functions for things like calculating conditional averages, or filament waiting time distributions. I could host this repo myself, but I thought maybe it could be included in PlasmaPy somehow?

I have a pretty specific idea of what this would look like, it would be a set of functions like in the MetPy calc module. One thing is that I want these functions to operate on xarray data objects - that's how my current workflow operates, and it's incredibly useful. I'm not sure if that would be a problem - I see you discussed using xarray in #87 . (As a separate point, I'm now one of the core devs on the xarray project, so I can probably help you with that integration if you did want to use xarray.)

Let me know what you think!

[StanczakDominik]

Hey there,

Thanks for the encouragement! It gets a little difficult coding into the
silent void sometimes :laughing:

I'd love to have SOL filament functionality within the main repo! I've been
meaning to learn a bit more about those, so I'd love to help out however I
can.

Speaking of things I've been meaning to pick up, xarray is one of them -
filaments are inherently spatial data, it makes little sense to process
them any other way with that package out there in the ecosystem. So...
Yeah, I'm all for including that.

Also, congrats on becoming a core dev for xarray! :party:

[TomNicholas]

Thanks!

Okay awesome, how exactly might this work? A new module I guess, but where would it go? I think most of the functions I'm thinking of are basically time-series analysis, but some would be for 2D image data too. So I guess they could technically be applied to data from multiple diagnostics (GPI, BES, Langmuir probes), so they shouldn't live in the diagnostics module?

Is the xarray integration a problem? Is it okay if I simply make a module in PlasmaPy to store the functions and they act on xarray DataArrays? (It probably wouldn't be too hard to generalise the functions to other input types later if needed)

How does contribution work? It's likely that I will be including and standardising functions that other people have written, so would you want me to submit a PR for each? Or could I maybe just have access for that module and not touch anything else? (I admit I haven't fully read your contributing guide yet)

[namurphy]

It would be great for this to be included in PlasmaPy! The two long term possibilities would be to include it within a subpackage within the main respository or in an affiliated package in a separate repository (following Astropy's practice). The decision would not need to be made anytime soon and I'd be fine with either approach for the time being.

I'm not sure which subpackage would be best to put this in. My inclination would be to figure out where one would intuitively look or it. I'm wondering about something along the lines of time_series, image_processing, or statistics (or some combination thereof).

With respect to xarray, the only potential problem that I can think of is that last I heard it was not yet compatible with units subpackages. We use astropy.units pretty heavily for compatibility with Astropy and SunPy, so we probably would need to include a workaround unless xarray gets units support (e.g., https://github.com/pydata/xarray/issues/525).

We'll be having our biweekly community meeting tomorrow (May 14) at 18 UT on PlasmaPy's Jitsi channel in case you would be able to join, and our Matrix channel for text-based chat is another good place to chat about the nuts and bolts of contributing.

Thanks! –Nick

edit: I added the link to the Jitsi channel because I noticed I forgot to yesterday.

[StanczakDominik]

Units are nice but if xarray has issues with those, developing those SOL filament functionalities first and adding unit support there later could be a viable strategy. I'd be okay with it.

I think doing this over a few not-too-large incremental PRs would be best - not necessarily one per function, of course. That way code review is less overwhelming.

We wanted to release 0.2 by the end of May, though, so I'll have to set up a branch for that so that filament PRs can be merged onto master instead.

[TomNicholas]

I'm wondering about something along the lines of time_series, image_processing, or statistics (or some combination thereof).

Yeah I think that's sensible. Distinguishing between functions intended for 1D or 2D data seems reasonable (even though with xarray you can always apply an ND function to n >= N dimensional data)

last I heard it was not yet compatible with units subpackages.

This is true, though unit support is one of the next things they we're planning to work on, at which point it should be able to wrap pint arrays or maybe astropy quantity arrays (not sure on the exact details of how astropy does units). I don't think this is prohibitive as we can either:

• just decorate the analysis functions to drop units for now

• just have them accept xarray dataarrays so the module would be a semi-separate part of PlasmaPy for now.

It looks like until you have a more concrete implementation of the plasma class then this new module doesn't particularly need to conform to a particular input structure? So for now xarray DataArrays alone are fine?

[namurphy]

It looks like until you have a more concrete implementation of the plasma class then this new module doesn't particularly need to conform to a particular input structure? So for now xarray DataArrays alone are fine?

Yes, this would be fine, I think.

• just decorate the analysis functions to drop units for now

• just have them accept xarray dataarrays so the module would be a semi-separate part of PlasmaPy for now.

Both of these sound like viable options. I particularly like the first option, but only because I think decorators are really fun to use. :slightly_smiling_face:

There are also plans by @tulasinandan to implement time series turbulence analysis tools, so it may be worth thinking about how to structure a time_series subpackage to include broad functionality, if that's the approach we end up taking. We described plans for this (along with a lot of other plans) in a proposal we submitted last month.

[tulasinandan]

I believe we should stick to pandas time series and not worry about
complicating anything on top. The quantities can simply be in astropy units
but no fancy stuff is needed on top of that. I am actually streamlining
some of my codes these days. I have ported the codes to python 3.0 and the
codes are in my repo called TurbAn (Turbulence Analysis). Once I have
streamlined these codes, I will open a PR for including these codes into
PlasmaPy which might be a lot of work because the codes will need to be
formatted and documented to PlasmaPy standards.

On Tue, May 14, 2019 at 11:11 AM Nick Murphy [e-mail]
wrote:

It looks like until you have a more concrete implementation of the plasma
class then this new module doesn't particularly need to conform to a
particular input structure? So for now xarray DataArrays alone are fine?

Yes, this would be fine, I think.

-

just decorate the analysis functions to drop units for now
-

just have them accept xarray dataarrays so the module would be a
semi-separate part of PlasmaPy for now.

Both of these sound like viable options.

There are also plans by @tulasinandan https://github.com/tulasinandan
to implement time series turbulence analysis tools, so it may be worth
thinking about how to structure a time_series subpackage to include broad
functionality, if that's the approach we end up taking. We described plans
for this (along with a lot of other plans) in a proposal
http://doi.org/10.5281/zenodo.2633287 we submitted last month.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/PlasmaPy/PlasmaPy/issues/621?email_source=notifications&email_token=AATJYZ6D4EEGWSW2B5I5IZLPVLJDPA5CNFSM4HMPAICKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVLZU7I#issuecomment-492280445,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AATJYZ3R4YG7FCUAQ6WXS6LPVLJDPANCNFSM4HMPAICA
.

--
Tulasi Nandan Parashar,
217 Sharp Laboratory,
104 The Grn,
University of Delaware,
Newark, DE 19716
Office: [phone]
Cell: [phone]
http://www.flickr.com/photos/tulasinandan

[namurphy]

I believe we should stick to pandas time series and not worry about complicating anything on top.

We would need xarray instead of pandas for multi-dimensional data sets, I believe.

I am actually streamlining some of my codes these days. I have ported the codes to python 3.0 and the codes are in my repo called TurbAn (Turbulence Analysis). Once I have streamlined these codes, I will open a PR for including these codes into PlasmaPy which might be a lot of work because the codes will need to be formatted and documented to PlasmaPy standards.

That's great to hear! We've learned the hard way (#191) that it works better to split up big changes into multiple, smaller, more manageable pull requests. For the moment, taking some time to plan the software architecture seems wise.

[TomNicholas]

We would need xarray instead of pandas for multi-dimensional data sets, I believe.

Yeah, I think if you used xarray instead you could get almost all the benefits of using pandas plus some very valuable extras:

• xarray is built upon the same core indexers as pandas, by some of the same developers. It deliberately has a similar API where possible, and is just as fast. There is close interoperability - pandas has a to_xarray method and vice versa.

• with xarray you would get broadcasting over multiple dimensions for no extra effort. So if you want to apply a time-domain analysis function across multiple positions, shots, simulations etc. then that's the default behaviour with no extra code:

from plasmapy.timeseries import pdf

n = load_data()  # data with dimensions ('time', 'position')

n_pdf = pdf(n, dim='time')  # result has dimensions ('bins', 'position')

• This broadcasting can also be parallelized easily with dask.

• xarray is basically the same as the netCDF data model, so you can load and save data to and from netCDF and HDF5 files very easily.

[StanczakDominik]

Having read through https://github.com/TomNicholas/CDG_xarray - yes, a thousand-times lazy-loaded yes to xarray! :smile:

Also, Sid on Matrix mentioned wanting to develop a simulation code for streamers, which - to a layman such as me - sound potentially related. This could potentially provide a synergy.