Plasmapy: What packages should we use to store data?

[namurphy]

@bla1089 and I were just talking in person and ran into the following question: What package or packages should we use to store data?

Some possibilities include:

• NumPy (which we are mostly using thus far)
• pandas
• xarray
• yt

Some of the qualities that I think would be very much worth having are:

• Units-aware or units-compatible
• Ability to label dimensions (like pandas does for 1D and xarray does for multidimensional arrays)
• Compatible with flexible data sources
  
  
  
  • Common data types used by lab, space, astro, and solar
  
  
  • Fluid and particle representations (yt has better particle capabilities since the "demeshening")
  
  
• Can be easily installed using tools like conda
• Wide user/developer community
  
  
  
  • Ideally, we be able to make upstream contributions too
  
  
• Fast and easy-to-use

This is related to #66 and #59, including this comment.

Maybe this topic would be a good thing to write our first PlasmaPy Enhancement Proposal about! (I'm hoping that our zeroeth PlasmaPy Enhancement Proposal will be to have a GitHub repository for bad poetry about eigenfunctions.)

[blalterman]

What about finding a way to be agnostic to some of these other packages? One way I get around subclassing things like numpy and pandas is doing the following.

    class Plasma(object):
        __init__(self, data):
            self.set_data(data)
        @property
        def data(self):
            return self._data
        def set_data(self, new):
             assert isinstance(new, (pd.DataFrame, xr.DataArray, ....))
             self._data = new

        def do_physics(self, *args, *kwargs):
              data = self.data
              out = physics(data)
              return out

If we do something like this, there are huge piles of code I can contribute to the project (10,000s of lines, across many files). There are also benefits of doing things like not assuming a specific particle species. For example, some solar wind people include ions of oxygen and nitrogen, among others, along with the typical protons. But some people really only care about protons or protons & electrons. Being agnostic to these by developing the code in appropriate ways would make the PlasmaPy project accessible to a much larger field of users.

[blalterman]

It's also worth noting that with something like Pandas we get to leverage all of the Cython level code that the package has completed.

[antoinetavant]

Hey all,

I dig into yt (on their site ). They seem to have everything we need !!

• units and dimensions label
• can load lots of common data type (see this list )
• grid and particle representations as well as data-frame like Pandas
• conda's supported
• has a large community
• seems fast and easy from the examples, I did not check it myself.

Moreover :

• They use Cython and can be used in parallel (see here)
• There are lots of implemented functions, from field analysis to particle to cell
• a lot of work on visualization as been done

On my opinion, yt seems to be the best solution. On the other hand, I never tried it myself, so I am not confident in building a whole package over it.

I don't know how would you like to make the decision (by a vote, by example ?). I think we can fix a "decision date" like September the 1rst, let everyone think about it and then make the decision.

PS : @bla1089 in your example : assert isinstance(new, (pd.DataFrame, xr.DataArray, ....)) do you expect new to be either a pd or a xr data, and PlasmaPy to be able to deal with both of them or did you used the two only for the sake of the example ?

[blalterman]

@antoinelpp I expected new to be any instance we wanted to account for.
DataFrame and DataArray were just the examples that immediately came to
mind.

On Wed, Aug 23, 2017 at 4:29 AM antoinelpp [e-mail] wrote:

Hey all,

I dig into yt (on their site http://yt-project.org/doc/index.html ).
They seem to have everything we need !!

• units and dimensions label
• can load lots of common data type (see this list
  http://yt-project.org/doc/examining/index.html )
• grid and particle representations as well as data-frame like Pandas
• conda's supported
• has a large community
• seems fast and easy from the examples, I did not check it myself.

Moreover :

• They use Cython and can be used in parallel (see here)
  http://yt-project.org/doc/analyzing/parallel_computation.html
• There are lots of implemented functions, from field analysis to
  particle to cell
• a lot of work on visualization as been done

On my opinion, yt seems to be the best solution. On the other hand, I
never tried it myself, so I am not confident in building a whole package
over it.

I don't know how would you like to make the decision (by a vote, by
example ?). I think we can fix a "decision date" like September the 1rst,
let everyone think about it and then make the decision.

PS : @bla1089 https://github.com/bla1089 in your example : assert
isinstance(new, (pd.DataFrame, xr.DataArray, ....)) do you expect new to
be either a pd or a xr data, and PlasmaPy to be able to deal with both of
them or did you used the two only for the sake of the example ?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/PlasmaPy/PlasmaPy/issues/87#issuecomment-324259453,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AMPWVZlUbKiX4p56rZ7xa4BijJbJciEUks5sa-LsgaJpZM4OxSZ8
.

>

Ben

[StanczakDominik]

@bla1089's abstraction idea seems reasonable. I'm not 100% sure how it'd work in practice just yet, but... I'm pretty sure I'd need pointers on implementing this kind of thing.

As for data structures, at a glance yt does seem to be the better fit at least for huge simulation etc data, given that it also seems to make grid visualization easier. But it seems like there's nobody among us but Cadair who's used it yet! I've started learning it, for what it's worth.

[Cadair]

@solardrew has used it a bit as well.

[SolarDrew]

Indeed. I've found it really good for storing and plotting simulation data, but I don't know what kind of support it has for timeseries-type data of the kind @bla1089 has been talking about.

That said, I think it's great for pretty much everything else though, so in practice I think the real solution is going to be to use both as appropriate and try to provide consistent wrappers so the user doesn't need to worry what's underneath.

[blalterman]

I agree. It's also worth looking at other developments in regards to
spacecraft data. For example:

pysat: https://github.com/rstoneback/pysat
pysatCDF: https://github.com/rstoneback/pysatCDF

Part of why I like the idea of pandas & xarray is that they are designed to
be very similar, simplifying the early part of the learning curve.

Just to point out, we could decided that simulation and experimental data
just need to be treated differently. For example, we might decide that a
plasmapy/sim and plasmapy/experiment might be necessary to keep things
clean and we develop routines to format simulation output for comparing to
experimental data and vice versa.

On Thu, Aug 24, 2017 at 2:20 PM Drew Leonard [e-mail]
wrote:

Indeed. I've found it really good for storing and plotting simulation
data, but I don't know what kind of support it has for timeseries-type data
of the kind @bla1089 https://github.com/bla1089 has been talking about.

That said, I think it's great for pretty much everything else though, so
in practice I think the real solution is going to be to use both as
appropriate and try to provide consistent wrappers so the user doesn't need
to worry what's underneath.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/PlasmaPy/PlasmaPy/issues/87#issuecomment-324716186,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AMPWVa8zYIGgJq0dCakZsfVnbHpYq7Umks5sbb8FgaJpZM4OxSZ8
.

>

Ben

[StanczakDominik]

Sure, that sounds pretty reasonable. I wouldn't mind this being the semi-final consensus we reach, as we'll probably eventually need to split simulation and experiment up anyway.

[blalterman]

Then the question inevitably becomes: how do we want to name it? Do we want
it to be PlasmaPy and have two separate parts in it, an experiment
(PlasmaExp) and a simulation (PlasmaSim) module with potentially common
routines or do we want the modules to be entirely separate and retire the
PlasmaPy name?

On Thu, Aug 24, 2017 at 3:16 PM Dominik Stańczak [e-mail]
wrote:

Sure, that sounds pretty reasonable. I wouldn't mind this being the
semi-final consensus we reach, as we'll probably eventually need to split
simulation and experiment up anyway.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/PlasmaPy/PlasmaPy/issues/87#issuecomment-324730691,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AMPWVcZ6PgpNNNzGGfx03SFE7EULx_nlks5sbcwpgaJpZM4OxSZ8
.

>

Ben

[SolarDrew]

Yeah that's pretty reasonable, I had considered we might have to do that. If possible I would like to keep a single mostly-consistent API, or at least have a single contructor that returns an appropriate subclass of Plasma, a la SunPy Maps. But yes, if necessary having two would be a decent compromise.

[SolarDrew]

And with regards to the name I'd say we still want to keep everything within PlasmaPy and have experiment and simulation (or whatever) as top-level submodules.

[lemmatum]

What about HDF5? We can use PyTables, Pandas, and h5py to handle HDF5 files in different ways (with different learning curves). There is already archiving and compression functionality built-in. It's cross-platform, highly-highly-optimized (fast read/write for massive simulation/experimental data files), and language agnostic. This latter point would be important for converting to other formats like CDF.

[Cadair]

Also, just for the record here, someone is working on a pure-python implementation of a CDF package: https://github.com/MAVENSDC/cdflib

I believe it is currently read only.

[blalterman]

I know of several people working on a pure python CDF library. It seems
like there are a lot of people duplicating this work. Perhaps it's best to
stay away from this one.

Ben

---

B. L. Alterman
Candidate, Applied Physics
Solar and Heliospheric Research Group
Climate and Space Sciences and Engineering
University of Michigan
[email protected]

On Mon, Oct 30, 2017 at 8:21 AM, Stuart Mumford [e-mail]
wrote:

Also, just for the record here, someone is working on a pure-python
implementation of a CDF package: https://github.com/MAVENSDC/cdflib

I believe it is currently read only.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/PlasmaPy/PlasmaPy/issues/87#issuecomment-340428079,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AMPWVX-CRYktN0cPy-POkCSZGja0hRH-ks5sxb85gaJpZM4OxSZ8
.

[Cadair]

Do you have links to the others? Perhaps we should get them talking? This one is being written by someone I know at GSFC.

[blalterman]

I not worth it. I know the people working on several. It's one of those
things that's just going to happen and settle out in time.

On Oct 31, 2017 03:40, "Stuart Mumford" [e-mail] wrote:

Do you have links to the others? Perhaps we should get them talking? This
one is being written by someone I know at GSFC.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/PlasmaPy/PlasmaPy/issues/87#issuecomment-340684209,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AMPWVcCvwPNHqyYAznuwxFnZ29dAjaGcks5sxs8BgaJpZM4OxSZ8
.