Pyvista: Parallel I/O and computation? (with Dask, etc.)

[JiaweiZhuang]

I am getting ~1 Terabyte of VTK files (for a class project). Processing them takes forever... I wonder if vtki is able to perform parallel I/O (assuming that you have a parallel file system like Lustre and GPFS)?

I notice that VTK/ParaView supports parallel I/O:
https://www.paraview.org/ParaView/index.php/Parallel_I/O
Is this interface exposed to vtki? VTK seems to have both MPI and non-MPI implementations, so MPI probably doesn't have to be a dependency.

An alternative way is using Dask. This is how Xarray handles parallel I/O and computation (http://xarray.pydata.org/en/stable/dask.html).

This is a follow-up of #175, but is about the performance side.

[banesullivan]

First, does the ~1 TB of files make one single mesh?

And what kind of processing will you perform?

[JiaweiZhuang]

does the ~1 TB of files make one single mesh?

They are actually individual VTK files, each represents a time step of an evolving system. I can probably hack it with multiprocessing or Dask on my own. I just wonder if vtki has any sort of built-in parallelism.

The computation pattern is pretty simple. Mostly just read the data and compute the sum.

[banesullivan]

Just to note, the vtkXMLP<type>Reader implemented for parallel file formats to solve #175 actually reads the parallel files in series, from the VTK docs:

vtkXMLPUnstructuredGridReader reads the PVTK XML UnstructuredGrid
file format. This reads the parallel format's summary file and then
uses vtkXMLUnstructuredGridReader to read data from the individual
UnstructuredGrid piece files. Streaming is supported. The standard
extension for this reader's file format is "pvtu".

Since vtki simply wraps a call to that reader in a standard routine, this means vtki currently does not support parallel IO/processing. I do think we could expand on the fact that those Parallel format readers support streaming to implement a parallel pipeline - the VTK discourse would have folks that are much more experienced than I on how to actually implement this.

With ~1 TB of data, I'd personally recommend setting up ParaView on your parallel file system to do this processing as this is exactly what ParaView is built for... otherwise, this would be a feature we could probably outline implementing for future development. I'd recommend posting this question on the VTK discourse to see if 1) you need to compile VTK with MPI support yourself since the wheels on PyPI and Anaconda do not have MPI support enabled, and 2) if this big of a dataset is able to be processed in VTK alone without ParaView.

[banesullivan]

I just wonder if vtki has any sort of built-in parallelism.

There's no built-in parallelism support in vtki currently - perhaps a new feature request?

They are actually individual VTK files, each represents a time step of an evolving system. I can probably hack it with multiprocessing or Dask on my own.

Perhaps you could define a function that you fire off in parallel that only cares about a single file, pass that file to the function, have vtki read/process that single file, save the results

[JiaweiZhuang]

@banesullivan Thanks, that's very helpful information! I am not sure how many VTK users are interested in "big data" problems and whether a new feature is worth the effort. But if you want to hack some parallelism I am happy to discuss. Xarray has excellent parallel I/O performance and can probably serve as a reference & benchmark (see my blog post for a real-world example).

Feel free to close or keep this issue. Thanks again :)

[banesullivan]

I am not sure how many VTK users are interested in "big data" problems

Generally speaking, a big portion of the VTK community is interested in "big data" problems - these folks tend to use ParaView though because ParaView is built for use on supercomputers in a client-server manner.

But if you want to hack some parallelism I am happy to discuss.

At this time, I don't think I'll be able to go at this... 😞 Parallelism is something I think vtki should definitely support down the road but I don't see @akaszynski or I being able to take on that effort on our own.

My primary focus for development lately has been to make an intuitive, easy to use tool for researchers, students, and generally novice programmers to use in visualizing generally large spatial datasets (large in the sense of millions to 10s of billions of points/cells) but not so large that the dataset needs to be processed on a supercomputer - in my experience datasets of this size can be handled by a laptop with a standard GPU quite easily.

Worth Noting

The folks making VTK have put considerable thought into this when building their dataset adapter and numpy wrapping module directly in the VTK Python package - there's a suite of algorithms in that module that can handle numerical operations in parallel. Perhaps we could adapt some of those features down the road.

There's a series of blog posts that dive into this (see this 4 part series). I can't remember if the blog posts talk much about the parallel capabilities of those algorithms, but they are definitely worth checking out for this effort.

Moving forward

I'm going to close this issue for now with a tag to mark it for reevaluation down the road.