Visidata: View Pandas DataFrames while in iPython [wishlist]

[baogorek]

Seven months ago I started a project called datascroller and recruited two fine developers to help me. Last week I discovered Visidata and absolutely, positively could not believe that this existed.

While the existence of Visidata has taken the wind out of datascroller developent, I wanted to share a feature of datascroller that will keep me using it instead of switching to Visidata: the ability to view Pandas DataFrames while in iPython. With datascroller you can do this:

import pandas as pd
from datascroller import scroll

my_df = pd.read_csv('<path to your csv>')
scroll(my_df)

I.e, you can use it in the midst of other data science activities in an iPython session.

I would not be surprised if this is already a feature of Visidata, but I had a heard time finding it in the documentation. I do see that Visidata is importable but it's nice to be able to operate directly on the DataFrame (and not have to save a csv first).

[saulpw]

Hi @baogorek! This is a really useful feature, I agree. In fact it's # 7 on the list of "10 Ways to Use VisiData" on the front page of visidata.org: vd.view_pandas(df). Where did you look in the docs that you couldn't find it? We should probably put a reference to it there.

As for datascroller, I know how hard is it to stumble across a 'competing' project for something you've put a lot of effort into. (It's happened to me with a past project and I can't say I was super thrilled about it.) So I really appreciate your comments and suggestions, and also the work you've put into datascroller as an open-source software project. I think we are onto something here with this whole "view data easily in the terminal" thing :)

I went and read through your blog post announcing datascroller and it seems we share the same vision and aesthetic. You've also articulated the business case for these kinds of tools a lot better than I am able to. I'd be interested in hearing about the big plans you have/had for datascroller, and maybe we can make them a reality in VisiData.

Also we have several open issues to do with the pandas loader (#223 #248 #264), which have been languishing because I am not that fluent with pandas. If you're interested, we would absolutely love your help. The VisiData pandas loader is pretty popular and it would be fantastic to have it work as well as the other data sources.

Thanks for reaching out!

[anjakefala]

Welcome @baogorek! ^^

I wanted to sneak this in: https://github.com/saulpw/visidata/issues/460

If you are planning to engage with the pandas loader, this is the issue where we began chatting about that!

[baogorek]

Hi @saulpw , @anjakefala ,

Yes, mine is clearly reader error, as on the front page I now see visidata.view_pandas(df). I think what happened was that I didn't scroll down all the way on the front page, looked at the gifs for a minute, and then clicked the "docs" link. The first things I saw were shell commands, and instead of scrolling down, I went back to Google to search for new pages about Visidata and iPython. Along the way I found the Developer Quickstart Guide and Issue 10 which led me to the incorrect conclusion that this was a command line tool primarily with libraries aimed at developers.

Well, considering lack of research got me to work on datascroller, I guess it's fitting that my first Visidata issue would be made on the basis of poor research. Thanks for the kind words in spite of that and introductions to issues #223, #248, and #264. I'll relay them to my datascroller colleagues @johncmerfeld and @kjmerf. Right now, I personally need a break from anything datascroller related, but on next Monday I'm still on for a speaking slot at the Mid Ohio Python User's Group about datascroller. After I tell the story I'll introduce these issues and see if I can drum up some interest.

Cheers!
Ben