Pandas: DEPR: Clean up list of deprecations from prior versions

[jsexauer]

Log for previous deprecations, once they are actually removed, move the issue to #13777

We try to keep these for three major versions as actual deprecations. e.g. deprecate in 0.17, 0.18 & 0.19 get the warning, removed in 0.20.

0.24.0

• [x] #24596 __array__ for tzaware Series/Index (0.24.0)

[TomAugspurger]

Just FYI, cols is used in to_csv and to_excel as well.

[jreback]

@TomAugspurger good point!

@jsexauer would you do another PR to deprecate cols in to_csv / to_excel (e.g. do what you did in pivot, still accept it, but provide the warning, and substitue columns)

[jsexauer]

Sure no problem.

[jorisvandenbossche]

@jreback I was thinking, now that we are alse deprecating this in to_csv and to_excel, shouldn't we keep this deprecation cycle longer than one release? Because I don't know if the majority of the users update their version every release.

[jtratner]

I agree with @jorisvandenbossche - giving an additional release cycle as buffer would be prudent.

[jreback]

certainly
just keeping open issues as reminders
I just removed the old syntax in in sort I think from like < 0.8!

though maybe will change them to say 0.16 so are at the correct point

[TomAugspurger]

I added sections by version to the original post. Let's keep this as the running deprecation issue (unless there are objections. I'm open to having specific issues per release)

[jreback]

@jsexauer want to do a pr for those indicated for 0.15.0

[jsexauer]

Sure will do

[jankatins]

There are a few deprecation in categorical, which will result in failing unittests.

Maybe that would be a nice way to track this deprecations:

# in testing.py
def fail_after_whatever_comes_first(date, pandas_version):
     from dateutil import parser
     import datetime
     latest = parser.parse(date)
     self.assertFalse(datetime.now() >= latest)
     self.assertFalse(LooseVersion(pd.__version__) >= pandas_version)

and then use that as the last test in the unittests for deprecated methods/properties/arguments

[jreback]

@JanSchulz what are you suggesting here?

[jankatins]

My suggestion was to put the deprecation handling into the code itself and not into a bugreport :-)

[jreback]

putting in the actual code is not a good idea (if its the ONLY place), and in general just clutters things. You have to have someway of tracking something that would occur (with an explicit reminder), hence an issue is MUCH better, and how we do it.

[jreback]

@jsexauer any deliverables for 0.15.0? (or pls edit the top and move to 0.16)

[jsexauer]

@jreback I wrote the code but it is failing unit tests right now. Need to go back and figure out if the tests need to be updated or if I removed the deprecations incorrectly. Do you have a timeline you'd like the changes by?

[jreback]

@jsexauer what are the changes that you have partially working?

[jsexauer]

Here is the branch they are on: https://github.com/jsexauer/pandas/tree/fix6581_015
Here is the unit test fail report: https://travis-ci.org/jsexauer/pandas/jobs/35994074

[jreback]

ahh ok, how about we just do the delevel and move the boxplot to 0.16.0 then

@TomAugspurger, @jorisvandenbossche

[jreback]

@jsexauer want do just put up a PR for the delevel one?

[jreback]

@jsexauer let's push this to 0.16 (though I'll accept a delevel removal now if you can)

[jsexauer]

I'll write one up real quick and submit it within the hour

[jreback]

@jsexauer gr8 thanks. Pls edit the issues above and move to 0.16.0 the remaining 0.15.0 ones

[jsexauer]

I was going to start working on the 0.16 deprecations soon. Wanted to check in and see if you still wanted everything that is listed in 0.16 to be deprecated. @jreback

[jorisvandenbossche]

I think everything in the list is good to go.

[jreback]

I updated all of the to be deprecated issues. I think think anything <= 0.14.0 should be done in 0.17.0. All require mentions in the Previous Deprecations section. Some may need a doc note if its 'important' enough (e.g. this was removed in version ???) and/or more prominent mention in the whatsnew (e.g. i the highlites)

[jreback]

Ok with doing the 0.15.0 ones to be honest as well.

[jreback]

anyone interested in putting up a PR for any of these deprecation removals. Anything < 0.14 is ok for sure. anything else pls lmk.

[jreback]

@TomAugspurger

can we do a PR for this? (first one up top)

 boxplot #7096 (0.14.0)
 change default of return_type from None to 'axes'
 update return_type section in visualization.rst

[TomAugspurger]

Sure thing, will do a PR tonight or Sunday.

[jorisvandenbossche]

I updated the list at the top, and there are some deprecations from 0.15 that can be removed in 0.19.0 (and some of 0.16.0 are maybe also worth considering)

[gfyoung]

@jorisvandenbossche , @jreback : What do the "maybe" deprecation removals represent? Are they ones that could be done in 0.19.0 but are not necessary? Or that we're not sure yet about them?

[jreback]

I think let's try to prioritize the ones that are marked for 0.19.0, then do the maybes if possible (and other ones on a case-by-case).

[gfyoung]

Little confused by what's going with the '+' / '-' Index ops deprecation. Besides removing all of the warnings, are those ops just being re-purposed to do arithmetic addition / subtraction --> the change is just an API change to do arithmetic ops? Or are these functions being removed? My guess is the former, but I wanted to clarify.

[gfyoung]

@jreback, @jorisvandenbossche : you can check off the flavor deprecation removal on this list now that #13611 has been merged!

[gfyoung]

Also, just curious, any reason why we are removing SparsePanel now?

[jreback]

well SparsePanel is causing some addtl deprecation warnings that are annoying.

[gfyoung]

@jreback : Could you checkoff the return_type='frame'|'series' box in the "maybe" list? My PR to remove that was merged in a couple of days ago (#13701).

[gfyoung]

Another nit: @jorisvandenbossche 's #13735 should be moved into the For 0.19.0 section IINM (not "Future Release")

[jorisvandenbossche]

Little confused by what's going with the '+' / '-' Index ops deprecation. Besides removing all of the warnings, are those ops just being re-purposed to do arithmetic addition / subtraction --> the change is just an API change to do arithmetic ops? Or are these functions being removed? My guess is the former, but I wanted to clarify.

Yes, indeed, re-purposing to be arithmetic ops. If you would have time to do a PR, always welcome :-)

[gfyoung]

@jreback : For the 0.17 section, generate_bq_schema is no longer in the code-base. Probably removed in the migration to pandas-gbq?

[jreback]

@gfyoung

For the 0.17 section, generate_bq_schema is no longer in the code-base. Probably removed in the migration to pandas-gbq?

marked it

[gfyoung]

@jreback , @jorisvandenbossche : Could something regarding the policy of deprecation removal be added in the original issue? That might be useful in case anyone wants to work on this front (e.g. how many versions back does the deprecation need to be before it can removed?)

[jorisvandenbossche]

Good idea. I think we currently remove the deprecations of eg 0.17 in 0.20. This means we keep the deprcations for 3 releases (deprecation in 0.17 -> warns in 0.17, 0.18, 0.19 -> changed in 0.20) ?

[gfyoung]

Yes, three major releases back...that's correct IINM.

[gfyoung]

@jorisvandenbossche : Do we want to keep changing the milestone version each time? Your call.

[jorisvandenbossche]

It's not much work to change it :-)
I would put it at that milestone, then it is clear that there is work here to be done for that release. Once everything for that release is done, or when it's time for rc and we don't further put in deprecations, we just change it to the next release.
I see that @jreback had put it at an "Admin" milestone, but I think the workflow described above is OK?

[gfyoung]

@jorisvandenbossche : This workflow works me. It's a lot clearer now what deprecations need to be tackled and when (hopefully others will agree! 😄 )

[jreback]

@gfyoung FYI I want to see when we deprecated pd.options.line_width/height. They can be removed (in 0.21.0) am pretty sure

[gfyoung]

@jreback : You're absolutely right. 0.11 and 0.12 respectively. 😢

[jreback]

if anyone interested can remove any of .17/.18/.19 in 0.22

[jbrockmendel]

Noticing a couple of functions/methods not hit in tests, might merit consideration:

indexing.is_index_slice
Block.reindex_axis
_most_ of SingleBlockManager.reindex

[gfyoung]

@jbrockmendel : Might merit consideration? Absolutely! Are all of these deprecated though? I don't see all of them on the list. I would definitely would want to make sure the non-deprecated ones get more coverage if possible.

[jreback]

cc @gfyoung I updated to remove most of the 0.20.0 ones

[jorisvandenbossche]

I don't think we should already remove things deprecated in 0.20. I know we bumped 0.22 to 0.23, but for deprecations I don't think we need to see the released 0.22 as a "major release" in this regard (as it has no new features, and in principle should not delay 0.23).
0.20 was only released in May 2017, so those deprecations will not even be a year old.

[gfyoung]

Hmm...fair enough. I only started because I noticed that @jreback had moved a bunch under 0.23.

[jreback]

no we need to move the schedule up
otherwise it adds too many things in a single version

[gfyoung]

no we need to move the schedule up

@jreback : So...resume removing deprecated things from 0.20.0 ? I see that the original issue has not been modified since @jorisvandenbossche changed it.

[jorisvandenbossche]

@jreback you were not being fully clear, I can interprete "no need to move schedule up" as "no need to already start removing things from 0.20". But I think it was you who marked those 0.20 deprecations as ready for removal?

[jreback]

@jorisvandenbossche yes I changed that. I don't see any reason to wait on removing those (except some I specifically marked). if you have ones that you don't think should be removed until 1.0, then pls move them to where ix/Panel are.

[jorisvandenbossche]

As I said above, I think none of the deprecations of 0.20 should be removed in 0.23 (speaking about those in the first 0.20 list, so not about ix/Panel). IMO they should only be removed in the next major release, whether this is 0.24 or 1.0 that does not really matter.

Our implicit rule has been to keep deprecation at minimum for two major releases (but maybe we should document this expectation more explicitly), and in counting those major releases, I don't think we should count 0.22. Those deprecations will not even be a year old when 0.23 is released.

[TomAugspurger]

I don't think we should count 0.22. Those deprecations will not even be a year old when 0.23 is released.

Agreed with @jorisvandenbossche. The thing that matters most to downstream libraries is the calendar time between deprecating something and removing it. Since 0.22 accelerated our release schedule, I think we should hold off on removing stuff that was scheduled for 0.23.

[jreback]

and the 1 in question atm was deprecated in 0.19
so should be removed now

if you want to move forward on cleaning things and extending them removing deprecations is very important

[gfyoung]

@TomAugspurger : the deprecation done in #16617...looks like it got removed at some point? Is this still on the docket for removal in 0.25.0, or has the plan changed?

[jorisvandenbossche]

@gfyoung the PR you link to got never merged?

[gfyoung]

@jorisvandenbossche : Good point - oops 🙂. In that case, looks like we won't be removing this anytime soon. That being said, my question still stands regarding a future deprecation and removal of those methods.

[jorisvandenbossche]

That being said, my question still stands regarding a future deprecation and removal of those methods.

The whole SparseSeries class is being deprecated, so I don't think there is a need to further deprecate a certain case in its constructor.
There is DataFrame.sparse.from_spmatrix now to do this.

[WillAyd]

Note that I pinned this on GH after our DEV chat yesterday. I believe we want to actually remove anything that was deprecated prior to 1.0 for our 1.0 release, so PRs to do that from the community would certainly be welcome

[jbrockmendel]

@mroeschke would it be convenient for you to take point on removing #20584?

[mroeschke]

Most definitely @jbrockmendel. Can do.

[jbrockmendel]

@mroeschke heads up this and 13777 have been behaving poorly w/r/t getting multiple edits in a short period of time. in this case, I think it silently reverted the rolling.apply edit you made

[mroeschke]

Thanks for the headsup @jbrockmendel. Let's see if 2nd's time a charm.

[jbrockmendel]

@TomAugspurger do you have comparative advantage in #23767?

@mroeschke IIRC #22644 was yours, want to finish it off?

@WillAyd didn't you already take care of #18731? if so pls update the checkbox lists

@pandas-dev/pandas-core A few of these that are ambiguous and may need discussion/clarification:

• [ ] #18347 .add_suffix/.add_prefix - the discussion there makes it look like this may not be deprecated after all. Can we confirm the status on this?
• [ ] #21930 Series.compress - removing this breaks np.compress(series). Do we care?
• [ ] #27112 Series.item & Index.item - there was some recent discussion of un-deprecating item. Where did we land on that?

[jreback]

• +1 on re-enabling .item()
• +0 on deprecating .add_suffix/.add_prefix, IIRC don't believe was actually deprecated
• ok with removing .compress() its not very useful anyhow even in numpy, esp for 1-D where its just indexing.

[jorisvandenbossche]

add_prefix/add_suffix were never deprecated, so there is no discussion about removing them. Someone can open a new issue (or find an existing one) to further discuss deprecating it, but since it is here about removal of actually deprecated things, removed this from the list above.

I think I am fine with np.compress() breaking.

Let's have the discussion about item in https://github.com/pandas-dev/pandas/issues/29250 (I am in favor of at least keeping part of it)

[jbrockmendel]

@jorisvandenbossche there are a couple of these I tried and couldn't figure out. Can I get you to take a look or suggest someone I should bug? First is the how fill_method in resample (needs GH ref). Second is the one for #26403.

[jschendel]

Second is the one for #26403.

Just opened a PR for that one, see #29955

[jorisvandenbossche]

First is the how fill_method in resample (needs GH ref)

That deprecation was introduced in https://github.com/pandas-dev/pandas/pull/11841. This might be left-overs that were not removed in https://github.com/pandas-dev/pandas/pull/20782

[jbrockmendel]

@mroeschke any others you want to call dibs on before i go to town?

[mroeschke]

@jbrockmendel I'll probably be slow on deprecations this week. Go for it.

[jorisvandenbossche]

I moved the deprecations we are doing in 1.0 to a new issue: https://github.com/pandas-dev/pandas/issues/30228 (so a similar issue as this to keep a log of deprecations, but starting a fresh one for 1.x)

[TomAugspurger]

All the listed deprecations have been cleaned up. Thanks all!