Pandas: "Concatenation will no longer sort" wording is very inconsistent

[quant9]

The wording in the update docs is (emphasis added):

In a future version of pandas pandas.concat() will no longer sort the non-concatenation axis when it is not already aligned. (...)

To keep the previous behavior (sorting) and silence the warning, pass sort=True.
To accept the future behavior (no sorting), pass sort=False

https://pandas-docs.github.io/pandas-docs-travis/whatsnew.html#concatenation-will-no-longer-sort

But the wording in the FutureWarning messages is (emphasis added):

Sorting because non-concatenation axis is not aligned. A future version
of pandas will change to not sort by default.

To accept the future behavior, pass sort=True.
To retain the current behavior and silence the warning, pass sort=False

https://github.com/pandas-dev/pandas/blob/c4da79b5b322c73d8e61d1cb98ac4ab1e2438b40/pandas/core/indexes/api.py

It needs to be clearer that current default is still sort=True but will revert to sort=False in a future version.

[TomAugspurger]

It's confusing because it's wrong :)

Duplicate of https://github.com/pandas-dev/pandas/issues/21101 if you want to take a look at fixing it.

[techvslife]

also: Currently the warning doesn't get silenced if you pass "sort=False" (i.e. the true future default behavior)--presumably there should be no warning as long as the sort parameter is explicitly set (passed).

[TomAugspurger]

Do you have an example of a warning being emitted with sort=False?

On Mon, May 21, 2018 at 11:56 AM, techvslife [e-mail]
wrote:

(also: Currently the warning doesn't get silenced if you pass "sort=False"
(i.e. the true future default behavior)--presumably there should be no
warning as long as the sort parameter is explicitly set (passed).)

—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/pandas-dev/pandas/issues/21112#issuecomment-390715220,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABQHIpAD9lxmu-3099jbpALhS3ZyfSEAks5t0vG3gaJpZM4UD6_G
.

[techvslife]

Yes, what happens is that it does not produce the line of code in the warning (as it does when you omit the sort parameter), but it does still print the warning (this is the exact text and line spacing, not sure why the last line looks so strange):

C:\Users\mlopez\Anaconda3\lib\site-packages\pandas\core\frame.py:6201: FutureWarning: Sorting because non-concatenation axis is not aligned. A future version
of pandas will change to not sort by default.

To accept the future behavior, pass 'sort=True'.

To retain the current behavior and silence the warning, pass sort=False

  sort=sort)

[TomAugspurger]

Can you make a small reproducible example (with data) that emits the
warning with sort=False?

On Mon, May 21, 2018 at 1:30 PM, techvslife [e-mail]
wrote:

Yes, what happens is that it does not produce the line in the warning (as
it does when you omit the sort parameter), but it does print the warning
(this is the exact text and line spacing, not sure why the last line looks
so strange):

C:UsersmlopezAnaconda3libsite-packagespandascoreframe.py:6201: FutureWarning: Sorting because non-concatenation axis is not aligned. A future version
of pandas will change to not sort by default.

To accept the future behavior, pass 'sort=True'.

To retain the current behavior and silence the warning, pass sort=False

sort=sort)

—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/pandas-dev/pandas/issues/21112#issuecomment-390741837,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABQHIgdb-iGD6Vf-ngWHubuYQP9gdN4uks5t0wfcgaJpZM4UD6_G
.

[techvslife]

Sorry, I'm not having luck reproducing with a small example; may not be a concat command, but a merge command instead. Unfortunately I don't know what line in code is triggering the warning from the warning.

[TomAugspurger]

You can make warnings raise. Something like

import warnings

warnings.simplefilter("error")
<your code>

On Mon, May 21, 2018 at 2:06 PM, techvslife [e-mail]
wrote:

Sorry, I'm not having luck reproducing with a small example; may not be a
concat command, but a merge command instead. Unfortunately I don't know
what line in code is triggering the warning from the warning.

—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/pandas-dev/pandas/issues/21112#issuecomment-390751519,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABQHIgfJzYneJoqF0VaPK1WMQhQlSp9lks5t0xAngaJpZM4UD6_G
.

[techvslife]

ok thanks, I'll try re-running it, but I won't be able to set up the full script today.

[techvslife]

Ok, I found it. It actually was an append command (not concat or merge). Adding the sort parameter did remove the warning. I added the sort=False parameter to this:
df_B = df_A.append(df_B, verify_integrity=True, sort=False).copy()

[TomAugspurger]

Thanks for checking.

On Wed, May 23, 2018 at 5:17 PM, techvslife [e-mail]
wrote:

Ok, I found it. It actually was an append command (not concat or merge).
Adding the sort parameter did remove the warning. I added the sort=False
parameter to this:
df_B = df_A.append(df_B, verify_integrity=True, sort=False).copy()

—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/pandas-dev/pandas/issues/21112#issuecomment-391514421,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABQHItsIjgL3fXCxhDf7n0p4bNPqsMQnks5t1d_tgaJpZM4UD6_G
.

[allayli]

It seems that sort=None is NOT backward compatible.
Why NOT make it True by default? Or any Global config can set it be True?

[TomAugspurger]

Do you have an example where it isn't backwards compatible?

On Thu, Jun 7, 2018 at 10:04 AM, Li Sixiang [e-mail]
wrote:

It seems that sort=None is NOT backward compatible.
Why NOT make it True by default? Or any Global config can set it be True?

—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/pandas-dev/pandas/issues/21112#issuecomment-395454312,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABQHIiaP5dcDeW2YvPtCfKxOfa_AcVZ6ks5t6UEFgaJpZM4UD6_G
.

[TasneemVKhan]

i had the same error I resolved it using
store_items = store_items.append(new_store , sort=True)

You just have to add sort =True to your code and the warning disappears

[Dougz00]

If you are using glob and a loop to read a large number of csv files this also works:
files = glob.glob(pattern)
df = pd.DataFrame()
for f in files:
df = df.append(pd.read_csv(f),sort=True)

[jreback]

If you are using glob and a loop to read a large number of csv files this also works:
files = glob.glob(pattern)
df = pd.DataFrame()
for f in files:
df = df.append(pd.read_csv(f),sort=True)

@Dougz00 that is exponential time

concatenation a list of data frames ; don’t append
see the append warnings in the docs