Pandas: RLS: 1.1.1

[simonjayhawkins]

Tracking issue for the 1.1.1 release.

https://github.com/pandas-dev/pandas/milestone/75

Please do not remove/change milestones from these issues without a note explaining the reasoning (changing milestones doesn't trigger notification)

All issues raised that are regressions from 1.0.5 should be milestoned 1.1.1. Post 1.1.1 release, unclosed issues from this list can be changed to 1.1.2 to maintain the tracking or reviewed.

[jreback]

@simonjayhawkins i think all of the PR's are merged, except for the excel encoding thing. any issues left @simonjayhawkins ?

[simonjayhawkins]

i closed #35764. need to decide what to do there. but no other PRs in progress that I can think of other than the encoding one.

was originally planning release end of this week (i'll review the milestone and see if we can bring it forward)

[jreback]

i closed #35764. need to decide what to do there. but no other PRs in progress that I can think of other than the encoding one.

was originally planning release end of this week (i'll review the milestone and see if we can bring it forward)

yeah i think can just push a change to return self.copy() and call it a day

[simonjayhawkins]

ok. i'll create a 1.1.2 milestone and move the bulk of the open issues across. i'll ought to look into #35460!

[simonjayhawkins]

@simonjayhawkins i think all of the PR's are merged, except for the excel encoding thing. any issues left @simonjayhawkins ?

29442 is milestoned 1.1.1, there is an open PR #35688 without a milestone (and no release note yet). There is ongoing discussion regarding a suitable location for the fix so may not be resolved in next few days.

[TomAugspurger]

https://github.com/pandas-dev/pandas/pull/35688 shouldn't block the release.

[jreback]

agreed
release can be anytime

[simonjayhawkins]

@TomAugspurger getting failure on pip test. will run conda test next. is this something to be concerned about?

================================================================== FAILURES ===================================================================
______________________________________________________ test_missing_required_dependency _______________________________________________________

    def test_missing_required_dependency():
        # GH 23868
        # To ensure proper isolation, we pass these flags
        # -S : disable site-packages
        # -s : disable user site-packages
        # -E : disable PYTHON* env vars, especially PYTHONPATH
        # https://github.com/MacPython/pandas-wheels/pull/50

        pyexe = sys.executable.replace("\\", "/")
        call = [pyexe, "-sSE", "-c", "import pandas"]

        msg = (
            rf"Command '\['{pyexe}', '-sSE', '-c', 'import pandas'\]' "
            "returned non-zero exit status 1."
        )

        with pytest.raises(subprocess.CalledProcessError, match=msg) as exc:
>           subprocess.check_output(call, stderr=subprocess.STDOUT)
E           Failed: DID NOT RAISE <class 'subprocess.CalledProcessError'>

/opt/conda/envs/pip-test/lib/python3.7/site-packages/pandas/tests/test_downstream.py:161: Failed
============================================================== warnings summary ===============================================================
/opt/conda/envs/pip-test/lib/python3.7/site-packages/_pytest/config/__init__.py:1040
  /opt/conda/envs/pip-test/lib/python3.7/site-packages/_pytest/config/__init__.py:1040: PytestAssertRewriteWarning: Module already imported so cannot be rewritten: hypothesis
    self._mark_plugins_for_rewrite(hook)

tests/arrays/masked/test_arithmetic.py::test_array_scalar_like_equivalence[boolean-__pow__]
tests/extension/test_boolean.py::TestArithmeticOps::test_arith_series_with_scalar[__pow__]
tests/extension/test_boolean.py::TestArithmeticOps::test_arith_frame_with_scalar[__pow__]
  /opt/conda/envs/pip-test/lib/python3.7/site-packages/pandas/core/arrays/boolean.py:735: DeprecationWarning: In future, it will be an error for 'np.bool_' scalars to be interpreted as an index
    result = op(self._data, other)

tests/indexes/test_index_new.py::TestIndexConstructorInference::test_constructor_infer_nat_dt_like[<NA>-DatetimeIndex-datetime64[ns]-ctor0-0]
tests/indexes/test_index_new.py::TestIndexConstructorInference::test_constructor_infer_nat_dt_like[<NA>-DatetimeIndex-datetime64[ns]-ctor0-0]
tests/indexes/test_index_new.py::TestIndexConstructorInference::test_constructor_infer_nat_dt_like[<NA>-DatetimeIndex-datetime64[ns]-ctor0-1]
tests/indexes/test_index_new.py::TestIndexConstructorInference::test_constructor_infer_nat_dt_like[<NA>-DatetimeIndex-datetime64[ns]-ctor0-1]
tests/indexes/test_index_new.py::TestIndexConstructorInference::test_constructor_infer_nat_dt_like[<NA>-TimedeltaIndex-timedelta64[ns]-ctor1-0]
tests/indexes/test_index_new.py::TestIndexConstructorInference::test_constructor_infer_nat_dt_like[<NA>-TimedeltaIndex-timedelta64[ns]-ctor1-0]
tests/indexes/test_index_new.py::TestIndexConstructorInference::test_constructor_infer_nat_dt_like[<NA>-TimedeltaIndex-timedelta64[ns]-ctor1-1]
tests/indexes/test_index_new.py::TestIndexConstructorInference::test_constructor_infer_nat_dt_like[<NA>-TimedeltaIndex-timedelta64[ns]-ctor1-1]
  /opt/conda/envs/pip-test/lib/python3.7/site-packages/pandas/_testing.py:790: DeprecationWarning: elementwise comparison failed; this will raise an error in the future.
    diff = np.sum((left.values != right.values).astype(int)) * 100.0 / len(left)

-- Docs: https://docs.pytest.org/en/stable/warnings.html
=========================================================== short test summary info ===========================================================
FAILED ../opt/conda/envs/pip-test/lib/python3.7/site-packages/pandas/tests/test_downstream.py::test_missing_required_dependency - Failed: DI...
==================== 1 failed, 64188 passed, 7303 skipped, 225 deselected, 1017 xfailed, 12 warnings in 1022.93s (0:17:02) ====================
(base) root@cf6757b4e4bb:/pandas-release#

[TomAugspurger]

Mmm you can probably ignore that? As long as it's passing on MacPython/pandas-wheels I think we're OK.

[simonjayhawkins]

same test failed on conda

=========================== short test summary info ============================
FAILED ../_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_p/lib/python3.8/site-packages/pandas/tests/test_downstream.py::test_missing_required_dependency
= 1 failed, 64192 passed, 7301 skipped, 225 deselected, 1016 xfailed, 39 warnings in 999.59s (0:16:39) =
Tests failed for pandas-1.1.1-py38h2d6d810_0.tar.bz2 - moving package to /opt/conda/conda-bld/broken
WARNING:conda_build.build:Tests failed for pandas-1.1.1-py38h2d6d810_0.tar.bz2 - moving package to /opt/conda/conda-bld/broken
WARNING conda_build.build:tests_failed(2231): Tests failed for pandas-1.1.1-py38h2d6d810_0.tar.bz2 - moving package to /opt/conda/conda-bld/broken
TESTS FAILED: pandas-1.1.1-py38h2d6d810_0.tar.bz2

I'll pick this up again tomorrow.

[simonjayhawkins]

As long as it's passing on MacPython/pandas-wheels I think we're OK.

that test seems to be ok. but seeing some other failures on some builds https://github.com/MacPython/pandas-wheels/pull/93

=========================== short test summary info ===========================
FAILED test_venv/lib/site-packages/pandas/tests/io/parser/test_common.py::test_file_descriptor_leak[c_high]
FAILED test_venv/lib/site-packages/pandas/tests/window/test_grouper.py::TestGrouperGrouping::test_groupby_rolling_center_center
FAILED test_venv/lib/site-packages/pandas/tests/window/test_grouper.py::TestGrouperGrouping::test_groupby_subselect_rolling
FAILED test_venv/lib/site-packages/pandas/tests/window/test_grouper.py::TestGrouperGrouping::test_groupby_rolling_custom_indexer
FAILED test_venv/lib/site-packages/pandas/tests/window/test_grouper.py::TestGrouperGrouping::test_groupby_rolling_subset_with_closed
FAILED test_venv/lib/site-packages/pandas/tests/window/test_grouper.py::TestGrouperGrouping::test_groupby_subset_rolling_subset_with_closed
= 6 failed, 63972 passed, 7374 skipped, 1044 xfailed, 14 warnings in 574.95s (0:09:34) =

and all Mac builds are failing

error: $MACOSX_DEPLOYMENT_TARGET mismatch: now "10.9" but "10.14" during configure

[simonjayhawkins]

and all Mac builds are failing

error: $MACOSX_DEPLOYMENT_TARGET mismatch: now "10.9" but "10.14" during configure

also on nightly builds

[simonjayhawkins]

The groupby tests (failing on backport branch test) started failing on the nightly builds between 6/7 August.

test_groupby_subselect_rolling was added in #35513 @mroeschke

test_groupby_rolling_center_center failing on nightly build added in #35562

test_groupby_rolling_custom_indexer failing on nightly build added in #35647

test_groupby_rolling_subset_with_closed and test_groupby_subset_rolling_subset_with_closed failing on nightly build added in #35639

[simonjayhawkins]

AFAICT test_file_descriptor_leak[c_high] not failing on nightly builds. failing on windows py_3.6_32, windows py_3.8_32 and windows py_3.8_64 only

all three crashed workers

worker 'gw1' crashed while running 'test_venv/lib/site-packages/pandas/tests/io/parser/test_common.py::test_file_descriptor_leak[c_high]'

@jbrockmendel you made some changes to check_file_leaks (called from test_file_descriptor_leak) in #35693. This PR was labelled CI. was CI failing and should that PR have been backported.

[TomAugspurger]

error: $MACOSX_DEPLOYMENT_TARGET mismatch: now "10.9" but "10.14" during configure

Huh, I haven't seen that. Is it just while building psutil? Perhaps we can pin for an older version.

[simonjayhawkins]

Huh, I haven't seen that. Is it just while building psutil? Perhaps we can pin for an older version.

the failure is in the pip install pytest pytest-xdist hypothesis stage

AFAICT this is from pip install $TEST_DEPENDS in extra_functions.sh

so does that mean changing TEST_DEPENDS

[simonjayhawkins]

@TomAugspurger is the mac builds the only blocker for 1.1.1 release here

[simonjayhawkins]

@TomAugspurger is the mac builds the only blocker for 1.1.1 release here

looking at the nightly builds, if the test fails the wheel is not uploaded even if it is built. Is that correct?

[TomAugspurger]

looking at the nightly builds, if the test fails the wheel is not uploaded even if it is built. Is that correct?

I think that's correct.

[simonjayhawkins]

the mac builds on nightly started to fail on Saturday

eg macOS py_3.7_64

2020-08-15T03:42:41.7780780Z + pip install pytest pytest-xdist hypothesis
2020-08-15T03:42:42.9501640Z Collecting pytest
2020-08-15T03:42:42.9759240Z   Downloading pytest-6.0.1-py3-none-any.whl (270 kB)
2020-08-15T03:42:43.2320350Z Collecting pytest-xdist
2020-08-15T03:42:43.2428960Z   Downloading pytest_xdist-2.0.0-py2.py3-none-any.whl (36 kB)
2020-08-15T03:42:46.2895450Z Collecting hypothesis
2020-08-15T03:42:46.3031150Z   Downloading hypothesis-5.24.4-py3-none-any.whl (323 kB)
2020-08-15T03:42:46.3639550Z Collecting iniconfig
2020-08-15T03:42:46.3733180Z   Downloading iniconfig-1.0.1-py3-none-any.whl (4.2 kB)
2020-08-15T03:42:46.4161670Z Collecting toml
2020-08-15T03:42:46.4256890Z   Downloading toml-0.10.1-py2.py3-none-any.whl (19 kB)
2020-08-15T03:42:46.4925710Z Collecting pluggy<1.0,>=0.12
2020-08-15T03:42:46.5023630Z   Downloading pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
2020-08-15T03:42:46.5850510Z Collecting attrs>=17.4.0
2020-08-15T03:42:46.5983580Z   Downloading attrs-19.3.0-py2.py3-none-any.whl (39 kB)
2020-08-15T03:42:46.7620900Z Collecting more-itertools>=4.0.0
2020-08-15T03:42:46.7728000Z   Downloading more_itertools-8.4.0-py3-none-any.whl (43 kB)
2020-08-15T03:42:46.9142270Z Collecting packaging
2020-08-15T03:42:46.9219650Z   Downloading packaging-20.4-py2.py3-none-any.whl (37 kB)
2020-08-15T03:42:47.0893640Z Collecting importlib-metadata>=0.12; python_version < "3.8"
2020-08-15T03:42:47.0943990Z   Using cached importlib_metadata-1.7.0-py2.py3-none-any.whl (31 kB)
2020-08-15T03:42:47.1769970Z Collecting py>=1.8.2
2020-08-15T03:42:47.1861410Z   Downloading py-1.9.0-py2.py3-none-any.whl (99 kB)
2020-08-15T03:42:47.2450150Z Collecting pytest-forked
2020-08-15T03:42:47.2536370Z   Downloading pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
2020-08-15T03:42:47.7529280Z Collecting psutil>=3.0.0
2020-08-15T03:42:47.7662740Z   Downloading psutil-5.7.2.tar.gz (460 kB)
2020-08-15T03:42:48.4232370Z Collecting execnet>=1.1
2020-08-15T03:42:48.4301800Z   Downloading execnet-1.7.1-py2.py3-none-any.whl (39 kB)
2020-08-15T03:42:48.5120450Z Collecting sortedcontainers<3.0.0,>=2.1.0
2020-08-15T03:42:48.5218510Z   Downloading sortedcontainers-2.2.2-py2.py3-none-any.whl (29 kB)
2020-08-15T03:42:48.7571200Z Collecting pyparsing>=2.0.2
2020-08-15T03:42:48.7670150Z   Downloading pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
2020-08-15T03:42:48.9306660Z Collecting six
2020-08-15T03:42:48.9358110Z   Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
2020-08-15T03:42:49.0952230Z Collecting zipp>=0.5
2020-08-15T03:42:49.1026340Z   Using cached zipp-3.1.0-py3-none-any.whl (4.9 kB)
2020-08-15T03:42:49.1404540Z Collecting apipkg>=1.4
2020-08-15T03:42:49.1507280Z   Downloading apipkg-1.5-py2.py3-none-any.whl (4.9 kB)
2020-08-15T03:42:49.1632650Z Building wheels for collected packages: psutil
2020-08-15T03:42:49.1643460Z   Building wheel for psutil (setup.py): started
2020-08-15T03:42:50.6391420Z   Building wheel for psutil (setup.py): finished with status 'error'
2020-08-15T03:42:50.6392750Z   ERROR: Command errored out with exit status 1:

on Friday

2020-08-14T03:44:18.2608020Z + pip install pytest pytest-xdist hypothesis
2020-08-14T03:44:19.1547510Z Collecting pytest
2020-08-14T03:44:19.1781660Z   Downloading pytest-6.0.1-py3-none-any.whl (270 kB)
2020-08-14T03:44:19.3924390Z Collecting pytest-xdist
2020-08-14T03:44:19.4034840Z   Downloading pytest_xdist-1.34.0-py2.py3-none-any.whl (36 kB)
2020-08-14T03:44:21.8781040Z Collecting hypothesis
2020-08-14T03:44:21.8871660Z   Downloading hypothesis-5.24.3-py3-none-any.whl (321 kB)
2020-08-14T03:44:22.0705180Z Collecting more-itertools>=4.0.0
2020-08-14T03:44:22.0803940Z   Downloading more_itertools-8.4.0-py3-none-any.whl (43 kB)
2020-08-14T03:44:22.1337380Z Collecting pluggy<1.0,>=0.12
2020-08-14T03:44:22.1412590Z   Downloading pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
2020-08-14T03:44:22.2080810Z Collecting py>=1.8.2
2020-08-14T03:44:22.2172710Z   Downloading py-1.9.0-py2.py3-none-any.whl (99 kB)
2020-08-14T03:44:22.2496160Z Collecting iniconfig
2020-08-14T03:44:22.2564690Z   Downloading iniconfig-1.0.1-py3-none-any.whl (4.2 kB)
2020-08-14T03:44:22.2899750Z Collecting toml
2020-08-14T03:44:22.2975770Z   Downloading toml-0.10.1-py2.py3-none-any.whl (19 kB)
2020-08-14T03:44:22.3051970Z Requirement already satisfied: packaging in ./test_venv/lib/python3.7/site-packages (from pytest) (20.4)
2020-08-14T03:44:22.3533380Z Collecting attrs>=17.4.0
2020-08-14T03:44:22.3601170Z   Downloading attrs-19.3.0-py2.py3-none-any.whl (39 kB)
2020-08-14T03:44:22.5516180Z Collecting importlib-metadata>=0.12; python_version < "3.8"
2020-08-14T03:44:22.5572160Z   Using cached importlib_metadata-1.7.0-py2.py3-none-any.whl (31 kB)
2020-08-14T03:44:22.6162780Z Collecting pytest-forked
2020-08-14T03:44:22.6245200Z   Downloading pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
2020-08-14T03:44:22.6330440Z Requirement already satisfied: six in ./test_venv/lib/python3.7/site-packages (from pytest-xdist) (1.15.0)
2020-08-14T03:44:22.6849220Z Collecting execnet>=1.1
2020-08-14T03:44:22.6957910Z   Downloading execnet-1.7.1-py2.py3-none-any.whl (39 kB)
2020-08-14T03:44:22.7535990Z Collecting sortedcontainers<3.0.0,>=2.1.0
2020-08-14T03:44:22.7615310Z   Downloading sortedcontainers-2.2.2-py2.py3-none-any.whl (29 kB)
2020-08-14T03:44:22.7695500Z Requirement already satisfied: pyparsing>=2.0.2 in ./test_venv/lib/python3.7/site-packages (from packaging->pytest) (2.4.7)
2020-08-14T03:44:22.8457100Z Collecting zipp>=0.5
2020-08-14T03:44:22.8499370Z   Using cached zipp-3.1.0-py3-none-any.whl (4.9 kB)
2020-08-14T03:44:22.8858000Z Collecting apipkg>=1.4
2020-08-14T03:44:22.8932760Z   Downloading apipkg-1.5-py2.py3-none-any.whl (4.9 kB)
2020-08-14T03:44:23.0412380Z Installing collected packages: more-itertools, zipp, importlib-metadata, pluggy, py, iniconfig, toml, attrs, pytest, pytest-forked, apipkg, execnet, pytest-xdist, sortedcontainers, hypothesis
2020-08-14T03:44:23.8842830Z Successfully installed apipkg-1.5 attrs-19.3.0 execnet-1.7.1 hypothesis-5.24.3 importlib-metadata-1.7.0 iniconfig-1.0.1 more-itertools-8.4.0 pluggy-0.13.1 py-1.9.0

so looks like maybe pytest_xdist-1.34.0 -> pytest_xdist-2.0.0 (although hypothesis had small bump)

[simonjayhawkins]

@TomAugspurger getting failure on pip test. will run conda test next. is this something to be concerned about?

test_missing_required_dependency is not run on MacPython

https://github.com/MacPython/pandas-wheels/blob/f607d12b25060b0cb07e25dbbb16bfb87d7cbdcf/config.sh#L36

so that must have failed in the past. https://github.com/pandas-dev/pandas/issues/33999

[simonjayhawkins]

so looks like maybe pytest_xdist-1.34.0 -> pytest_xdist-2.0.0 (although hypothesis had small bump)

pinning fixes the Mac builds. see latest build at https://github.com/MacPython/pandas-wheels/pull/93

known issue https://github.com/pytest-dev/pytest-xdist/issues/583 and https://github.com/pytest-dev/pytest-xdist/issues/585

[mroeschke]

The groupby rolling failures may still be related to https://github.com/pandas-dev/pandas/issues/35294

Which I had a PR for: https://github.com/pandas-dev/pandas/pull/35228

But I haven't had a moment to revisit and test locally via Docker yet.

[simonjayhawkins]

and maybe related to https://github.com/pandas-dev/pandas/issues/31856

The failures are on 32 bit py36 and py38 only

[simonjayhawkins]

so looks like maybe pytest_xdist-1.34.0 -> pytest_xdist-2.0.0 (although hypothesis had small bump)

pinning fixes the Mac builds. see latest build at MacPython/pandas-wheels#93

known issue pytest-dev/pytest-xdist#583 and pytest-dev/pytest-xdist#585

I think this also fixed the test_file_descriptor_leak fail.

[simonjayhawkins]

The groupby rolling failures may still be related to #35294

OK to xfail those tests using the same ref as #35295 i.e. @pytest.mark.xfail(not compat.IS64, reason="GH-35294")

[simonjayhawkins]

planning to tag locally once #35801 merged and backported

[simonjayhawkins]

open regressions moved to 1.1.2 https://github.com/pandas-dev/pandas/milestone/76 (will open tracking issue shortly)

[jreback]

@simonjayhawkins you should be good on PyPi.

[simonjayhawkins]

docs uploaded https://pandas.pydata.org/pandas-docs/version/1.1.1/index.html

(once wheels are uploaded will switch to stable)

[simonjayhawkins]

wheels on PyPi

[simonjayhawkins]

stable docs switched to 1.1.1

link on home page autoupdated for the past few releases https://pandas.pydata.org/

will pick this up again and investigate tomorrow

[simonjayhawkins]

link on home page autoupdated for the past few releases https://pandas.pydata.org/

will pick this up again and investigate tomorrow

sorted itself out

[jorisvandenbossche]

@simonjayhawkins Thanks a lot for the release!

[TomAugspurger]

Thanks Simon!