we have several regressions reported since 1.0.3 (not due to 1.0.3, since 0.25.3) and a couple that had not been fixed.
https://github.com/pandas-dev/pandas/issues?page=1&q=is%3Aopen+is%3Aissue+label%3ARegression
Is there any that warrant a 1.0.4 release?
@pandas-dev/pandas-core
simonjayhawkins
All 53 comments
I probably won't be able to help with one this week, but the scripts at https://github.com/pandas-dev/pandas-release should be reasonably up to date.
We'll need to manually verify that all the necessary PRs have been backported (I don't think they have). It might be easier to do a 1.1 release soon.
TomAugspurger
on 6 Apr 2020
I probably won't be able to help with one this week
I don't think this is urgent. It's more a matter of whether we should be labelling PRs that fix regressions as 1.0.4 and backporting.
It might be easier to do a 1.1 release soon.
IMO we should be ensuring that the 1.0.x fixes regressions from 0.25.3. 1.1 could introduce new regressions. so this is about ensuring an upgrade path for affected users.
simonjayhawkins
on 6 Apr 2020
@simonjayhawkins I think in the call last week we decided to leave it up to you. If you have time to go through and backport the necessary PRs for a release we can go ahead and do it.
In addition to backporting, I think that
- https://github.com/MacPython/pandas-wheels/pull/82
- https://github.com/pandas-dev/pandas/issues/33210
are blockers.
TomAugspurger
on 15 Apr 2020
potential backports (add to this list)
- [x] #33462 (regression in 1.0.2)
simonjayhawkins
on 17 Apr 2020
@simonjayhawkins still working on this?
jreback
on 4 May 2020
When this issue was opened there was no PRs merged to master that fixed regressions. There were also blockers to doing a release, https://github.com/pandas-dev/pandas/issues/33300#issuecomment-614069516.
I've not been working on this, but now we are a few weeks on and the blockers are resolved, we now have a few PRs that _could_ be backported.
I could start backporting these if we are likely to do a 1.0.4 release.
simonjayhawkins
on 4 May 2020
At this point, I think pushing for a 1.1 release in a week or two is probably easier.
TomAugspurger
on 4 May 2020
I actually would be +1 on a 1.0.4; I also think this is orthogonal to doing 1.1, and good for others to try a release.
jreback
on 4 May 2020
In addition to backporting, I think that
- MacPython/pandas-wheels#82
- #33210
are blockers.
32737 wasn't backported, so #33210 shouldn't be an issue here?
simonjayhawkins
on 4 May 2020
@jorisvandenbossche there is a to_feather test failing on 1.0.x for Linux py37_locale. pyarrow 0.17 is being installed. see https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=34788&view=logs&j=a3a13ea8-7cf0-5bdb-71bb-6ac8830ae35c
The logs for the previous builds on this branch where this test passed are unavailable to confirm the pyarrow version.
I can't find any recent PRs to fix ci for this. maybe #33422 fixes this.
on windows, conda is not installing pyarrow 0.17 so can't reproduce failure, tests pass with 0.16
$ conda install pyarrow==0.17.0 -c conda-forge
Collecting package metadata (repodata.json): done
Solving environment: \
The environment is inconsistent, please check the package plan carefully
The following packages are causing the inconsistency:
- conda-forge/win-64::fastparquet==0.3.3=py38hc8d92b1_0
- conda-forge/win-64::pyarrow==0.16.0=py38h57df961_2
- conda-forge/win-64::pyreadstat==0.3.3=py38h11373af_0
- conda-forge/noarch::seaborn==0.10.1=py_0
- conda-forge/win-64::statsmodels==0.11.1=py38h9de7a3e_1
- conda-forge/noarch::xarray==0.15.1=py_0
done
## Package Plan ##
environment location: C:\Users\simon\Anaconda3\envs\pandas-dev
added / updated specs:
- pyarrow==0.17.0
The following packages will be downloaded:
package | build
---------------------------|-----------------
arrow-cpp-0.16.0 |py38h5a970ca_5_cpu 11.8 MB conda-forge
openssl-1.1.1g | he774522_0 5.7 MB conda-forge
------------------------------------------------------------
Total: 17.5 MB
The following NEW packages will be INSTALLED:
pandas conda-forge/win-64::pandas-1.0.3-py38he6e81aa_1
The following packages will be UPDATED:
arrow-cpp 0.16.0-py38h5a970ca_4 --> 0.16.0-py38h5a970ca_5_cpu
ca-certificates pkgs/main::ca-certificates-2020.1.1-0 --> conda-forge::ca-certificates-2020.4.5.1-hecc5488_0
The following packages will be SUPERSEDED by a higher-priority channel:
certifi pkgs/main::certifi-2020.4.5.1-py38_0 --> conda-forge::certifi-2020.4.5.1-py38h32f6830_0
openssl pkgs/main --> conda-forge
Proceed ([y]/n)? y
Downloading and Extracting Packages
openssl-1.1.1g | 5.7 MB | ################################################################################################################# | 100%
arrow-cpp-0.16.0 | 11.8 MB | ################################################################################################################# | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
(pandas-dev)
$ conda list pyarrow
# packages in environment at C:\Users\simon\Anaconda3\envs\pandas-dev:
#
# Name Version Build Channel
pyarrow 0.16.0 py38h57df961_2 conda-forge
(pandas-dev)
If Simon is up to it, also +1 on doing a 1.0.4 release.
there is a to_feather test failing on 1.0.x for Linux py37_locale. pyarrow 0.17 is being installed
https://github.com/pandas-dev/pandas/pull/33422 might indeed fix this, as I changed some tests there. I think that PR should be OK to backport (the actual code changes are limited, just about passing through keywords, or could also limit to only backport the test fixes)
I can take a look at this if you want.
jorisvandenbossche
on 4 May 2020
or could also limit to only backport the test fixes
will do that, just aiming to get ci passing on the 1.0.x branch for now. once the branch is ready to accept backports, we can discuss what can/should be backported. (for instance, #32611 fixed a regression (#33058), but may not be suitable for backporting.)
simonjayhawkins
on 5 May 2020
Guys, can you push the release of 1.0.4 (at least, please do not to postpone it to be after 1.1.0, as I saw at the last dev meeting minutes) or apply some more love to it?
The memory leak in rolling,min/max is serious, do you understand that thousands of apps using pandas crash worldwide in the meantime, till the fix is available for us, users of pandas?
hroff-1902
on 15 May 2020
Guys, can you push the release of 1.0.4 (at least, please do not to postpone it to be after 1.1.0, as I saw at the last dev meeting minutes) or apply some more love to it?
The memory leak in rolling,min/max is serious, do you understand that thousands of apps using pandas crash worldwide in the meantime, till the fix is available for us, users of pandas?
@hroff-1902
pandas is an all volunteer project; you are welcome to help out
comment like the above are simply not helpful
jreback
on 15 May 2020
yes, a volunteer project, as many others, that's clear
the release procedure/regressions is not a part of the development process where a novice in a project could really help, I guess
sometimes (often) devs do not understand the severity of the issues so explicit expressions of concernments from the users of the project can help, at least sometimes.
please understand our concerns
hroff-1902
on 15 May 2020
@hroff-1902 have you been following the huge amount of work @simonjayhawkins has been doing to prep a 1.0.4 release?
Meanwhile, a release is blocked by our wheel building infrastructure at https://github.com/macpython/pandas-wheels. https://github.com/pandas-dev/pandas/issues/34114 will need to be fixed before we can make a release. Can you look into it?
TomAugspurger
on 15 May 2020
yes, a volunteer project, as many others, that's clear
the release procedure/regressions is not a part of the development process where a novice in a project could really help, I guess
sometimes (often) devs do not understand the severity of the issues so explicit expressions of concernments from the users of the project can help, at least sometimes.
please understand our concerns
there are many areas you could help, why are you not volunteering there?
I do understand severity of issues very well and am sympathetic but w/o any way to actually direct resources, your comments are not meaningful.
Its like, let’s not pay for anything, not contribute in-kind and yet have the audacity to demand something.
jreback
on 15 May 2020
@simonjayhawkins I would be in favor of releasing this asap. no need to wait for any other patches. the timing is orthogonal to 1.1. If really really needed / wanted could always do a 1.0.5.
The only real requirements are passing CI & ability to build wheels (which i think are satisfied).
jreback
on 25 May 2020
I think both #32479 and #33644 are ready to be merged. These can then be backported. I think backporting #33644 is important as incorrect results are being produced.
simonjayhawkins
on 25 May 2020
we can finish the second (first is already merged) but generally tiny edge cases even if a regression are not worth holding things up
jreback
on 25 May 2020
@simonjayhawkins remaining issues are merged.
jreback
on 25 May 2020
@simonjayhawkins remaining issues are merged.
backported.
TestToGBQIntegrationWithServiceAccountKeyPath failures on 1.0.x ci, otherwise green.
I noticed an extant lint error while backporting and included in #34369. github actions is not run on the 1.0.x branch. is this a blocker?
simonjayhawkins
on 26 May 2020
@TomAugspurger
I probably won't be able to help with one this week, but the scripts at https://github.com/pandas-dev/pandas-release should be reasonably up to date.
I'm using a Windows laptop and so cannot run the scripts out-of-the-box.
Have been making progress over the last couple of days and can probably start the release process.
I'm not in a position to improve the release scripts to work with Windows at this time since I'm on a slow metered mobile connection and this makes doing things from scratch to test things impracticable. i.e. building the pandas-docs container takes over 3 hours.
So, it's a question of risk. I am quite happy to start the release process or otherwise let someone else do it.
The main gotchas seem to be dealing with line-endings when sharing volumes for the docker containers and not being able to share volumes from git bash (to be able to use make)
simonjayhawkins
on 28 May 2020
Mmm OK. It's too bad the scripts don't work out of the box :/
I'll try to work through things off and on over the next day or two. Anything else need to be backported?
TomAugspurger
on 28 May 2020
Anything else need to be backported?
ac44c5de19c018a06d5345e800235bcb645cc842 hopefully should be the last commit before tag.
simonjayhawkins
on 28 May 2020
@TomAugspurger
from the release docs
Now manually create a release https://github.com/pandas-dev/pandas/releases
Make sure to upload the sdist that's in pandas/dist/ as the "binary". Conda-forge uses it.
for previous releases there are three assets. are the other two created automatically?
simonjayhawkins
on 28 May 2020
Yeah, github makes those.
TomAugspurger
on 28 May 2020
@TomAugspurger
scripts\conda-forge.sh is failing with
Traceback (most recent call last):
File "C:\Users\simon\Anaconda3\envs\pandas-release\Scripts\conda-smithy-script.py", line 9, in <module>
sys.exit(main())
File "C:\Users\simon\Anaconda3\envs\pandas-release\lib\site-packages\conda_smithy\cli.py", line 584, in main
args.subcommand_func(args)
File "C:\Users\simon\Anaconda3\envs\pandas-release\lib\site-packages\conda_smithy\cli.py", line 391, in __call__
self._call(args, tmpdir)
File "C:\Users\simon\Anaconda3\envs\pandas-release\lib\site-packages\conda_smithy\cli.py", line 402, in _call
temporary_directory=temporary_directory,
File "C:\Users\simon\Anaconda3\envs\pandas-release\lib\site-packages\conda_smithy\configure_feedstock.py", line 1893, in main
conda_build_version,
File "C:\Users\simon\Anaconda3\envs\pandas-release\lib\site-packages\conda_smithy\configure_feedstock.py", line 1611, in commit_changes
subprocess.check_call(git_args, cwd=forge_file_directory)
File "C:\Users\simon\Anaconda3\envs\pandas-release\lib\subprocess.py", line 363, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['git', 'commit', '-m', 'MNT: Re-rendered with conda-build 3.19.2, conda-smithy 3.7.3, and conda-forge-pinning 2020.05.27.14.19.32', '--edit', '--status', '--verbose']' returned non-zero exit status 1.
make: *** [Makefile:136: conda-forge] Error 1
i've had a few git related issues so far.
I think it's because it's in the conda environment
if I remove it, conda-forge.sh reinstalls it.
will try the wheels next and i'll come back to this. either maybe you can do the conda stuff or I could try building a docker container to do this step.
simonjayhawkins
on 28 May 2020
maybe worked on re-run https://github.com/conda-forge/pandas-feedstock/pull/81
simonjayhawkins
on 28 May 2020
Thanks. I think I've also had issues with git in the conda env.
I'll start on the doc build now.
TomAugspurger
on 28 May 2020
I'll start on the doc build now.
cool. just got rsync installed for git bash. would have needed ssh access to continue but calling it a day. will carry-on tomorrow with what's left to do.
simonjayhawkins
on 28 May 2020
Website and docs are uploaded. Let me know if you see any strange formatting issues. The layout was messed up the first time I visited a docs page, but seems to be OK now.
Wheels are on PyPI and conda-forge packages are building.
@simonjayhawkins do you want to send an email to the mailing lists with the announcement?
TomAugspurger
on 29 May 2020
I still see the messed up layout for the symlinked ones, like https://pandas.pydata.org/docs/ (https://pandas.pydata.org/pandas-docs/version/1.0.4/ is fine).
jorisvandenbossche
on 29 May 2020
Hmm, I have to run for a bit so I'll revert the symlink for now.
TomAugspurger
on 29 May 2020
@simonjayhawkins do you want to send an email to the mailing lists with the announcement?
happy to do this when we're happy all is OK
What's new in 1.0.4 on https://pandas.pydata.org/ points to https://pandas.pydata.org/docs/whatsnew/v1.0.0.html
Is this something that I need to change?
simonjayhawkins
on 29 May 2020
No, it’s been fixed on master so I’m not worried about that.
TomAugspurger
on 29 May 2020
FYI, I'm debugging this at https://pandas.pydata.org/pandas-docs/debug/. It seems to only occur when the server has served a 1.0.3 request at that URL. So almost surely a CSS asset is being cached somewhere.
TomAugspurger
on 29 May 2020
https://pandas.pydata.org/pandas-docs/debug/ now looks fine for me. Was that url supposed to be a "broken" one? (in which case it might be solved now)
jorisvandenbossche
on 30 May 2020
Not sure if anyone has noticed but to @simonjayhawkins point clicking on the 1.0.4 whatsnew from the home page takes me to 1.0.0 with no option to see 1.0.4. It also mentions 1.0.1 in the top menu:
WillAyd
on 1 Jun 2020
That's because the link seems to be hardcoded in the home page (while the text is automatically updated):
So that needs a fix there.
(I also seem to remember some discussion about whether we found linking to 1.0.0 instead of 1.0.x more relevant, but in any case now the text and the link is inconsistent)
jorisvandenbossche
on 1 Jun 2020
It's fixed on master: https://pandas.pydata.org/pandas-docs/dev/
TomAugspurger
on 1 Jun 2020
I don't know what to do about the CSS issue. I set up https://pandas.pydata.org/pandas-docs/debug/ on Friday to replicate /docs and it did: I saw the messed up formatting for a while. Visiting now shows that it looks fine, so whatever was caching the CSS has since expired.
TomAugspurger
on 1 Jun 2020
@TomAugspurger that's the docs (where we indeed removed the "Whatsnew" link in the top bar), but AFAIU it is about a wrong link on the home page (/web/)
jorisvandenbossche
on 1 Jun 2020
Ah, thanks. Should that just link to the whatsnew index?
On Mon, Jun 1, 2020 at 6:10 AM Joris Van den Bossche <
[email protected]> wrote:
@TomAugspurger https://github.com/TomAugspurger that's the docs (where
we indeed removed the "Whatsnew" link in the top bar), but AFAIU it is
about a wrong link on the home page (/web/)—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pandas-dev/pandas/issues/33300#issuecomment-636791563,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAKAOITYNVIWYMGP5OXZTDTRUOEEDANCNFSM4MATQACQ
.
TomAugspurger
on 1 Jun 2020
@aterrel I think this may be cached at the cloudflare CDN sitting between pandas' webserver and the internet.
$ http https://pandas.pydata.org/pandas-docs/debug/_static/css/index.css
HTTP/1.1 200 OK
Age: 123
CF-Cache-Status: HIT
CF-RAY: 59ca0e5778575c43-STL
Cache-Control: max-age=14400
Connection: keep-alive
Content-Encoding: gzip
Content-Type: text/css
Date: Mon, 01 Jun 2020 15:44:26 GMT
ETag: W/"5e724acc-156c"
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Last-Modified: Wed, 18 Mar 2020 16:22:36 GMT
Server: cloudflare
Set-Cookie: __cfduid=d1bde881be99bc9d3922b6701123103c61591026266; expires=Wed, 01-Jul-20 15:44:26 GMT; path=/; domain=.pydata.org; HttpOnly; SameSite=Lax
Transfer-Encoding: chunked
Vary: Accept-Encoding
cf-request-id: 0312274aa800005c43bab24200000001
...
Is giving me access to cloudflare so that I can purge some caches an option? https://support.cloudflare.com/hc/en-us/articles/200169246-Purging-cached-resources-from-Cloudflare.
TomAugspurger
on 1 Jun 2020
Actually, hold on that. I might be able to instruct nginx to not use CF's cache. Won't have time to look into that until later though.
TomAugspurger
on 1 Jun 2020
My idea was to add a cache-control to nginx to tell cloudflare to not cache anything, but I don't want to accidentally take down our webserver if the load is too high. I don't have a good idea.
@aterrel I think getting access to Cloudflare to manually purge the cache is best. Can you grant me access or can we sync up sometime for you to do it?
TomAugspurger
on 3 Jun 2020
@TomAugspurger you didn't find out how to change the /doc symlink? (to link directly to the correct version, instead of through stable. Not sure if this would fix it of course)
jorisvandenbossche
on 3 Jun 2020
@TomAugspurger yes, create a cloudflare account and send me the email address you used.
aterrel
on 4 Jun 2020
For those watching along, invite has been sent. Let me know if I can be of any other service.
aterrel
on 4 Jun 2020
Thanks Andy.
Possibly fixed with this purge list through the cloudflare console.
https://pandas.pydata.org/docs/_static/css/getting_started.css
https://pandas.pydata.org/docs/_static/css/index.css
https://pandas.pydata.org/docs/_static/css/pandas.css
https://pandas.pydata.org/pandas-docs/stable/_static/css/getting_started.css
https://pandas.pydata.org/pandas-docs/stable/_static/css/index.css
https://pandas.pydata.org/pandas-docs/stable/_static/css/pandas.css
https://pandas.pydata.org/docs/_static/js/index.js
https://pandas.pydata.org/pandas-docs/stable/_static/js/index.js
@jorisvandenbossche anything else you can think of that should be purged?
TomAugspurger
on 4 Jun 2020
That should be all pandas doc specific js/css files, I think
jorisvandenbossche
on 4 Jun 2020
Thanks.
And a huge thanks to @simonjayhawkins for managing the release!
TomAugspurger
on 4 Jun 2020
Related issues
quicknir
·
58Comments
jorisvandenbossche
·
50Comments
ShaharNaveh
·
137Comments
jreback
·
61Comments
dragoljub
·
56Comments
Most helpful comment
there are many areas you could help, why are you not volunteering there?
I do understand severity of issues very well and am sympathetic but w/o any way to actually direct resources, your comments are not meaningful.
Its like, let’s not pay for anything, not contribute in-kind and yet have the audacity to demand something.