According to this article, seems that build process have some issues in Alpine Python image
https://pythonspeed.com/articles/alpine-docker-python/

Standard PyPI wheels don鈥檛 work on Alpine
Alpine Linux can cause unexpected runtime bugs
Alpine uses a different libc: musl, compared to glibc. And the .whl format is not entirely compatible with musl. And some packages for pip are only offered in .whl format
https://github.com/docker-library/docs/issues/904
https://github.com/pypa/manylinux/issues/37
https://github.com/pypa/pip/issues/3969
https://github.com/pypa/manylinux
The manylinux1 tags allow projects to distribute wheels that are automatically installed (and work!) on the vast majority of desktop and server Linux distributions.
In the docs it's mentioned that Alpine may behave differently from other Linux variants https://github.com/docker-library/docs/tree/master/python#pythonversion-alpine
The size difference is also improved by https://github.com/docker-library/python/pull/508:
$ docker pull python:3.8
3.8: Pulling from library/python
Digest: sha256:2c1045587e4452d49544c6dce92efe21c3b4b33864cfb56fdee66a2c8585c769
Status: Image is up to date for python:3.8
docker.io/library/python:3.8
$ docker pull python:3.8-slim
3.8-slim: Pulling from library/python
Digest: sha256:a9e6f501d8330b2b00d87f3ff17cc0ef8c1c5031f8dfe428983d1c6fe7080338
Status: Image is up to date for python:3.8-slim
docker.io/library/python:3.8-slim
$ docker pull python:3.8-alpine
3.8-alpine: Pulling from library/python
Digest: sha256:bd7f0719874c7de4ca47418de143ab4de79731dc0b01fdcf2425b8a32f4823ce
Status: Image is up to date for python:3.8-alpine
docker.io/library/python:3.8-alpine
$ docker images python:3.8
REPOSITORY TAG IMAGE ID CREATED SIZE
python 3.8 79cc46abd78d 7 days ago 882MB
$ docker images python:3.8-slim
REPOSITORY TAG IMAGE ID CREATED SIZE
python 3.8-slim 38cd21c9e1a8 7 days ago 113MB
$ docker images python:3.8-alpine
REPOSITORY TAG IMAGE ID CREATED SIZE
python 3.8-alpine 44fceb565b2a 7 days ago 42.7MB