Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pandas/pyArrow/read_parquet error #17

Open
phwuil opened this issue Jul 21, 2023 · 8 comments
Open

Pandas/pyArrow/read_parquet error #17

phwuil opened this issue Jul 21, 2023 · 8 comments

Comments

@phwuil
Copy link

phwuil commented Jul 21, 2023

  • condastats version: 0.2.1
  • Python version: Python 3.11.3
  • Operating System: linux (Manjaro/Plasma)

Description

Unable to use condastats.cli.overall (internal error on pandas->pyArrow)

    dataconda = condastats.cli.overall([conda_module], monthly=True)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[...]/lib/python3.11/site-packages/condastats/cli.py", line 62, in overall
    df = dd.read_parquet(
         ^^^^^^^^^^^^^^^^
  File "[...]/python3.11/site-packages/dask/backends.py", line 138, in wrapper
    raise type(e)(
ValueError: An error occurred while calling the read_parquet method registered to the pandas backend.
Original Message: ArrowStringArray requires a PyArrow (chunked) array of string type
@ragonneau
Copy link

Hi @sophiamyang,

I am facing the same issue here.

Cheers,
Tom.

@ragonneau
Copy link

Hi @sophiamyang,

The error seems to be persistent. I now get the following:

Traceback (most recent call last):
  File "/home/tmragon/PycharmProjects/pdfo-stats/venv/lib/python3.10/site-packages/dask/backends.py", line 136, in wrapper
    return func(*args, **kwargs)
  File "/home/tmragon/PycharmProjects/pdfo-stats/venv/lib/python3.10/site-packages/dask/dataframe/io/parquet/core.py", line 543, in read_parquet
    read_metadata_result = engine.read_metadata(
  File "/home/tmragon/PycharmProjects/pdfo-stats/venv/lib/python3.10/site-packages/dask/dataframe/io/parquet/arrow.py", line 549, in read_metadata
    meta = cls._create_dd_meta(dataset_info)
  File "/home/tmragon/PycharmProjects/pdfo-stats/venv/lib/python3.10/site-packages/dask/dataframe/io/parquet/arrow.py", line 1266, in _create_dd_meta
    meta = clear_known_categories(
  File "/home/tmragon/PycharmProjects/pdfo-stats/venv/lib/python3.10/site-packages/dask/dataframe/utils.py", line 293, in clear_known_categories
    raise ValueError("Not all columns are categoricals")
ValueError: Not all columns are categoricals

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/tmragon/PycharmProjects/pdfo-stats/venv/lib/python3.10/site-packages/condastats/cli.py", line 62, in overall
    df = dd.read_parquet(
  File "/home/tmragon/PycharmProjects/pdfo-stats/venv/lib/python3.10/site-packages/dask/backends.py", line 138, in wrapper
    raise type(e)(
ValueError: An error occurred while calling the read_parquet method registered to the pandas backend.
Original Message: Not all columns are categoricals

Is there anything we can do?

Cheers,
Tom.

@phwuil
Copy link
Author

phwuil commented Oct 2, 2023

Hi,

Does anyone know of a workaround to this bug ? Or an alternative to condastats ?

Cheers,
PH.

@sophiamyang
Copy link
Contributor

Thanks for the issue. Could you open the issue in this repo instead? condastats depends on the data pipeline in this repo: https://github.com/ContinuumIO/anaconda-package-data cc @cappadona

@phwuil
Copy link
Author

phwuil commented Oct 2, 2023

Hi @sophiamyang and thanks a lot for this package and this answer.

But, as I only experiment this error when using condastats, I don't see how to reproducibly describe the bug in this repo ...

@nicrie
Copy link

nicrie commented Oct 13, 2023

Hey Pierre-Henri, I respond to your message posted in #18 here to keep the threads separated and focused.

Quick inspection of my environment (see below) indicates that the main differences are in the pandas and pyarrow version. Specifically, I have pyarrow version 12.0.1 and pandas 1.5.3. You could try downgrading to see if it solves your problem.

Environment
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
aiobotocore               2.5.0              pyhd8ed1ab_0    conda-forge
aiohttp                   3.8.5           py311h459d7ec_0    conda-forge
aioitertools              0.11.0             pyhd8ed1ab_0    conda-forge
aiosignal                 1.3.1              pyhd8ed1ab_0    conda-forge
alsa-lib                  1.2.9                hd590300_0    conda-forge
anyio                     3.7.1              pyhd8ed1ab_0    conda-forge
asttokens                 2.2.1              pyhd8ed1ab_0    conda-forge
async-timeout             4.0.2              pyhd8ed1ab_0    conda-forge
attr                      2.5.1                h166bdaf_1    conda-forge
attrs                     23.1.0             pyh71513ae_1    conda-forge
aws-c-auth                0.7.0                hf8751d9_2    conda-forge
aws-c-cal                 0.6.0                h93469e0_0    conda-forge
aws-c-common              0.8.23               hd590300_0    conda-forge
aws-c-compression         0.2.17               h862ab75_1    conda-forge
aws-c-event-stream        0.3.1                h9599702_1    conda-forge
aws-c-http                0.7.11               hbe98c3e_0    conda-forge
aws-c-io                  0.13.28              h3870b5a_0    conda-forge
aws-c-mqtt                0.8.14               h2e270ba_2    conda-forge
aws-c-s3                  0.3.13               heb0bb06_2    conda-forge
aws-c-sdkutils            0.1.11               h862ab75_1    conda-forge
aws-checksums             0.1.16               h862ab75_1    conda-forge
aws-crt-cpp               0.20.3               he9c0e7f_4    conda-forge
aws-sdk-cpp               1.10.57             hbc2ea52_17    conda-forge
backcall                  0.2.0              pyh9f0ad1d_0    conda-forge
backports                 1.0                pyhd8ed1ab_3    conda-forge
backports.functools_lru_cache 1.6.5              pyhd8ed1ab_0    conda-forge
bokeh                     3.2.1              pyhd8ed1ab_0    conda-forge
botocore                  1.29.76            pyhd8ed1ab_0    conda-forge
brotli                    1.0.9                h166bdaf_9    conda-forge
brotli-bin                1.0.9                h166bdaf_9    conda-forge
brotlipy                  0.7.0           py311hd4cff14_1005    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
c-ares                    1.19.1               hd590300_0    conda-forge
ca-certificates           2023.7.22            hbcca054_0    conda-forge
cairo                     1.16.0            hbbf8b49_1016    conda-forge
certifi                   2023.7.22          pyhd8ed1ab_0    conda-forge
cffi                      1.15.1          py311h409f033_3    conda-forge
chardet                   5.1.0           py311h38be061_0    conda-forge
charset-normalizer        3.2.0              pyhd8ed1ab_0    conda-forge
click                     8.1.6           unix_pyh707e725_0    conda-forge
cloudpickle               2.2.1              pyhd8ed1ab_0    conda-forge
comm                      0.1.4              pyhd8ed1ab_0    conda-forge
condastats                0.2.1              pyhd8ed1ab_0    conda-forge
contourpy                 1.1.0           py311h9547e67_0    conda-forge
cryptography              41.0.2          py311h63ff55d_0    conda-forge
cycler                    0.11.0             pyhd8ed1ab_0    conda-forge
cytoolz                   0.12.2          py311h459d7ec_0    conda-forge
dask                      2023.7.1           pyhd8ed1ab_0    conda-forge
dask-core                 2023.7.1           pyhd8ed1ab_0    conda-forge
dataproperty              1.0.1              pyhd8ed1ab_0    conda-forge
dbus                      1.13.6               h5008d03_3    conda-forge
debugpy                   1.6.8           py311hb755f60_0    conda-forge
decorator                 5.1.1              pyhd8ed1ab_0    conda-forge
distributed               2023.7.1           pyhd8ed1ab_0    conda-forge
dominate                  2.8.0              pyhd8ed1ab_0    conda-forge
exceptiongroup            1.1.2              pyhd8ed1ab_0    conda-forge
executing                 1.2.0              pyhd8ed1ab_0    conda-forge
expat                     2.5.0                hcb278e6_1    conda-forge
font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
font-ttf-inconsolata      3.000                h77eed37_0    conda-forge
font-ttf-source-code-pro  2.038                h77eed37_0    conda-forge
font-ttf-ubuntu           0.83                 hab24e00_0    conda-forge
fontconfig                2.14.2               h14ed4e7_0    conda-forge
fonts-conda-ecosystem     1                             0    conda-forge
fonts-conda-forge         1                             0    conda-forge
fonttools                 4.41.1          py311h459d7ec_0    conda-forge
freetype                  2.12.1               hca18f0e_1    conda-forge
frozenlist                1.4.0           py311h459d7ec_0    conda-forge
fsspec                    2023.6.0           pyh1a96a4e_0    conda-forge
gettext                   0.21.1               h27087fc_0    conda-forge
gflags                    2.2.2             he1b5a44_1004    conda-forge
glib                      2.76.4               hfc55251_0    conda-forge
glib-tools                2.76.4               hfc55251_0    conda-forge
glog                      0.6.0                h6f12383_0    conda-forge
graphite2                 1.3.13            h58526e2_1001    conda-forge
gst-plugins-base          1.22.5               hf7dbed1_0    conda-forge
gstreamer                 1.22.5               h98fc4e7_0    conda-forge
h11                       0.14.0             pyhd8ed1ab_0    conda-forge
h2                        4.1.0              pyhd8ed1ab_0    conda-forge
harfbuzz                  7.3.0                hdb3a94d_0    conda-forge
hpack                     4.0.0              pyh9f0ad1d_0    conda-forge
httpcore                  0.17.3             pyhd8ed1ab_0    conda-forge
httpx                     0.24.1             pyhd8ed1ab_0    conda-forge
hyperframe                6.0.1              pyhd8ed1ab_0    conda-forge
icu                       72.1                 hcb278e6_0    conda-forge
idna                      3.4                pyhd8ed1ab_0    conda-forge
importlib-metadata        6.8.0              pyha770c72_0    conda-forge
importlib_metadata        6.8.0                hd8ed1ab_0    conda-forge
ipykernel                 6.25.0             pyh71e2992_0    conda-forge
ipython                   8.14.0             pyh41d4057_0    conda-forge
jedi                      0.19.0             pyhd8ed1ab_0    conda-forge
jinja2                    3.1.2              pyhd8ed1ab_1    conda-forge
jmespath                  1.0.1              pyhd8ed1ab_0    conda-forge
jupyter_client            8.3.0              pyhd8ed1ab_0    conda-forge
jupyter_core              5.3.1           py311h38be061_0    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
kiwisolver                1.4.4           py311h4dd048b_1    conda-forge
krb5                      1.21.1               h659d440_0    conda-forge
lame                      3.100             h166bdaf_1003    conda-forge
lcms2                     2.15                 haa2dc70_1    conda-forge
ld_impl_linux-64          2.40                 h41732ed_0    conda-forge
lerc                      4.0.0                h27087fc_0    conda-forge
libabseil                 20230125.3      cxx17_h59595ed_0    conda-forge
libarrow                  12.0.1           h657c46f_7_cpu    conda-forge
libblas                   3.9.0           17_linux64_openblas    conda-forge
libbrotlicommon           1.0.9                h166bdaf_9    conda-forge
libbrotlidec              1.0.9                h166bdaf_9    conda-forge
libbrotlienc              1.0.9                h166bdaf_9    conda-forge
libcap                    2.69                 h0f662aa_0    conda-forge
libcblas                  3.9.0           17_linux64_openblas    conda-forge
libclang                  15.0.7          default_h7634d5b_3    conda-forge
libclang13                15.0.7          default_h9986a30_3    conda-forge
libcrc32c                 1.1.2                h9c3ff4c_0    conda-forge
libcups                   2.3.3                h4637d8d_4    conda-forge
libcurl                   8.2.1                hca28451_0    conda-forge
libdeflate                1.18                 h0b41bf4_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 h516909a_1    conda-forge
libevent                  2.1.12               hf998b51_1    conda-forge
libexpat                  2.5.0                hcb278e6_1    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libflac                   1.4.3                h59595ed_0    conda-forge
libgcc-ng                 13.1.0               he5830b7_0    conda-forge
libgcrypt                 1.10.1               h166bdaf_0    conda-forge
libgfortran-ng            13.1.0               h69a702a_0    conda-forge
libgfortran5              13.1.0               h15d22d2_0    conda-forge
libglib                   2.76.4               hebfc3b9_0    conda-forge
libgomp                   13.1.0               he5830b7_0    conda-forge
libgoogle-cloud           2.12.0               h840a212_1    conda-forge
libgpg-error              1.47                 h71f35ed_0    conda-forge
libgrpc                   1.56.2               h3905398_0    conda-forge
libiconv                  1.17                 h166bdaf_0    conda-forge
libjpeg-turbo             2.1.5.1              h0b41bf4_0    conda-forge
liblapack                 3.9.0           17_linux64_openblas    conda-forge
libllvm15                 15.0.7               h5cf9203_3    conda-forge
libnghttp2                1.52.0               h61bc06f_0    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libnuma                   2.0.16               h0b41bf4_1    conda-forge
libogg                    1.3.4                h7f98852_1    conda-forge
libopenblas               0.3.23          pthreads_h80387f5_0    conda-forge
libopus                   1.3.1                h7f98852_1    conda-forge
libpng                    1.6.39               h753d276_0    conda-forge
libpq                     15.3                 hfc447b1_2    conda-forge
libprotobuf               4.23.3               hd1fb520_0    conda-forge
libsndfile                1.2.0                hb75c966_0    conda-forge
libsodium                 1.0.18               h36c2ea0_1    conda-forge
libsqlite                 3.42.0               h2797004_0    conda-forge
libssh2                   1.11.0               h0841786_0    conda-forge
libstdcxx-ng              13.1.0               hfd8a6a1_0    conda-forge
libsystemd0               254                  h3516f8a_0    conda-forge
libthrift                 0.18.1               h8fd135c_2    conda-forge
libtiff                   4.5.1                h8b53f26_0    conda-forge
libutf8proc               2.8.0                h166bdaf_0    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libvorbis                 1.3.7                h9c3ff4c_0    conda-forge
libwebp-base              1.3.1                hd590300_0    conda-forge
libxcb                    1.15                 h0b41bf4_0    conda-forge
libxkbcommon              1.5.0                h5d7e998_3    conda-forge
libxml2                   2.11.4               h0d562d8_0    conda-forge
libzlib                   1.2.13               hd590300_5    conda-forge
locket                    1.0.0              pyhd8ed1ab_0    conda-forge
lz4                       4.3.2           py311h9f220a4_0    conda-forge
lz4-c                     1.9.4                hcb278e6_0    conda-forge
markupsafe                2.1.3           py311h459d7ec_0    conda-forge
matplotlib                3.7.2           py311h38be061_0    conda-forge
matplotlib-base           3.7.2           py311h54ef318_0    conda-forge
matplotlib-inline         0.1.6              pyhd8ed1ab_0    conda-forge
matplotx                  0.3.10             pyhd8ed1ab_0    conda-forge
mbstrdecoder              1.1.3              pyhd8ed1ab_1    conda-forge
mpg123                    1.31.3               hcb278e6_0    conda-forge
msgpack-python            1.0.5           py311ha3edf6b_0    conda-forge
multidict                 6.0.4           py311h2582759_0    conda-forge
munkres                   1.1.4              pyh9f0ad1d_0    conda-forge
mysql-common              8.0.33               hf1915f5_2    conda-forge
mysql-libs                8.0.33               hca2cd23_2    conda-forge
ncurses                   6.4                  hcb278e6_0    conda-forge
nest-asyncio              1.5.6              pyhd8ed1ab_0    conda-forge
networkx                  3.1                pyhd8ed1ab_0    conda-forge
nspr                      4.35                 h27087fc_0    conda-forge
nss                       3.89                 he45b914_0    conda-forge
numpy                     1.25.1          py311h64a7726_0    conda-forge
openjpeg                  2.5.0                hfec8fc6_2    conda-forge
openssl                   3.1.2                hd590300_0    conda-forge
orc                       1.9.0                h385abfd_1    conda-forge
packaging                 23.1               pyhd8ed1ab_0    conda-forge
pandas                    1.5.3           py311h2872171_1    conda-forge
parso                     0.8.3              pyhd8ed1ab_0    conda-forge
partd                     1.4.0              pyhd8ed1ab_0    conda-forge
pathvalidate              3.1.0              pyhd8ed1ab_0    conda-forge
patsy                     0.5.3              pyhd8ed1ab_0    conda-forge
pcre2                     10.40                hc3806b6_0    conda-forge
pexpect                   4.8.0              pyh1a96a4e_2    conda-forge
pickleshare               0.7.5                   py_1003    conda-forge
pillow                    10.0.0          py311h0b84326_0    conda-forge
pip                       23.2.1             pyhd8ed1ab_0    conda-forge
pixman                    0.40.0               h36c2ea0_0    conda-forge
platformdirs              3.10.0             pyhd8ed1ab_0    conda-forge
ply                       3.11                       py_1    conda-forge
pooch                     1.7.0              pyha770c72_3    conda-forge
prettytable               3.8.0              pyhd8ed1ab_0    conda-forge
prompt-toolkit            3.0.39             pyha770c72_0    conda-forge
prompt_toolkit            3.0.39               hd8ed1ab_0    conda-forge
psutil                    5.9.5           py311h2582759_0    conda-forge
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
pulseaudio-client         16.1                 hb77b528_4    conda-forge
pure_eval                 0.2.2              pyhd8ed1ab_0    conda-forge
pyarrow                   12.0.1          py311h39c9aba_7_cpu    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pygments                  2.15.1             pyhd8ed1ab_0    conda-forge
pyopenssl                 23.2.0             pyhd8ed1ab_1    conda-forge
pyparsing                 3.0.9              pyhd8ed1ab_0    conda-forge
pypistats                 1.4.0              pyhd8ed1ab_0    conda-forge
pyqt                      5.15.9          py311hf0fb5b6_4    conda-forge
pyqt5-sip                 12.12.2         py311hb755f60_4    conda-forge
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
pytablewriter             1.0.0              pyhd8ed1ab_0    conda-forge
python                    3.11.4          hab00c5b_0_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python-slugify            8.0.1              pyhd8ed1ab_0    conda-forge
python-snappy             0.6.1           py311h48ad69a_0    conda-forge
python-tzdata             2023.3             pyhd8ed1ab_0    conda-forge
python_abi                3.11                    3_cp311    conda-forge
pytz                      2023.3             pyhd8ed1ab_0    conda-forge
pyyaml                    6.0             py311hd4cff14_5    conda-forge
pyzmq                     25.1.0          py311h75c88c4_0    conda-forge
qt-main                   5.15.8              h7fe3ca9_15    conda-forge
rdma-core                 28.9                 h59595ed_1    conda-forge
re2                       2023.03.02           h8c504da_0    conda-forge
readline                  8.2                  h8228510_1    conda-forge
requests                  2.31.0             pyhd8ed1ab_0    conda-forge
s2n                       1.3.46               h06160fa_0    conda-forge
s3fs                      2023.6.0           pyhd8ed1ab_0    conda-forge
scipy                     1.11.1          py311h64a7726_0    conda-forge
seaborn                   0.12.2               hd8ed1ab_0    conda-forge
seaborn-base              0.12.2             pyhd8ed1ab_0    conda-forge
setuptools                68.0.0             pyhd8ed1ab_0    conda-forge
sip                       6.7.10          py311hb755f60_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
snappy                    1.1.10               h9fff704_0    conda-forge
sniffio                   1.3.0              pyhd8ed1ab_0    conda-forge
sortedcontainers          2.4.0              pyhd8ed1ab_0    conda-forge
stack_data                0.6.2              pyhd8ed1ab_0    conda-forge
statsmodels               0.14.0          py311h1f0f07a_1    conda-forge
tabledata                 1.3.1              pyhd8ed1ab_0    conda-forge
tblib                     1.7.0              pyhd8ed1ab_0    conda-forge
tcolorpy                  0.1.3              pyhd8ed1ab_0    conda-forge
termcolor                 2.3.0              pyhd8ed1ab_0    conda-forge
text-unidecode            1.3                        py_0    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
toml                      0.10.2             pyhd8ed1ab_0    conda-forge
tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
toolz                     0.12.0             pyhd8ed1ab_0    conda-forge
tornado                   6.3.2           py311h459d7ec_0    conda-forge
traitlets                 5.9.0              pyhd8ed1ab_0    conda-forge
typepy                    1.3.1              pyhd8ed1ab_0    conda-forge
typing-extensions         4.7.1                hd8ed1ab_0    conda-forge
typing_extensions         4.7.1              pyha770c72_0    conda-forge
tzdata                    2023c                h71feb2d_0    conda-forge
ucx                       1.14.1               hf587318_2    conda-forge
unidecode                 1.3.6              pyhd8ed1ab_0    conda-forge
urllib3                   1.26.15            pyhd8ed1ab_0    conda-forge
wcwidth                   0.2.6              pyhd8ed1ab_0    conda-forge
wheel                     0.41.0             pyhd8ed1ab_0    conda-forge
wrapt                     1.15.0          py311h2582759_0    conda-forge
xcb-util                  0.4.0                hd590300_1    conda-forge
xcb-util-image            0.4.0                h8ee46fc_1    conda-forge
xcb-util-keysyms          0.4.0                h8ee46fc_1    conda-forge
xcb-util-renderutil       0.3.9                hd590300_1    conda-forge
xcb-util-wm               0.4.1                h8ee46fc_1    conda-forge
xkeyboard-config          2.39                 hd590300_0    conda-forge
xorg-kbproto              1.0.7             h7f98852_1002    conda-forge
xorg-libice               1.1.1                hd590300_0    conda-forge
xorg-libsm                1.2.4                h7391055_0    conda-forge
xorg-libx11               1.8.6                h8ee46fc_0    conda-forge
xorg-libxau               1.0.11               hd590300_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xorg-libxext              1.3.4                h0b41bf4_2    conda-forge
xorg-libxrender           0.9.11               hd590300_0    conda-forge
xorg-renderproto          0.11.1            h7f98852_1002    conda-forge
xorg-xextproto            7.3.0             h0b41bf4_1003    conda-forge
xorg-xf86vidmodeproto     2.3.1             h7f98852_1002    conda-forge
xorg-xproto               7.0.31            h7f98852_1007    conda-forge
xyzservices               2023.7.0           pyhd8ed1ab_0    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
yaml                      0.2.5                h7f98852_2    conda-forge
yarl                      1.9.2           py311h459d7ec_0    conda-forge
zeromq                    4.3.4                h9c3ff4c_1    conda-forge
zict                      3.0.0              pyhd8ed1ab_0    conda-forge
zipp                      3.16.2             pyhd8ed1ab_0    conda-forge
zlib                      1.2.13               hd590300_5    conda-forge
zstd                      1.5.2                hfc55251_7    conda-forge

@phwuil
Copy link
Author

phwuil commented Oct 13, 2023

ok, that's it !! pandas<2.0.0 is required !

Thanks @nicrie

@phwuil phwuil closed this as completed Oct 17, 2023
@phwuil
Copy link
Author

phwuil commented Sep 22, 2024

Hi @sophiamyang and @nicrie,

The requirement pandas<2.0.0 no longer works: other libraries (such as dask-expr) now require pandas>2.0.0 ... :-(

Any comments or ideas ? Thanks !

shell>> # python 3.12.5 with brand new virtual env

shell>>  pip install condastat
shell>>  condastat overall numpy --monthly
      Traceback (most recent call last):
      [...]
      ValueError: An error occurred while calling the read_parquet method registered to the pandas backend.

shell>>  pip install "pandas<2.0.0"
      Installing collected packages: pandas
        Attempting uninstall: pandas
          Found existing installation: pandas 2.2.3
          Uninstalling pandas-2.2.3:
            Successfully uninstalled pandas-2.2.3
      ERROR: pip dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
      dask-expr 1.1.14 requires pandas>=2, but you have pandas 1.5.3 which is incompatible.
      Successfully installed pandas-1.5.3

shell>>  condastat overall  numpy --monthly
      Traceback (most recent call last):
      [...]
      ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject

@phwuil phwuil reopened this Sep 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants