close out protobuf 4.23.3 migration; drop protobuf 3.21

[h-vetinari]

This has been going on for a while now, it's almost 3 months since #4482.

The remaining feedstocks are

• TODO:
  • or-tools: first compatible version just released¹, which does not support 3.x anymore, so it effectively assumes this PR already.
  • pytorch: moves at its own speed (due needing to CFEP-3)
  • tensorflow: moves at its own speed (due needing to CFEP-3)
• Blocked:
  • mysql-connector-python: pins libprotobuf 3.11! (xref Unpin protobuf mysql-connector-python-feedstock#80)
• Essentially dead feedstocks:
  • bear
  • go-cockroach
  • tomviz

Given that we have protobuf 4.23.4 and 4.24.0 ready to migrate (plan), I think we should move on and drop 3.x.

CC @conda-forge/core

Footnotes

1. I'm working on some ABI issues but solvable ↩

[conda-forge-webservices]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

[h-vetinari]

I just noticed that closing this migration effectively means requiring MacOS >=10.13, because a protobuf 4.x-compatible grpc requires C11's aligned_alloc at runtime.

Just collecting dependencies of libgrpc, grpcio and one of its main dependents (libarrow and pyarrow), we get:

mamba repoquery whoneeds libgrpc -c conda-forge -p linux-64 > proto4.txt
mamba repoquery whoneeds grpcio -c conda-forge -p linux-64 >> proto4.txt
mamba repoquery whoneeds libarrow -c conda-forge -p linux-64 >> proto4.txt
mamba repoquery whoneeds pyarrow -c conda-forge -p linux-64 >> proto4.txt
# light editing to remove headers
python
>>> q = open("proto4.txt", "r").readlines()
>>> p = {x.strip().split(" ")[0] for x in q} - {"", "grpc-cpp", "arrow-cpp"}  # remove aliases
>>> len(p)
277
>>> print("\n".join(sorted(list(p))))

This results in ~280 affected packages, and that's not even counting transitive dependencies (much less the fact that pandas is planning to depend on pyarrow). So I'm afraid this is now coupled with the MacOS discussion. 😑

List of affected packages (first-order dependent on grpc/arrow)

_dvc-hdfs
activitysim
adaptive-scheduler
aim
airflow-provider-xlsx
airflow-with-grpc
airflow-with-pandas
airflow-with-papermill
alchemlyb
ansys-api-mapdl-v0
apache-airflow-providers-google
apache-airflow-providers-grpc
apache-airflow-providers-papermill
apache-beam
api2db
arcos-gui
argopandas
arize-phoenix
arrow-c-glib
arrow_odbc
autogluon.core
av2
awswrangler
bazel
bear
bentoml
bertopic
bigearthnet-common
bigearthnet-gdf-builder
biobear
brax
bufrtools
catalystcoop.pudl
census-parquet
clouddrift
coffea
coiled-runtime
compass-interface
compass-interface-pandas
condastats
cox
cpp-opentelemetry-sdk
dagster
dagster-duckdb-polars
dagster-duckdb-pyspark
dask
dask-bigquery
dask-deltatable
dask-expr
dask-ms
dask-snowflake
dask-yarn
databricks-sql-connector
datafusion
datapane
dataprofiler
datasets
db-dtypes
delta-sharing-python
deltalake
descarteslabs
dirty_cat
dremio-arrow
dsc
dvc
dvc-hdfs
dye-score
earthformer
eo-dataflow-report
etcdpy
feast
feather-format
featherhelper
featurewiz
felyx-processor
felyx-report
fletcher
flwr
flytekit
flytekitplugins-modin
foundry-dev-tools
fugue
gdptools
geosnap
gmpyinfr_dbutils
google-ads
google-api-core-grpc
google-cloud-bigquery
google-cloud-bigquery-core
google-cloud-bigquery-storage
google-cloud-bigquery-storage-core
google-cloud-container
google-cloud-core
google-cloud-cpp
google-cloud-memcache
google-cloud-pubsub
google-cloud-pubsublite
google-cloud-speech
google-cloud-workflows
googleapis-common-protos-grpc
graphistry
graphysio
grpc-google-iam-v1
grpc-stubs
grpcio
grpcio-gcp
grpcio-health-checking
grpcio-reflection
grpcio-status
grpcio-tools
hangar
heavyai
hydrofunctions
hypernets
ibis-bigquery
ibis-dask
ibis-framework
ibis-framework-core
ibis-heavyai
ibis-omniscidb
ibis-parquet
ibis-postgres
ibis-pyspark
ibis-trino
ilamb
incognita
intake-dremio
intake-odbc
intake-parquet
ipumspy
jaxlib
jina
jina-core
jina-perf
json2parquet
jupyter-hdfscm
kartothek
kedro
kedro-vertexai
koalas
koopa-viz
larch
lenskit
libarrow
libgdal-arrow-parquet
libgoogle-cloud
libopentelemetry-cpp
libtensorflow
libtensorflow_cc
lsst_dashboard
lsstdesc-env
lsstdesc-gcr-catalogs
mage-ai
mapshader
mlflow
mlflow-pipelines
modin-hdk
modin-omnisci
modin-ray
modzy-sdk
molfeat
mosartwmpy
nannyml
napari-locpix
nlp
nteract-scrapbook
nutpie
ocsmesh
opencensus-proto
openml
opentelemetry-exporter-jaeger-proto-grpc
opentelemetry-exporter-opencensus
opentelemetry-exporter-otlp-proto-grpc
pado
pandas-gbq
pandera
pandera-base
pandera-core
parquet-tools
plateau
ploomber
premise
provenance
pyarrow
pyarrow-tests
pydiverse-pipedag
pyfitparquet
pygadm
pygaul
pygeohydro
pygwalker
pyhdk
pyiceberg
pyiem
pyinaturalist-convert
pyinterpolate
pymapd
pymilvus
pymongoarrow
pymovements
pymrio
pynhd
pyomnisci
pyomniscidbe
pyquokka
pyramids
pyspark
pysurvival
python-dremio-client
python-duckdb
qcfractal-core
qcportal
qp
qp-prob-full
quilt
quilt3
r-arrow
ray-client
ray-core
ray-data
ray-tune
rerun-sdk
river-cpp
river-ingester
robotframework-browser
robustness
rpy2-arrow
rubicon-ml
rubin-env
rubin-env-nosysroot
rubinenv
saqc
sas7bdat-converter
scmorph
scrapbook
scrapbook-build
sepal-ui
sharrow
skein
sliderule
smif
snowflake-connector-python
spatialpandas
sqlalchemy-bigquery
sqlalchemy-dremio
squirrel-core
stac-geoparquet
stglib
streamlit
superset
tables-io
tables-io-full
tensorboard
tensorflow
tensorflow-base
tensorflow-estimator
text_extensions_for_pandas
tiled-base
tiled-dataframe
tiled-sparse
tip
towhee
traffic
triad
tritonclient-grpc
turbodbc
unfccc_di_api
vaex-arrow
vaex-core
vegafusion
verta
wetterdienst
woodwork
xbbg
xlntpyarrow
yandexcloud
zcollection

[hmaarrfk]

not sure if i want to fully close out the migrations if pytorch/tensorflow are still running. can we soft-close instead? i.e. bump the global pins but leave the migrator running.

[h-vetinari]

@hmaarrfk, I tried to do that now - could you PTAL?

I left the zip in because we still need it for the old migrator, and we'll need it as well for the introduction of 4.24.x.

[hmaarrfk]

thanks.

[h-vetinari]

TBH, I didn't expect we'd merge yet given

So I'm afraid this is now coupled with the MacOS discussion. 😑

But at least it seems everyone's on board and we'll be able to go ahead with the announcement after the core call. 🤞

[hmaarrfk]

MacOS discussion

I thought everybody was on the same page to move forward with it (albeit with some announcements)

[hmaarrfk]

I guess we might as well close things off. I'm not too interested in building both protobufs for pytorch and tensorflow, they are pretty difficult as is.