Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update PUDL to SQLAlchemy 2.0 #2267

Merged
merged 2 commits into from
Nov 18, 2023
Merged

Update PUDL to SQLAlchemy 2.0 #2267

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6f603e9
Update PUDL to use SQLAlchemy 2.0
zaneselvans Nov 17, 2023
831d569
Update conda-lock.yml and rendered conda environment files.
zaneselvans Nov 17, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions environments/conda-linux-64.lock.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Generated by conda-lock.
# platform: linux-64
# input_hash: 62bcd0fc7d50df96982d048631ad45bff7910db618111621e212e7f3c28a44bc
# input_hash: 3488f1f8b42f706745fb582d78e40be692fed5eb08accbfb1ebf539a089d87bd

channels:
- conda-forge
Expand Down Expand Up @@ -101,7 +101,7 @@ dependencies:
- libsqlite=3.44.0=h2797004_0
- libssh2=1.11.0=h0841786_0
- libxcb=1.15=h0b41bf4_0
- libxml2=2.11.5=h232c23b_1
- libxml2=2.11.6=h232c23b_0
- libzip=1.10.1=h2629f0a_3
- pcre2=10.42=hcad00b1_0
- readline=8.2=h8228510_1
Expand Down Expand Up @@ -384,7 +384,6 @@ dependencies:
- rfc3339-validator=0.1.4=pyhd8ed1ab_0
- rsa=4.9=pyhd8ed1ab_0
- ruamel.yaml=0.18.5=py311h459d7ec_0
- sqlalchemy=1.4.49=py311h459d7ec_1
- terminado=0.18.0=pyh0d859eb_0
- tinycss2=1.2.1=pyhd8ed1ab_0
- tqdm=4.66.1=pyhd8ed1ab_0
Expand Down Expand Up @@ -438,6 +437,7 @@ dependencies:
- python-build=1.0.3=pyhd8ed1ab_0
- requests=2.31.0=pyhd8ed1ab_0
- rich=13.7.0=pyhd8ed1ab_0
- sqlalchemy=2.0.23=py311h459d7ec_0
- stack_data=0.6.2=pyhd8ed1ab_0
- starlette=0.32.0.post1=pyhd8ed1ab_0
- tiledb=2.16.3=h8c794c1_3
Expand Down
119 changes: 61 additions & 58 deletions environments/conda-lock.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,9 +15,9 @@
version: 1
metadata:
content_hash:
linux-64: 62bcd0fc7d50df96982d048631ad45bff7910db618111621e212e7f3c28a44bc
osx-64: bcf38db86781af22df7f496ce0dafe43a905c359e9c7d6d038429569b874c305
osx-arm64: 8d8eb4c72944efa02bab31f0274acad445146594b4cad129cc341a304e08ebb2
linux-64: 3488f1f8b42f706745fb582d78e40be692fed5eb08accbfb1ebf539a089d87bd
osx-64: b0e3c33170a43614c2f811caf4b3f1572d74d3cfac308aaa1d7b7440f6b126a0
osx-arm64: 4741766e82a5d953aa4d5bc129ecfb5a9580d7cc918942e6ef8d36fc0bf4daa4
channels:
- url: conda-forge
used_env_vars: []
Expand Down Expand Up @@ -1223,7 +1223,7 @@ package:
category: main
optional: false
- name: libxml2
version: 2.11.5
version: 2.11.6
manager: conda
platform: linux-64
dependencies:
Expand All @@ -1232,10 +1232,10 @@ package:
libiconv: ">=1.17,<2.0a0"
libzlib: ">=1.2.13,<1.3.0a0"
xz: ">=5.2.6,<6.0a0"
url: https://conda.anaconda.org/conda-forge/linux-64/libxml2-2.11.5-h232c23b_1.conda
url: https://conda.anaconda.org/conda-forge/linux-64/libxml2-2.11.6-h232c23b_0.conda
hash:
md5: f3858448893839820d4bcfb14ad3ecdf
sha256: 1b3cb6864de1a558ea5fb144c780121d52507837d15df0600491d8ed92cff90c
md5: 427a3e59d66cb5d145020bd9c6493334
sha256: e6183d5e57ee48cc1fc4340477c31a6bd8be4d3ba5dded82cbca0d5280591086
category: main
optional: false
- name: libzip
Expand Down Expand Up @@ -5073,21 +5073,6 @@ package:
sha256: c92e7bbb1d02286bcd3d3292208006f796ae45df82af3deec940339493415c04
category: main
optional: false
- name: sqlalchemy
version: 1.4.49
manager: conda
platform: linux-64
dependencies:
greenlet: "!=0.4.17"
libgcc-ng: ">=12"
python: ">=3.11,<3.12.0a0"
python_abi: 3.11.*
url: https://conda.anaconda.org/conda-forge/linux-64/sqlalchemy-1.4.49-py311h459d7ec_1.conda
hash:
md5: 17392bcb4ceac1b2c95db9d54b4ac018
sha256: 542dea4823e2e1283936fbd25c9f3fa960ec6df2dd54589b192b4dac68af7295
category: main
optional: false
- name: terminado
version: 0.18.0
manager: conda
Expand Down Expand Up @@ -5892,6 +5877,22 @@ package:
sha256: 4bb25bf1f5664772b2c4c2e3878aa6e7dc2695f97e3da4ee8e47c51e179913bb
category: main
optional: false
- name: sqlalchemy
version: 2.0.23
manager: conda
platform: linux-64
dependencies:
greenlet: "!=0.4.17"
libgcc-ng: ">=12"
python: ">=3.11,<3.12.0a0"
python_abi: 3.11.*
typing-extensions: ">=4.2.0"
url: https://conda.anaconda.org/conda-forge/linux-64/sqlalchemy-2.0.23-py311h459d7ec_0.conda
hash:
md5: caccc840985d972796a3c94e69376177
sha256: b616e46d0e4c914d29a9860384a6e44e33106cef565ba238d669766e658faa80
category: main
optional: false
- name: stack_data
version: 0.6.2
manager: conda
Expand Down Expand Up @@ -9009,18 +9010,18 @@ package:
category: main
optional: false
- name: libxml2
version: 2.11.5
version: 2.11.6
manager: conda
platform: osx-64
dependencies:
icu: ">=73.2,<74.0a0"
libiconv: ">=1.17,<2.0a0"
libzlib: ">=1.2.13,<1.3.0a0"
xz: ">=5.2.6,<6.0a0"
url: https://conda.anaconda.org/conda-forge/osx-64/libxml2-2.11.5-h3346baf_1.conda
url: https://conda.anaconda.org/conda-forge/osx-64/libxml2-2.11.6-hc0ae0f7_0.conda
hash:
md5: 7584dee6af7de378aed0ae49aebedb8a
sha256: d901fab32e57a43c44e630fb1c4d0a163d23b109eecd6c68b9ee371800760bca
md5: 2b6ec8c6366ea74db4b910469addad1d
sha256: b5b1c3df3e6d0d294764938e79d7f413191cc5b1af2ede49f42b1df04d068a18
category: main
optional: false
- name: lz4-c
Expand Down Expand Up @@ -12972,20 +12973,6 @@ package:
sha256: afdaab0d0f5a288b31450c3da260381da5916c61f122a0b3f5dea76d1ca863bb
category: main
optional: false
- name: sqlalchemy
version: 1.4.49
manager: conda
platform: osx-64
dependencies:
greenlet: "!=0.4.17"
python: ">=3.11,<3.12.0a0"
python_abi: 3.11.*
url: https://conda.anaconda.org/conda-forge/osx-64/sqlalchemy-1.4.49-py311he705e18_1.conda
hash:
md5: 04845132cd5d140b638c4927e1895e12
sha256: 39569dfc13e742e1b61fcc92948e02bc7f361d05bb5100e5226251b70fcdf1af
category: main
optional: false
- name: terminado
version: 0.18.0
manager: conda
Expand Down Expand Up @@ -13768,6 +13755,21 @@ package:
sha256: 4bb25bf1f5664772b2c4c2e3878aa6e7dc2695f97e3da4ee8e47c51e179913bb
category: main
optional: false
- name: sqlalchemy
version: 2.0.23
manager: conda
platform: osx-64
dependencies:
greenlet: "!=0.4.17"
python: ">=3.11,<3.12.0a0"
python_abi: 3.11.*
typing-extensions: ">=4.2.0"
url: https://conda.anaconda.org/conda-forge/osx-64/sqlalchemy-2.0.23-py311he705e18_0.conda
hash:
md5: f06f912df000cc6bd840011c703c607a
sha256: da5ab07c9148d561586f7fa8110a0794b136e96e168cd591cb7aa87e9805f1da
category: main
optional: false
- name: stack_data
version: 0.6.2
manager: conda
Expand Down Expand Up @@ -16835,18 +16837,18 @@ package:
category: main
optional: false
- name: libxml2
version: 2.11.5
version: 2.11.6
manager: conda
platform: osx-arm64
dependencies:
icu: ">=73.2,<74.0a0"
libiconv: ">=1.17,<2.0a0"
libzlib: ">=1.2.13,<1.3.0a0"
xz: ">=5.2.6,<6.0a0"
url: https://conda.anaconda.org/conda-forge/osx-arm64/libxml2-2.11.5-h25269f3_1.conda
url: https://conda.anaconda.org/conda-forge/osx-arm64/libxml2-2.11.6-h0d0cfa8_0.conda
hash:
md5: 627b5d1377536b5b632ba53cd1455555
sha256: 8291549b87aca48e9cd4aec124af01b5037acd16f8ad14083d7af23c8bb6bebe
md5: 37e112ce9494adfcee6c0c7bf3b5d98d
sha256: 07d2da8f3fb00fb6f84cd36b5329174b878105889f0fe21e79981f27573b47af
category: main
optional: false
- name: lz4-c
Expand Down Expand Up @@ -20785,20 +20787,6 @@ package:
sha256: 33c770e213c233e80b48256d17ce0e7bfe504576f2778307826cf1fd1db058d6
category: main
optional: false
- name: sqlalchemy
version: 1.4.49
manager: conda
platform: osx-arm64
dependencies:
greenlet: "!=0.4.17"
python: ">=3.11,<3.12.0a0"
python_abi: 3.11.*
url: https://conda.anaconda.org/conda-forge/osx-arm64/sqlalchemy-1.4.49-py311h05b510d_1.conda
hash:
md5: 1460f703e4ed3a7bda01ef1baee3a2ea
sha256: 99fbaaa9374f694a2fa86ca37b9c7d5044b3a21f2fdaff23048f1423031f03ac
category: main
optional: false
- name: terminado
version: 0.18.0
manager: conda
Expand Down Expand Up @@ -21581,6 +21569,21 @@ package:
sha256: 4bb25bf1f5664772b2c4c2e3878aa6e7dc2695f97e3da4ee8e47c51e179913bb
category: main
optional: false
- name: sqlalchemy
version: 2.0.23
manager: conda
platform: osx-arm64
dependencies:
greenlet: "!=0.4.17"
python: ">=3.11,<3.12.0a0"
python_abi: 3.11.*
typing-extensions: ">=4.2.0"
url: https://conda.anaconda.org/conda-forge/osx-arm64/sqlalchemy-2.0.23-py311h05b510d_0.conda
hash:
md5: 33795a9c237e7c3ec9cf01a2e89f11dd
sha256: ccf2046118ab2d32d41dc8e90aa3e701e9938522533e39332738f8654d9268cb
category: main
optional: false
- name: stack_data
version: 0.6.2
manager: conda
Expand Down
6 changes: 3 additions & 3 deletions environments/conda-osx-64.lock.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Generated by conda-lock.
# platform: osx-64
# input_hash: bcf38db86781af22df7f496ce0dafe43a905c359e9c7d6d038429569b874c305
# input_hash: b0e3c33170a43614c2f811caf4b3f1572d74d3cfac308aaa1d7b7440f6b126a0

channels:
- conda-forge
Expand Down Expand Up @@ -67,7 +67,7 @@ dependencies:
- libspatialindex=1.9.3=he49afe7_4
- libsqlite=3.44.0=h92b6c6a_0
- libxcb=1.15=hb7f2c08_0
- libxml2=2.11.5=h3346baf_1
- libxml2=2.11.6=hc0ae0f7_0
- lz4-c=1.9.4=hf0c8a7f_0
- ncurses=6.4=h93d8f39_2
- nspr=4.35=hea0b92c_0
Expand Down Expand Up @@ -364,7 +364,6 @@ dependencies:
- rfc3339-validator=0.1.4=pyhd8ed1ab_0
- rsa=4.9=pyhd8ed1ab_0
- ruamel.yaml=0.18.5=py311he705e18_0
- sqlalchemy=1.4.49=py311he705e18_1
- terminado=0.18.0=pyh31c8845_0
- tinycss2=1.2.1=pyhd8ed1ab_0
- tqdm=4.66.1=pyhd8ed1ab_0
Expand Down Expand Up @@ -418,6 +417,7 @@ dependencies:
- python-build=1.0.3=pyhd8ed1ab_0
- requests=2.31.0=pyhd8ed1ab_0
- rich=13.7.0=pyhd8ed1ab_0
- sqlalchemy=2.0.23=py311he705e18_0
- stack_data=0.6.2=pyhd8ed1ab_0
- starlette=0.32.0.post1=pyhd8ed1ab_0
- tiledb=2.16.3=hd3a41d5_3
Expand Down
6 changes: 3 additions & 3 deletions environments/conda-osx-arm64.lock.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Generated by conda-lock.
# platform: osx-arm64
# input_hash: 8d8eb4c72944efa02bab31f0274acad445146594b4cad129cc341a304e08ebb2
# input_hash: 4741766e82a5d953aa4d5bc129ecfb5a9580d7cc918942e6ef8d36fc0bf4daa4

channels:
- conda-forge
Expand Down Expand Up @@ -68,7 +68,7 @@ dependencies:
- libspatialindex=1.9.3=hbdafb3b_4
- libsqlite=3.44.0=h091b4b1_0
- libxcb=1.15=hf346824_0
- libxml2=2.11.5=h25269f3_1
- libxml2=2.11.6=h0d0cfa8_0
- lz4-c=1.9.4=hb7217d7_0
- ncurses=6.4=h463b476_2
- nspr=4.35=hb7217d7_0
Expand Down Expand Up @@ -364,7 +364,6 @@ dependencies:
- rfc3339-validator=0.1.4=pyhd8ed1ab_0
- rsa=4.9=pyhd8ed1ab_0
- ruamel.yaml=0.18.5=py311h05b510d_0
- sqlalchemy=1.4.49=py311h05b510d_1
- terminado=0.18.0=pyh31c8845_0
- tinycss2=1.2.1=pyhd8ed1ab_0
- tqdm=4.66.1=pyhd8ed1ab_0
Expand Down Expand Up @@ -418,6 +417,7 @@ dependencies:
- python-build=1.0.3=pyhd8ed1ab_0
- requests=2.31.0=pyhd8ed1ab_0
- rich=13.7.0=pyhd8ed1ab_0
- sqlalchemy=2.0.23=py311h05b510d_0
- stack_data=0.6.2=pyhd8ed1ab_0
- starlette=0.32.0.post1=pyhd8ed1ab_0
- tiledb=2.16.3=he15c4da_3
Expand Down
4 changes: 2 additions & 2 deletions pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ dependencies = [
"grpcio==1.59.2", # Required by dagster, binary dependencies are flaky
"grpcio-health-checking==1.59.2", # Required by dagster, binary dependencies are flaky
"grpcio-status==1.59.2", # Required by dagster, binary dependencies are flaky
"hypothesis>=6.87,<7.0",
"hypothesis>=6.87",
"jinja2>=3.1",
"jupyter",
"matplotlib>=3.8",
Expand Down Expand Up @@ -75,7 +75,7 @@ dependencies = [
"sphinx-reredirects>=0.1.2",
"sphinx>=7",
"sphinxcontrib_bibtex>=2.6",
"sqlalchemy>=1.4,<2",
"sqlalchemy>=2",
"timezonefinder>=6.2",
"xlrd>=2.0.1", # pandas[excel]
"xlsxwriter>=3.1", # pandas[excel]
Expand Down
6 changes: 3 additions & 3 deletions src/pudl/analysis/allocate_gen_fuel.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1012,7 +1012,7 @@ def _allocate_unassociated_pm_records(
eia_generators_connected = gen_assoc.loc[connected_mask].assign(
capacity_mw_minus_one=lambda x: x.groupby(idx_minus_one)[
"capacity_mw"
].transform(sum),
].transform("sum"),
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressing a deprecation warning. Future pandas will use the callable here directly, which is not what we want.

frac_cap_minus_one=lambda x: x.capacity_mw / x.capacity_mw_minus_one,
)

Expand Down Expand Up @@ -1192,7 +1192,7 @@ def prep_alloction_fraction(gen_assoc: pd.DataFrame) -> pd.DataFrame:
)
gen_pm_fuel["capacity_mw_fuel_in_bf_tbl_group"] = gen_pm_fuel.groupby(
IDX_PM_ESC + ["in_bf_tbl"], dropna=False
)[["capacity_mw"]].transform(sum, min_count=1)
)[["capacity_mw"]].transform("sum", min_count=1)

return gen_pm_fuel

Expand Down Expand Up @@ -1534,7 +1534,7 @@ def assign_plant_year(df):
| np.isclose(reporters[data_column_name], 0)
)
.groupby(key_columns_annual, dropna=False)[["missing_data"]]
.transform(sum)
.transform("sum")
)

# seperate annual and monthly reporters
Expand Down
Loading