Simplify network not working #1308
Comments
|
Hello @GbotemiB , this is a new error. In particular, the line/cable tag in simplify network is likely not relevant for the modelling, yet the check is still performed. |
|
@davide-f, I tried to not reuse past files. I also created a new environment. I will try out your suggestions and revert back. |
|
Hello, same issue is happining when running the model for Saudi Arabia. |
|
Welcome @mohmJR :D great for your interest and we welcome you in the initiative, we have weekly meetings :) @GbotemiB and @mohmJR , maybe a fix could be to drop "tag_area" from pypsa-earth/scripts/build_osm_network.py Lines 32 to 40 in e29386d Maybe that solves the issue |
|
Hey @GbotemiB @mohmJR, thanks a lot for sharing! Really helpful to get the model up and running, and very much appreciated. Agree with @davide-f that a likely reason is some specifics of the power grid data, and dropping Unfortunately, the error is poorly reproducible due to some specifics of the clustering algorithm we are using. @GbotemiB @mohmJR if you would have an opportunity to try this fix and report the results it would be incredibly helpful. |
|
To make the whole picture even more interesting, I'm getting another error when running the model for Saudi Arabia. The workflow claims namely that frequency values do not agree for some lines (the full listing is attached bellow). That doesn't feel right, and probably there are some issues in the network model to be fixed. Not sure however if this mismatch is really meaningful for the grid model which we are using in the optimisation runs. I also suspect that the similar issue may arise also for other countries as the updated PyPSA version in much more restrictive towards the data structure. To have a more general fix, it could be a geed idea to drop all the columns from Error listingrule cluster_network:
input: networks/SA/elec_s.nc, resources/SA/shapes/country_shapes.geojson, resources/SA/bus_regions/regions_onshore_elec_s.geojson, resources/SA/bus_regions/regions_offshore_elec_s.geojson, resources/SA/shapes/gadm_shapes.geojson, resources/SA/costs_2030.csv
output: networks/SA/elec_s_10.nc, resources/SA/bus_regions/regions_onshore_elec_s_10.geojson, resources/SA/bus_regions/regions_offshore_elec_s_10.geojson, resources/SA/bus_regions/busmap_elec_s_10.csv, resources/SA/bus_regions/linemap_elec_s_10.csv
log: logs/SA/cluster_network/elec_s_10.log
jobid: 4
benchmark: benchmarks/SA/cluster_network/elec_s_10
reason: Missing output files: networks/SA/elec_s_10.nc; Input files updated by another job: networks/SA/elec_s.nc, resources/SA/bus_regions/regions_offshore_elec_s.geojson, resources/SA/bus_regions/regions_onshore_elec_s.geojson
wildcards: simpl=, clusters=10
resources: tmpdir=/var/folders/qn/vpndfm21795ckkq89np1ckp40000gn/T, mem_mb=3000, mem_mib=2862
INFO:pypsa.io:Imported network elec_s.nc has buses, carriers, generators, lines, loads
Overwriting marginal_cost of Index(['solar', 'onwind', 'offwind', 'hydro', 'H2', 'electrolysis',
'fuel cell', 'battery', 'battery inverter'],
dtype='object') to [0.01 0.015 0.015 0. 0. 0. 0. 0. 0. ]
INFO:add_electricity:Overwriting marginal_cost of Index(['solar', 'onwind', 'offwind', 'hydro', 'H2', 'electrolysis',
'fuel cell', 'battery', 'battery inverter'],
dtype='object') to [0.01 0.015 0.015 0. 0. 0. 0. 0. 0. ]
ERROR:_helpers:An error happened in module '~/python3.10/site-packages/pypsa/clustering/spatial.py', function 'consense': In Line cluster tag_frequency, the values of attribute tag_frequency do not agree:
Line
317 50.0
439 60.0
498 60.0
499 60.0
Name: tag_frequency, dtype: float64
Traceback (most recent call last):
File "/pypsa-earth/.snakemake/scripts/tmp56nf9s9u.cluster_network.py", line 759, in <module>
clustering = clustering_for_n_clusters(
File "/pypsa-earth/.snakemake/scripts/tmp56nf9s9u.cluster_network.py", line 609, in clustering_for_n_clusters
clustering = get_clustering_from_busmap(
File "~/python3.10/site-packages/pypsa/clustering/spatial.py", line 473, in get_clustering_from_busmap
lines, lines_t, linemap = aggregatelines(
File "~/python3.10/site-packages/pypsa/clustering/spatial.py", line 416, in aggregatelines
df = df.groupby(grouper).agg(static_strategies)
File "~/python3.10/site-packages/pandas/core/groupby/generic.py", line 1432, in aggregate
result = op.agg()
File "~/python3.10/site-packages/pandas/core/apply.py", line 190, in agg
return self.agg_dict_like()
File "~/python3.10/site-packages/pandas/core/apply.py", line 423, in agg_dict_like
return self.agg_or_apply_dict_like(op_name="agg")
File "~/python3.10/site-packages/pandas/core/apply.py", line 1608, in agg_or_apply_dict_like
result_index, result_data = self.compute_dict_like(
File "~/python3.10/site-packages/pandas/core/apply.py", line 496, in compute_dict_like
results = [
File "~/python3.10/site-packages/pandas/core/apply.py", line 497, in <listcomp>
getattr(obj._gotitem(key, ndim=1), op_name)(how, **kwargs)
File "~/python3.10/site-packages/pandas/core/groupby/generic.py", line 294, in aggregate
return self._python_agg_general(func, *args, **kwargs)
File "~/python3.10/site-packages/pandas/core/groupby/generic.py", line 327, in _python_agg_general
result = self._grouper.agg_series(obj, f)
File "~/python3.10/site-packages/pandas/core/groupby/ops.py", line 864, in agg_series
result = self._aggregate_series_pure_python(obj, func)
File "~/python3.10/site-packages/pandas/core/groupby/ops.py", line 885, in _aggregate_series_pure_python
res = func(group)
File "~/python3.10/site-packages/pandas/core/groupby/generic.py", line 324, in <lambda>
f = lambda x: func(x, *args, **kwargs)
File "~/python3.10/site-packages/pypsa/clustering/spatial.py", line 135, in consense
assert (x == v).all() or x.isnull().all(), (
AssertionError: In Line cluster tag_frequency, the values of attribute tag_frequency do not agree:
Line
317 50.0
439 60.0
498 60.0
499 60.0
Name: tag_frequency, dtype: float64 |
|
Have drafted a possible solution in #1315. It works for the frequency-related error, but I can't test it for tag_area error due to a low reproducibility. Any additional inputs on that would be very much welcome. |
|
Hi @ekatef, thank you creating a possible solution. But just like you mentioned, reproducing the issue is also an issue. I encountered the error while running on cluster, I am running into a different error locally in add_electricity. Here is the log to have an idea of what is going on. Building DAG of jobs...
Using shell: /bin/bash
Provided cores: 10
Rules claiming more threads will be scaled down.
Job stats:
job count
-------------------- -------
add_electricity 1
add_extra_components 1
cluster_network 1
prepare_network 1
simplify_network 1
solve_all_networks 1
solve_network 1
total 7
Select jobs to execute...
[Tue Jan 21 01:09:00 2025]
rule add_electricity:
input: submodules/pypsa-earth/resources/US_2021/renewable_profiles/profile_hydro.nc, submodules/pypsa-earth/resources/US_2021/renewable_profiles/profile_onwind.nc, submodules/pypsa-earth/resources/US_2021/renewable_profiles/profile_offwind-ac.nc, submodules/pypsa-earth/resources/US_2021/renewable_profiles/profile_offwind-dc.nc, submodules/pypsa-earth/resources/US_2021/renewable_profiles/profile_solar.nc, submodules/pypsa-earth/resources/US_2021/renewable_profiles/profile_csp.nc, submodules/pypsa-earth/networks/US_2021/base.nc, submodules/pypsa-earth/resources/US_2021/costs_2030.csv, submodules/pypsa-earth/resources/US_2021/powerplants.csv, submodules/pypsa-earth/resources/US_2021/shapes/gadm_shapes.geojson, submodules/pypsa-earth/data/hydro_capacities.csv, submodules/pypsa-earth/resources/US_2021/demand_profiles.csv
output: submodules/pypsa-earth/networks/US_2021/elec.nc
log: submodules/pypsa-earth/logs/US_2021/add_electricity.log
jobid: 6
benchmark: submodules/pypsa-earth/benchmarks/US_2021/add_electricity
reason: Missing output files: submodules/pypsa-earth/networks/US_2021/elec.nc; Updated input files: submodules/pypsa-earth/resources/US_2021/renewable_profiles/profile_offwind-dc.nc, submodules/pypsa-earth/resources/US_2021/costs_2030.csv, submodules/pypsa-earth/resources/US_2021/renewable_profiles/profile_csp.nc, submodules/pypsa-earth/resources/US_2021/shapes/gadm_shapes.geojson, submodules/pypsa-earth/resources/US_2021/demand_profiles.csv, submodules/pypsa-earth/resources/US_2021/renewable_profiles/profile_offwind-ac.nc, submodules/pypsa-earth/resources/US_2021/renewable_profiles/profile_onwind.nc, submodules/pypsa-earth/resources/US_2021/renewable_profiles/profile_solar.nc, submodules/pypsa-earth/networks/US_2021/base.nc, submodules/pypsa-earth/resources/US_2021/powerplants.csv, submodules/pypsa-earth/resources/US_2021/renewable_profiles/profile_hydro.nc
resources: tmpdir=/var/folders/c3/657p__892k72qxh6r8fdkqsh0000gn/T, mem_mb=3000, mem_mib=2862
INFO:pypsa.io:Imported network base.nc has buses, lines, links, transformers
Overwriting marginal_cost of Index(['solar', 'onwind', 'offwind', 'hydro', 'H2', 'electrolysis',
'fuel cell', 'battery', 'battery inverter'],
dtype='object') to [0.01 0.015 0.015 0. 0. 0. 0. 0. 0. ]
INFO:__main__:Overwriting marginal_cost of Index(['solar', 'onwind', 'offwind', 'hydro', 'H2', 'electrolysis',
'fuel cell', 'battery', 'battery inverter'],
dtype='object') to [0.01 0.015 0.015 0. 0. 0. 0. 0. 0. ]
No renewables found in config entry `extendable_carriers`. In future versions, these have to be explicitly listed. Falling back to all renewables.
WARNING:__main__:No renewables found in config entry `extendable_carriers`. In future versions, these have to be explicitly listed. Falling back to all renewables.
Adding 12859 generators with capacities [GW]
carrier
CCGT 320.68
OCGT 245.70
biomass 6.91
coal 205.45
geothermal 3.97
nuclear 99.44
oil 35.53
Name: p_nom, dtype: float64
INFO:__main__:Adding 12859 generators with capacities [GW]
carrier
CCGT 320.68
OCGT 245.70
biomass 6.91
coal 205.45
geothermal 3.97
nuclear 99.44
oil 35.53
Name: p_nom, dtype: float64
Added connection cost of 0-0 Eur/MW/a to offwind-ac
INFO:__main__:Added connection cost of 0-0 Eur/MW/a to offwind-ac
Added connection cost of 0-173034 Eur/MW/a to offwind-dc
INFO:__main__:Added connection cost of 0-173034 Eur/MW/a to offwind-dc
Assuming max_hours=6 for hydro reservoirs in the countries: US
WARNING:__main__:Assuming max_hours=6 for hydro reservoirs in the countries: US
tech_colors for carriers ['csp', 'PHS'] not defined in config.
WARNING:__main__:tech_colors for carriers ['csp', 'PHS'] not defined in config.
INFO:pypsa.io:Exported network elec.nc has buses, links, lines, generators, transformers, loads, storage_units, carriers
ERROR:_helpers:An error happened in module '/Users/gbotemi/miniconda3/envs/pypsa-earth-linopy/lib/python3.10/site-packages/netCDF4/_netCDF4.cpython-310-darwin.so', function 'netCDF4._netCDF4._ensure_nc_success': NetCDF: HDF error
Traceback (most recent call last):
File "/Users/gbotemi/Documents/code/PYPSA/US/efuels-supply-potentials/.snakemake/scripts/tmp4_nf2jmw.add_electricity.py", line 908, in <module>
n.export_to_netcdf(snakemake.output[0])
File "/Users/gbotemi/miniconda3/envs/pypsa-earth-linopy/lib/python3.10/site-packages/pypsa/io.py", line 733, in export_to_netcdf
with ExporterNetCDF(path, compression, float32) as exporter:
File "/Users/gbotemi/miniconda3/envs/pypsa-earth-linopy/lib/python3.10/site-packages/pypsa/io.py", line 72, in __exit__
self.finish()
File "/Users/gbotemi/miniconda3/envs/pypsa-earth-linopy/lib/python3.10/site-packages/pypsa/io.py", line 421, in finish
self.ds.to_netcdf(self.path)
File "/Users/gbotemi/miniconda3/envs/pypsa-earth-linopy/lib/python3.10/site-packages/xarray/core/dataset.py", line 2280, in to_netcdf
return to_netcdf( # type: ignore # mypy cannot resolve the overloads:(
File "/Users/gbotemi/miniconda3/envs/pypsa-earth-linopy/lib/python3.10/site-packages/xarray/backends/api.py", line 1259, in to_netcdf
dump_to_store(
File "/Users/gbotemi/miniconda3/envs/pypsa-earth-linopy/lib/python3.10/site-packages/xarray/backends/api.py", line 1306, in dump_to_store
store.store(variables, attrs, check_encoding, writer, unlimited_dims=unlimited_dims)
File "/Users/gbotemi/miniconda3/envs/pypsa-earth-linopy/lib/python3.10/site-packages/xarray/backends/common.py", line 356, in store
self.set_variables(
File "/Users/gbotemi/miniconda3/envs/pypsa-earth-linopy/lib/python3.10/site-packages/xarray/backends/common.py", line 398, in set_variables
writer.add(source, target)
File "/Users/gbotemi/miniconda3/envs/pypsa-earth-linopy/lib/python3.10/site-packages/xarray/backends/common.py", line 243, in add
target[...] = source
File "/Users/gbotemi/miniconda3/envs/pypsa-earth-linopy/lib/python3.10/site-packages/xarray/backends/netCDF4_.py", line 79, in __setitem__
data[key] = value
File "src/netCDF4/_netCDF4.pyx", line 5630, in netCDF4._netCDF4.Variable.__setitem__
File "src/netCDF4/_netCDF4.pyx", line 5917, in netCDF4._netCDF4.Variable._put
File "src/netCDF4/_netCDF4.pyx", line 2164, in netCDF4._netCDF4._ensure_nc_success
RuntimeError: NetCDF: HDF error
[Tue Jan 21 01:10:29 2025]
Error in rule add_electricity:
jobid: 6
input: submodules/pypsa-earth/resources/US_2021/renewable_profiles/profile_hydro.nc, submodules/pypsa-earth/resources/US_2021/renewable_profiles/profile_onwind.nc, submodules/pypsa-earth/resources/US_2021/renewable_profiles/profile_offwind-ac.nc, submodules/pypsa-earth/resources/US_2021/renewable_profiles/profile_offwind-dc.nc, submodules/pypsa-earth/resources/US_2021/renewable_profiles/profile_solar.nc, submodules/pypsa-earth/resources/US_2021/renewable_profiles/profile_csp.nc, submodules/pypsa-earth/networks/US_2021/base.nc, submodules/pypsa-earth/resources/US_2021/costs_2030.csv, submodules/pypsa-earth/resources/US_2021/powerplants.csv, submodules/pypsa-earth/resources/US_2021/shapes/gadm_shapes.geojson, submodules/pypsa-earth/data/hydro_capacities.csv, submodules/pypsa-earth/resources/US_2021/demand_profiles.csv
output: submodules/pypsa-earth/networks/US_2021/elec.nc
log: submodules/pypsa-earth/logs/US_2021/add_electricity.log (check log file(s) for error details)
RuleException:
CalledProcessError in file /Users/gbotemi/Documents/code/PYPSA/US/efuels-supply-potentials/submodules/pypsa-earth/Snakefile, line 551:
Command 'set -euo pipefail; /Users/gbotemi/miniconda3/envs/pypsa-earth-linopy/bin/python3.10 /Users/gbotemi/Documents/code/PYPSA/US/efuels-supply-potentials/.snakemake/scripts/tmp4_nf2jmw.add_electricity.py' returned non-zero exit status 1.
File "/Users/gbotemi/Documents/code/PYPSA/US/efuels-supply-potentials/submodules/pypsa-earth/Snakefile", line 551, in __rule_add_electricity
File "/Users/gbotemi/miniconda3/envs/pypsa-earth-linopy/lib/python3.10/concurrent/futures/thread.py", line 58, in run
Removing output files of failed job add_electricity since they might be corrupted:
submodules/pypsa-earth/networks/US_2021/elec.nc
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2025-01-21T010855.996361.snakemake.log |
|
This last error is actually weird... it seems an environment issue actually. |
I will try to use pinned environment. |
|
Thank you all for your help! @ekatef I have tried your modified codes and it seems working but I got error with solve_network rule as following : Set parameter Username
INFO:gurobipy: INFO:gurobipy: INFO:gurobipy: RuleException: |
@mohmJR Thank you so much for updating the status. The error in |
Just to understand, is this issue still alive? |
|
The issue has been fixed somehow, as the error no longer appears after running with latest commit. |
Checklist
mainbranch or the latest release. Please indicate.pypsa-earthenvironment. Update viaconda env update -f envs/environment.yaml.Describe the Bug
Simiplify network is running into an issue. I encountered this issue while running the model for US.
Please provide a description of what the bug is and add a minimal example/command for reproducing the bug.
Error Message
If applicable, paste any terminal output to help illustrating your problem.
In some cases it may also be useful to share your list of installed packages:
conda list.Here is my conda list
The text was updated successfully, but these errors were encountered: