KeyError 'p' in rule simplify_network / PyPSA-Earth India

[Friederike22]

Hello everyone,
I have been trying to set up PyPSA-Earth for India. When executing the rule "simplify_network" I have encountered the error "key error 'p' ".
I can run the tutorial model successfully and when comparing the data from the tutorial with the data for india, I was not able to identify what the problem is in the indian model.

Does anyone have an idea where this problem is coming from?
I would highly appreciate any help to fix this.
Thank you!

The error description in the Terminal looks like this:

warnings.warn(
D:\pypsa-earth-Indien_test.snakemake\scripts\tmphionssfi.simplify_network.py:839: DeprecationWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass include_groups=False to exclude the groupings or
explicitly select the grouping columns after groupby to silence this warning.
.apply(lambda d: gpd.sjoin_nearest(islands_bcountry[d["country"].values[0]], d))
ERROR:_helpers:An error happened in module 'D:\miniconda3\pypsa-earth\lib\site-packages\pandas\core\indexes\base.py', function 'get_loc': 'p'
Traceback (most recent call last):
File "D:\miniconda3\pypsa-earth\lib\site-packages\pandas\core\indexes\base.py", line 3805, in get_loc
return self._engine.get_loc(casted_key)
File "index.pyx", line 167, in pandas._libs.index.IndexEngine.get_loc
File "index.pyx", line 196, in pandas._libs.index.IndexEngine.get_loc
File "pandas\_libs\hashtable_class_helper.pxi", line 7081, in pandas._libs.hashtable.PyObjectHashTable.get_item
File "pandas\_libs\hashtable_class_helper.pxi", line 7089, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'p'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "D:\pypsa-earth-Indien_test.snakemake\scripts\tmphionssfi.simplify_network.py", line 1155, in
n, fetched_nodes_map = merge_into_network(
File "D:\pypsa-earth-Indien_test.snakemake\scripts\tmphionssfi.simplify_network.py", line 866, in merge_into_network
clustering = get_clustering_from_busmap(
File "D:\miniconda3\pypsa-earth\lib\site-packages\pypsa\clustering\spatial.py", line 519, in get_clustering_from_busmap
new_df, new_pnl = aggregateoneport(
File "D:\miniconda3\pypsa-earth\lib\site-packages\pypsa\clustering\spatial.py", line 266, in aggregateoneport
data = n.get_switchable_as_dense(c, attr)
File "D:\miniconda3\pypsa-earth\lib\site-packages\pypsa\descriptors.py", line 124, in get_switchable_as_dense
vals = np.repeat([df.loc[fixed_i, attr].values], len(snapshots), axis=0)
File "D:\miniconda3\pypsa-earth\lib\site-packages\pandas\core\indexing.py", line 1184, in getitem
return self._getitem_tuple(key)
File "D:\miniconda3\pypsa-earth\lib\site-packages\pandas\core\indexing.py", line 1368, in _getitem_tuple
return self._getitem_lowerdim(tup)
File "D:\miniconda3\pypsa-earth\lib\site-packages\pandas\core\indexing.py", line 1065, in _getitem_lowerdim
section = self._getitem_axis(key, axis=i)
File "D:\miniconda3\pypsa-earth\lib\site-packages\pandas\core\indexing.py", line 1431, in _getitem_axis
return self._get_label(key, axis=axis)
File "D:\miniconda3\pypsa-earth\lib\site-packages\pandas\core\indexing.py", line 1381, in _get_label
return self.obj.xs(label, axis=axis)
File "D:\miniconda3\pypsa-earth\lib\site-packages\pandas\core\generic.py", line 4287, in xs
return self[key]
File "D:\miniconda3\pypsa-earth\lib\site-packages\pandas\core\frame.py", line 4102, in getitem
indexer = self.columns.get_loc(key)
File "D:\miniconda3\pypsa-earth\lib\site-packages\pandas\core\indexes\base.py", line 3812, in get_loc
raise KeyError(key) from err
KeyError: 'p'
[Mon Jan 20 11:58:25 2025]
Error in rule simplify_network:
jobid: 5
input: networks/elec.nc, resources/costs_2030.csv, resources/bus_regions/regions_onshore.geojson, resources/bus_regions/regions_offshore.geojson
output: networks/elec_s.nc, resources/bus_regions/regions_onshore_elec_s.geojson, resources/bus_regions/regions_offshore_elec_s.geojson, resources/bus_regions/busmap_elec_s.csv, resources/bus_regions/connection_costs_s.csv
log: logs/simplify_network/elec_s.log (check log file(s) for error details)

RuleException:
CalledProcessError in file D:\pypsa-earth-Indien_test\Snakefile, line 592:
Command 'D:/miniconda3/pypsa-earth/python.exe "D:\pypsa-earth-Indien_test.snakemake\scripts\tmphionssfi.simplify_network.py"' returned non-zero exit status 1.
File "D:\pypsa-earth-Indien_test\Snakefile", line 592, in __rule_simplify_network
File "D:\miniconda3\pypsa-earth\lib\concurrent\futures\thread.py", line 58, in run
Removing output files of failed job simplify_network since they might be corrupted:
resources/bus_regions/connection_costs_s.csv
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: .snakemake\log\2025-01-20T104335.994757.snakemake.log

[ekatef]

Hello everyone, I have been trying to set up PyPSA-Earth for India. When executing the rule "simplify_network" I have encountered the error "key error 'p' ". I can run the tutorial model successfully and when comparing the data from the tutorial with the data for india, I was not able to identify what the problem is in the indian model.

Does anyone have an idea where this problem is coming from? I would highly appreciate any help to fix this. Thank you!

The error description in the Terminal looks like this:

warnings.warn( D:\pypsa-earth-Indien_test.snakemake\scripts\tmphionssfi.simplify_network.py:839: DeprecationWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass include_groups=False to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning. .apply(lambda d: gpd.sjoin_nearest(islands_bcountry[d["country"].values[0]], d)) ERROR:_helpers:An error happened in module 'D:\miniconda3\pypsa-earth\lib\site-packages\pandas\core\indexes\base.py', function 'get_loc': 'p' Traceback (most recent call last): File "D:\miniconda3\pypsa-earth\lib\site-packages\pandas\core\indexes\base.py", line 3805, in get_loc return self._engine.get_loc(casted_key) File "index.pyx", line 167, in pandas._libs.index.IndexEngine.get_loc File "index.pyx", line 196, in pandas._libs.index.IndexEngine.get_loc File "pandas_libs\hashtable_class_helper.pxi", line 7081, in pandas._libs.hashtable.PyObjectHashTable.get_item File "pandas_libs\hashtable_class_helper.pxi", line 7089, in pandas._libs.hashtable.PyObjectHashTable.get_item KeyError: 'p'

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "D:\pypsa-earth-Indien_test.snakemake\scripts\tmphionssfi.simplify_network.py", line 1155, in n, fetched_nodes_map = merge_into_network( File "D:\pypsa-earth-Indien_test.snakemake\scripts\tmphionssfi.simplify_network.py", line 866, in merge_into_network clustering = get_clustering_from_busmap( File "D:\miniconda3\pypsa-earth\lib\site-packages\pypsa\clustering\spatial.py", line 519, in get_clustering_from_busmap new_df, new_pnl = aggregateoneport( File "D:\miniconda3\pypsa-earth\lib\site-packages\pypsa\clustering\spatial.py", line 266, in aggregateoneport data = n.get_switchable_as_dense(c, attr) File "D:\miniconda3\pypsa-earth\lib\site-packages\pypsa\descriptors.py", line 124, in get_switchable_as_dense vals = np.repeat([df.loc[fixed_i, attr].values], len(snapshots), axis=0) File "D:\miniconda3\pypsa-earth\lib\site-packages\pandas\core\indexing.py", line 1184, in getitem return self._getitem_tuple(key) File "D:\miniconda3\pypsa-earth\lib\site-packages\pandas\core\indexing.py", line 1368, in _getitem_tuple return self._getitem_lowerdim(tup) File "D:\miniconda3\pypsa-earth\lib\site-packages\pandas\core\indexing.py", line 1065, in _getitem_lowerdim section = self._getitem_axis(key, axis=i) File "D:\miniconda3\pypsa-earth\lib\site-packages\pandas\core\indexing.py", line 1431, in _getitem_axis return self._get_label(key, axis=axis) File "D:\miniconda3\pypsa-earth\lib\site-packages\pandas\core\indexing.py", line 1381, in _get_label return self.obj.xs(label, axis=axis) File "D:\miniconda3\pypsa-earth\lib\site-packages\pandas\core\generic.py", line 4287, in xs return self[key] File "D:\miniconda3\pypsa-earth\lib\site-packages\pandas\core\frame.py", line 4102, in getitem indexer = self.columns.get_loc(key) File "D:\miniconda3\pypsa-earth\lib\site-packages\pandas\core\indexes\base.py", line 3812, in get_loc raise KeyError(key) from err KeyError: 'p' [Mon Jan 20 11:58:25 2025] Error in rule simplify_network: jobid: 5 input: networks/elec.nc, resources/costs_2030.csv, resources/bus_regions/regions_onshore.geojson, resources/bus_regions/regions_offshore.geojson output: networks/elec_s.nc, resources/bus_regions/regions_onshore_elec_s.geojson, resources/bus_regions/regions_offshore_elec_s.geojson, resources/bus_regions/busmap_elec_s.csv, resources/bus_regions/connection_costs_s.csv log: logs/simplify_network/elec_s.log (check log file(s) for error details)

RuleException: CalledProcessError in file D:\pypsa-earth-Indien_test\Snakefile, line 592: Command 'D:/miniconda3/pypsa-earth/python.exe "D:\pypsa-earth-Indien_test.snakemake\scripts\tmphionssfi.simplify_network.py"' returned non-zero exit status 1. File "D:\pypsa-earth-Indien_test\Snakefile", line 592, in __rule_simplify_network File "D:\miniconda3\pypsa-earth\lib\concurrent\futures\thread.py", line 58, in run Removing output files of failed job simplify_network since they might be corrupted: resources/bus_regions/connection_costs_s.csv Shutting down, this might take some time. Exiting because a job execution failed. Look above for error message Complete log: .snakemake\log\2025-01-20T104335.994757.snakemake.log

Hello @Friederike22, and thanks a lot for the reporting!

The problem is caused by using a feature which fetches isolated networks:

pypsa-earth/config.default.yaml

Line 87 in e29386d

s_threshold_fetch_isolated: 0.05 # [-] a share of the national load for merging an isolated network into a backbone network

A quick solution is to set s_threshold_fetch_isolated: false.

[Friederike22]

Hi Ekaterina, it worked! Thank you so much for the quick and helpful reply!

[ekatef]

Hey @Friederike22 have created the issue #1311 to track and fix this issue. Feel free to comment there!
Could you please also share your conda list for your working environment?