Merge branch 'yerbol_main' into daniele_main

pypsa-meets-earth · Jan 21, 2025 · f437ddb · f437ddb
2 parents 52e6636 + e29386d
commit f437ddb
Show file tree

Hide file tree

Showing 25 changed files with 939 additions and 675 deletions.
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
@@ -31,8 +31,8 @@ jobs:
           env_file: envs/linux-pinned.yaml
         - os: macos
           env_file: envs/macos-pinned.yaml
-        # - os: windows
-        #   env_file: envs/windows-pinned.yaml
+        - os: windows
+          env_file: envs/windows-pinned.yaml
 
     defaults:
       run:

diff --git a/.readthedocs.yaml b/.readthedocs.yaml
@@ -4,6 +4,9 @@
 
 version: 2
 
+sphinx:
+  configuration: doc/conf.py
+
 build:
   os: ubuntu-22.04
   tools:

diff --git a/README.md b/README.md
@@ -58,41 +58,22 @@ The diagram below depicts one representative clustered node for the sector-coupl
     <img src="https://api.star-history.com/svg?repos=pypsa-meets-earth/pypsa-earth,OSeMOSYS/osemosys_global,niclasmattsson/Supergrid,SGIModel/MUSE_OS,etsap-TIMES/TIMES_model&type=Date" width="60%">
 <a/>
 
-## Get involved
-
-There are multiple ways to get involved and learn more about our work. That's how we organise ourselves:
-
-- [**Discord NEW! (Open)**](https://discord.gg/AnuJBk23FU)
-  - chat with the community, team up on features, exchange with developers, code in voice channels
-  - registration and usage is for free
-        <p align="left">
-            <a href="https://discord.gg/AnuJBk23FU">
-            <img src="https://discord.com/assets/cb48d2a8d4991281d7a6a95d2f58195e.svg" width="150">
-            <a/>
-        </p>
-- **General initiative meeting (Open)**
-  - every forth Thursday each month Thursday 16-17:00 (UK time)
-    <a href="https://drive.google.com/file/d/1naH4WwW9drkOkOJ3PLO4fyWdkZQi5-_w/view?usp=share_link">
-    `download .ics`
-    </a>
-  - join for project news and high-level code updates
-  - meeting hosted on Discord
-  - [open agenda](https://docs.google.com/document/d/1r6wm2RBe0DWFngmItpFfSFHA-CnUmVcVTkIKmthdW3g/edit?usp=sharing). See what we will discuss. Invited members have edit rights.
-- **Buddy talk (Open)**
-  - book a 30min meeting with Max to discuss anything you like
-  - booking link: [calendly.com/pypsa-meets-earth](https://calendly.com/max-parzen/pypsa-meets-earth-exchange-30min)
-- **Specific code meeting (Open)**
-  - meeting hosted on Discord
-  - join updates, demos, Q&A's, discussions and the coordination of each work package
-    1. Demand creation and prediction meeting, on demand
-    2. AI asset detection meeting, on demand
-    3. Sector coupling meeting, every Thursday 09:00 (UK time), <a href="https://drive.google.com/file/d/1PDdmjsKhzyGRo0_YrP4wPQkn2XTNh6jA/view?usp=share_link" >`download .ics`</a>
-    4. PyPSA-Earth meeting, every Thursday 16:00 (UK time), <a href="https://drive.google.com/file/d/1gaLmyV4qGPXsogkeRcAPWjC0ESebUxU-/view?usp=share_link" >`download .ics`</a>
-- **Outreach meeting (Open)**
-  - every second week, Tuesday 17:00 (UK time)
-  - planning, discussing events, workshops, communication, community activities
-- [**Google Drive**](https://drive.google.com/drive/folders/13Z8Y9zgsh5IZaDNkkRyo1wkoMgbdUxT5?usp=sharing)
-  - access to minutes, presentations, lists, documents (access to minutes)
+## How to get involved
+
+There are multiple ways to get involved and learn more about our work:
+1. **Join our forum** and communication platform on [**PyPSA-meets-Earth**](https://discord.gg/AnuJBk23FU) Discord Server
+2. **Chat on Discord with us** in the following open meetings:
+    - **General initiative meeting** for project news and [high-level code updates](https://docs.google.com/document/d/1r6wm2RBe0DWFngmItpFfSFHA-CnUmVcVTkIKmthdW3g/edit?usp=sharing). Held every [fourth Thursday 16-17:00 (UK time)](https://drive.google.com/file/d/1naH4WwW9drkOkOJ3PLO4fyWdkZQi5-_w/view?usp=share_link) and is a perfect place to meet the community and get a high-level update on PyPSA ecosystem relevant for PyPSA-Earth developments.
+    - **Weekly developers meetings**
+        - Eastern-Hemisphere friendly *Morning meeting* every [Thursday at 09:00 (UK time)](https://drive.google.com/file/d/1PDdmjsKhzyGRo0_YrP4wPQkn2XTNh6jA/view?usp=share_link).
+        - Western-Hemisphere friendly *Evening meeting* every [Thursday 16:00 (UK time)](https://drive.google.com/file/d/1gaLmyV4qGPXsogkeRcAPWjC0ESebUxU-/view?usp=share_link). Every forth Thursday is replaced by the General initiative meeting which has a more high-level perspective, but you can also join to discuss more particular questions.
+3. **Look at public materials** at [**google Drive**](https://drive.google.com/drive/folders/13Z8Y9zgsh5IZaDNkkRyo1wkoMgbdUxT5?usp=sharing) to share to minutes, presentations, lists and documents. Feel gree to get a look!
+4. **Notify your interest** to on-demand meetings:
+    - On-demand meetings
+        - Demand creation and prediction meeting
+        - AI asset detection meeting
+        - Outreach meeting for planning, discussing events, workshops, communication, community activities
+5. Join us and **propose your stream**.
 
 ## Installation
 
@@ -140,10 +121,10 @@ There are multiple ways to get involved and learn more about our work. That's ho
 
 ## Running the model in previous versions
 
-The model can be run in previous versions by checking out the respective tag. For instance, to run the model in version 0.4.1, which is the last version before the repo `pypsa-earth-sec` was merged, the following command can be used:
+The model can be run in previous versions by checking out the respective tag. For instance, to run the model in version 0.6.0, which is the last version before the repo `pypsa-earth-sec` was merged, the following command can be used:
 
 ```bash
-git checkout v0.4.1
+git checkout v0.6.0
 ```
 After checking out the tag, the model can be run as usual. Please make sure to install the required packages for the respective version.
 

diff --git a/Snakefile b/Snakefile
@@ -16,6 +16,7 @@ from _helpers import (
     get_last_commit_message,
     check_config_version,
     copy_default_files,
+    BASE_DIR,
 )
 from build_demand_profiles import get_load_paths_gegis
 from retrieve_databundle_light import datafiles_retrivedatabundle
@@ -552,6 +553,7 @@ rule add_electricity:
 
 rule simplify_network:
     params:
+        aggregation_strategies=config["cluster_options"]["aggregation_strategies"],
         renewable=config["renewable"],
         geo_crs=config["crs"]["geo_crs"],
         cluster_options=config["cluster_options"],
@@ -594,6 +596,7 @@ if config["augmented_line_connection"].get("add_to_snakefile", False) == True:
 
     rule cluster_network:
         params:
+            aggregation_strategies=config["cluster_options"]["aggregation_strategies"],
             build_shape_options=config["build_shape_options"],
             electricity=config["electricity"],
             costs=config["costs"],
@@ -679,6 +682,7 @@ if config["augmented_line_connection"].get("add_to_snakefile", False) == False:
 
     rule cluster_network:
         params:
+            aggregation_strategies=config["cluster_options"]["aggregation_strategies"],
             build_shape_options=config["build_shape_options"],
             electricity=config["electricity"],
             costs=config["costs"],
@@ -732,7 +736,10 @@ if config["augmented_line_connection"].get("add_to_snakefile", False) == False:
 
 
 rule add_extra_components:
+    params:
+        transmission_efficiency=config["sector"]["transmission_efficiency"],
     input:
+        overrides="data/override_component_attrs",
         network="networks/" + RDIR + "elec_s{simpl}_{clusters}.nc",
         tech_costs=COSTS,
     output:
@@ -806,6 +813,7 @@ if config["monte_carlo"]["options"].get("add_to_snakefile", False) == False:
             solving=config["solving"],
             augmented_line_connection=config["augmented_line_connection"],
         input:
+            overrides=BASE_DIR + "/data/override_component_attrs",
             network="networks/" + RDIR + "elec_s{simpl}_{clusters}_ec_l{ll}_{opts}.nc",
         output:
             "results/" + RDIR + "networks/elec_s{simpl}_{clusters}_ec_l{ll}_{opts}.nc",
@@ -872,6 +880,7 @@ if config["monte_carlo"]["options"].get("add_to_snakefile", False) == True:
             solving=config["solving"],
             augmented_line_connection=config["augmented_line_connection"],
         input:
+            overrides=BASE_DIR + "/data/override_component_attrs",
             network="networks/"
             + RDIR
             + "elec_s{simpl}_{clusters}_ec_l{ll}_{opts}_{unc}.nc",
@@ -901,7 +910,7 @@ if config["monte_carlo"]["options"].get("add_to_snakefile", False) == True:
         resources:
             mem_mb=memory,
         shadow:
-            "shallow"
+            "copy-minimal" if os.name == "nt" else "shallow"
         script:
             "scripts/solve_network.py"
 
@@ -1141,7 +1150,7 @@ rule build_ship_profile:
 
 rule add_export:
     params:
-        gadm_level=config["sector"]["gadm_level"],
+        gadm_layer_id=config["build_shape_options"]["gadm_layer_id"],
         alternative_clustering=config["cluster_options"]["alternative_clustering"],
         store=config["export"]["store"],
         store_capital_costs=config["export"]["store_capital_costs"],
@@ -1615,7 +1624,7 @@ if config["foresight"] == "overnight":
             solving=config["solving"],
             augmented_line_connection=config["augmented_line_connection"],
         input:
-            overrides="data/override_component_attrs",
+            overrides=BASE_DIR + "/data/override_component_attrs",
             # network=RESDIR
             # + "prenetworks/elec_s{simpl}_{clusters}_ec_l{ll}_{opts}_{sopts}_{planning_horizons}_{discountrate}.nc",
             network=RESDIR
@@ -1626,7 +1635,7 @@ if config["foresight"] == "overnight":
             RESDIR
             + "postnetworks/elec_s{simpl}_{clusters}_ec_l{ll}_{opts}_{sopts}_{planning_horizons}_{discountrate}_{demand}_{h2export}export.nc",
         shadow:
-            "shallow"
+            "copy-minimal" if os.name == "nt" else "shallow"
         log:
             solver=RESDIR
             + "logs/elec_s{simpl}_{clusters}_ec_l{ll}_{opts}_{sopts}_{planning_horizons}_{discountrate}_{demand}_{h2export}export_solver.log",
@@ -1824,7 +1833,7 @@ rule prepare_db:
 rule build_industrial_distribution_key:  #default data
     params:
         countries=config["countries"],
-        gadm_level=config["sector"]["gadm_level"],
+        gadm_layer_id=config["build_shape_options"]["gadm_layer_id"],
         alternative_clustering=config["cluster_options"]["alternative_clustering"],
         industry_database=config["custom_data"]["industry_database"],
     input:
@@ -2081,7 +2090,7 @@ if config["foresight"] == "myopic":
                 "co2_sequestration_potential", 200
             ),
         input:
-            overrides="data/override_component_attrs",
+            overrides=BASE_DIR + "/data/override_component_attrs",
             network=RESDIR
             + "prenetworks-brownfield/elec_s{simpl}_{clusters}_l{ll}_{opts}_{sopts}_{planning_horizons}_{discountrate}_{demand}_{h2export}export.nc",
             costs=CDIR + "costs_{planning_horizons}.csv",
@@ -2092,7 +2101,7 @@ if config["foresight"] == "myopic":
             # config=RESDIR
             # + "configs/config.elec_s{simpl}_{clusters}_ec_l{ll}_{opts}_{sopts}_{planning_horizons}_{discountrate}_{demand}_{h2export}export.yaml",
         shadow:
-            "shallow"
+            "copy-minimal" if os.name == "nt" else "shallow"
         log:
             solver=RESDIR
             + "logs/elec_s{simpl}_{clusters}_ec_l{ll}_{opts}_{sopts}_{planning_horizons}_{discountrate}_{demand}_{h2export}export_solver.log",

diff --git a/config.default.yaml b/config.default.yaml
@@ -98,6 +98,7 @@ cluster_options:
       p_nom_max: sum
       p_nom_min: sum
       p_min_pu: mean
+      p_max_pu: weighted_average
       marginal_cost: mean
       committable: any
       ramp_limit_up: max
@@ -602,6 +603,9 @@ sector:
   transmission_efficiency:
     electricity distribution grid:
       efficiency_static: 0.97 # efficiency of distribution grid (i.e. 3% loses)
+    H2 pipeline:
+      efficiency_per_1000km: 1
+      compression_per_1000km: 0.017 # DEA technology data. Mean of  Energy losses, lines 5000-20000 MW and lines >20000 MW for 2020, 2030 and 2050, [%/1000 km]
 
   dynamic_transport:
     enable: false # If "True", then the BEV and FCEV shares are obtained depending on the "Co2L"-wildcard (e.g. "Co2L0.70: 0.10"). If "False", then the shares are obtained depending on the "demand" wildcard and "planning_horizons" wildcard as listed below (e.g. "DF_2050: 0.08")
@@ -657,7 +661,6 @@ sector:
   co2_network: true
   co2_sequestration_potential: 200 #MtCO2/a sequestration potential for Europe
   co2_sequestration_cost: 10 #EUR/tCO2 for sequestration of CO2
-  hydrogen_underground_storage: true
   shipping_hydrogen_liquefaction: false
   shipping_average_efficiency: 0.4 #For conversion of fuel oil to propulsion in 2011
 
@@ -672,7 +675,6 @@ sector:
     NZ_2050: 0.36
     DF_2050: 0.12
 
-  gadm_level: 1
   h2_cavern: true
   marginal_cost_storage: 0
   methanation: true
@@ -695,7 +697,6 @@ sector:
     biomass: biomass
   keep_existing_capacities: true
 
-
 solving:
   options:
     formulation: kirchhoff

diff --git a/doc/index.rst b/doc/index.rst
@@ -78,39 +78,19 @@ The `website <https://pypsa-meets-earth.github.io/>`_ provides more context of t
 Get Involved
 ==============
 
-The PyPSA meets Earth team is currently running four types of meetings:
-
-
-- `**Discord NEW! (Open)** <https://discord.gg/AnuJBk23FU>`_
-
-  - Chat with the community, team up on features, exchange with developers, code in voice channels
-
-- **General code meeting (Open)**
-
-  - every forth Thursday each month 16-17:00 (UK time) `download .ics file <https://drive.google.com/file/d/1naH4WwW9drkOkOJ3PLO4fyWdkZQi5-_w/view?usp=share_link>`_
-  - updates on overall project and code blocks
-  - meeting hosted on `Discord <https://discord.gg/AnuJBk23FU>`_; join us, we are waiting for you!
-  - `open agenda <https://docs.google.com/document/d/1r6wm2RBe0DWFngmItpFfSFHA-CnUmVcVTkIKmthdW3g/edit?usp=sharing>`_. See what we will discuss. Invited members have edit rights.
-
-- **Specific code meeting (by invitation)**
-
-  - meeting hosted on Discord
-  - join updates, demos, Q&A's, discussions and the coordination of each work package
-
-    1. Demand creation and prediction meeting, on demand basis
-    2. AI asset detection meeting, on demand basis
-    3. Sector coupling meeting, every Thursday 09:00 (UK time), `download .ics file <https://drive.google.com/file/d/1PDdmjsKhzyGRo0_YrP4wPQkn2XTNh6jA/view?usp=share_link>`__
-    4. PyPSA-Earth meeting, every Thursday 16:00 (UK time), `download .ics file <https://drive.google.com/file/d/1gaLmyV4qGPXsogkeRcAPWjC0ESebUxU-/view?usp=share_link>`__
-
-- **Outreach meeting (by invitation)**
-
-  - every second week
-  - planning, discussing events, workshops, communication, community activities
-
-- **Buddy talk (Open)**
-
-  - book a 30min meeting with Max to discuss anything you like
-  - booking link: `calendly.com/pypsa-meets-earth <https://calendly.com/max-parzen/pypsa-meets-earth-exchange-30min>`_
+There are multiple ways to get involved and learn more about our work:
+    1. **Join our forum** and communication platform on `PyPSA-meets-Earth <https://discord.gg/AnuJBk23FU>`__ Discord Server
+    2. **Chat on Discord with us** in the following meetings:
+        - General initiative meeting for project news and `high-level code updates <https://docs.google.com/document/d/1r6wm2RBe0DWFngmItpFfSFHA-CnUmVcVTkIKmthdW3g/edit?usp=sharing>`__. Held every `fourth Thursday 16-17:00 (UK time) <https://drive.google.com/file/d/1naH4WwW9drkOkOJ3PLO4fyWdkZQi5-_w/view?usp=share_link>`__ and is a perfect place to meet the community and get a high-level update on PyPSA ecosystem relevant for PyPSA-Earth developments.
+        - Weekly developers meetings
+            - Eastern-Hemisphere friendly *Morning meeting* every `Thursday at 09:00 (UK time) <https://drive.google.com/file/d/1PDdmjsKhzyGRo0_YrP4wPQkn2XTNh6jA/view?usp=share_link>`__.
+            - Western-Hemisphere friendly *Evening meeting* every `Thursday 16:00 (UK time) <https://drive.google.com/file/d/1gaLmyV4qGPXsogkeRcAPWjC0ESebUxU-/view?usp=share_link>`__. Every forth Thursday is replaced by the General initiative meeting which has a more high-level perspective, but you can also join to discuss more particular questions.
+    3. **Look at public materials** at `google Drive <https://drive.google.com/drive/folders/13Z8Y9zgsh5IZaDNkkRyo1wkoMgbdUxT5?usp=sharing>`__ to share to minutes, presentations, lists and documents. Feel gree to get a look!
+    4. **Notify your interest** to on-demand meetings:
+        - Demand creation and prediction meeting
+        - AI asset detection meeting
+        - Outreach meeting for planning, discussing events, workshops, communication, community activities
+    5. Join us and **propose your stream**.
 
 =============
 Documentation

diff --git a/doc/release_notes.rst b/doc/release_notes.rst
@@ -17,12 +17,24 @@ This part of documentation collects descriptive release notes to capture the mai
 
 * In alternative clustering, generate hydro inflows by shape and avoid hydro inflows duplication for plants installed in the same node `PR #1120 <https://github.com/pypsa-meets-earth/pypsa-earth/pull/1120>`
 
+* Add a function to calculate length-based efficiencies and apply it to the H2 pipelines. `PR #1192 <https://github.com/pypsa-meets-earth/pypsa-earth/pull/1192>`__
+
+* Support of Linopy for Power and Sector-Coupled Modelling and latest PyPSA version `PR #1172 <https://github.com/pypsa-meets-earth/pypsa-earth/pull/1172>`__
+
+* Update workflow to geopandas >= 1.0 `PR #1276 <https://github.com/pypsa-meets-earth/pypsa-earth/pull/1276>`__
+
 **Minor Changes and bug-fixing**
 
 * Prevent computation of powerplantmatching if replace option is selected for custom_powerplants `PR #1281 <https://github.com/pypsa-meets-earth/pypsa-earth/pull/1281>`__
 
 * Fix overlapping bus regions when alternative clustering is selected `PR #1287 <https://github.com/pypsa-meets-earth/pypsa-earth/pull/1287>`__
 
+* Fix readthedocs by explicitly specifying the location of the Sphinx config `PR #1292 <https://github.com/pypsa-meets-earth/pypsa-earth/pull/1292>`__
+
+* Fix lossy bidirectional links, especially H2 pipelines, which would sometimes gain H2 instead of losing it.  `PR #1192 <https://github.com/pypsa-meets-earth/pypsa-earth/pull/1192>`__
+
+* Fix the need for administrative rights on Windows by changing all shadow directory settings for Windows in the Snakefile `PR #1295 <https://github.com/pypsa-meets-earth/pypsa-earth/pull/1295>`__ and  `PR #1301 <https://github.com/pypsa-meets-earth/pypsa-earth/pull/1301>`__
+
 PyPSA-Earth 0.6.0
 =================
 
@@ -148,6 +160,7 @@ PyPSA-Earth 0.4.0
 
 * Add an option to use csv format for custom demand imports. `PR #995 <https://github.com/pypsa-meets-earth/pypsa-earth/pull/995>`__
 
+
 **Minor Changes and bug-fixing**
 
 * Minor bug-fixing to run the cluster wildcard min `PR #1019 <https://github.com/pypsa-meets-earth/pypsa-earth/pull/1019>`__

diff --git a/envs/environment.yaml b/envs/environment.yaml
@@ -12,7 +12,8 @@ dependencies:
 - pip
 - mamba   # esp for windows build
 
-- pypsa>=0.24, <0.25
+- pypsa>=0.25
+# - atlite>=0.2.4  # until https://github.com/PyPSA/atlite/issues/244 is not merged
 - dask
 # currently the packages are being installed with pip
 # need to move back to conda once the issues will be resolved
@@ -28,13 +29,14 @@ dependencies:
 - memory_profiler
 - ruamel.yaml<=0.17.26
 - pytables
+- pyscipopt # added to compy with the quadratic objective requirement of the clustering script
 - lxml
 - numpy
 # starting from 1.3.5 numpoly requires numpy>2.0 which leads to issues
 - numpoly<=1.3.4
 - pandas
-- geopandas>=0.11.0, <=0.14.3
-- fiona!=1.8.22, <1.10.0
+- geopandas>=1
+- fiona>=1.10
 - xarray>=2023.11.0, <2023.12.0
 - netcdf4
 - networkx
@@ -49,6 +51,7 @@ dependencies:
 - pyogrio
 - numba
 - py7zr
+- tsam>=1.1.0
 
   # Keep in conda environment when calling ipython
 - ipython
@@ -80,11 +83,9 @@ dependencies:
 
 # Default solver for tests (required for CI)
 - glpk
-- ipopt
 - gurobi
 
 - pip:
   - git+https://github.com/davide-f/google-drive-downloader@master  # google drive with fix for virus scan
-  - tsam>=1.1.0
   - chaospy  # lastest version only available on pip
   - fake_useragent
-Original file line number
+Diff line change
@@ Expand Up / @@ -4,6 +4,9 @@ @@
     version: 2
+    sphinx:
+      configuration: doc/conf.py
     build:
       os: ubuntu-22.04
       tools:
@@ Expand Down @@