Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use submodule files. #87

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions .github/workflows/build_docs.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,9 @@ jobs:
os: [ubuntu-latest]
python-version: ['3.10']
steps:
- uses: actions/checkout@v3
- name: Set up Python ${{ matrix.python-version }}
- uses: actions/checkout@v4
with:
submodules: true
uses: actions/setup-python@v3
with:
python-version: ${{ matrix.python-version }}
Expand Down
10 changes: 3 additions & 7 deletions .github/workflows/deploy_to_pypi.yaml
Original file line number Diff line number Diff line change
@@ -1,30 +1,26 @@
name: Deploy to PyPI

on:
release:
types: [created]

jobs:
deploy:
runs-on: ubuntu-latest
steps:
- name: Check out code
uses: actions/checkout@v3

uses: actions/checkout@v4
with:
submodules: true
- name: Set up Python
uses: actions/setup-python@v3
with:
python-version: '3.10'

- name: Install dependencies
run: |
python -m pip install --upgrade pip
python -m pip install setuptools wheel twine

- name: Build package
run: |
python setup.py sdist bdist_wheel

- name: Publish package to PyPI
uses: pypa/gh-action-pypi-publish@release/v1
with:
Expand Down
10 changes: 3 additions & 7 deletions .github/workflows/docs_check.yaml
Original file line number Diff line number Diff line change
@@ -1,32 +1,28 @@
name: Check for Sphinx Warnings

on:
pull_request:
paths:
- "doc/**"
- "**/*.rst"
- ".github/workflows/docs_check.yaml"
- "setup.py"

jobs:
build:
runs-on: ubuntu-latest

steps:
- name: Check out the repository
uses: actions/checkout@v3

uses: actions/checkout@v4
with:
submodules: true
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: '3.x'

- name: Install dependencies
run: |
sudo apt-get update
sudo apt-get install -y pandoc
python -m pip install -e .[development]

- name: Check for Sphinx warnings
run: |
sphinx-build -M html ./doc/source ./doc/_build --fail-on-warning
2 changes: 2 additions & 0 deletions .github/workflows/format_check.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ jobs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
submodules: true
- uses: chartboost/ruff-action@v1
with:
args: 'format --check'
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/lint.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ jobs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
submodules: true
- uses: chartboost/ruff-action@v1
with:
version: 0.7.0
9 changes: 3 additions & 6 deletions .github/workflows/spell_check.yml
Original file line number Diff line number Diff line change
@@ -1,15 +1,12 @@
name: Spell Check

on: [push, pull_request]

jobs:
spell-check:
runs-on: ubuntu-latest

steps:
- name: Checkout code
uses: actions/checkout@v2

uses: actions/checkout@v4
with:
submodules: true
- name: Run codespell
uses: codespell-project/actions-codespell@v2

6 changes: 3 additions & 3 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
@@ -1,9 +1,7 @@
name: Tests

on:
- push
- pull_request

jobs:
test:
runs-on: ${{ matrix.os }}
Expand All @@ -12,7 +10,9 @@ jobs:
os: [ubuntu-latest, windows-latest, macos-latest]
python-version: ['3.9', '3.10', '3.11','3.12']
steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v4
with:
submodules: true
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v2
with:
Expand Down
10 changes: 8 additions & 2 deletions pelicun/assessment.py
Original file line number Diff line number Diff line change
Expand Up @@ -218,7 +218,10 @@ def get_default_data(self, data_name: str) -> pd.DataFrame:
'Please use `loss_repair_DB` instead.'
)

data_path = f'{base.pelicun_path}/resources/SimCenterDBDL/{data_name}.csv'
data_path = file_io.substitute_default_path(
[f'PelicunDefault/{data_name}.csv']
)[0]
assert isinstance(data_path, str)

data = file_io.load_data(
data_path, None, orientation=1, reindex=False, log=self.log
Expand Down Expand Up @@ -249,7 +252,10 @@ def get_default_metadata(self, data_name: str) -> dict:
'`fragility_DB` is deprecated and will be dropped in '
'future versions of pelicun. Please use `damage_DB` instead.'
)
data_path = f'{base.pelicun_path}/resources/SimCenterDBDL/{data_name}.json'
data_path = file_io.substitute_default_path(
[f'PelicunDefault/{data_name}.json']
)[0]
assert isinstance(data_path, str)

with Path(data_path).open(encoding='utf-8') as f:
data = json.load(f)
Expand Down
78 changes: 53 additions & 25 deletions pelicun/file_io.py
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@

from __future__ import annotations

import json
from pathlib import Path

import numpy as np
Expand Down Expand Up @@ -250,30 +251,33 @@ def substitute_default_path(
data_paths: list[str | pd.DataFrame],
) -> list[str | pd.DataFrame]:
"""
Substitute the default directory path with a specified path.
Substitute the default directory path.

This function iterates over a list of data paths and replaces
occurrences of the 'PelicunDefault/' substring with the path
specified by `base.pelicun_path` concatenated with
'/resources/SimCenterDBDL/'. This operation is performed to update
paths that are using a default location to a user-defined location
within the pelicun framework. The updated list of paths is then
returned.
occurrences of the 'PelicunDefault/' substring by looking up the
filename in a resource mapping file located at
`{base.pelicun_path}/resources/dlml_resource_paths.json`. If a
match is found, the file path is replaced with the value found in
the `resource_paths` dictionary. The updated list of paths is
then returned.

Parameters
----------
data_paths: list of str
data_paths: list of str or pd.DataFrame
A list containing the paths to data files. These paths may
include a placeholder directory 'PelicunDefault/' that needs
to be substituted with the actual path specified in
`base.pelicun_path`.
to be substituted with the actual path specified in the
resource mapping.

Returns
-------
list of str
The list with updated paths where 'PelicunDefault/' has been
replaced with the specified path in `base.pelicun_path`
concatenated with '/resources/SimCenterDBDL/'.
list of str or pd.DataFrame

Raises
------
KeyError
If the file after 'PelicunDefault/' does not exist in the
`resource_paths` keys.

Notes
-----
Expand All @@ -282,26 +286,50 @@ def substitute_default_path(
are located.
- If a path in the input list does not contain 'PelicunDefault/',
it is added to the output list unchanged.
- If the file after 'PelicunDefault/' does not exist in the
`resource_paths` keys, a `KeyError` is raised.

Examples
--------
>>> data_paths = ['PelicunDefault/data/file1.txt',
'data/file2.txt']
>>> data_paths = ['PelicunDefault/damage_DB_SimCenter_HU.csv', 'data/file2.txt']
>>> substitute_default_path(data_paths)
['{base.pelicun_path}/resources/SimCenterDBDL/data/file1.txt',
'data/file2.txt']
['{base.pelicun_path}/resources/DamageAndLossModelLibrary/'
'hurricane/building/component/fragility.csv',
'data/file2.txt']

"""
# Load the resource paths from the JSON file
resource_file_path = (
Path(base.pelicun_path) / 'resources' / 'dlml_resource_paths.json'
)
with resource_file_path.open('r') as file:
resource_paths = json.load(file)

updated_paths: list[str | pd.DataFrame] = []
for data_path in data_paths:
if isinstance(data_path, str) and 'PelicunDefault/' in data_path:
path = data_path.replace(
'PelicunDefault/',
f'{base.pelicun_path}/resources/SimCenterDBDL/',
for data_path_str in data_paths:
if isinstance(data_path_str, str) and 'PelicunDefault/' in data_path_str:
data_path = Path(data_path_str)
# Extract the filename after 'PelicunDefault/'
file_name = data_path.parts[-1]

# Check if the filename exists in the resource paths
# dictionary
if file_name not in resource_paths:
msg = f'File `{file_name}` not found in resource paths.'
raise KeyError(msg)

# Substitute the path with the corresponding value from
# the dictionary
updated_path = str(
Path(base.pelicun_path)
/ 'resources'
/ 'DamageAndLossModelLibrary'
/ resource_paths[file_name]
)
updated_paths.append(path)
updated_paths.append(updated_path)
else:
updated_paths.append(data_path)
updated_paths.append(data_path_str)

return updated_paths


Expand Down
2 changes: 1 addition & 1 deletion pelicun/model/damage_model.py
Original file line number Diff line number Diff line change
Expand Up @@ -1310,7 +1310,7 @@ def map_ds(values: np.ndarray, offset: int) -> np.ndarray:
else:
# if there are contents, ensure they are valid.
# See docstring for an example of what is expected.
parsed_scaling_specification = {}
parsed_scaling_specification: dict = {}
# validate contents
for key, value in scaling_specification.items():
css = 'capacity adjustment specification'
Expand Down
2 changes: 1 addition & 1 deletion pelicun/resources/DamageAndLossModelLibrary
Submodule DamageAndLossModelLibrary updated 57 files
+4 −0 .flake8
+47 −0 .github/workflows/build_docs.yaml
+36 −0 .github/workflows/format_and_lint.yaml
+5 −0 .gitignore
+572 −0 .pylintrc
+20 −0 doc/Makefile
+35 −0 doc/make.bat
+7 −0 doc/source/_extensions/compute_global_hash.py
+642 −0 doc/source/_extensions/generate_dl_doc.py
+40 −0 doc/source/_extensions/sphinx_generate_dl_doc.py
+1,114 −0 doc/source/_extensions/visuals.py
+33 −0 doc/source/_static/css/custom.css
+32 −0 doc/source/_templates/custom-class-template.rst
+63 −0 doc/source/_templates/custom-module-template.rst
+11 −0 doc/source/about/about.rst
+100 −0 doc/source/conf.py
+20 −0 doc/source/index.rst
+0 −0 doc/source/references.bib
+16 −0 doc/source/release_notes/index.rst
+10 −0 doc/source/release_notes/v1.0.rst
+819 −0 flood/building/portfolio/Hazus v6.1/consequence_repair.csv
+182 −0 flood/building/portfolio/Hazus v6.1/data_sources/generate_library_files.py
+25,821 −0 hurricane/building/portfolio/Hazus v4.2/consequence_repair_fitted.csv
+198 −0 hurricane/building/portfolio/Hazus v4.2/data_sources/damage_DB_SimCenter_Hazus_HU_bldg_template.json
+25,821 −0 hurricane/building/portfolio/Hazus v4.2/fragility_fitted.csv
+556 −0 hurricane/building/portfolio/Hazus v4.2/fragility_fitted.json
+11 −0 hurricane/building/portfolio/Hazus v5.1/combined_loss_matrices/Wind_Flood.csv
+1 −0 hurricane/building/portfolio/Hazus v5.1/combined_loss_matrices/note.md
+25,821 −0 hurricane/building/portfolio/Hazus v5.1/consequence_repair_fitted.csv
+535,265 −0 hurricane/building/portfolio/Hazus v5.1/consequence_repair_fitted.json
+2,011 −0 hurricane/building/portfolio/Hazus v5.1/data_sources/generate_library_files.py
+ hurricane/building/portfolio/Hazus v5.1/data_sources/input_files/huDamLossFun.xlsx
+ hurricane/building/portfolio/Hazus v5.1/data_sources/input_files/huDamLossFunEF.xlsx
+ hurricane/building/portfolio/Hazus v5.1/data_sources/input_files/huListOfWindBldgTypes.xlsx
+ hurricane/building/portfolio/Hazus v5.1/data_sources/input_files/huListOfWindBldgTypesEF.xlsx
+198 −0 hurricane/building/portfolio/Hazus v5.1/data_sources/input_files/metadata.json
+27,621 −0 hurricane/building/portfolio/Hazus v5.1/fragility_fitted.csv
+535,265 −0 hurricane/building/portfolio/Hazus v5.1/fragility_fitted.json
+27,621 −0 hurricane/building/portfolio/Hazus v5.1/fragility_unmodified.csv
+535,265 −0 hurricane/building/portfolio/Hazus v5.1/fragility_unmodified.json
+25,821 −0 hurricane/building/portfolio/Hazus v5.1/loss_repair_unmodified.csv
+129,205 −0 hurricane/building/portfolio/Hazus v5.1/loss_repair_unmodified.json
+2 −0 pyproject.toml
+13 −0 requirements.txt
+2 −2 seismic/building/component/FEMA P-58 2nd Edition/consequence_repair.json
+1,235 −0 seismic/building/component/FEMA P-58 2nd Edition/data_sources/generate_library_files.py
+2 −2 seismic/building/component/FEMA P-58 2nd Edition/data_sources/input_files/FEMA_P58_meta.json
+2 −2 seismic/building/component/FEMA P-58 2nd Edition/fragility.json
+871 −0 seismic/building/portfolio/Hazus v5.1/data_sources/generate_library_files.py
+113 −0 seismic/building/subassembly/Hazus v5.1/consequence_repair.csv
+1,781 −0 seismic/building/subassembly/Hazus v5.1/consequence_repair.json
+707 −0 seismic/building/subassembly/Hazus v5.1/data_sources/generate_library_files.py
+68 −0 seismic/building/subassembly/Hazus v5.1/fragility.csv
+2,053 −0 seismic/building/subassembly/Hazus v5.1/fragility.json
+2 −2 seismic/transportation_network/portfolio/Hazus v5.1/fragility.csv
+12 −2 seismic/transportation_network/portfolio/Hazus v5.1/fragility.json
+5 −0 seismic/water_network/portfolio/Hazus v6.1/fragility.csv
31 changes: 0 additions & 31 deletions pelicun/resources/SimCenterDBDL/README.md

This file was deleted.

This file was deleted.

This file was deleted.

Loading
Loading