Clover hpc to merge #27

sheridanfew · 2021-03-11T16:20:59Z

Proposal to merge hpc-enabling changes back into the master branch as discussed

…oblems with some compilers. Clean up anomylous naming and spacing. This is in order to allow automatic editing of scripts matching given patterns in later changes. Used the following commands to achieve this in shell terminal in CLOVER directory: # Clean up inconsistency in type of inverted commas. This has caused problems with some compilers. SRC="‘" DST="'" sed -i "" -e "s/$SRC/$DST/g" Scripts/*/*py SRC="’" DST="'" sed -i "" -e "s/$SRC/$DST/g" Scripts/*/*py # Clean up anomylous naming and spacing. This is in order to allow automatic editing of scripts matching given patterns in later changes. sed -i "" -e "s/location_input_data/location_inputs/g" Scripts/*/*py sed -i "" -e "s/_inputs = pd.read_csv/_inputs = pd.read_csv/g" Scripts/*/*py sed -i "" -e "s/_inputs = pd.read_csv/_inputs = pd.read_csv/g" Scripts/*/*py

Reconfigure to take '.' as filepath - necessary for HPC where CLOVER is copied to the node before processing. (May wish to amend if CLOVER dir gets very big) Commands used: SRC="\/\*\*\*YOUR LOCAL FILE PATH\*\*\*\/CLOVER 4.0" DST="." sed -i "" -e "s/$SRC/$DST/g" Scripts/*/*py SRC="\/\*\*\*YOUR LOCAL FILE PATH\*\*\*\/CLOVER" DST="." sed -i "" -e "s/$SRC/$DST/g" Scripts/*/*py # Reconfigure to take kwargs for location sed -i "" -e 's/__init__(self)/__init__(self,**kwargs)/g' Scripts/*/*py SRC="self.location = 'Bahraich'" DST="self.location = kwargs.get('location')" sed -i "" -e "s/$SRC/$DST/g" Scripts/*/*py SRC="self.location = ‘Bahraich’" # This is required too as there is some inconsistency in type of inverted commas used in original CLOVER scripts. DST="self.location = kwargs.get('location')" sed -i "" -e "s/$SRC/$DST/g" Scripts/*/*py

This reverts commit 599cc85.

# Reconfigure to take '.' as filepath - necessary for HPC where CLOVER is copied to the node before processing. (May wish to amend if CLOVER dir gets very big) Bash commands: SRC="\/\*\*\*YOUR LOCAL FILE PATH\*\*\*\/CLOVER 4.0" DST="." sed -i "" -e "s/$SRC/$DST/g" Scripts/*/*py SRC="\/\*\*\*YOUR LOCAL FILE PATH\*\*\*\/CLOVER" DST="." sed -i "" -e "s/$SRC/$DST/g" Scripts/*/*py

# Reconfigure to take kwargs for location sed -i "" -e 's/__init__(self)/__init__(self,**kwargs)/g' Scripts/*/*py SRC="self.location = 'Bahraich'" DST="self.location = kwargs.get('location')" sed -i "" -e "s/$SRC/$DST/g" Scripts/*/*py SRC="self.location = ‘Bahraich’" # This was required too as there was some inconsistency in type of inverted commas used in original CLOVER scripts - may have been fixed by earlier commands DST="self.location = kwargs.get('location')" sed -i "" -e "s/$SRC/$DST/g" Scripts/*/*py # Set up dependencies in taking kwargs for location sed -i "" -e "s/Diesel()\./Diesel(kwargs)./g" Scripts/*/*py sed -i "" -e "s/Grid()\./Grid(kwargs)./g" Scripts/*/*py sed -i "" -e "s/Solar()\./Solar(kwargs)./g" Scripts/*/*py sed -i "" -e "s/Finance()\./Finance(kwargs)./g" Scripts/*/*py sed -i "" -e "s/GHGs()\./GHGs(kwargs)./g" Scripts/*/*py sed -i "" -e "s/Load()\./Load(kwargs)./g" Scripts/*/*py sed -i "" -e "s/Energy_System()\./Energy_System(kwargs)./g" Scripts/*/*py

Replace spaces with underscores in filenames because spaces cause problems in file handling This stops github from recognising that files are the same, which is a pity! This will need to be resolved to merge hpc CLOVER with new features in main branch Bash commands: rename -s ' scripts' '_scripts' Scripts/*/ for file in Scripts/*/*py; do sed "s| scripts|_scripts|" $file > temp; mv temp $file; done

…s those specified in location) # Set up scripts to allow specification of inputs with kwargs (overrides those specified in location) bash commands used: for name in grid location finance GHG device optimisation energy_system scenario diesel do cat > ${name}_line.py << EOF # Replace input values with keywords if specified if kwargs.get('${name}_inputs'): for i in kwargs.get('${name}_inputs'): if not i[0] in self.${name}_inputs.index: print("Couldn't find entry",i[0],"in ${name}_inputs. Perhaps it's misspelt in kwargs? Printing list of possible variables and exiting.") print(self.${name}_inputs.index) exit(1) self.${name}_inputs[i[0]] = i[1] EOF for file in Scripts/*/*py do sed "/ self.${name}_inputs = pd.read_csv/r ${name}_line.py" $file > tmp mv tmp $file done rm ${name}_line.py done

Relating to specific projects - I think unlikely to be harmful, but may not be useful

Bhinjpur

Accomodate revised file structuring (keep Jobs in separate dir and repository from main CLOVER files

…sis script

Reduced number of devices for example run

Clean up and improve comments

…to CLOVER-hpc

New hpc users don't seem to have a work directory of the same format

Directory structure changed in 2018

Changed file format to remove \r newlines which were causing problems

- Moved launch file & added it's new home to the path in .bash_profile (readme instructions, could also add to setup file) - Corrected time between renewables ninja requests in Load_Solar_Bhinjpur.py

Hoping this will encourage these necessary folders to be recognised in upload & download of repo

Added indication of which kwargs can/must be specified per script (some way towards Ben's suggested safer way of specifying arguments) Removed some of my obsolete options (quick battery recharge, efficient diesel - introduced in the earlier "miscellaneous" commit): Left a few other additions in (which were also introduced in the earlier "miscellaneous" commit) : - In Optimisation.py: Additional metrics around storage performance reported: LCSE, Cumulative storage cost - In Load.py: optional arguments for different approaches to load generation - In Optimisation.py: optional argument to dump spare battery capacity at the end of a simulation period

… redundant script

sheridanfew · 2021-03-11T16:23:15Z

Not sure what the issue with the Load script is... Perhaps more info here when logged in as Phil?

BenWinchester · 2021-03-15T13:47:59Z

Not sure what the issue with the Load script is... Perhaps more info here when logged in as Phil?

Hi Sheridan, did you merge in the latest changes from the master branch? I think Phil can see what the merge conflicts are when he's logged in maybe 😄 .

BenWinchester

I added a few comments 😄 . There are lots of .csv files cropping up everywhere, do these need to be committed? There are also the solar files, which I thought needed to be generated by the user? I don't know whether it makes sense to commit these is all?

BenWinchester · 2021-03-15T13:50:15Z

HPC_setup.sh

@@ -0,0 +1,14 @@
+# coding: utf-8


Not a biggie, but adding a shebang to the start of the file will tell the terminal that it should execute the code in bash. It probably won't ever cause an issue, but is good practice for avoiding potential bash vs term etc. issues 😄
https://linuxize.com/post/bash-shebang/

BenWinchester · 2021-03-15T13:51:08Z

Launch_Script/launch_CLOVER_jobs.sh

@@ -0,0 +1,184 @@
+#!/bin/sh


For instance, this is the Shebang in this file which tells the terminal interpreter to use /bin/sh to execute the script.

BenWinchester · 2021-03-15T13:52:27Z

Load_Solar_Bhinjpur.py

@@ -0,0 +1,36 @@
+# coding: utf-8


You can also add shebangs to python files, though this isn't necessary provided they're executed with python my_script.py 😄
(For python, it would be /usr/bin/python3.7 or something similar. Using which python3 or which python in the command line/terminal will point you to the python installation that's used by default.)

BenWinchester · 2021-03-15T13:55:12Z

Load_Solar_Bhinjpur.py

+
+# Import scripts
+print('Importing Scripts...')
+exec(open("Scripts/Load_scripts/Load.py").read())


As the os module is imported, it might be good to use its inbuilt functionality to avoid issues when switching between Windows/Mac/Linux file systems in terms of how the / vs \ is used as the path separator.

os.path.join("Scripts", "Load_scripts", "Load.py")

will return "Scripts/Load_scripts/Load.py" on a linux system and "Scripts\Load_scripts\Load.py" if someone is using an operating system that uses backslashes 😄

BenWinchester · 2021-03-15T13:59:14Z

Scripts/Conversion_scripts/Conversion.py

@@ -19,7 +19,7 @@
 import scipy

 class Conversion():
-    def __init__(self):
+    def __init__(self,**kwargs):


It doesn't look like **kwargs is used in the Conversion class at all..? Using the location: str=None syntax for specifying keyword arguments picks up on things like this 😄

BenWinchester · 2021-03-15T14:08:01Z

Scripts/Impact_scripts/Finance.py

    def get_connections_expenditure(self,households,year=0):
        '''
        Function:
            Calculates cost of connecting households to the system
        Inputs:
-            households          DataFrame of households from Energy_System().simulation(...)
+            households          DataFrame of households from Energy_System(**self.kwargs).simulation(...)


Unneeded change? 😛

BenWinchester · 2021-03-15T14:08:53Z

Scripts/Simulation_scripts/Energy_System.py

+        # kwargs can include scenario_inputs, location_inputs, device_inputs, diesel_inputs, grid_inputs, finance_inputs, GHG_inputs (passed on to other scripts to override inputs in location files)
+        self.kwargs = kwargs
+        self.location = kwargs.get('location')
+        self.CLOVER_filepath = '.'


Here, the os.getcwd() would work well. Hard-coding in something like . shouldn't really be needed 😄

BenWinchester · 2021-03-15T14:10:51Z

Scripts/Simulation_scripts/Energy_System.py

            filename            Name of .csv file to be saved as (defaults to timestamp)
        Outputs:
            Simulation saved to .csv file
        """
+        if not os.path.exists(self.simulation_storage):


For this block, the os module provides a method that covers this case without needing an if statement:

os.makedirs(self.simulation_storage, exist_ok=True)

This will make the folder if it doesn't exist already, and won't throw an error, or overwrite the folder, if it already exists.

BenWinchester · 2021-03-15T14:11:19Z

Scripts/Simulation_scripts/Energy_System.py

@@ -359,7 +386,7 @@ def get_storage_profile(self, start_year = 0, end_year = 4, PV_size = 10, **opti

 #   Initialise power generation, including degradation of PV
        PV_generation = PV_size * pd.DataFrame(self.get_PV_generation()[start_hour:end_hour].values 
-                                     * Solar().solar_degradation()[0:(end_hour-start_hour)].values)
+                                     * Solar(**self.kwargs).solar_degradation()[0:(end_hour-start_hour)].values)


Not best sticking kwargs on the class as it's unclear what's stored in it and what's being passed through 🐱

BenWinchester · 2021-03-15T14:11:36Z

run_info.sh

@@ -0,0 +1,31 @@
+wd=$(pwd)


Shebang 😱

sheridanfew added 30 commits November 25, 2019 16:46

no message

9f17f6c

Revert "Reconfigure to take '.' as filepath"

a4a263f

This reverts commit 599cc85.

Add bash file used to generate CLOVER-hpc from (old) version of CLOVER

a154374

Miscellanious other changes

46ec6c9

Relating to specific projects - I think unlikely to be harmful, but may not be useful

Put in CLOVER-master dir

dbec171

Put example jobs in place

bba686f

Bhinjpur

Update README.md

f05a7b6

Update README.md

0f6c986

Update Location inputs.csv

ec06793

Update README.md

73c9f4f

Rearranging files & revising launch scripts

6285c3c

Delete Bhinjpur_LiB_cyc1000_cost1270_It2yrs_Opt.py

eb748e1

Update launch_CLOVER_job_multi_share.sh

b1985c0

Accomodate revised file structuring (keep Jobs in separate dir and repository from main CLOVER files

Remove Jobs (added to separate repo)

23dbe59

Update to launch scripts & readme, inclusion of simple run_info analy…

35e6b5d

…sis script

Update Devices.csv

5a68ef3

Reduced number of devices for example run

Update README.md

5e4d489

Update README.md

6c3b3a0

Update README.md

0d369a7

Update README.md

04846ce

Update README.md

ed3f71c

Update README.md

241ff90

Update launch_CLOVER_job_multi_share.sh

17c6385

Corrected launch script

055ab98

sheridanfew added 24 commits January 15, 2021 14:36

Update launch_CLOVER_job_multi_share.sh

9f4eb54

Clean up and improve comments

Update README.md

8091026

Update HPC_setup.sh

529ca8e

Merge branch 'CLOVER-hpc' of https://github.com/sheridanfew/CLOVER in…

c625f36

…to CLOVER-hpc

Change instructions & launch script from 'work' to 'home' dir

179ed42

New hpc users don't seem to have a work directory of the same format

Corrections to path for home directory

ec2e596

Directory structure changed in 2018

Update HPC_setup.sh

9cfd360

Update launch_CLOVER_job_multi_share.sh

c2c7627

Update HPC_setup.sh

4e796dd

Changed file format to remove \r newlines which were causing problems

Changes following training session

ae4a943

- Moved launch file & added it's new home to the path in .bash_profile (readme instructions, could also add to setup file) - Corrected time between renewables ninja requests in Load_Solar_Bhinjpur.py

Update README.md

d39dbe0

Update Load_Solar_Bhinjpur.py

0384f68

Remove old mess

38e7272

Create folder Saved optimisations

be4d31c

Add placeholder file in saved opt & sim

da4b089

Hoping this will encourage these necessary folders to be recognised in upload & download of repo

Make directory when saving opt/sim if they don't exist

13853ca

Update HPC_setup.sh

07e9664

Revert underscores to spaces to allow comparison with main clover branch

87820db

Merge branch 'master' into CLOVER-hpc_to_merge

829a9d2

Re-add Bahraich location, update readme, rename launch script, remove…

ebcbd62

… redundant script

Revert " " to "_" in paths to scripts

c930b09

Update UPDATE REGISTER.rtf

abaaa98

Update README.md

3f60b78

BenWinchester requested changes Mar 15, 2021

View reviewed changes

BenWinchester linked an issue Jul 1, 2021 that may be closed by this pull request

Add HPC functionality to CLOVER #32

Open

BenWinchester added enhancement New feature or request duplicate This issue or pull request already exists labels Jul 13, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Clover hpc to merge #27

Clover hpc to merge #27

sheridanfew commented Mar 11, 2021

sheridanfew commented Mar 11, 2021

BenWinchester commented Mar 15, 2021

BenWinchester left a comment

BenWinchester Mar 15, 2021

BenWinchester Mar 15, 2021

BenWinchester Mar 15, 2021

BenWinchester Mar 15, 2021

BenWinchester Mar 15, 2021

BenWinchester Mar 15, 2021

BenWinchester Mar 15, 2021

BenWinchester Mar 15, 2021

BenWinchester Mar 15, 2021

BenWinchester Mar 15, 2021

Clover hpc to merge #27

Are you sure you want to change the base?

Clover hpc to merge #27

Conversation

sheridanfew commented Mar 11, 2021

sheridanfew commented Mar 11, 2021

BenWinchester commented Mar 15, 2021

BenWinchester left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment