A dashboard for the monitoring of energy metrics and such as price, demand, or carbon production. See how much of your electricity is generated by nuclear fission, check the cheapest time to charge your electric campervan, or just have a curious glance at what's going on in the grid we all depend upon.
-
MacOS
-
Python 3.12
-
Amazon AWS (with an access key and secret key)
-
Brew (to install Terraform)
-
For badges and reports to be generated you will also need a CodeCov and CodeClimate account. These are free and will need to be linked to your own git repo using their respective browser-based GUIs.
! We'll add some visuals here maybe, using Asciinema or using ttygif and a recording. !
Clone the repo from github.
In terminal, navigate to the root directory of the repository energy_monitor.
Run the following commands:
python3 -m venv .venvbash setup.sh! we might want a simple setup script !source .venv/bin/activateadd_root_to_path.shpip3 install requirements.txt
You then need to setup files containing your personal secrets.
Create a .env file in the root directory with the following:
AWS_ACCESS_KEY=myaccesskey
AWS_SECRET_KEY=mysecretkey
AWS_REGION="my-region-#"
Create a terraform.tfvars file in the root directory with the following:
AWS_REGION = "my-region-#"
AWS_ACCESS_KEY = "myaccesskey"
AWS_SECRET_KEY = "mysecretkey"
DB_HOST = "energy-monitor.????????.my-region-#.rds.amazonaws.com"
DB_PORT = "5432"
DB_PASSWORD = "mypassword"
DB_USER = "myusername"
DB_NAME = "mydatabasename"
To then build the infrastructure navigate into infrastructure/ run terraform plan then terraform apply.
Once you have done this, you should be able to deploy the modules to the cloud using bash build_and_push.sh.
For workflows (used by tests and badges) to run you will need to add to your repo's secrets. There are found in the repo's settings, first select 'Secrets and variables'. Under 'actions', 'secrets' select 'New repository secrets'. Add the following:
- AWS_ACCESS_KEY (as above)
- AWS_REGION (as above)
- AWS_SECRET_KEY (as above)
- ECR_URL (the URL to your AWS Elastic Container Registry)
- CODECOV_TOKEN (From the CodeCov website once you have created an account)
- CC_TEST_REPORTER_ID (From the CodeClimate website once you have created an account)
The names must match perfectly as the code is configured to search for these exact names.
Energy production, demand, and cost data is sources from Elexions. Carbon tracking data is taken from carbon_intensity.org.
Our Extract, Transform, Load [ETL] pipleine works in the following way; firstly, Extract scripts for each API endpoint run regularly on Amazon Web Service [AWS] Lambdas requesting the latest data. Lambdas are quick to fire up and shut down, making them the ideal choice to quickly execute lightweight, intermittent tasks. This fresh data is then transferred to an AWS 's3 bucket', offering a quick and cost-effective storage. Some modest data curation is done by a Transform script, and a Load script loads the clean data into a PostGres SQL database on the AWS. For a full-scale and not a proof-of-concept project, we would ideally use an Amazon RedShift database which better supports read-heavy operations. In our limited instance, the inexpensive PostGres database is the best solution.
Another script runs our StreamLit dashboard. StreamLit is an open-source dashboard solution that is easy to work with and lightweight to run. The dashboard, running perpetually as a service on an EC2 instance, reads data from our database and plots it in a pleasing set of graphs.
Where could we go next?
- Automatic location detection
- Switch to RedShift
- Tariffs of ‘big 6’ energy suppliers (Price Comparison)
- News feed of energy-relevant articles
- Smart meter integration
Elexon offers a lot more data that we could integrate, and energy suppliers like Octopus offer detailed insight which might allow users to add their personal energy usage to the dashboard.
If you are having trouble using our app please open a ticket we'll get back to you as soon as possible.
If people have pushed to this repo, they are authors.
Elexon License requires the following statement:
Contains BMRS data © Exelon Limited copyright and database right 2024.
MIT License
Copyright (c) 2024 J J Marden
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
Testing Coverage Visualised (CodeCov)
