Skip to content

Commit

Permalink
Merge branch 'main' into testing
Browse files Browse the repository at this point in the history
  • Loading branch information
dan-mm committed Feb 15, 2024
2 parents 988236b + 32ea8e1 commit 4215d54
Show file tree
Hide file tree
Showing 10 changed files with 69 additions and 36 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ on:
- main
schedule:
# only run once a week to show the action is working and preserve as much energy as possible
# Reason being that we pull our ML model and this could have changed in the meantim
# Reason being that we pull our ML model and this could have changed in the meantime
- cron: '22 4 * * 6'
workflow_dispatch:

Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/update-tag.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ jobs:
- name: Git config
run: |
git config user.name eco-ci
git config user.email eco-ci@green-coding.berlin
git config user.email eco-ci@green-coding.io
- name: Update tag
run: |
Expand Down
2 changes: 1 addition & 1 deletion .gitlab-ci.yml.example
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
image: ubuntu:22.04
include:
remote: 'https://raw.githubusercontent.com/green-coding-berlin/eco-ci-energy-estimation/main/eco-ci-gitlab.yml'
remote: 'https://raw.githubusercontent.com/green-coding-solutions/eco-ci-energy-estimation/main/eco-ci-gitlab.yml'

stages:
- test
Expand Down
35 changes: 19 additions & 16 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Initialize Energy Estimation
uses: green-coding-berlin/eco-ci-energy-estimation@v2 # use hash or @vX here (See note below)
uses: green-coding-solutions/eco-ci-energy-estimation@v2 # use hash or @vX here (See note below)
with:
task: start-measurement

Expand All @@ -37,7 +37,7 @@ jobs:
submodules: 'true'

- name: Checkout Repo Measurement
uses: green-coding-berlin/eco-ci-energy-estimation@v2 # use hash or @vX here (See note below)
uses: green-coding-solutions/eco-ci-energy-estimation@v2 # use hash or @vX here (See note below)
with:
task: get-measurement
label: 'repo checkout'
Expand All @@ -54,7 +54,7 @@ jobs:
pip install -r requirements.txt
- name: Setup Python Measurment
uses: green-coding-berlin/eco-ci-energy-estimation@v2 # use hash or @vX here (See note below)
uses: green-coding-solutions/eco-ci-energy-estimation@v2 # use hash or @vX here (See note below)
with:
task: get-measurement
label: 'python setup'
Expand All @@ -65,13 +65,13 @@ jobs:
pytest
- name: Tests measurement
uses: green-coding-berlin/eco-ci-energy-estimation@v2 # use hash or @vX here (See note below)
uses: green-coding-solutions/eco-ci-energy-estimation@v2 # use hash or @vX here (See note below)
with:
task: get-measurement
label: 'pytest'

- name: Show Energy Results
uses: green-coding-berlin/eco-ci-energy-estimation@v2 # use hash or @vX here (See note below)
uses: green-coding-solutions/eco-ci-energy-estimation@v2 # use hash or @vX here (See note below)
with:
task: display-results
```
Expand All @@ -89,7 +89,7 @@ jobs:
- `label`: (optional) (default: 'measurement ##')
- Used with `get_measurement` and `display_results` to identify the measurement
- `send-data`: (optional) (default: true)
- Send metrics data to metrics.green-coding.berlin to create and display badge, and see an overview of the energy of your CI runs. Set to false to send no data. The data we send are: the energy value and duration of measurement; cpu model; repository name/branch/workflow_id/run_id; commit_hash; source (GitHub or GitLab). We use this data to display in our green-metrics-tool front-end here: https://metrics.green-coding.berlin/ci-index.html
- Send metrics data to metrics.green-coding.io to create and display badge, and see an overview of the energy of your CI runs. Set to false to send no data. The data we send are: the energy value and duration of measurement; cpu model; repository name/branch/workflow_id/run_id; commit_hash; source (GitHub or GitLab). We use this data to display in our green-metrics-tool front-end here: https://metrics.green-coding.io/ci-index.html
- `display-table`: (optional) (default: true)
- call during the `display-graph` step to either show/hide the energy reading table results in the output
- `display-graph`: (optional) (default: true)
Expand All @@ -102,6 +102,9 @@ jobs:
- used with display-results
- if on, will post a comment on the PR issue with the Eco-CI results. only occurs if the triggering event is a pull_request
- remember to set `pull-requests: write` to true in your workflow file
- `api-base`: (optional) (default: 'api.github.com')
- Eco-CI uses the github api to post/edit PR comments
- set to github's default api, but can be changed if you are using github enterprise


#### Continuing on Errors
Expand All @@ -110,7 +113,7 @@ We recommend running our action with `continue-on-error:true`, as it is not crit

```yaml
- name: Eco CI Energy Estimation
uses: green-coding-berlin/eco-ci-energy-estimation@v2
uses: green-coding-solutions/eco-ci-energy-estimation@v2
with:
task: final-measurement
continue-on-error: true
Expand All @@ -131,7 +134,7 @@ Here is an example demonstrating how this can be achieved:
submodules: 'true'
- name: Checkout Repo Measurment
uses: green-coding-berlin/eco-ci-energy-estimation@v2
uses: green-coding-solutions/eco-ci-energy-estimation@v2
id: checkout-step
with:
task: get-measurement
Expand All @@ -142,7 +145,7 @@ Here is an example demonstrating how this can be achieved:
echo "total json: ${{ steps.checkout-step.outputs.data-lap-json }}"
- name: Show Energy Results
uses: green-coding-berlin/eco-ci-energy-estimation@v2
uses: green-coding-solutions/eco-ci-energy-estimation@v2
id: total-measurement-step
with:
task: display-results
Expand All @@ -164,7 +167,7 @@ jobs:
actions: read
steps:
- name: Eco CI - Initialize
uses: green-coding-berlin/eco-ci-energy-estimation@v2
uses: green-coding-solutions/eco-ci-energy-estimation@v2
with:
task: start-measurement
```
Expand All @@ -173,7 +176,7 @@ jobs:
To use Eco-CI in your GitLab pipeline, you must first include a reference to the eco-ci-gitlab.yml file as such:
```
include:
remote: 'https://raw.githubusercontent.com/green-coding-berlin/eco-ci-energy-estimation/main/eco-ci-gitlab.yml'
remote: 'https://raw.githubusercontent.com/green-coding-solutions/eco-ci-energy-estimation/main/eco-ci-gitlab.yml'
```

and you call the various scripts in your pipeline with call like this:
Expand All @@ -186,7 +189,7 @@ where function name is one of the following:
`get_measurement` - make a spot measurment here. If you wish to label the measurement, you need to set the ECO_CI_LABEL environment variable right before this call.
`display_results` - will print all the measurement values to the jobs-output and prepare the artifacts, which must be exported in the normal GitLab way.

By default, we send data to our API, which will allow us to present you with a badge, and a front-end display to review your results. The data we send are: the energy value and duration of measurement; cpu model; repository name/branch/workflow_id/run_id; commit_hash; source (GitHub or GitLab). We use this data to display in our green-metrics-tool front-end here: https://metrics.green-coding.berlin/ci-index.html
By default, we send data to our API, which will allow us to present you with a badge, and a front-end display to review your results. The data we send are: the energy value and duration of measurement; cpu model; repository name/branch/workflow_id/run_id; commit_hash; source (GitHub or GitLab). We use this data to display in our green-metrics-tool front-end here: https://metrics.green-coding.io/ci-index.html

If you do not wish to send us data, you can set this global variable in your pipeline:

Expand All @@ -211,7 +214,7 @@ Here is a sample .gitlab-ci.yml example file to illustrate:
```
image: ubuntu:22.04
include:
remote: 'https://raw.githubusercontent.com/green-coding-berlin/eco-ci-energy-estimation/main/eco-ci-gitlab.yml'
remote: 'https://raw.githubusercontent.com/green-coding-solutions/eco-ci-energy-estimation/main/eco-ci-gitlab.yml'
stages:
- test
Expand Down Expand Up @@ -241,7 +244,7 @@ test-job:


### How does it work?
- The Eco-CI at its core makes its energy estimations based on an XGBoost Machine Learning model we have created based on the SpecPower database. The model and further information can be found here: https://github.com/green-coding-berlin/spec-power-model
- The Eco-CI at its core makes its energy estimations based on an XGBoost Machine Learning model we have created based on the SpecPower database. The model and further information can be found here: https://github.com/green-coding-solutions/spec-power-model
- When you initialize the Eco-CI, it downloads the XGBoost model onto the machine, as well as a small program to track the cpu utilization over a period of time. This tracking begins when you call the start_measurement function. Then, each time you call get-measurement, it will take the cpu-utilization data collected (either from the start, or since the last get-measurement call) and make an energy estimation based on the detected hardware (mainly cpu data) and utilization.

### Limitations
Expand All @@ -255,10 +258,10 @@ test-job:

### Note on the integration
- If you use dependabot and want to get updates, we recommend using the hash notation
+ `uses: green-coding-berlin/eco-ci-energy-estimation@06837b0b3b393a04d055979e1305852bda82f044 #v2.2`
+ `uses: green-coding-solutions/eco-ci-energy-estimation@06837b0b3b393a04d055979e1305852bda82f044 #v2.2`
+ Note that this hash is just an example. You find the latest current hash under *Tags*

- If you want the extension to automatically update within a version number, use the convenient @v2 form
+ `uses: green-coding-berlin/eco-ci-energy-estimation@v2 # will pick the latest minor v2. for example v2.2`
+ `uses: green-coding-solutions/eco-ci-energy-estimation@v2 # will pick the latest minor v2. for example v2.2`


32 changes: 26 additions & 6 deletions action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ inputs:
default: null
required: false
send-data:
description: 'Send metrics data to metrics.green-coding.berlin to create and display badge, and see an overview of the energy of your CI runs. Set to false to send no data.'
description: 'Send metrics data to metrics.green-coding.io to create and display badge, and see an overview of the energy of your CI runs. Set to false to send no data.'
default: true
required: false
display-table:
Expand All @@ -31,6 +31,10 @@ inputs:
pr-comment:
description: 'Add a comment to the PR with the results during display-results step'
default: false
api-base:
description: 'Base URL of the Github API to send data to. Default is api.github.com, but can be changed to your hostname if you have Github Enterprise'
default: 'api.github.com'
required: false
outputs:
data-total-json:
description: "Contains the data of the total measurement which is retrieved by the 'display-results' task."
Expand Down Expand Up @@ -152,18 +156,34 @@ runs:
total_data_file="/tmp/eco-ci/total-data.json"
echo "data-total-json=$(cat $total_data_file)" >> $GITHUB_OUTPUT
- if: github.event_name == 'pull_request' && inputs.task == 'display-results' && inputs.pr-comment == 'true'
name: Comment on Pull Request
- if: github.event_name == 'pull_request' && inputs.pr-comment == 'true'
name: Minimize Old Comment and Post New Comment
id: pr-comment
shell: bash
env:
PR_NUMBER: ${{ github.event.pull_request.number }}
run: |
COMMENT=$(cat "/tmp/eco-ci/output-pr.txt" | jq -Rs '.')
COMMENTS=$(curl -s -H "Authorization: Bearer ${{github.token}}" "https://${{inputs.api-base}}/repos/${{ github.repository }}/issues/$PR_NUMBER/comments")
API_URL="${{ github.api_url }}/repos/${{ github.repository }}/issues/${PR_NUMBER}/comments"
echo "$COMMENTS" | jq -c --arg username "github-actions[bot]" '.[] | select(.user.login == $username and (.body | index("Eco-CI") // false))' | while read -r comment; do
COMMENT_ID=$(echo "$comment" | jq -r '.id')
COMMENT_BODY=$(echo "$comment" | jq -r '.body')
INNER_BODY=$(echo "$COMMENT_BODY" | sed 's/<details><summary>Old Energy Estimation<\/summary>//g' | sed 's/<\/details>//g')
## indentation here is important, otherwise newlines are not properly sent/processed
PAYLOAD=$(jq --null-input --arg body "<details><summary>Old Energy Estimation</summary>
$INNER_BODY
</details>" '{"body": $body}')
curl -s -H "Authorization: Bearer ${{github.token}}" -X PATCH -d "$PAYLOAD" "https://${{inputs.api-base}}/repos/${{ github.repository }}/issues/comments/$COMMENT_ID"
echo "Comment $COMMENT_ID collapsed."
done
NEW_COMMENT=$(cat "/tmp/eco-ci/output-pr.txt" | jq -Rs '.')
API_URL="https://${{inputs.api-base}}/repos/${{ github.repository }}/issues/${PR_NUMBER}/comments"
curl -X POST -H "Authorization: Bearer ${{github.token}}" -d @- $API_URL <<EOF
{
"body": $COMMENT
"body": $NEW_COMMENT
}
EOF
8 changes: 2 additions & 6 deletions eco-ci-gitlab.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,6 @@ variables:
ECO_CI_DISPLAY_TABLE: "true" # true
ECO_CI_DISPLAY_GRAPH: "false" # false

cache:
paths:
- /tmp/eco-ci/venv/

.initialize_energy_estimator:
script:
- |
Expand All @@ -20,7 +16,7 @@ cache:
if [[ -d /tmp/eco-ci ]]; then
rm -rf /tmp/eco-ci
fi
git clone --depth 1 --single-branch --branch main https://github.com/green-coding-berlin/eco-ci-energy-estimation /tmp/eco-ci/main
git clone --depth 1 --single-branch --branch main https://github.com/green-coding-solutions/eco-ci-energy-estimation /tmp/eco-ci/main
/tmp/eco-ci/main/scripts/setup.sh initialize -r false -g false
/tmp/eco-ci/main/scripts/setup.sh setup_python
Expand Down Expand Up @@ -63,4 +59,4 @@ cache:
-sd $ECO_CI_SEND_DATA \
-s "gitlab" \
- echo -e "$FORMAT_CLR$(cat /tmp/eco-ci/output.txt)$TXT_CLEAR"
- cp /tmp/eco-ci/output.txt ./eco-ci-output.txt
- cp /tmp/eco-ci/output.txt ./eco-ci-output.txt
3 changes: 2 additions & 1 deletion scripts/display_results.sh
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ function display_results {
if [[ $PREVIOUS_VENV != '' ]]; then
source $PREVIOUS_VENV/bin/activate
fi
max_measurement_number=1
fi

cpu_avg=$(awk '{ total += $1; count++ } END { print total/count }' /tmp/eco-ci/cpu-util-total.txt)
Expand Down Expand Up @@ -120,7 +121,7 @@ function display_results {

if [[ ${send_data} == 'true' && ${display_badge} == 'true' ]]; then
get_endpoint=$API_BASE"/v1/ci/measurement/get"
metrics_url="https://metrics.green-coding.berlin"
metrics_url="https://metrics.green-coding.io"

echo "Badge for your README.md:" >> $output
echo ' ```' >> $output
Expand Down
2 changes: 1 addition & 1 deletion scripts/make_measurement.sh
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ function make_measurement() {

if [[ $send_data == 'true' ]]; then
add_endpoint=$API_BASE"/v1/ci/measurement/add"
metrics_url="https://metrics.green-coding.berlin"
metrics_url="https://metrics.green-coding.io"

value=$total_energy
value_mJ=$(echo "$value*1000" | bc -l | cut -d '.' -f 1)
Expand Down
4 changes: 2 additions & 2 deletions scripts/setup.sh
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ function initialize {
mkdir /tmp/eco-ci
fi

git clone --depth 1 --single-branch --branch main https://github.com/green-coding-berlin/spec-power-model /tmp/eco-ci/spec-power-model
git clone --depth 1 --single-branch --branch main https://github.com/green-coding-solutions/spec-power-model /tmp/eco-ci/spec-power-model

## Reimplement ascii graph when we find a better library
# install go ascii
Expand Down Expand Up @@ -62,7 +62,7 @@ function start_measurement {
# call init_variables
source "$(dirname "$0")/vars.sh" cpu_vars

source "$(dirname "$0")/vars.sh" add_var API_BASE "https://api.green-coding.berlin"
source "$(dirname "$0")/vars.sh" add_var API_BASE "https://api.green-coding.io"
source "$(dirname "$0")/vars.sh" add_var INIT "DONE"

# start measurement
Expand Down
15 changes: 14 additions & 1 deletion scripts/vars.sh
Original file line number Diff line number Diff line change
Expand Up @@ -119,8 +119,21 @@ function cpu_vars_fill {
add_var "CPU_CHIPS" 1;
add_var "VHOST_RATIO" $(echo "2/52" | bc -l);

elif [[ "$model_name" == *"AMD EPYC 7763"* ]]; then
echo "Found EPYC 7763 model";
add_var "MODEL_NAME" "EPYC_7763";

add_var "TDP" 280;
add_var "CPU_THREADS" 128;
add_var "CPU_CORES" 64;
add_var "CPU_MAKE" "amd";
add_var "RELEASE_YEAR" 2021;
add_var "RAM" 512;
add_var "CPU_FREQ" 2450;
add_var "CPU_CHIPS" 1;
add_var "VHOST_RATIO" $(echo "4/128" | bc -l);

# gitlab uses this one
# double check these values with someone
elif [[ "$model_name" == *"AMD EPYC 7B12"* ]]; then
echo "Found EPYC 7B12 model"
add_var "MODEL_NAME" "EPYC_7B12";
Expand Down

0 comments on commit 4215d54

Please sign in to comment.