Skip to content

Commit

Permalink
Version 0.5 release - Squashed commit of the following:
Browse files Browse the repository at this point in the history
commit b94d1cb
Author: Arne Tarara <[email protected]>
Date:   Tue Aug 16 16:47:45 2022 +0200

    Differential moved to AVG. and disabled unused channels for now

commit 5fcb537
Author: Arne Tarara <[email protected]>
Date:   Tue Aug 16 14:57:09 2022 +0200

    Introduced measurement time as s in overview

commit c4916d1
Author: Arne Tarara <[email protected]>
Date:   Tue Aug 16 10:02:57 2022 +0200

    setuid bit now used, so we can execute still with root rights but without calling sudo. chown for root in make. removed sudoers file usage; small fixes to printf error calls which were not on Stderr

commit 16ef1de
Author: Arne Tarara <[email protected]>
Date:   Mon Aug 15 19:43:09 2022 +0200

    Added cpu_utilization_procfs_system_provider

commit d44950f
Author: Arne Tarara <[email protected]>
Date:   Mon Aug 15 19:41:08 2022 +0200

    Renamed cpu_time_system provider to cpu_time_procfs_system

commit 2533083
Merge: 4676f71 805016e
Author: Arne Tarara <[email protected]>
Date:   Mon Aug 15 15:40:23 2022 +0200

    Merge branch 'main' into dev

    * main:
      Adding missing time and atx metrics to lookup
      Release 1.0: Squashed commit of the following:

commit 4676f71
Author: dan-mm <[email protected]>
Date:   Mon Aug 15 15:26:51 2022 +0200

    Bugfixes rapl c improvements (#19)

    * small refactor of code; use enum for measurement modes; moved all setup call outside of main measurement loop

    * get userid dynamically; makefile now chmod +s on static-binary

    * changed enum to #define for measurement mode

    * missed a variable during refactor

    * implemented user_id fix to the rest of the metric providers which use cgroups

    Co-authored-by: Dan M <[email protected]>

commit 79bb522
Author: Arne Tarara <[email protected]>
Date:   Sat Aug 13 14:37:22 2022 +0200

    Metrics have now dimension added in their chart displays

commit ab2dc17
Author: Arne Tarara <[email protected]>
Date:   Sat Aug 13 14:03:36 2022 +0200

    RAM reporter renamed to Memory

commit 66c6830
Author: Arne Tarara <[email protected]>
Date:   Sat Aug 13 13:55:40 2022 +0200

    Explanations added for network formulas and accounting

commit ae2e615
Author: Arne Tarara <[email protected]>
Date:   Sat Aug 13 13:55:23 2022 +0200

    Time reporters inactive by default

commit f590ec4
Author: Arne Tarara <[email protected]>
Date:   Sat Aug 13 12:37:00 2022 +0200

    Network IO was not correctly added up, but only last container was accounted for

commit 6a30388
Author: Arne Tarara <[email protected]>
Date:   Sat Aug 13 11:08:18 2022 +0200

    CLI is now very pretty

commit f483dd9
Author: Arne Tarara <[email protected]>
Date:   Fri Aug 12 20:23:57 2022 +0200

    Added all channels to importer

commit ab486a4
Author: Arne Tarara <[email protected]>
Date:   Fri Aug 12 13:35:08 2022 +0200

    DC converter now operates with mV and new measurement resistor of 0.005 W

commit e455ce1
Author: Arne Tarara <[email protected]>
Date:   Fri Aug 12 12:47:38 2022 +0200

    Added Watt power reading and renamed atx channel with DC added

commit ed9752e
Author: Arne Tarara <[email protected]>
Date:   Fri Aug 12 12:31:51 2022 +0200

    Moved dc_converter to mV

commit 448ea1e
Author: Arne Tarara <[email protected]>
Date:   Fri Aug 12 12:30:09 2022 +0200

    Clarifying comment

commit e25c273
Author: Arne Tarara <[email protected]>
Date:   Sun Aug 7 22:42:44 2022 +0200

    Removed obsolte TODOs and comments

commit a895f73
Author: Arne Tarara <[email protected]>
Date:   Sun Aug 7 22:41:22 2022 +0200

    Clarifying comment

commit a58c3eb
Author: Arne Tarara <[email protected]>
Date:   Sun Aug 7 15:12:47 2022 +0200

    Introduced typo

commit 4100461
Author: Arne Tarara <[email protected]>
Date:   Sun Aug 7 15:10:47 2022 +0200

    also renamed field to MB

commit 65a242c
Author: Arne Tarara <[email protected]>
Date:   Sun Aug 7 15:09:51 2022 +0200

    Network IO moved to MB

commit f7a5068
Author: Arne Tarara <[email protected]>
Date:   Sun Aug 7 15:09:01 2022 +0200

    ATX is now using channel as identifier

commit 573d199
Author: Arne Tarara <[email protected]>
Date:   Sun Aug 7 14:48:33 2022 +0200

    DC measurement converter and display added
  • Loading branch information
ArneTR committed Aug 17, 2022
1 parent 805016e commit 0372acd
Show file tree
Hide file tree
Showing 22 changed files with 378 additions and 165 deletions.
7 changes: 4 additions & 3 deletions config.yml.example
Original file line number Diff line number Diff line change
Expand Up @@ -23,12 +23,13 @@ measurement:
metric-providers:
cpu.utilization.cgroup.container.provider.CpuUtilizationCgroupContainerProvider: 100
cpu.energy.RAPL.MSR.system.provider.CpuEnergyRaplMsrSystemProvider: 100
memory.total.cgroup.container.provider.MemoryTotalCgroupContainerProvider: 100
cpu.utilization.procfs.system.provider.CpuUtilizationProcfsSystemProvider: 100
# cpu.time.cgroup.container.provider.CpuTimeCgroupContainerProvider: 100
# cpu.time.cgroup.system.provider.CpuTimeCgroupSystemProvider: 100
# cpu.time.system.provider.CpuTimeSystemProvider: 100
network.io.cgroup.container.provider.NetworkIoCgroupContainerProvider: 100
# cpu.time.procfs.system.provider.CpuTimeProcfsSystemProvider: 100
memory.total.cgroup.container.provider.MemoryTotalCgroupContainerProvider: 100
memory.energy.RAPL.MSR.system.provider.MemoryEnergyRaplMsrSystemProvider: 100
network.io.cgroup.container.provider.NetworkIoCgroupContainerProvider: 100

admin:
# This address will get an email, when a new project was added through the frontend
Expand Down
43 changes: 28 additions & 15 deletions frontend/js/stats.js
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,11 @@ const metrics_info = {
SI_conversion_factor: 100, // CPU comes as ratio, but since stored as integer is was multiplicated with 100
unit_after_conversion: '%'
},
cpu_utilization_procfs_system: {
unit: 'Ratio',
SI_conversion_factor: 100, // CPU comes as ratio, but since stored as integer is was multiplicated with 100
unit_after_conversion: '%'
},
cpu_energy_rapl_msr_system: {
unit: 'mJ',
SI_conversion_factor: 1000,
Expand Down Expand Up @@ -39,7 +44,7 @@ const metrics_info = {
SI_conversion_factor: 1,
unit_after_conversion: 'us'
},
cpu_time_system: {
cpu_time_procfs_system: {
unit: 'us',
SI_conversion_factor: 1,
unit_after_conversion: 'us'
Expand Down Expand Up @@ -122,7 +127,6 @@ const getEChartsOptions = () => {
}

const fillProjectData = (project, key = null) => {

for (item in project) {
if (item == 'machine_specs') {
fillProjectTab('#machine-specs', project[item])
Expand All @@ -132,6 +136,7 @@ const fillProjectData = (project, key = null) => {
} else if(item == 'measurement_config') {
fillProjectTab('#measurement-config', project[item])
} else {

document.querySelector('#project-data').insertAdjacentHTML('beforeend', `<tr><td><strong>${item}</strong></td><td>${project?.[item]}</td></tr>`)
}
}
Expand All @@ -148,7 +153,7 @@ const fillProjectTab = (selector, data) => {
}

const getMetrics = (stats_data, style='apex') => {
const metrics = {cpu_load: [], mem_total: [], network_io: {}, series: {}, atx_energy: 0, cpu_energy: 0, memory_energy: 0}
const metrics = {cpu_utilization_containers: [], cpu_utilization_system: [], mem_total: [], network_io: {}, series: {}, atx_energy: 0, cpu_energy: 0, memory_energy: 0}

let accumulate = 0;

Expand All @@ -172,7 +177,9 @@ const getMetrics = (stats_data, style='apex') => {
value = value / metrics_info[metric_name].SI_conversion_factor;

if (metric_name == 'cpu_utilization_cgroup_container') {
if (accumulate === 1) metrics.cpu_load.push(value);
if (accumulate === 1) metrics.cpu_utilization_containers.push(value);
} else if (metric_name == 'cpu_utilization_procfs_system') {
if (accumulate === 1) metrics.cpu_utilization_system.push(value);
} else if (metric_name == 'cpu_energy_rapl_msr_system') {
if (accumulate === 1) metrics.cpu_energy += value;
} else if (metric_name == 'atx_energy_dc_channel') {
Expand Down Expand Up @@ -338,9 +345,6 @@ const createGraph = (element, data, labels, title) => {

const fillAvgContainers = (stats_data, metrics) => {

// timestamp is in microseconds, therefore divide by 10**6
const measurement_duration_in_s = (stats_data.project.end_measurement - stats_data.project.start_measurement) / 1000000;

const atx_energy_in_mWh = ((metrics.atx_energy) / 3600) * 1000;
const cpu_energy_in_mWh = ((metrics.cpu_energy) / 3600) * 1000;
const memory_energy_in_mWh = ((metrics.memory_energy) / 3600) * 1000;
Expand All @@ -367,21 +371,25 @@ const fillAvgContainers = (stats_data, metrics) => {
if(memory_energy_in_mWh) document.querySelector("#memory-energy").innerText = memory_energy_in_mWh.toFixed(2) + " mWh"
if(cpu_energy_in_mWh) document.querySelector("#total-energy").innerText = (cpu_energy_in_mWh+memory_energy_in_mWh+network_io_in_mWh).toFixed(2) + " mWh"

if(cpu_energy_in_mWh) document.querySelector("#component-power").innerText = ((metrics.cpu_energy+metrics.memory_energy)/measurement_duration_in_s).toFixed(2) + " W"
if(atx_energy_in_mWh) document.querySelector("#atx-power").innerText = (metrics.atx_energy / measurement_duration_in_s).toFixed(2) + " W"
if(cpu_energy_in_mWh) document.querySelector("#component-power").innerText = ((metrics.cpu_energy+metrics.memory_energy)/stats_data.project.measurement_duration_in_s).toFixed(2) + " W"
if(atx_energy_in_mWh) document.querySelector("#atx-power").innerText = (metrics.atx_energy / stats_data.project.measurement_duration_in_s).toFixed(2) + " W"


if(network_io) document.querySelector("#network-io").innerText = network_io.toFixed(2) + " MB"
if(network_io_in_mWh) document.querySelector("#network-energy").innerHTML = network_io_in_mWh.toFixed(2) + " mWh"

if(co2_display.value) document.querySelector("#total-co2-internal").innerHTML = `${(co2_display.value).toFixed(2)} ${co2_display.unit}`
if(co2_budget_utilization) document.querySelector("#co2-budget-utilization").innerHTML = (co2_budget_utilization).toFixed(2) + " %"
if (co2_display.value) document.querySelector("#total-co2-internal").innerHTML = `${(co2_display.value).toFixed(2)} ${co2_display.unit}`
if (co2_budget_utilization) document.querySelector("#co2-budget-utilization").innerHTML = (co2_budget_utilization).toFixed(2) + " %"

if(metrics.cpu_load.length) {
document.querySelector("#max-cpu-load").innerText = (Math.max.apply(null, metrics.cpu_load)) + " %"
document.querySelector("#avg-cpu-load").innerText = ((metrics.cpu_load.reduce((a, b) => a + b, 0) / metrics.cpu_load.length)).toFixed(2) + " %"
if (metrics.cpu_utilization_containers.length) {
document.querySelector("#max-cpu-load-containers").innerText = (Math.max.apply(null, metrics.cpu_utilization_containers)) + " %"
document.querySelector("#avg-cpu-load-containers").innerText = ((metrics.cpu_utilization_containers.reduce((a, b) => a + b, 0) / metrics.cpu_utilization_containers.length)).toFixed(2) + " %"
}
if (metrics.cpu_utilization_system.length) {
document.querySelector("#max-cpu-load-system").innerText = (Math.max.apply(null, metrics.cpu_utilization_system)) + " %"
document.querySelector("#avg-cpu-load-system").innerText = ((metrics.cpu_utilization_system.reduce((a, b) => a + b, 0) / metrics.cpu_utilization_system.length)).toFixed(2) + " %"
}
if(metrics.mem_total.length) document.querySelector("#avg-mem-load").innerText = ((metrics.mem_total.reduce((a, b) => a + b, 0) / metrics.mem_total.length)).toFixed(2) + " MB"
if (metrics.mem_total.length) document.querySelector("#avg-mem-load").innerText = ((metrics.mem_total.reduce((a, b) => a + b, 0) / metrics.mem_total.length)).toFixed(2) + " MB"

upscaled_CO2_in_kg = total_CO2_in_kg * 100 * 30 ; // upscaled by 30 days for 10.000 requests (or runs) per day

Expand Down Expand Up @@ -411,6 +419,11 @@ $(document).ready( (e) => {
$('.ui.secondary.menu .item').tab();

const metrics = getMetrics(stats_data, 'echarts');

// create new custom field
// timestamp is in microseconds, therefore divide by 10**6
stats_data.project['measurement_duration_in_s'] = (stats_data.project?.end_measurement - stats_data.project?.start_measurement) / 1000000

fillProjectData(stats_data.project)
displayGraphs(metrics.series, notes_json.data, 'echarts');
fillAvgContainers(stats_data, metrics);
Expand Down
75 changes: 55 additions & 20 deletions frontend/stats.html
Original file line number Diff line number Diff line change
Expand Up @@ -88,16 +88,18 @@ <h3>Project Data</h3>
</div>
</div>
</div><!-- end ui horizontal card -->
<h3>container level metrics</h3>
<div class="ui five cards stackable no-transform-statistics">
<div class="ui card">
<div class="ui content">
<div class="ui top attached yellow label">AVG CPU Load</div>
<div class="description">
<div class="ui tiny statistic">
<div class="value">
<i class="microchip icon"></i> <span id="avg-cpu-load">N/A</span>
<i class="microchip icon"></i> <span id="avg-cpu-load-containers">N/A</span>
</div>
</div>
<div class="ui bottom right attached label">all containers</div>
</div>
</div>
</div>
Expand All @@ -107,9 +109,10 @@ <h3>Project Data</h3>
<div class="description">
<div class="ui tiny statistic">
<div class="value">
<i class="angle double up icon"></i> <span id="max-cpu-load">N/A</span>
<i class="angle double up icon"></i> <span id="max-cpu-load-containers">N/A</span>
</div>
</div>
<div class="ui bottom right attached label">all containers</div>
</div>
</div>
</div>
Expand Down Expand Up @@ -138,53 +141,56 @@ <h3>Project Data</h3>
</div>
</div>
</div>
</div><!-- end ui five cards stackable -->
<h3>system level metrics</h3>
<div class="ui five cards stackable no-transform-statistics">
<div class="ui card">
<div class="ui content">
<div class="ui top attached orange label">CPU Energy</div>
<div class="ui bottom right attached label">Intel RAPL CPU Package</div>
<div class="ui top attached yellow label">AVG CPU Load</div>
<div class="description">
<div class="ui tiny statistic">
<div class="value">
<i class="power off icon"></i> <span id="cpu-energy">N/A</span>
<i class="microchip icon"></i> <span id="avg-cpu-load-system">N/A</span>
</div>
</div>
<div class="ui bottom right attached label">system</div>
</div>
</div>
</div>
<div class="ui card">
<div class="ui content">
<div class="ui top attached orange label">Memory Energy</div>
<div class="ui bottom right attached label">Intel RAPL DRAM</div>
<div class="ui top attached teal label">Max. CPU Load</div>
<div class="description">
<div class="ui tiny statistic">
<div class="value">
<i class="power off icon"></i> <span id="memory-energy">N/A</span>
<i class="angle double up icon"></i> <span id="max-cpu-load-system">N/A</span>
</div>
</div>
<div class="ui bottom right attached label">system</div>
</div>
</div>
</div>
<div class="ui card">
<div class="ui content">
<div class="ui top attached orange label">Component Energy</div>
<div class="ui bottom right attached label">RAPL CPU+Memory</div>
<div class="ui top attached orange label">CPU Energy</div>
<div class="ui bottom right attached label">Intel RAPL CPU Package</div>
<div class="description">
<div class="ui tiny statistic">
<div class="value">
<i class="power off icon"></i> <span id="component-energy">N/A</span>
<i class="power off icon"></i> <span id="cpu-energy">N/A</span>
</div>
</div>
</div>
</div>
</div>
<div class="ui card">
<div class="ui content">
<div class="ui top attached orange label">Component Power (avg.)</div>
<div class="ui bottom right attached label">RAPL CPU+Memory</div>
<div class="ui top attached orange label">Memory Energy</div>
<div class="ui bottom right attached label">Intel RAPL DRAM</div>
<div class="description">
<div class="ui tiny statistic">
<div class="value">
<i class="power off icon"></i> <span id="component-power">N/A</span>
<i class="power off icon"></i> <span id="memory-energy">N/A</span>
</div>
</div>
</div>
Expand All @@ -204,27 +210,43 @@ <h3>Project Data</h3>
</div>
</div>
</div>
</div>
<h3>compound metrics</h3>
<div class="ui five cards stackable no-transform-statistics">
<div class="ui card">
<div class="ui content">
<div class="ui top attached orange label">Total Energy</div>
<div class="ui bottom right attached label">CPU + Memory + Network</div>
<div class="ui top attached orange label">Component Energy</div>
<div class="ui bottom right attached label">RAPL CPU+Memory</div>
<div class="description">
<div class="ui tiny statistic">
<div class="value">
<i class="power off icon"></i> <span id="total-energy">N/A</span>
<i class="power off icon"></i> <span id="component-energy">N/A</span>
</div>
</div>
</div>
</div>
</div>
<div class="ui card">
<div class="ui content">
<div class="ui top black attached label">Total co2 </div>
<div class="ui top attached orange label">Component Power (avg.)</div>
<div class="ui bottom right attached label">RAPL CPU+Memory</div>
<div class="description">
<div class="ui tiny statistic">
<div class="value">
<i class="power off icon"></i> <span id="component-power">N/A</span>
</div>
</div>
</div>
</div>
</div>
<div class="ui card">
<div class="ui content">
<div class="ui top attached orange label">Total Energy</div>
<div class="ui bottom right attached label">CPU + Memory + Network</div>
<div class="description">
<div class="ui tiny statistic">
<div class="value">
<i class="burn icon"></i> <span id="total-co2-internal">N/A</span>
<i class="power off icon"></i> <span id="total-energy">N/A</span>
</div>
</div>
</div>
Expand Down Expand Up @@ -256,8 +278,21 @@ <h3>Project Data</h3>
</div>
</div>
</div>
<div class="ui card">
<div class="ui content">
<div class="ui top black attached label">Total co2 </div>
<div class="ui bottom right attached label">CPU + Memory + Network</div>
<div class="description">
<div class="ui tiny statistic">
<div class="value">
<i class="burn icon"></i> <span id="total-co2-internal">N/A</span>
</div>
</div>
</div>
</div>
</div>
</div><!-- end ui five cards stackable -->

</div><!-- end ui four cards stackable -->
<div class="ui icon message">
<i class="question circle icon"></i>
<div class="content">
Expand Down
25 changes: 0 additions & 25 deletions install.sh
Original file line number Diff line number Diff line change
Expand Up @@ -24,37 +24,12 @@ while IFS= read -r subdir; do
echo "Installing $subdir/static-binary ..."
rm -f $subdir/static-binary 2> /dev/null
make -C $subdir
chmod +x $subdir/static-binary
fi
done

sudo_line="$USER ALL=(ALL) NOPASSWD: $PWD/tools/metric_providers/cpu/energy/RAPL/MSR/system/static-binary -i *"
sudo_line_2="$USER ALL=(ALL) NOPASSWD: $PWD/tools/metric_providers/memory/energy/RAPL/MSR/system/static-binary -i * -d"
sudo_line_3="$USER ALL=(ALL) NOPASSWD: $PWD/tools/metric_providers/network/io/cgroup/container/static-binary -i * -s *"

etc_hosts_line_1="127.0.0.1 green-coding-postgres-container"
etc_hosts_line_2="127.0.0.1 api.green-coding.local metrics.green-coding.local"

echo "Writing to /etc/sudoers file..."
if ! sudo grep -Fxq "$sudo_line" /etc/sudoers; then
echo "$sudo_line" | sudo tee -a /etc/sudoers
else
echo "Entry was already present..."
fi

if ! sudo grep -Fxq "$sudo_line_2" /etc/sudoers; then
echo "$sudo_line_2" | sudo tee -a /etc/sudoers
else
echo "Entry was already present..."
fi

if ! sudo grep -Fxq "$sudo_line_3" /etc/sudoers; then
echo "$sudo_line_3" | sudo tee -a /etc/sudoers
else
echo "Entry was already present..."
fi


echo "Writing to /etc/hosts file..."
if ! sudo grep -Fxq "$etc_hosts_line_1" /etc/hosts; then
echo "$etc_hosts_line_1" | sudo tee -a /etc/hosts
Expand Down
Loading

0 comments on commit 0372acd

Please sign in to comment.