Likwid Powermeter

likwid-powermeter: Tool for accessing RAPL counters on Intel processor

Introduction

Intel introduced with the SandyBridge architecture an interface to configure and readout energy consumption of different domains like processors and memory. This so called RAPL interface is controlled through MSR registers. likwid-powermeter is a small tool which allows you to query the energy consumed within domains for a given time period and computes the resulting power consumption.

Additionally you can query the supported Turbo Mode steps of all Turbo mode equipped processors (except the EX variants). This information is also queried from MSR registers.

The RAPL counters are also available as events in likwid-perfctr. There is a ENERGY group on recent Intel systems to measure common metrics.

NOTICE You have to setup access to the msr device files to use likwid-powermeter.

Options

-h, --help	 Help message
-v, --version	 Version information
-V, --verbose <level>	 Verbose output, 0 (only errors), 1 (info), 2 (details), 3 (developer)
-M <0|1>		 Set how MSR registers are accessed, 0=direct, 1=accessDaemon
-c <list>		 Specify sockets to measure
-i, --info	 Print information from MSR_PKG_POWER_INFO register and Turbo mode
-s <duration>	 Set measure duration in us, ms or s. (default 2s)
-p		 Print dynamic clocking and CPI values, uses likwid-perfctr
-t		 Print current temperatures of all CPU cores
-f		 Print current temperatures in Fahrenheit

Common domains

Intel

Domain	Description
PKG	all CPU cores of a socket
DRAM	memory DIMMs attached to a socket

More details can be found in Intel's Architectures Software Developer Manuals in chapter 'Power And Thermal Management'.

AMD

Domain	Description
CORE	CPU core
PKG	all CPU cores of a socket
DRAM	memory DIMMs attached to a socket
L3	all CPU cores of a L3 segment

Further details can be found in the AMD Documentation Hub in the architecture-specific Programmer Reference manuals.

Examples

As usual you can get a short help message with

$ likwid-powermeter -h

Do get info about RAPL and Turbo Mode call

$ likwid-powermeter -i

Output should be something like the following:

--------------------------------------------------------------------------------
CPU name:	Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz
CPU type:	Intel Xeon IvyBridge EN/EP/EX processor
CPU clock:	3.00 GHz
--------------------------------------------------------------------------------
Base clock:	3000.00 MHz
Minimal clock:	1200.00 MHz
Turbo Boost Steps:
C0 3600.00 MHz
C1 3500.00 MHz
C2 3400.00 MHz
C3 3300.00 MHz
C4 3300.00 MHz
C5 3300.00 MHz
C6 3300.00 MHz
C7 3300.00 MHz
C8 3300.00 MHz
C9 3300.00 MHz
--------------------------------------------------------------------------------
Info for RAPL domain PKG:
Thermal Spec Power: 130 Watt
Minimum Power: 68 Watt
Maximum Power: 130 Watt
Maximum Time Window: 31232 micro sec

Info for RAPL domain DRAM:
Thermal Spec Power: 68.5 Watt
Minimum Power: 36 Watt
Maximum Power: 68.5 Watt
Maximum Time Window: 31232 micro sec
--------------------------------------------------------------------------------

This means the processor has a TDP of 130 Watt, the memory modules have a TDP of 68.5 Watt. With 1 Core active it can overclock up to 3.6 GHz. With all cores active it still may overclock to 3.3 GHz.

LIKWID offers no stand-alone tool to get the temperature of the CPU cores. The functionality is embedded into likwid-powermeter but can also be retrieved through likwid-perfctr. In order to print the temperatures call:

$ likwid-powermeter -t

This prints something like this:

--------------------------------------------------------------------------------
CPU name:	Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz
CPU type:	Intel Xeon IvyBridge EN/EP/EX processor
CPU clock:	3.00 GHz
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
Current core temperatures:
Socket 0 Core 0: 47 C
Socket 0 Core 20: 47 C
Socket 0 Core 1: 46 C
Socket 0 Core 21: 46 C
Socket 0 Core 2: 37 C
Socket 0 Core 22: 37 C
Socket 0 Core 3: 48 C
Socket 0 Core 23: 48 C
Socket 0 Core 4: 44 C
Socket 0 Core 24: 44 C
Socket 0 Core 5: 44 C
Socket 0 Core 25: 44 C
Socket 0 Core 6: 40 C
Socket 0 Core 26: 40 C
Socket 0 Core 7: 41 C
Socket 0 Core 27: 41 C
Socket 0 Core 8: 45 C
Socket 0 Core 28: 45 C
Socket 0 Core 9: 42 C
Socket 0 Core 29: 42 C
Socket 1 Core 10: 42 C
Socket 1 Core 30: 42 C
Socket 1 Core 11: 43 C
Socket 1 Core 31: 43 C
Socket 1 Core 12: 40 C
Socket 1 Core 32: 40 C
Socket 1 Core 13: 41 C
Socket 1 Core 33: 41 C
Socket 1 Core 14: 43 C
Socket 1 Core 34: 43 C
Socket 1 Core 15: 42 C
Socket 1 Core 35: 42 C
Socket 1 Core 16: 43 C
Socket 1 Core 36: 43 C
Socket 1 Core 17: 42 C
Socket 1 Core 37: 42 C
Socket 1 Core 18: 41 C
Socket 1 Core 38: 41 C
Socket 1 Core 19: 41 C
Socket 1 Core 39: 41 C
--------------------------------------------------------------------------------

If you use -f instead of -t or set it additional to -t, the temperatures are printed in Fahrenheit instead of Celsius.

To measure power in a given duration (Stethoscope mode) use

$ likwid-powermeter -s 3s

which gives you something like

--------------------------------------------------------------------------------
CPU name:	Intel(R) Xeon(R) Platinum 8360Y CPU @ 2.40GHz
CPU type:	Intel Icelake SP processor
CPU clock:	2.39 GHz
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
Runtime: 3.00012 s
Measure for socket 0 on CPU 0
Domain PKG:
Energy consumed: 128.384 Joules
Power consumed: 42.7928 Watt
Domain PP0:
Energy consumed: 0 Joules
Power consumed: 0 Watt
Domain DRAM:
Energy consumed: 13.5585 Joules
Power consumed: 4.51931 Watt
Domain PLATFORM:
Energy consumed: 0 Joules
Power consumed: 0 Watt

Measure for socket 1 on CPU 36
Domain PKG:
Energy consumed: 124.495 Joules
Power consumed: 41.4966 Watt
Domain PP0:
Energy consumed: 0 Joules
Power consumed: 0 Watt
Domain DRAM:
Energy consumed: 14.1665 Joules
Power consumed: 4.72197 Watt
Domain PLATFORM:
Energy consumed: 0 Joules
Power consumed: 0 Watt
--------------------------------------------------------------------------------

The output starts with a header describing the current system. Afterwards, it tells you the exact measurement time. The system has two CPU sockets. Each socket is measured individually by one CPU (in this case 0 and 36) and prints one block of output. Each block lists the RAPL domains provided by the platform with the vendor-specified names (PKG, DRAM, PP0, ...) with the corresponding energy and power measurements.

In summary: All cores of socket 0 (PKG domain) consumed during the 3s measurement time 128.384 J. All cores of socket 1 consumed 124.495 J in the same time. Moreover, the memory DIMMs (DRAM domain) attached to socket 0 consumed 13.5585 J while the DIMMs at socket 1 consumed 14.1665 J.

Next you can use likwid-powermeter as a wrapper to output the energy consumed by all domains for the runtime of an application:

$ likwid-powermeter  ./a.out

It gives you similar output as above for likwid-powermeter -s 3s but measures as long as ./a.out is running.

Finally you can also output the clock and CPI values for the socket you are currently measuring:

$ likwid-powermeter -c 0 -p ./a.out

It uses likwid-perfctr to get the clock and CPI values (performance group CLOCK):

--------------------------------------------------------------------------------
CPU name:	Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz
CPU type:	Intel Xeon IvyBridge EN/EP/EX processor
CPU clock:	3.00 GHz
--------------------------------------------------------------------------------
YOUR PROGRAM OUTPUT
--------------------------------------------------------------------------------
Group 1:
+-----------------------+---------+--------------+-------------+-------------+-------------+
|         Event         | Counter |    Core 0    |    Core 1   |    Core 2   |    Core 3   |
+-----------------------+---------+--------------+-------------+-------------+-------------+
|   INSTR_RETIRED_ANY   |  FIXC0  |  11724451762 | 10842844517 | 10561467656 | 10280194250 |
| CPU_CLK_UNHALTED_CORE |  FIXC1  |  25944160048 | 25515554594 | 25261453331 | 25175563204 |
|  CPU_CLK_UNHALTED_REF |  FIXC2  |  23562189510 | 23196392160 | 22965415440 | 22887349770 |
|     PWR_PKG_ENERGY    |   PWR0  | 7.988363e+02 |      0      |      0      |      0      | 
+-----------------------+---------+--------------+-------------+-------------+-------------+

+----------------------------+---------+--------------+------------+-------------+----------------+
|            Event           | Counter |      Sum     |     Min    |     Max     |       Avg      | 
+----------------------------+---------+--------------+------------+-------------+----------------+
|   INSTR_RETIRED_ANY STAT   |  FIXC0  | 226186141729 |  155188041 | 11724451762 | 5654653543.225 | 
| CPU_CLK_UNHALTED_CORE STAT |  FIXC1  | 683928549398 | 1795236569 | 25944160048 | 17098213734.95 | 
|  CPU_CLK_UNHALTED_REF STAT |  FIXC2  | 622578115140 | 1683415500 | 23562189510 |  15564452878.5 | 
|     PWR_PKG_ENERGY STAT    |   PWR0  |   1326.3737  |      0     |   798.8363  |   33.1593425   | 
+----------------------------+---------+--------------+------------+-------------+----------------+

+----------------------+--------------+--------------+--------------+--------------+
|        Metric        |    Core 0    |    Core 1    |    Core 2    |    Core 3    | 
+----------------------+--------------+--------------+--------------+--------------+
|  Runtime (RDTSC) [s] | 1.318434e+01 | 1.318434e+01 | 1.318434e+01 | 1.318434e+01 |
| Runtime unhalted [s] | 8.648601e+00 | 8.505723e+00 | 8.421018e+00 | 8.392386e+00 |
|      Clock [MHz]     | 3.303070e+03 | 3.299729e+03 | 3.299725e+03 | 3.299723e+03 | 
|          CPI         | 2.212825e+00 | 2.353216e+00 | 2.391851e+00 | 2.448938e+00 | 
|      Energy [J]      | 7.988363e+02 |       0      |       0      |       0      | 
|       Power [W]      | 6.058979e+01 |       0      |       0      |       0      |
+----------------------+--------------+--------------+--------------+--------------+

+---------------------------+-------------+-----------+----------+--------------+
|           Metric          |     Sum     |    Min    |    Max   |      Avg     | 
+---------------------------+-------------+-----------+----------+--------------+
|  Runtime (RDTSC) [s] STAT |   527.3736  |  13.18434 | 13.18434 |   13.18434   | 
| Runtime unhalted [s] STAT | 227.9906161 | 0.5984501 | 8.648601 | 5.6997654025 | 
|      Clock [MHz] STAT     |  131573.127 |  3199.073 |  3303.07 |  3289.328175 | 
|          CPI STAT         |  152.276829 |  2.212825 | 11.56814 |  3.806920725 | 
|      Energy [J] STAT      |  1326.3737  |     0     | 798.8363 |  33.1593425  | 
|       Power [W] STAT      |  100.60221  |     0     | 60.58979 |  2.51505525  | 
+---------------------------+-------------+-----------+----------+--------------+

--------------------------------------------------------------------------------
Runtime: 14.8027 s
Measure for socket 0 on CPU 0
Energy consumed: 854.958 Joules
Power consumed: 57.7567 Watts
Energy consumed: 611.597 Joules
Power consumed: 41.3165 Watts
Energy consumed: 403.18 Joules
Power consumed: 27.2369 Watts

Measure for socket 1 on CPU 10
Energy consumed: 573.944 Joules
Power consumed: 38.7729 Watts
Energy consumed: 368.422 Joules
Power consumed: 24.8888 Watts
Energy consumed: 235.443 Joules
Power consumed: 15.9054 Watts
--------------------------------------------------------------------------------

The output was truncated to 4 cores, the sum, min, max and average values are calculated using all cores. The system is clocked (in average) at 3.3 GHz.

The likwid-powermeter application always runs executables on all CPUs of the selected sockets. If you want to pin your application, you can use likwid-pin:

$ likwid-powermeter likwid-pin -c S0:0-3 ./a.out

likwid-powermeter still measures all CPU sockets but the application runs only on the first four physical cores of socket 0. You can limit likwid-powermeter by setting a socket with the -c commandline switch

$ likwid-powermeter -c 0 likwid-pin -c S0:0-3 ./a.out