Article originally posted on InfoQ. Visit InfoQ

SapMachine, a downstream distribution of OpenJDK, has introduced a new monitoring tool, SapMachine Vitals, that keeps a condensed history of operating system and JVM statistics. . The monitoring feature is activated by default and may be used to retrieve information such as heap usage, metaspace size, container memory statistics and limits, the number of classes loaded and the amount of threads spawned.

Vitals is a combination of sysstat operating system tools and Java Virtual Machine (JVM) statistics. The data is retrieved every ten seconds, by default, and buffered for ten days inside the JVM. The buffer contains more detailed information about the last hour. While the overhead of Vitals is relatively low with about 300 Kb of memory consumption and little CPU consumption, it’s still possible to disable the feature with the -XX:-EnableVitals command.

The jcmd VM.vitals command may be used to create the report for a running application. The report is also part of the hs_err_pid.log which the JVM generates as part of an abnormal exit. Lastly, when the JVM is started with the -XX:+PrintVitalsAtExit argument, the Vitals report is printed to stdout before the exit is completed. Alternatively, the -XX:+DumpVitalsAtExit argument may be used to create the following files: sapmachine_vitals_pid.txt and sapmachine_vitals_pid.csv. The names of the files may be customized with the argument -XX:VitalsFile=.

The various suboptions of VM.vitals are displayed with the help argument:

jcmd 11818 help VM.vitals
…
Options: (options must be specified using the  or = syntax)
scale : [optional] Memory usage in which to scale. Valid values are: k, m, g 
    (fixed scale) or "dynamic" for a dynamically chosen scale. (STRING, dynamic)
csv : [optional] csv format. (BOOLEAN, false)
no-legend : [optional] Omit legend. (BOOLEAN, false)
reverse : [optional] Reverse printing order. (BOOLEAN, false)
raw : [optional] Print raw values. (BOOLEAN, false)
now : [optional] Sample now values (BOOLEAN, false)

The Vitals report has three subsections: system including container stats, process stats and JVM stats. Each subsection contains a list of the metrics and their abbreviations and at the end of the report each metric is listed in a column and the measurements are displayed in the rows. In order to improve readability, we combine the metrics and measurements in this article per subsection from the jcmd 11818 VM.vitals command.

First the list of system and container metrics and their corresponding abbreviations is displayed:

------------system------------
avail: Memory available without swapping [host] [krn]
comm: Committed memory [host]
crt: Committed-to-Commit-Limit ratio (percent) [host]
swap: Swap space used [host]
si: Number of pages swapped in [host] [delta]
so: Number of pages pages swapped out [host] [delta]
p: Number of processes
t: Number of threads
tr: Number of threads running
tb: Number of threads blocked on disk IO
cpu-us: CPU user time [host]
cpu-sy: CPU system time [host]
cpu-id: CPU idle time [host]
cpu-st: CPU time stolen [host]
cpu-gu: CPU time spent on guest [host]
cgroup-lim: cgroup memory limit [cgrp]
cgroup-slim: cgroup memory soft limit [cgrp]
cgroup-usg: cgroup memory usage [cgrp]
cgroup-kusg: cgroup kernel memory usage (cgroup v1 only) [cgrp]

The measurements related to the system and container metrics:

                  	---------------------------------system----------------------------------
                                                           	-----cpu------ -----cgroup------
                  	avail comm  crt swap si so p   t   tr tb us sy id st gu lim slim usg kusg
2023-02-14 19:51:57   25,7g 11,4g  64   0k  0  0 305 804  1  0  3  1 95  0  0   	0k 64m	 
2023-02-14 19:51:47   25,7g 11,4g  64   0k  0  0 304 802  5  0  5  1 94  0  0   	0k 64m	 
2023-02-14 19:51:37   25,7g 11,3g  64   0k  0  0 304 800  2  0  5  2 92  0  0   	0k 64m    
2023-02-14 19:51:27   25,7g 11,4g  64   0k  0  0 305 800  4  0  5  2 92  0  0   	0k 64m    
2023-02-14 19:51:17   25,7g 11,4g  64   0k  0  0 305 804  2  0  9  2 88  0  0   	0k 64m    
2023-02-14 19:51:07   23,8g 13,4g  76   0k   	 308 915  2  0                  	0k 26m   

The list of process metrics and their corresponding abbreviations is displayed:

-----------process------------
virt: Virtual size
rss-all: Resident set size, total
rss-anon: Resident set size, anonymous memory [krn]
rss-file: Resident set size, file mappings [krn]
rss-shm: Resident set size, shared memory [krn]
swdo: Memory swapped out
cheap-usd: C-Heap, in-use allocations (may be unavailable if RSS > 4G) [glibc]
cheap-free: C-Heap, bytes in free blocks (may be unavailable if RSS > 4G) [glibc]
cpu-us: Process cpu user time
cpu-sy: Process cpu system time
io-of: Number of open files
io-rd: IO bytes read from storage or cache
io-wr: IO bytes written
thr: Number of native threads

The measurements related to the process metrics:

--------------------------process--------------------------
  	-------rss-------  	-cheap-- -cpu- ----io-----    
virt  all anon file shm swdo usd free us sy of rd   wr  thr
10,9g 85m  50m  35m  0k   0k 24m   9m  0  0  6  76k  0k  20
10,9g 85m  50m  35m  0k   0k 24m   9m  0  0  6  76k  0k  20
10,9g 85m  50m  35m  0k   0k 24m   9m  0  0  6  76k  0k  20
10,9g 85m  50m  35m  0k   0k 24m   9m  0  0  6  76k  0k  20
10,9g 85m  50m  35m  0k   0k 24m   9m  1  0  6 724k <1k  20
10,6g 36m  12m  24m  0k   0k 21m   2m    	 5           18

The list of JVM metrics and their corresponding abbreviations is displayed:

-------------jvm--------------
heap-comm: Java Heap Size, committed
heap-used: Java Heap Size, used
meta-comm: Meta Space Size (class+nonclass), committed
meta-used: Meta Space Size (class+nonclass), used
meta-csc: Class Space Size, committed [cs]
meta-csu: Class Space Size, used [cs]
meta-gctr: GC threshold
code: Code cache, committed
nmt-mlc: Memory malloced by hotspot [nmt]
nmt-map: Memory mapped by hotspot [nmt]
nmt-gc: NMT "gc" (GC-overhead, malloc and mmap) [nmt]
nmt-oth: NMT "other" (typically DBB or Unsafe.allocateMemory) [nmt]
nmt-ovh: NMT overhead [nmt]
jthr-num: Number of java threads
jthr-nd: Number of non-demon java threads
jthr-cr: Threads created [delta]
jthr-st: Total reserved size of java thread stacks [nmt] [linux]
cldg-num: Classloader Data
cldg-anon: Anonymous CLD
cls-num: Classes (instance + array)
cls-ld: Class loaded [delta]
cls-uld: Classes unloaded [delta]

The measurements related to the JVM metrics:

----------------------------------jvm----------------------------------
--heap--- ---------meta----------  	--jthr--- --cldg-- -----cls-----
comm used comm used csc  csu gctr code num nd cr num anon num  ld   uld
508m  23m   9m   9m   1m  1m  21m   7m  11  1  0  36   32 2515	0   0
508m  23m   9m   9m   1m  1m  21m   7m  11  1  0  36   32 2515	0   0
508m  23m   9m   9m   1m  1m  21m   7m  11  1  0  36   32 2515	0   0
508m  23m   9m   9m   1m  1m  21m   7m  11  1  0  36   32 2515	0   0
508m  23m   9m   9m   1m  1m  21m   7m  11  1  2  36   32 2515 1841 0
508m   6m 448k 217k 128k  5k  21m   7m  11  1      4    1  674 

The Vitals feature is only available as part of the SapMachine builds of OpenJDK. Thomas Stuefe, JVM engineer at SAP, indicated that the feature was not accepted upstream as its features overlap with Java Flight Recorder. More information can be found on GitHub and in the SapMachine Vitals blog.