acct_gather.conf

NAME

acct_gather.conf - Slurm configuration file for the acct_gather plugins

DESCRIPTION

acct_gather.conf is a UTF8 formatted file which defines parameters used by Slurm's acct_gather related plugins. The file will always be located in the same directory as the slurm.conf. Parameter names are case insensitive but parameter values are case sensitive. Any text following a "#" in the configuration file is treated as a comment through the end of that line. The size of each line in the file is limited to 1024 characters. Changes to the configuration file take effect upon restart of the Slurm daemons. The following acct_gather.conf parameters are defined to control the general behavior of various plugins in Slurm. The acct_gather.conf file is different than other Slurm .conf files. Each plugin defines which options are available. Each plugin to be loaded must be specified in the slurm.conf under the following configuration entries: • AcctGatherEnergyType (plugin type= acct_gather_energy) • AcctGatherInterconnectType (plugin type= acct_gather_interconnect) • AcctGatherFilesystemType (plugin type= acct_gather_filesystem) • AcctGatherProfileType (plugin type= acct_gather_profile) If the respective plugin for an option is not loaded then that option will appear to be unknown by Slurm and silently ignored. If you decide to change plugin types you may also have to change the related options.

acct_gather_energy/IPMI

Required entry in slurm.conf: Options used for acct_gather_energy/ipmi are as follows: Datasets provided by the plugin have name: <IPMI_SENSOR_LABEL>Power.

acct_gather_energy/rapl

Required entry in slurm.conf: This plugin doesn't read any options from acct_gather.conf. Dataset provided by the plugin is: Power.

acct_gather_energy/XCC

Required entry in slurm.conf: Options used for acct_gather_energy/xcc include only in-band communications with XClarity Controller, thus a reduced set of configurations is supported: Datasets provided by the plugin are: Energy, CurrPower.

acct_gather_filesystem/lustre

Required entry in slurm.conf: This plugin doesn't read any options from acct_gather.conf. Datasets provided by the plugin are: Reads, ReadMB, Writes, WriteMB.

acct_gather_profile/HDF5

Required entry in slurm.conf: Options used for acct_gather_profile/hdf5 are as follows:

acct_gather_profile/InfluxDB

Required entry in slurm.conf: The InfluxDB plugin provides the same information as the HDF5 plugin but will instead send information to the configured InfluxDB server. The InfluxDB plugin is designed against 1.x protocol of InfluxDB. Any site running a v2.x InfluxDB server will need to configure a v1.x compatibility endpoint along with the correct user and password authorization. Token authentication is not currently supported.

Options:

ProfileInfluxDBDatabase

InfluxDB v1.x database name where profiling information is to be written. InfluxDB v2.x bucket name where profiling information is to be written.

ProfileInfluxDBDefault

A comma-delimited list of data types to be collected for each job submission. Allowed values are:

ProfileInfluxDBHost=<hostname>:<port>

The hostname of the machine where the InfluxDB instance is executed and the port used by the HTTP API. The port used by the HTTP API is the one configured through the bind-address influxdb.conf option in the [http] section. Example:

ProfileInfluxDBHost=myinfluxhost:8086
    

ProfileInfluxDBPass

Password for username configured in ProfileInfluxDBUser. Required in v2.x and optional in v1.x InfluxDB.

ProfileInfluxDBRTPolicy

The InfluxDB v1.x retention policy name for the database configured in ProfileInfluxDBDatabase option. The InfluxDB v2.x retention policy bucket name for the database configured in ProfileInfluxDBDatabase option.

ProfileInfluxDBUser

InfluxDB username that should be used to gain access to the database configured in ProfileInfluxDBDatabase. Required in v2.x and optional in v1.x InfluxDB. This is only needed if InfluxDB v1.x is configured with authentication enabled in the [http] config section and a user has been granted at least WRITE access to the database. See also ProfileInfluxDBPass.

NOTES:

This plugin requires the libcurl development files to be installed and linkable at configure time. The plugin will not build otherwise. Information on how to install and configure InfluxDB and manage databases, retention policies and such is available on the official webpage. Collected information is written from every compute node where a job runs to the InfluxDB instance listening on the ProfileInfluxDBHost. In order to avoid overloading the InfluxDB instance with incoming connection requests, the plugin uses an internal buffer which is filled with samples. Once the buffer is full, a HTTP API write request is performed and the buffer is emptied to hold subsequent samples. A final request is also performed when a task ends even if the buffer isn't full. Failed HTTP API write requests are silently discarded. This means that collected profile information in the plugin buffer is lost if it can't be written to the InfluxDB database for any reason. Plugin messages are logged along with the slurmstepd logs to SlurmdLogFile. In order to troubleshoot any issues, it is recommended to temporarily increase the slurmd debug level to debug3 and add Profile to the debug flags. This can be accomplished by setting the slurm.conf SlurmdDebug and DebugFlags respectively or dynamically through scontrol setdebug and setdebugflags. Grafana can be used to create charts based on the data held by InfluxDB. This kind of tool permits one to create dashboards, tables and other graphics using the stored time series.

acct_gather_interconnect/OFED

Required entry in slurm.conf: Options used for acct_gather_interconnect/ofed are as follows: Datasets provided by the plugin: PacketsIn, PacketsOut, InMB, OutMB

acct_gather_interconnect/sysfs

Required entry in slurm.conf: Options used for acct_gather_interconnect/sysfs are as follows: Datasets provided by the plugin: PacketsIn, PacketsOut, InMB, OutMB

EXAMPLE

###
# Slurm acct_gather configuration file
###
# Parameters for acct_gather_energy/impi plugin
EnergyIPMIFrequency=10
EnergyIPMICalcAdjustment=yes
#
# Parameters for acct_gather_profile/hdf5 plugin
ProfileHDF5Dir=/app/slurm/profile_data
# Parameters for acct_gather_interconnect/ofed plugin
InfinibandOFEDPort=1

COPYING

Copyright (C) 2012-2013 Bull. Copyright (C) 2012-2022 SchedMD LLC. Produced at Bull (cf, DISCLAIMER). This file is part of Slurm, a resource management program. For details, see <https://slurm.schedmd.com/>. Slurm is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. Slurm is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

SEE ALSO

slurm.conf(5)