MPE - MultiProcessing Environment
The Multi-Processing Environment (MPE) attempts to provide programmers with a
complete suite of performance analysis tools for their MPI programs based on
post processing approach. These tools include a set of profiling libraries, a
set of utility programs, and a set of graphical visualization tools.
The first set of tools to be used with user MPI programs is profiling libraries
which provide a collection of routines that create log files. These log files
can be created manually by inserting MPE calls in the MPI program, or
automatically by linking with the appropriate MPE libraries, or by combining
the above two methods. Currently, the MPE offers the following 3 profiling
libraries.
- Tracing Library
- - This traces all MPI calls. Each MPI call is preceded by a
line that contains the rank in MPI_COMM_WORLD of the calling process, and
followed by another line indicating that the call has completed. Most send
and receive routines also indicate the values of count, tag, and partner
(destination for sends, source for receives). Output is to standard
output.
- Animation Library
- - This is a simple form of real-time program animation and
requires X window routines.
- Logging Library
- - This is the most useful and widely used profiling
libraries in MPE. They form the basis to generate log files from user MPI
programs. There are currently 3 different log file formats allowed in MPE.
The default log file format is CLOG. It is basically a collection of
events with single timestamps. And there is ALOG which is provided for
backward compatibility reason and it is not being developed. And the most
powerful one is SLOG, stands for Scalable LOGfile format, which can be
converted from CLOG after CLOG file has been generated (preferred
approach), or can be generated directly when MPI program is executed
(through setting the environmental variable MPE_LOG_FORMAT to SLOG).
The set of utility programs in MPE includes log format converter (e.g.
clog2slog), logfile print (e.g. slog_print) and logfile viewer wrappers,
logviewer, which selects the correct graphical tool to display the logfile
based on the logfile's file extension.
Currently, MPE's graphical tools includes 3 display programs, upshot for
ALOG, jumpshot-2 for CLOG and jumpshot-3 for SLOG. The logviewer script
eliminates the need for user to remember the relationship of logfile
formats to display programs.
MPE can be configured and installed as an extension to most MPI standard
compliant MPI implementations, e.g. MPICH, LAM, SGI's MPI, HP-UX's MPI and
IBM's MPI. It has been integrated seamlessly into MPICH distribution, so MPE
will be installed automatically during MPICH's installation process.
For details of configuring and building of MPE, please refer to the README file
in the MPE source directory.
As previously noted, the
mpe library is composed of 3 different profiling
libraries. Each MPI implementation requires a slightly different way in which
to link with these libraries. During configure, the link path and appropriate
libraries are determined and assigned to variables. These variables are first
substituted in the Makefile in the directory
mpe/contrib/test . The
Makefile is then installed into directory
share/examples during the
final installation process. The following is a list of these variables.
LOG_LIBS = link path needed to link with the logging library
TRACE_LIBS = link path needed to link with the tracing library
ANIM_LIBS = link path needed to link with the animation library
The variable FLIB_PATH is the link path needed to link fortran MPI programs with
the logging library.
During make, small test programs cpi.c and fpi.f will be linked with each of the
above libraries. In the output from Make, a message will be written regarding
the success of each attempted link test. The success of these linkage tests
will also be included in the Make output. If the linkage tests are successful,
then these library link paths can be used for your programs as well.
In addition to using the predefined MPE logging libraries to log all MPI calls,
MPE logging calls can be inserted into user's MPI program to define and log
states. These states are called User-Defined states. States may be nested,
allowing one to define a state describing a user routine that contains several
MPI calls, and display both the user-defined state and the MPI operations
contained within it.
The routine
MPE_Log_get_event_number() has to be used to get unique event
numbers (this is important if you are writing a library that uses the MPE
logging routines) from the MPE system. The routines
MPE_Describe_state() and
MPE_Log_event() are then used to
describe user-defined states.
int eventID_begin, eventID_end;
.
.
.
eventID_begin = MPE_Log_get_event_number();
eventID_end = MPE_Log_get_event_number();
.
.
.
MPE_Describe_state( eventID_begin, eventID_end, "Amult", "bluegreen" );
.
.
.
MyAmult( Matrix m, Vector v )
{
/* Log the start event along with the size of the matrix */
MPE_Log_event( eventID_begin, m->n, (char *)0 );
.
.
.
Amult code, including MPI calls ...
MPE_Log_event( eventID_end, 0, (char *)0 );
}
The log file generated by this code will have the MPI routines within the
routine MyAmult() indicated by a containing bluegreen rectangle.
If the MPE logging library, liblmpe.a, are NOT linked with the user program,
MPE_Init_log() and
MPE_Finish_log() need to be used before and
after all the MPE calls. Sample programs cpilog.c and fpi.f are available in
MPE source directory contrib/test or the installed directory share/examples to
illustrate the use of these MPE routines.
There are 2 environmental variables,
TMPDIR and
MPE_LOG_FORMAT ,
that user
- MPE_LOG_FORMAT
- - determines the format of the logfile generated from the
execution of application linked with MPE logging libraries. The allowed
value for MPE_LOG_FORMAT are CLOG, SLOG and ALOG. When
MPE_LOG_FORMAT is NOT set, CLOG is assumed.
- TMPDIR
- - specifies a directory to be used as temporary storage for
each process. By default, when TMPDIR is NOT set, /tmp will be
used. When user needs to generate a very large logfile for long-running
MPI job, user needs to make sure that TMPDIR is big enough to hold
the temporary logfile which will be deleted if the merged logfile can be
created successfully. In order to minimize the overhead of the logging to
the MPI program, it is highly recommended user to use a *local* file
system for TMPDIR .
Note : The final merged logfile will be written back to the file system
where process 0 is.
In bin/, user can find several useful utility programs when manipulating
logfiles. These includes log format converters, e.g. clog2slog, log format
print programs, e.g. slog_print, and a script to launch display program,
logviewer.
- clog2slog
- - a CLOG to SLOG logfile converter. Since the automatic
generation of SLOG file through setting of environmental variable
MPE_LOG_FORMAT to SLOG may NOT work for some non well-behaved MPI
programs, using the logfile format converter can generate extra diagnostic
information about the condition of the logfile. Also the converter allows
one to adjust certain parameters of the logfile, like frame size which is
the segment of the logfile to be displayed by Jumpshot-3's time line
window. For non well behaved MPI program, one may need to increase the
frame size from the default 64KB to a bigger value. For more information
about the converter, do "clog2slog -h"
- clog2alog
- - a CLOG to ALOG logfile converter. It is not being
developed. It is provided here for backward compatibility purpose.
- slog_print
- - a stdout print program for SLOG file. It serves to check
the content of the logfile. If the SLOG file is too big, it may NOT be
useful to use slog_print. Also, when slog is NOT complete, slog_print
won't work. So it serves as a simple test to check if the SLOG file is
generated completely.
- clog_print
- - stdout print program for CLOG file.
- logviewer
- - the script which involves appropriate viewer based on the
file extension of logfile. For instance, if the logfile is foo.slog,
logviewer will invoke jumpshot-3 to display the logfile. Jumpshot-3
resides in share/. For more information of logviewer, do "logviewer
-help" to list all available options.
To view a list of configure options, type
./configure--help in the
top-level
mpe directory.
In the Installation Manual, there is a section on MPE which gives instructions
and examples for each type of installation.
The README in the top-level
mpe directory.
In the Users Guide, there is additional information describing the MPE routines
and the profiling libraries.
The man pages included in the distribution has information regarding the MPE
routines. The command
mpich/bin/mpiman is a script that runs
xman on these man pages.
/home/MPI/mansrc/mpepage