amavis-logwatch

NAME

amavis-logwatch - An Amavisd-new log parser and analysis utility

SYNOPSIS

amavis-logwatch [options] [logfile ...]

DESCRIPTION

The utility is an Amavisd-new log parser that produces summaries, details, and statistics regarding the operation of Amavisd-new (henceforth, simply called Amavis).

This utility can be used as a standalone program, or as a Logwatch filter module to produce Amavisd-new summary and detailed reports from within Logwatch.

Amavis-logwatch is able to produce a wide range of reports with data grouped and sorted as much as possible to reduce noise and highlight patterns. Brief summary reports provide a quick overview of general Amavis operations and message delivery, calling out warnings that may require attention. Detailed reports provide easy to scan, hierarchically-arranged and organized information, with as much or little detail as desired.

Much of the interesting data is available when Amavis' $log_level is set to at least 2. See Amavis Log Level below.

Amavis-logwatch outputs two principal sections: a Summary section and a Detailed section. For readability and quick scanning, all event or hit counts appear in the left column, followed by brief description of the event type, and finally additional statistics or count representations may appear in the rightmost column.

The following segment from a sample Summary report illustrates:


****** Summary ********************************************

       9   Miscellaneous warnings 

   20313   Total messages scanned ----------------  100.00%
1008.534M  Total bytes scanned                1,057,524,252
========   ================================================

    1190   Blocked -------------------------------    5.86%
      18     Malware blocked                          0.09%
       4     Banned name blocked                      0.02%
     416     Spam blocked                             2.05%
     752     Spam discarded (no quarantine)           3.70%

   19123   Passed --------------------------------   94.14%
      47     Bad header passed                        0.23%
   19076     Clean passed                            93.91%
========   ================================================

      18   Malware -------------------------------    0.09%
      18     Malware blocked                          0.09%

       4   Banned --------------------------------    0.02%
       4     Banned file blocked                      0.02%

    1168   Spam ----------------------------------    5.75%
     416     Spam blocked                             2.05%
     752     Spam discarded (no quarantine)           3.70%

   19123   Ham -----------------------------------   94.14%
      47     Bad header passed                        0.23%
   19076     Clean passed                            93.91%
========   ================================================

    1982   SpamAssassin bypassed 
      32   Released from quarantine 
       2   DSN notification (debug supplemental) 
       2   Bounce unverifiable   
    2369   Whitelisted           
       2   Blacklisted           
      12   MIME error            
      58   Bad header (debug supplemental) 
      40   Extra code modules loaded at runtime

The report indicates there were 9 general warnings, and Amavis scanned a total of 20313 messages for a total of 1008.53 megabytes or 1,057,524,252 bytes. The next summary groups shows the Blocked / Passed overview, with 1190 Blocked messages (broken down as 18 messages blocked as malware, 4 messages with banned names, 416 spam messages, and 752 discarded messages), and 19123 Passed messages (47 messages with bad headers and 19076 clean messages).

The next (optional) summary grouping shows message disposition by contents category. There were 18 malware messages and 4 banned file messages (all blocked), 1168 Spam messages, of which 416 were blocked (quarantined) and 752 discarded. Finally, there were 19123 messages considered to be Ham (i.e. not spam), 47 of which contained bad headers.

Additional count summaries for a variety of events are also listed.

There are dozens of sub-sections available in the Detailed report, each of whose output can be controlled in various ways. Each sub-section attempts to group and present the most meaningful data at superior levels, while pushing less useful or noisy data towards inferior levels. The goal is to provide as much benefit as possible from smart grouping of data, to allow faster report scanning, pattern identification, and problem solving. Data is always sorted in descending order by count, and then numerically by IP address or alphabetically as appropriate.

The following Spam blocked segment from a sample Detailed report illustrates the basic hierarchical level structure of amavis-logwatch:


****** Detailed *******************************************

   19346   Spam blocked -----------------------------------
     756      [email protected]
      12         10.0.0.2
      12            <>
      12         192.168.2.2
      12            <>
       5         192.168.2.1
     ...

The amavis-logwatch utility reads from STDIN or from the named Amavis logfile. Multiple logfile arguments may be specified, each processed in order. The user running amavis-logwatch must have read permission on each named log file.

Options

The options listed below affect the operation of amavis-logwatch. Options specified later on the command line override earlier ones. Any option may be abbreviated to an unambiguous length.

--[no]autolearn

--show_autolearn boolean: Enables (disables) output of the autolearn report. This report is only available if the default Amavis $log_templ has been modified to provide autolearn results in log entries. This can be done by uncommenting two lines in the Amavis program itself (where the default log templates reside), or by correctly adding the $log_templ variable to the amavisd.conf file. See Amavis' README.customize and search near the end of the Amavisd program for "autolearn".

--[no]by_ccat_summary

--show_by_ccat_summary boolean: Enables (disables) the by contents category summary in the Summary section. Default: enabled.

-f config_file

--config_file config_file: Use an alternate configuration file config_file instead of the default. This option may be used more than once. Multiple configuration files will be processed in the order presented on the command line. See CONFIGURATION FILE below.

--debug keywords: Output debug information during the operation of amavis-logwatch. The parameter keywords is one or more comma or space separated keywords. To obtain the list of valid keywords, use --debug xxx where xxx is any invalid keyword.

--detail level: Sets the maximum detail level for amavis-logwatch to level. This option is global, overriding any other output limiters described below. The amavis-logwatch utility produces a Summary section, a Detailed section, and additional report sections. With level less than 5, amavis-logwatch will produce only the Summary section. At level 5 and above, the Detailed section, and any additional report sections are candidates for output. Each incremental increase in level generates one additional hierarchical sub-level of output in the Detailed section of the report. At level 10, all levels are output. Lines that exceed the maximum report width (specified with max_report_width) will be cut. Setting level to 11 will prevent lines in the report from being cut (see also --line_style).

--[no]first_recip_only

--show_first_recip_only boolean: Specifies whether or not to sort by, and show, only the first recipient when a scanned messages contains multiple recipients.

--help: Print usage information and a brief description about command line options.

--ipaddr_width width: Specifies that IP addresses in address/hostname pairs should be printed with a field width of width characters. Increasing the default may be useful for systems using long IPv6 addresses.

-l limiter=levelspec

--limit limiter=levelspec: Sets the level limiter limiter with the specification levelspec.

--line_style style: Specifies how to handle long report lines. Three styles are available: full, truncate, and wrap. Setting style to full will prevent cutting lines to max_report_width; this is what occurs when detail is 11 or higher. When style is truncate (the default), long lines will be truncated according to max_report_width. Setting style to wrap will wrap lines longer than max_report_width such that left column hit counts are not obscured. This option takes precedence over the line style implied by the detail level. The options --full, --truncate, and --wrap are synonyms.

--nodetail: Disables the Detailed section of the report, and all supplemental reports. This option provides a convenient mechanism to quickly disable all sections under the Detailed report, where subsequent command line options may re-enable one or more sections to create specific reports.

--sarules `S,H´

--sarules default: Enables the SpamAssassin Rules Hit report. The comma-separated S and H arguments are top N values for the Spam and Ham reports, respectively, and can be any integer greater than or equal to 0, or the keyword all. The keyword default uses the built-in default values.

--nosarules: Disables the SpamAssassin Rules Hit report.

--sa_timings nrows: Enables the SpamAssassin Timings percentiles report. The report can be limited to the top N rows with the nrows argument. This report requires Amavis 2.6+ and SpamAssassin 3.3+.

--sa_timings_percentiles `P1 [P2 ...]´: Specifies the percentiles shown in the SpamAssassin Timings report. The arguments P1 ... are integers from 0 to 100 inclusive. Their order will be preserved in the report.

--nosa_timings: Disables the SpamAssassin Timings report.

--version: Print amavis-logwatch version information.

--score_frequencies `B1 [B2 ...]´

--score_frequencies default: Enables the Spam Score Frequency report. The arguments B1 ... are frequency distribution buckets, and can be any real numbers. Their order will be preserved in the report. The keyword default uses the built-in default values.

--noscore_frequencies: Disables the Spam Score Frequency report.

--score_percentiles `P1 [P2 ...]´

--score_percentiles default: Enables the Spam Score Percentiles report. The arguments P1 ... specify the percentiles shown in the report, and are integers from 0 to 100 inclusive. The keyword default uses the built-in default values.

--noscore_percentiles: Disables the Spam Score Percentiles report.

--[no]sect_vars

--show_sect_vars boolean: Enables (disables) supplementing each Detailed section title with the name of that section's level limiter. The name displayed is the command line option (or configuration file variable) used to limit that section's output. With the large number of level limiters available in amavis-logwatch, this a convenient mechanism for determining exactly which level limiter affects a section.

--[no]startinfo

--show_startinfo boolean: Enables (disables) the Amavis startup report showing most recent Amavis startup details.

--[no]summary

--show_summary: Enables (disables) displaying of the the Summary section of the report. The variable Amavis_Show_Summary in used in a configuration file.

--syslog_name namepat: Specifies the syslog service name that amavis-logwatch uses to match syslog lines. Only log lines whose service name matches the perl regular expression namepat will be used by amavis-logwatch; all non-matching lines are silently ignored. This is useful when a pre-installed Amavis package uses a name other than the default ( amavis). Note: if you use parenthesis in your regular expression, be sure they are cloistering and not capturing: use (?:pattern) instead of (pattern).

--timings percent: Enables the Amavis Scan Timings percentiles report. The report can be top N-percent limited with the percent argument.

--timings_percentiles `P1 [P2 ...]´: Specifies the percentiles shown in the Scan Timings report. The arguments P1 ... are integers from 0 to 100 inclusive. Their order will be preserved in the report.

--notimings: Disables the Amavis Scan Timings report.

--version: Print amavis-logwatch version information.

Level Limiters

The output of every section in the Detailed report is controlled by a level limiter. The name of the level limiter variable will be output when the sect_vars option is set. Level limiters are set either via command line in standalone mode with --limit limiter=levelspec option, or via configuration file variable $amavis_limiter= levelspec. Each limiter requires a levelspec argument, which is described below in LEVEL CONTROL.

The list of level limiters is shown below.

Amavis major contents category (ccatmajor) sections, listed in order of priority: VIRUS, BANNED, UNCHECKED, SPAM, SPAMMY, BADH, OVERSIZED, MTA, CLEAN.

MalwareBlocked

MalwarePassed: Blocked or passed messages that contain malware (ccatmajor: VIRUS).

BannedNameBlocked

BannedNamePassed: Blocked or passed messages that contain banned names in MIME parts (ccatmajor: BANNED).

UncheckedBlocked

UncheckedPassed: Blocked or passed messages that were not checked by a virus scanner or SpamAssassin (Amavis ccatmajor: UNCHECKED).

SpamBlocked

SpamPassed: Blocked or passed messages that were considered spam that reached kill level (Amavis ccatmajor: SPAM)

SpammyBlocked

SpammyPassed: Blocked or passed messages that were considered spam, but did not reach kill level (Amavis ccatmajor: SPAMMY)

BadHeaderBlocked

BadHeaderPassed: Blocked or passed messages that contain bad mail headers (ccatmajor: BAD-HEADER).

OversizedBlocked

OversizedPassed: Blocked or passed messages that were considered oversized (Amavis ccatmajor: OVERSIZED).

MtaBlocked

MtaPassed: Blocked or passed messages due to failure to re-inject to MTA (Amavis ccatmajor: MTA-BLOCKED). Occurrences of this event indicates a configuration problem. [ note: I don't believe mtapassed occurs, but exists for completeness.]

OtherBlocked

OtherPassed: Blocked or passed messages that are not any of other major contents categories (Amavis ccatmajor: OTHER).

TempFailBlocked

TempfailPassed: Blocked or passed messages that had a temporary failure (Amavis ccatmajor: TEMPFAIL)

CleanBlocked

CleanPassed: Messages blocked or passed which were considered clean (Amavis ccatmajor: CLEAN; i.e. non-spam, non-viral).

Other sections, arranged alphabetically:

AvConnectFailure: Problems connecting to Anti-Virus scanner(s).

AvTimeout: Timeouts awaiting responses from Anti-Virus scanner(s).

ArchiveExtract: Archive extraction problems.

BadHeaderSupp: Supplemental debug information regarding messages containing bad mail headers.

Bayes: Messages frequencies by Bayesian probability buckets.

BadAddress: Invalid mail address syntax.

Blacklisted: Messages that were (soft-)blacklisted. See also Whitelisted below.

BounceKilled

BounceRescued

BounceUnverifiable: Disposition of incoming bounce messages (DSNs).

ContentType: MIME attachment breakdown by type/subtype.

DccError: Errors encountered with or returned by DCC.

DefangError: Errors encountered during defang process.

Defanged: Messages defanged (rendered harmless).

DsnNotification: Errors encountered during attempt to send delivery status notification.

DsnSuppressed: Delivery status notification (DSN) intentionally suppressed.

ExtraModules: Additional code modules Amavis loaded during runtime.

FakeSender: Forged sender addresses, as determined by Amavis.

Fatal: Fatal events. These are presented at the top of the report, as they may require attention.

LocalDeliverySkipped: Failures delivering to a local address.

MalwareByScanner: Breakdown of malware by scanner(s) that detected the malware.

MimeError: Errors encountered during MIME extraction.

Panic: Panic events. These are presented at the top of the report, as they may require attention.

p0f: Passive fingerprint (p0f) hits, grouped by mail contents type (virus, unchecked, banned, spam, ham), next by operating system genre, and finally by IP address. Note: Windows systems are refined by Windows OS version, whereas versions of other operating systems are grouped generically.

Released: Messages that were released from Amavis quarantine.

SADiags: Diagnostics as reported from SpamAssassin.

SmtpResponse: SMTP responses received during dialog with MTA. These log entries are primarly debug.

TmpPreserved: Temporary directories preserved by Amavis when some component encounters a problem or failure. Directories listed and their corresponding log entries should be evaluated for problems.

VirusScanSkipped: Messages that could not be scanned by a virus scanner.

Warning: Warning events not categorized in specific warnings below. These are presented at the top of the report, as they may require attention.

WarningAddressModified: Incomplete email addresses modified by Amavis for safety.

WarningNoQuarantineId: Attempts to release a quarantined message that did not contain an X-Quarantine-ID header.

WarningSecurity levelspec: Insecure configuration or utility used by Amavis.

WarningSmtpShutdown: Failures during SMTP conversation with MTA.

WarningSql: Failures to communicate with, or error replies from, SQL service.

Whitelisted: Messages that were (soft-)whitelisted. See also Blacklisted above.

LEVEL CONTROL

The Detailed section of the report consists of a number of sub-sections, each of which is controlled both globally and independently. Two settings influence the output provided in the Detailed report: a global detail level (specified with --detail) which has final (big hammer) output-limiting control over the Detailed section, and sub-section specific detail settings (small hammer), which allow further limiting of the output for a sub-section. Each sub-section may be limited to a specific depth level, and each sub-level may be limited with top N or threshold limits. The levelspec argument to each of the level limiters listed above is used to accomplish this.

It is probably best to continue explanation of sub-level limiting with the following well-known outline-style hierarchy, and some basic examples:


    level 0
       level 1
          level 2
             level 3
                level 4
                level 4
          level 2
             level 3
                level 4
                level 4
                level 4
             level 3
                level 4
             level 3
       level 1
          level 2
             level 3
                level 4

The simplest form of output limiting suppresses all output below a specified level. For example, a levelspec set to "2" shows only data in levels 0 through 2. Think of this as collapsing each sub-level 2 item, thus hiding all inferior levels (3, 4, ...), to yield:


    level 0
       level 1
          level 2
          level 2
       level 1
          level 2

Sometimes the volume of output in a section is too great, and it is useful to suppress any data that does not exceed a certain threshold value. Consider a dictionary spam attack, which produces very lengthy lists of hit-once recipient email or IP addresses. Each sub-level in the hierarchy can be threshold-limited by setting the levelspec appropriately. Setting levelspec to the value "2::5" will suppress any data at level 2 that does not exceed a hit count of 5.

Perhaps producing a top N list, such as top 10 senders, is desired. A levelspec of "3:10:" limits level 3 data to only the top 10 hits.

With those simple examples out of the way, a levelspec is defined as a whitespace- or comma-separated list of one or more of the following:

l: Specifies the maximum level to be output for this sub-section, with a range from 0 to 10. if l is 0, no levels will be output, effectively disabling the sub-section (level 0 data is already provided in the Summary report, so level 1 is considered the first useful level in the Detailed report). Higher values will produce output up to and including the specified level.

l.n: Same as above, with the addition that n limits this section's level 1 output to the top n items. The value for n can be any integer greater than 1. (This form of limiting has less utility than the syntax shown below. It is provided for backwards compatibility; users are encouraged to use the syntax below).

l:n:t: This triplet specifies level l, top n, and minimum threshold t. Each of the values are integers, with l being the level limiter as described above, n being a top n limiter for the level l, and t being the threshold limiter for level l. When both n and t are specified, n has priority, allowing top n lists (regardless of threshold value). If the value of l is omitted, the specified values for n and/or t are used for all levels available in the sub-section. This permits a simple form of wildcarding (eg. place minimum threshold limits on all levels). However, specific limiters always override wildcard limiters. The first form of level limiter may be included in levelspec to restrict output, regardless of how many triplets are present.

All three forms of limiters are effective only when amavis-logwatch's detail level is 5 or greater (the Detailed section is not activated until detail is at least 5).

See the EXAMPLES section for usage scenarios.

CONFIGURATION FILE

Amavis-logwatch can read configuration settings from a configuration file. Essentially, any command line option can be placed into a configuration file, and these settings are read upon startup.

Because amavis-logwatch can run either standalone or within Logwatch, to minimize confusion, amavis-logwatch inherits Logwatch's configuration file syntax requirements and conventions. These are:

•: White space lines are ignored.

•: Lines beginning with # are ignored

•

Settings are of the form:


         option = value

•: Spaces or tabs on either side of the = character are ignored.

•: Any value protected in double quotes will be case-preserved.

•: All other content is reduced to lowercase (non-preserving, case insensitive).

•: All amavis-logwatch configuration settings must be prefixed with " $amavis_" or amavis-logwatch will ignore them.

•: When running under Logwatch, any values not prefixed with " $amavis_" are consumed by Logwatch; it only passes to amavis-logwatch (via environment variable) settings it considers valid.

•: The values True and Yes are converted to 1, and False and No are converted to 0.

•: Order of settings is not preserved within a configuration file (since settings are passed by Logwatch via environment variables, which have no defined order).

To include a command line option in a configuration file, prefix the command line option name with the word " $amavis_". The following configuration file setting and command line option are equivalent:


         $amavis_Line_Style = Truncate

         --line_style Truncate

Level limiters are also prefixed with $amavis_, but on the command line are specified with the --limit option:


         $amavis_SpamBlocked = 2

         --limit SpamBlocked=2

The order of command line options and configuration file processing occurs as follows: 1) The default configuration file is read if it exists and no --config_file was specified on a command line. 2) Configuration files are read and processed in the order found on the command line. 3) Command line options override any options already set either via command line or from any configuration file.

Command line options are interpreted when they are seen on the command line, and later options will override previously set options.

EXIT STATUS

The amavis-logwatch utility exits with a status code of 0, unless an error occurred, in which case a non-zero exit status is returned.

EXAMPLES

Running Standalone

Note: amavis-logwatch reads its log data from one or more named Amavis log files, or from STDIN. For brevity, where required, the examples below use the word file as the command line argument meaning /path/to/amavis.log. Obviously you will need to substitute file with the appropriate path.

To run  amavis-logwatch in standalone mode, simply run:

amavis-logwatch file

A complete list of options and basic usage is available via:

amavis-logwatch --help

To print a summary only report of Amavis log data:

amavis-logwatch --detail 1 file

To produce a summary report and a one-level detail report for May 25th:

grep 'May 25' file | amavis-logwatch --detail 5

To produce only a top 10 list of Sent email domains, the summary report and detailed reports are first disabled. Since commands line options are read and enabled left-to-right, the Sent section is re-enabled to level 1 with a level 1 top 10 limiter:

amavis-logwatch --nosummary --nodetail \
   --limit spamblocked '1 1:10:' file

The following command and its sample output shows a more complex level limiter example. The command gives the top 4 spam blocked recipients (level 1), and under with each recipient the top 2 sending IPs (level 2) and finally below that, only envelope from addresses (level 3) with hit counts greater than 6. Ellipses indicate top N or threshold-limited data:

amavis-logwatch --nosummary --nodetail \
        --limit spamblocked '1:4: 2:2: 3::6' file

19346   Spam blocked -----------------------------------
  756      [email protected]
   12         10.0.0.1
   12            <>
   12         10.99.99.99
   12            <>
	     ...
  640      [email protected]
    8         10.0.0.1
    8            <>
    8         192.168.3.19
    8            <>
	     ...
  595      [email protected]
    8         10.0.0.1
    8            <>
    7         192.168.3.3
    7            <>
	     ...
  547      [email protected]
    8         192.168.3.19
    8            <>
    7         10.0.0.1
    7            <>
 	      ...
	   ...

Running within Logwatch

Note: Logwatch versions prior to 7.3.6, unless configured otherwise, required the --print option to print to STDOUT instead of sending reports via email. Since version 7.3.6, STDOUT is the default output destination, and the --print option has been replaced by --output stdout. Check your configuration to determine where report output will be directed, and add the appropriate option to the commands below.

To print a summary report for today's Amavis log data:

logwatch --service amavis --range today --detail 1

To print a report for today's Amavis log data, with one level
of detail in the  Detailed section:

logwatch --service amavis --range today --detail 5

To print a report for yesterday, with two levels of detail in the Detailed section:

logwatch --service amavis --range yesterday --detail 6

To print a report from Dec 12th through Dec 14th, with four levels of detail in the Detailed section:

logwatch --service amavis --range \
        'between 12/12 and 12/14' --detail 8

To print a report for today, with all levels of detail:

logwatch --service amavis --range today --detail 10

Same as above, but leaves long lines uncropped:

logwatch --service amavis --range today --detail 11

Amavis Log Level

Amavis provides additional log information when the variable $log_level is increased above the default 0 value. This information is used by the amavis-logwatch utility to provide additional reports, not available with the default $log_level=0 value. A $log_level of 2 is suggested.

If you prefer not to increase the noise level in your main mail or Amavis logs, you can configure syslog to log Amavis' output to multiple log files, where basic log entries are routed to your main mail log(s) and more detailed entries routed to an Amavis-specific log file used to feed the amavis-logwatch utility.

A convenient way to accomplish this is to change the Amavis configuration variables in amavisd.conf as shown below:


    amavisd.conf:
        $log_level = 2;
        $syslog_facility = 'local5';
        $syslog_priority = 'debug';

This increases $log_level to 2, and sends Amavis' log entries to an alternate syslog facility (eg. local5, user), which can then be routed to one or more log files, including your main mail log file:


    syslog.conf:
        #mail.info                         -/var/log/maillog
        mail.info;local5.notice            -/var/log/maillog

        local5.info                        -/var/log/amavisd-info.log

Amavis' typical $log_level 0 messages will be directed to both your maillog and to the amavisd-info.log file, but higher $log_level messages will only be routed to the amavisd-info.log file. For additional information on Amavis' logging, search the file RELEASE_NOTES in the Amavis distribution for:


    "syslog priorities are now dynamically derived"

ENVIRONMENT

The amavis-logwatch program uses the following (automatically set) environment variables when running under Logwatch:

LOGWATCH_DETAIL_LEVEL: This is the detail level specified with the Logwatch command line argument --detail or the Detail setting in the ...conf/services/amavis.conf configuration file.

LOGWATCH_DEBUG: This is the debug level specified with the Logwatch command line argument --debug.

amavis_xxx: The Logwatch program passes all settings amavis_xxx in the configuration file ...conf/services/amavis.conf to the amavis filter (which is actually named .../scripts/services/amavis) via environment variable.

FILES

Standalone mode

/usr/local/bin/amavis-logwatch: The amavis-logwatch program

/usr/local/etc/amavis-logwatch.conf: The amavis-logwatch configuration file in standalone mode

Logwatch mode

/etc/logwatch/scripts/services/amavis: The Logwatch amavis filter

/etc/logwatch/conf/services/amavis.conf: The Logwatch amavis filter configuration file

README FILES

README, an overview of  amavis-logwatch
Changes, the version change list history
Bugs, a list of the current bugs or other inadequacies
Makefile, the rudimentary installer
LICENSE, the usage and redistribution licensing terms

LICENSE

Covered under the included MIT/X-Consortium License:
http://www.opensource.org/licenses/mit-license.php

AUTHOR(S)

Mike Cappella

The original amavis Logwatch filter was written by Jim O'Halloran, and has had many contributors over the years. They are entirely not responsible for any errors, problems or failures since the current author's hands have touched the source code.