varnishncsa - Display Varnish logs in Apache / NCSA combined log format
varnishncsa [-a] [-b] [-c] [-C] [-d] [-D] [-E] [-F <format>] [-f
<formatfile>] [-g <request|vxid>] [-h] [-j] [-L <limit>] [-n
<dir>] [-P <file>] [-Q <file>] [-q <query>] [-r
<filename>] [-R <limit[/duration]>] [-t <seconds|off>] [-V]
[-w <filename>]
The varnishncsa utility reads
varnishd(1) shared memory logs and presents them
in the Apache / NCSA "combined" log format.
Each log line produced is based on a single Request type transaction gathered
from the shared memory log. The Request transaction is then scanned for the
relevant parts in order to output one log line. To filter the log lines
produced, use the query language to select the applicable transactions.
Non-request transactions are ignored.
The following options are available:
- -a
- When writing output to a file, append to it rather than
overwrite it. This option has no effect without the -w option.
- -b
- Log backend requests. If -c is not specified, then only
backend requests will trigger log lines.
- -c
- Log client requests. This is the default. If -b is
specified, then -c is needed to also log client requests
- -C
- Do all regular expression and string matching
caseless.
- -d
- Process log records at the head of the log and exit.
- -D
- Daemonize.
- -E
- Show ESI requests, implies client mode.
-
-F <format>
- Set the output log format string.
-
-f <formatfile>
- Read output format from a file. Will read a single line
from the specified file, and use that line as the format.
-
-g <request|vxid>
- The grouping of the log records. The default is to group by
vxid.
- -h
- Print program usage and exit
- -j
- Make format-specifier replacements JSON-compatible. When
escaping characters, use JSON-style \uXXXX escape sequences instead of
C-style \xXX sequences. Empty strings will be replaced with ""
instead of "-", and empty integers will be replaced with null.
Use -F or -f in combination with -j to write JSON logs.
-
-L <limit>
- Sets the upper limit of incomplete transactions kept before
the oldest transaction is force completed. A warning record is synthesized
when this happens. This setting keeps an upper bound on the memory usage
of running queries. Defaults to 1000 transactions.
-
-n <dir>
- Specify the varnishd working directory (also known as
instance name) to get logs from. If -n is not specified, the host name is
used.
-
-P <file>
- Write the process' PID to the specified file.
-
-Q <file>
- Specifies the file containing the VSL query to use. When
multiple -Q or -q options are specified, all queries are considered as if
the 'or' operator was used to combine them.
-
-q <query>
- Specifies the VSL query to use. When multiple -q or -Q
options are specified, all queries are considered as if the 'or' operator
was used to combine them.
-
-r <filename>
- Read log in binary file format from this file. The file can
be created with varnishlog -w filename. If the filename is -, logs
are read from the standard input. and cannot work as a daemon.
-
-R <limit[/duration]>
- Restrict the output to the specified limit. Transactions
exceeding the limit will be suppressed. The limit is specified as the
maximum number of transactions (with respect to the chosen grouping
method) and an optional time period. If no duration is specified, a
default of s is used. The duration field can be formatted as in VCL
(e.g. -R 10/2m) or as a simple time period without the prefix (e.g.
-R 5/m). When in -g raw grouping mode, this setting can not
be used alongside -i, -I, -x or -X, and we
advise using -q instead.
-
-t <seconds|off>
- Timeout before returning error on initial VSM connection.
If set the VSM connection is retried every 0.5 seconds for this many
seconds. If zero the connection is attempted only once and will fail
immediately if unsuccessful. If set to "off", the connection
will not fail, allowing the utility to start and wait indefinetely for the
Varnish instance to appear. Defaults to 5 seconds.
- -V
- Print version information and exit.
-
-w <filename>
- Redirect output to file. The file will be overwritten
unless the -a option was specified. If the application receives a SIGHUP
in daemon mode the file will be reopened allowing the old one to be
rotated away. This option is required when running in daemon mode. If the
filename is -, varnishncsa writes to the standard output and cannot work
as a daemon.
- --optstring
- Print the optstring parameter to getopt(3) to help
writing wrapper scripts.
The default mode of varnishncsa is "client mode". In this mode, the
log will be similar to what a web server would produce in the absence of
varnish. Client mode can be explicitly selected by using -c.
If the -b switch is specified, varnishncsa will operate in "backend
mode". In this mode, requests generated by varnish to the backends will
be logged. Unless -c is also specified, client requests received by varnish
will be ignored.
When running varnishncsa in both backend and client mode, it is strongly advised
to include the format specifier %{Varnish:side}x to distinguish between
backend and client requests.
Client requests that results in a pipe (ie. return(pipe) in vcl), will not
generate logging in backend mode. This is because varnish is not generating
requests, but blindly passes on bytes in both directions. However, a
varnishncsa instance running in normal mode can see this case by using the
formatter %{Varnish:handling}x, which will be 'pipe'.
In backend mode, some of the fields in the format string get different meanings.
Most notably, the byte counting formatters (%b, %I, %O) considers varnish to
be the client.
It is possible to keep two varnishncsa instances running, one in backend mode,
and one in client mode, logging to different files.
Specify the log format to use. If no format is specified the default log format
is used:
%h %l %u %t "%r" %s %b "%{Referer}i" "%{User-agent}i"
Escape sequences \n and \t are supported.
Supported formatters are:
- %b
- In client mode, size of response in bytes, excluding HTTP
headers. In backend mode, the number of bytes received from the backend,
excluding HTTP headers. In CLF format, i.e. a '-' rather than a 0 when no
bytes are sent.
- %D
- In client mode, time taken to serve the request, in
microseconds. In backend mode, time from the request was sent to the
entire body had been received. This is equivalent to %{us}T.
- %H
- The request protocol. Defaults to HTTP/1.0 if not
known.
- %h
- Remote host. Defaults to '-' if not known. In backend mode
this is the IP of the backend server.
- %I
- In client mode, total bytes received from client. In
backend mode, total bytes sent to the backend.
- %{X}i
- The contents of request header X. If the header appears
multiple times in a single transaction, the last occurrence is used.
- %l
- Remote logname. Always '-'.
- %m
- Request method. Defaults to '-' if not known.
- %{X}o
- The contents of response header X. If the header appears
multiple times in a single transaction, the last occurrence is used.
- %O
- In client mode, total bytes sent to client. In backend
mode, total bytes received from the backend.
- %q
- The query string. Defaults to an empty string if not
present.
- %r
- The first line of the request. Synthesized from other
fields, so it may not be the request verbatim. See the NOTES section.
- %s
- Status sent to the client. In backend mode, status received
from the backend.
- %t
- In client mode, time when the request was received, in HTTP
date/time format. In backend mode, time when the request was sent.
- %{X}t
- In client mode, time when the request was received, in the
format specified by X. In backend mode, time when the request was sent.
The time specification format is the same as for strftime(3) with these
extensions:
- •
-
%{sec}: number of seconds since the Epoch
- •
-
%{msec}: number of milliseconds since the Epoch
- •
-
%{usec}: number of milliseconds since the Epoch
- •
-
%{msec_frac}: millisecond fraction
- •
-
%{usec_frac}: microsecond fraction
The extensions can not be combined with each other or
strftime(3) in the same
specification. Use multiple
%{X}t specifications instead.
- %T
- In client mode, time taken to serve the request, in
seconds. In backend mode, time from the request was sent to the entire
body had been received. This is equivalent to %{s}T.
- %{X}T
- In client mode, time taken to serve the request, in the
format specified by X. In backend mode, time from the request was sent to
the entire body had been received. The time specification format can be
one of the following: s (same as %T), ms or us (same as %D).
- %U
- The request URL without the query string. Defaults to '-'
if not known.
- %u
- Remote user from auth.
- %{X}x
- Extended variables. Supported variables are:
- Varnish:time_firstbyte
- Time from when the request processing starts until the
first byte is sent to the client, in seconds. For backend mode: Time from
the request was sent to the backend to the entire header had been
received.
- Varnish:hitmiss
- One of the 'hit' or 'miss' strings, depending on whether
the request was a cache hit or miss. Pipe, pass and synth are considered
misses.
- Varnish:handling
- One of the 'hit', 'miss', 'pass', 'pipe' or 'synth' strings
indicating how the request was handled.
- Varnish:side
- Backend or client side. One of two values, 'b' or 'c',
depending on where the request was made. In pure backend or client mode,
this field will be constant.
- Varnish:vxid
- The VXID of the varnish transaction.
- VCL_Log:key
- The value set by std.log("key:value") in
VCL.
- VSL:tag:record-prefix[field]
- The value of the VSL entry for the given tag-record
prefix-field combination. Tag is mandatory, the other components are
optional.
The record prefix will limit the matches to those records that have this
prefix as the first part of the record content followed by a colon.
The field will, if present, treat the log record as a white space separated
list of fields, and only the nth part of the record will be matched
against. Fields start counting at 1 and run up to 255.
Defaults to '-' when the tag is not seen, the record prefix does not match
or the field is out of bounds. If a tag appears multiple times in a single
transaction, the first occurrence is used.
- •
- SIGHUP
Rotate the log file (see -w option) in daemon mode, abort the loop and die
gracefully when running in the foreground.
- •
- SIGUSR1
Flush any outstanding transactions.
The %r formatter is equivalent to "%m
http://%{Host}i%U%q %H".
This differs from apache's %r behavior, equivalent to "%m %U%q %H".
Furthermore, when using the %r formatter, if the Host header appears multiple
times in a single transaction, the first occurrence is used.
Log the second field of the Begin record, corresponding to the VXID of the
parent transaction:
varnishncsa -F "%{VSL:Begin[2]}x"
Log the entire Timestamp record associated with the processing length:
varnishncsa -F "%{VSL:Timestamp:Process}x"
Log in JSON, using the -j flag to ensure that the output is valid JSON for all
inputs:
varnishncsa -j -F '{"size": %b, "time": "%t", "ua": "%{User-Agent}i"}'
varnishd(1) varnishlog(1) varnishstat(1) vsl(7)
The varnishncsa utility was developed by Poul-Henning Kamp in cooperation with
Verdens Gang AS and Varnish Software AS. This manual page was initially
written by Dag-Erling Smørgrav <
[email protected]>, and later
updated by Martin Blix Grydeland and Pål Hermunn Johansen.
This document is licensed under the same licence as Varnish itself. See LICENCE
for details.
- •
- Copyright (c) 2006 Verdens Gang AS
- •
- Copyright (c) 2006-2016 Varnish Software AS