vsl-query - Varnish VSL Query Expressions
The Varnish VSL Query Expressions extracts transactions from the Varnish shared
memory log, and perform queries on the transactions before reporting matches.
A transaction is a set of log lines that belongs together, e.g. a client request
or a backend request. The API monitors the log, and collects all log records
that make up a transaction before reporting on that transaction. Transactions
can also be grouped, meaning backend transactions are reported together with
the client transaction that initiated it.
A query is run on a group of transactions. A query expression is true if there
is a log record within the group that satisfies the condition. It is false
only if none of the log records satisfies the condition. Query expressions can
be combined using boolean functions. In addition to log records, it is
possible to query transaction ids (vxid) in query.
When grouping transactions, there is a hierarchy structure showing which
transaction initiated what. The level increases by one on an 'initiated by'
relation, so for example a backend transaction will have one higher level than
the client transaction that initiated it on a cache miss. Request restart
transactions don't get their level increased to make it predictable.
Levels start counting at 1, except when using raw where it will always be 0.
The grouping modes are:
- •
-
session
All transactions initiated by a client connection are reported together.
Client connections are open ended when using HTTP keep-alives, so it is
undefined when the session will be reported. If the transaction timeout
period is exceeded an incomplete session will be reported.
Non-transactional data (vxid == 0) is not reported.
- •
-
request
Transactions are grouped by request, where the set will include the request
itself as well as any backend requests or ESI-subrequests. Session data
and non-transactional data (vxid == 0) is not reported.
- •
-
vxid
Transactions are not grouped, so each vxid is reported in its entirety.
Sessions, requests, ESI-requests and backend requests are all reported
individually. Non-transactional data is not reported (vxid == 0). This is
the default.
- •
-
raw
Every log record will make up a transaction of its own. All data, including
non-transactional data will be reported.
Example transaction hierarchy using request grouping mode
Lvl 1: Client request (cache miss)
Lvl 2: Backend request
Lvl 2: ESI subrequest (cache miss)
Lvl 3: Backend request
Lvl 3: Backend request (VCL restart)
Lvl 3: ESI subrequest (cache miss)
Lvl 4: Backend request
Lvl 2: ESI subrequest (cache hit)
The API will use pointers to shared memory log data as long as possible to keep
memory usage at a minimum. But as the shared memory log is a ring buffer, data
will get overwritten eventually, so the API creates local copies of referenced
log data when varnishd comes close to overwriting still unreported content.
This process avoids loss of log data in many scenarios, but it is not failsafe:
Overruns where varnishd "overtakes" the log reader process in the
ring buffer can still happen when API clients cannot keep up reading and/or
copying, for instance due to output blocking.
Though being unrelated to grouping in principle, copying of log data is
particularly relevant for session grouping together with long lasting client
connections - for this grouping, the logging API client process is likely to
consume relevant amounts of memory. As the vxid grouping also logs
(potentially long lasting) sessions, it is also likely to require memory for
copies of log entries, but far less than session grouping.
A query expression consists of record selection criteria, and optionally an
operator and a value to match against the selected records.
<record selection criteria> <operator> <operand>
Additionally, a query expression can occur on the transaction itself rather than
log records belonging to the transaction.
vxid <numerical operator> <integer>
A
vxid query allows you to directly target a specific transacion, whose
id can be obtained from an
X-Varnish HTTP header, the default
"guru meditation" error page, or
Begin and
Link log
records.
A query must fit on a single line, but it is possible to pass multiple queries
at once, one query per line. Empty lines are ignored, and the list of queries
is treated as if the 'or' operator was used to combine them.
For example this list of queries:
# catch varnish errors
*Error
# catch backend errors
BerespStatus >= 500
is identical to this query:
(*Error) or (BerespStatus >= 500)
Comments can be used and will be ignored, they start with the
'#'
character, which may be more useful when the query is read from a file.
For very long queries that couldn't easily be split into multiple queries it is
possible to break them into multiple lines with a backslash preceding an end
of line.
For example this query:
is identical to this query:
A backslash-newline sequence doesn't continue a comment on the next line and
isn't allowed in a quoted string.
The record selection criteria determines what kind records from the transaction
group the expression applies to. Syntax:
{level}taglist:record-prefix[field]
Taglist is mandatory, the other components are optional.
The level limits the expression to a transaction at that level. If left
unspecified, the expression is applied to transactions at all levels. Level is
a positive integer or zero. If level is followed by a '+' character, it
expresses greater than or equal. If level is followed by a '-', it expresses
less than or equal.
The taglist is a comma-separated list of VSL record tags that this expression
should be checked against. Each list element can be a tag name or a tag glob.
Globs allow a '*' either in the beginning of the name or at the end, and will
select all tags that match either the prefix or subscript. A single '*' will
select all tags.
The record prefix will further limit the matches to those records that has this
prefix as their first part of the record content followed by a colon. The part
of the log record matched against will then be limited to what follows the
prefix and colon. This is useful when matching against specific HTTP headers.
The record prefix matching is done case insensitive.
The field will, if present, treat the log record as a white space separated list
of fields, and only the nth part of the record will be matched against. Fields
start counting at 1.
An expression using only a record selection criteria will be true if there is
any record in the transaction group that is selected by the criteria.
The following matching operators are available:
- •
- == != < <= > >=
Numerical comparison. The record contents will be converted to either an
integer or a float before comparison, depending on the type of the
operand.
- •
- eq ne
String comparison. 'eq' tests string equality, 'ne' tests for not
equality.
- •
- ~ !~
Regular expression matching. '~' is a positive match, '!~' is a
non-match.
The operand is the value the selected records will be matched against.
An operand can be quoted or unquoted. Quotes can be either single or double
quotes, and for quoted operands a backslash can be used to escape the quotes.
Unquoted operands can only consist of the following characters:
The following types of operands are available:
- •
- Integer
A number without any fractional part, valid for the numerical comparison
operators. The integer type is used when the operand does not contain any
period (.) nor exponent (e) characters. However if the record evaluates as
a float, only its integral part is used for the comparison.
- •
- Float
A number with a fractional part, valid for the numerical comparison
operators. The float type is used when the operand does contain a period
(.) or exponent (e) character.
- •
- String
A sequence of characters, valid for the string equality operators.
- •
- Regular expression
A PCRE2 regular expression. Valid for the regular expression operators.
Query expressions can be linked together using boolean functions. The following
are available, in decreasing precedence:
- •
- not <expr>
Inverts the result of <expr>
- •
- <expr1> and <expr2>
True only if both expr1 and expr2 are true
- •
- <expr1> or <expr2>
True if either of expr1 or expr2 is true
Expressions can be grouped using parenthesis.
- •
- Transaction group contains a request URL that equals to
"/foo"
- •
- Transaction group contains a request cookie header
- •
- Transaction group doesn't contain a request cookie
header
- •
- Client request where internal handling took more than
800ms.:
Timestamp:Process[2] > 0.8
- •
- Transaction group contains a request user-agent header that
contains "iPod" and the request delivery time exceeds 1
second
ReqHeader:user-agent ~ "iPod" and Timestamp:Resp[2] > 1.
- •
- Transaction group contains a backend response status larger
than or equal to 500
- •
- Transaction group contains a request response status of
304, but where the request did not contain an if-modified-since
header
RespStatus == 304 and not ReqHeader:if-modified-since
- •
- Transactions that have had backend failures or long
delivery time on their ESI subrequests. (Assumes request grouping
mode).
BerespStatus >= 500 or {2+}Timestamp:Process[2] > 1.
- •
- Log non-transactional errors. (Assumes raw grouping
mode).
This document was initially written by Martin Blix Grydeland and amended by
others.
This document is licensed under the same licence as Varnish itself. See LICENCE
for details.
- •
- Copyright (c) 2006 Verdens Gang AS
- •
- Copyright (c) 2006-2015 Varnish Software AS