innfeed.conf - Configuration file for innfeed
The
innfeed.conf file configures to which remote peers
innfeed
sends NNTP feeds.
A common entry to parameterize "news.server.com" as an outgoing feed
is:
peer news.server.com {
ip-name: "news.server.com"
}
If standard NNTP port 119 is not used, you may specify an alternate port as
follows:
peer news.server.com {
ip-name: "news.server.com"
port-number: 433
}
After any changes, run "inncheck" to perform basic syntax checks, and
reload this configuration file with the following command which makes
innd respawn a new instance of
innfeed (assuming
"innfeed!" is the name of the corresponding channel feed in
newsfeeds):
ctlinnd flush innfeed!
The configuration file
innfeed.conf in
pathetc is used to control
the
innfeed(8) program. It is a fairly free-format file that consists
of three types of entries:
key:
value,
peer and
group. Comments are from the hash character "#" to the end of
the line.
key:
value entries are a keyword and a value separated by a colon
(which can itself be surrounded by whitespace). For example:
max-connections: 10
A legal
key starts with a letter and contains only letters, digits, and
the "_" and "-" characters. There are 5 different types of
values: integers, floating-point numbers, characters, booleans, and strings.
Integer and floating-point numbers are as to be expected, except that exponents
in floating-point numbers are not supported. A boolean value is either
"true" or "false" (case is not significant). A character
value is a single-quoted character as defined by the C-language. A string
value is any other sequence of characters. If the string needs to contain
whitespace, then it must be quoted with double quotes, and uses the same
format for embedding non-printing characters as normal C-language string.
Peer entries look like:
peer <name> {
# body ...
}
The word "peer" is required. The
<name> is the same as
the site name in INN's
newsfeeds configuration file. The body of a peer
entry contains some number (possibly zero) of
key:
value entries.
Group entries look like:
group <name> {
# body ...
}
The word "group" is required. The
<name> is any string
valid as a key. The body of a group entry contains any number of the three
types of entries. So
key:
value pairs can be defined inside a
group, and peers can be nested inside a group, and other groups can be nested
inside a group.
key:
value entries that are defined outside of all
peer and
group entries are said to be at "global scope". There are
global
key:
value entries that apply to the process as a whole
(for example the location of the backlog file directory), and there are global
key:
value entries that act as defaults for peers. When
innfeed looks for a specific value in a peer entry (for example, the
maximum number of connections to set up), if the value is not defined in the
peer entry, then the enclosing groups are examined for the entry (starting at
the closest enclosing group). If there are no enclosing groups, or the
enclosing groups do not define the
key:
value, then the value at
global scope is used.
A small example could be:
# Global value applied to all peers that have
# no value of their own.
max-connections: 5
# A peer definition. "uunet" is the name used by innd
# in the newsfeeds configuration file.
peer uunet {
ip-name: usenet1.uu.net
}
peer vixie {
ip-name: gw.home.vix.com
max-connections: 10 # Override global value.
}
# A group of two peers which can handle more connections
# than normal.
group fast-sites {
max-connections: 15
# Another peer. The "max-connections" value from the
# "fast-sites" group scope is used. The "ip-name" value
# defaults to the peer's name.
peer data.ramona.vix.com {
}
peer bb.home.vix.com {
max-connections: 20 # He can really cook.
}
}
Given the above configuration file, the defined peers would have the following
values for the
max-connections key:
uunet 5
vixie 10
data.ramona.vix.com 15
bb.home.vix.com 20
innfeed ignores
key:
value pairs it is not interested in.
Some configuration file values can be set via a command-line option, in which
case that setting overrides the settings in the file.
Configuration files can be included in other configuration files via the syntax:
$INCLUDE filename
There is a maximum nesting depth of 10.
For a fuller example configuration file, see the supplied
innfeed.conf.
The following listing show all the keys that apply to the process as whole.
These are not required (compiled-in defaults are used where needed).
- news-spool
- This key requires a pathname value and defaults to
patharticles in inn.conf. It specifies where the top of the
article spool is. This corresponds to the -a command-line
option.
- input-file
- This key requires a pathname value. It specifies the
pathname (relative to the backlog-directory value) that should be
read in funnel-file mode. This corresponds to giving a filename as an
argument on the command-line (i.e. its presence also implies that
funnel-file mode should be used).
The default is unset; innfeed then runs in channel or batch
mode.
- pid-file
- This key requires a pathname value and defaults to
innfeed.pid. It specifies the pathname (relative to pathrun
in inn.conf) where the pid of the innfeed process should be
stored. This corresponds to the -p command-line option.
- debug-level
- This key defines the debug level for the process. Default
is 0. A non-zero number generates a lot of messages to stderr, or to the
config-defined log-file. This corresponds to the -d
command-line option.
If a file named innfeed.debug exists in the pathlog directory
(as set in inn.conf), then debug-level is automatically set
to 1. This is a cheap way of avoiding continual reloading of the
newsfeeds file when debugging. Note that debug messages still go to
log-file.
- debug-shrinking
- This key requires a boolean value and defaults to false
(the debug file is allowed to grow without bound). If set to true, this
file is truncated when its size reaches a certain limit. See
backlog-limit for more details.
- initial-sleep
- This key requires a positive integer. The default value is
2. It defines the number of seconds to wait when innfeed (or a
fork) starts, before beginning to open connections to remote hosts.
- fast-exit
- This key requires a boolean value and defaults to false. If
set to true, when innfeed receives a SIGTERM or SIGQUIT signal, it
will close its listeners as soon as it can, even if it means dropping
articles.
- use-mmap
- This key requires a boolean value and defaults to true.
When innfeed is given file names to send (a fairly rare use case)
instead of storage API tokens, it specifies whether mmaping should be used
if innfeed has been built with mmap(2) support. If article
data on disk is not in NNTP-ready format (CR/LF at the end of each line),
then after mmaping, the article is read into memory and fixed up, so
mmaping has no positive effect (and possibly some negative effect
depending on your system), and so in such a case this value should be
"false", which corresponds to the -M command-line
option.
- log-file
- This key requires a pathname value and defaults to
innfeed.log. It specifies where any logging messages that could not
be sent via syslog(3) should go (such as those generated when a
positive value for debug-value is used). This corresponds to the
-l command-line option.
This pathname is relative to pathlog in inn.conf.
- log-time-format
- This key requires a format string suitable for
strftime(3). It is used for messages sent via syslog(3) and
to the status-file. Default value is "%a %b %d %H:%M:%S
%Y".
- backlog-directory
- This key requires a pathname value and defaults to
innfeed. It specifies where the current innfeed process
should store backlog files. This corresponds to the -b command-line
option.
This pathname is relative to pathspool in inn.conf.
- backlog-highwater
- This key requires a positive integer value and defaults to
5. It specifies how many articles should be kept on the backlog file queue
before starting to write new entries to disk.
- backlog-ckpt-period
- This key requires a positive integer value and defaults to
30. It specifies how many seconds elapse between checkpoints of the input
backlog file. Too small a number will mean frequent disk accesses; too
large a number will mean after a crash, innfeed will re-offer more
already-processed articles than necessary.
- backlog-newfile-period
- This key requires a positive integer value and defaults to
600. It specifies how many seconds elapse before each check for externally
generated backlog files that are to be picked up and processed.
- backlog-rotate-period
- This key requires a positive integer value and defaults to
60. It specifies how many seconds elapse before innfeed checks for
a manually created backlog file and moves the output backlog file to the
input backlog file.
- dns-retry
- This key requires a positive integer value and defaults to
900. It defines the number of seconds between attempts to re-lookup host
information that previously failed to be resolved.
- dns-expire
- This key requires a positive integer value and defaults to
86400. It defines the number of seconds between refreshes of name to
address DNS translation. This is so long-running processes do not get
stuck with stale data, should peer IP addresses change.
- gen-html
- This key requires a boolean value and defaults to false. It
specifies whether the status-file should be HTML-ified.
- status-file
- This key requires a pathname value and defaults to
innfeed.status. An absolute pathname can be used. It specifies the
pathname (relative to pathhttp when gen-html is true;
otherwise, pathlog as set in inn.conf) where the periodic
status of the innfeed process should be stored. This corresponds to
the -S command-line option.
- connection-stats
- This key requires a boolean value and defaults to false. If
the value is true, then whenever the transmission statistics for a peer
are logged, each active connection logs its own statistics. This
corresponds to the -z command-line option.
- host-queue-highwater
- This key requires a positive integer value and defaults to
10. It defines how many articles will be held internally for a peer before
new arrivals cause article information to be spooled to the backlog
file.
- stats-period
- This key requires a positive integer value and defaults to
600. It defines how many seconds innfeed waits between generating
statistics on transfer rates.
- stats-reset
- This key requires a positive integer value and defaults to
43200. It defines how many seconds innfeed waits before resetting
all internal transfer counters back to zero (after logging one final
time). This is so a innfeed process running more than a day will
generate "final" stats that will be picked up by logfile
processing scripts.
- initial-reconnect-time
- This key requires a positive integer value and defaults to
30. It defines how many seconds to first wait before retrying to reconnect
after a connection failure. If the next attempt fails too, then the
reconnect time is approximately doubled until the connection succeeds, or
max-reconnect-time is reached.
- max-reconnect-time
- This key requires an integer value and defaults to 3600. It
defines the maximum number of seconds to wait between attempt to reconnect
to a peer. The initial value for reconnection attempts is defined by
initial-reconnect-time, and it is doubled after each failure, up to
this value.
- stdio-fdmax
- This key requires a non-negative integer value and defaults
to 0. If the value is greater than zero, then whenever a network socket
file descriptor is created and it has a value less than this, the
file descriptor will be dup'ed to bring the value up greater than this.
This is to leave lower numbered file descriptors free for stdio. Certain
systems, Sun's in particular, require this. SunOS 4.1.x usually requires a
value of 128 and Solaris requires a value of 256. The default if this is
not specified, is 0.
The following keys are used with
imapfeed to authenticate to a remote
host. Several parameters may be included at global scope:
- deliver-authname
- The authname is who you want to authenticate as.
- deliver-password
- This is the appropriate password for authname.
- deliver-username
- The username is who you want to "act" as, that
is, who is actually going to be using the server.
- deliver-realm
- In this case, the "realm" is the realm in which
the specified authname is valid. Currently this is only needed by the
DIGEST-MD5 SASL mechanism.
- deliver-rcpt-to
- A printf(3)-style format string for creating the
envelope recipient address. The pattern MUST include a single string
specifier which will be replaced with the newgroup (e.g.
"bb+%s"). The default is "+%s".
- deliver-to-header
- An optional printf(3)-style format string for
creating a To header field to be prepended to the article. The pattern
MUST include a single string specifier which will be replaced with the
newgroup (e.g. "post+%s@domain"). If not specified, the To
header field will not be prepended.
All the
key:
value pairs mentioned in this section can be specified
at global scope. They may also be specified inside a group or peer definition.
Note that when peers are added dynamically (i.e. when
innfeed receives
an article for an unspecified peer), it will add the peer site using the
parameters specified at global scope.
No keys are currently required. They all have a default value, if not present in
the configuration file.
The following keys are optional:
- article-timeout
- This key requires a non-negative integer value. The default
value is 600. If no articles need to be sent to the peer for this many
seconds, then the peer is considered idle and all its active connections
are torn down.
- response-timeout
- This key requires a non-negative integer value. The default
value is 300. It defines the maximum amount of time to wait for a response
from the peer after issuing a command.
- initial-connections
- This key requires a non-negative integer value. The default
value is 1. It defines the number of connections to be opened immediately
when setting up a peer binding. A value of 0 means no connections will be
created until an article needs to be sent.
- max-connections
- This key requires a positive integer value. The default
value is 2 but may be increased if needed or for large feeds. It defines
the maximum number of connections to run in parallel to the peer. A value
of 0 specifies an unlimited number of maximum connections. In general, use
of an unlimited number of maximum connections is not recommended. Do not
ever set max-connections to zero with dynamic-method 0 set,
as this will saturate peer hosts with connections.
- close-period
- This key requires a positive integer value and defaults to
86400. It is the maximum number of seconds a connection should be kept
open. Some NNTP servers do not deal well with connections being held open
for long periods.
- dynamic-method
- This key requires an integer value between 0 and 3. The
default value is 3. It controls how connections are opened, up to the
maximum specified by max-connections. In general (and specifically,
with dynamic-method 0), a new connection is opened when the current
number of connections is below max-connections, and an article is
to be sent while no current connections are idle. Without further
restraint (i.e. using dynamic-method 0), in practice this means
that max-connections connections are established while articles are
being sent. Use of other dynamic-method settings imposes a further
limit on the amount of connections opened below that specified by
max-connections. This limit is calculated in different ways,
depending of the value of dynamic-method.
Users should note that adding additional connections is not always
productive -- just because opening twice as many connections
results in a small percentage increase of articles accepted by the remote
peer, this may be at considerable resource cost both locally and at the
remote site, whereas the remote site might well have received the extra
articles sent from another peer a fraction of a second later. Opening
large numbers of connections is considered antisocial.
The meanings of the various settings are:
-
0 (no method)
- Increase of connections up to max-connections is
unrestrained.
-
1 (maximize articles per second)
- Connections are increased (up to max-connections)
and decreased so as to maximize the number of articles per second sent,
while using the fewest connections to do this.
-
2 (set target queue length)
- Connections are increased (up to max-connections)
and decreased so as to keep the queue of articles to be sent within the
bounds set by dynamic-backlog-low and dynamic-backlog-high,
while using the minimum resources possible. As the queue will tend to fill
if the site is not keeping up, this method ensures that the maximum number
of articles are offered to the peer while using the minimum number of
connections to achieve this.
-
3 (combination)
- This method uses a combination of methods 1 and 2 above.
For sites accepting a large percentage of articles, method 2 will be used
to ensure these sites are offered as complete a feed as possible. For
sites accepting a small percentage of articles, method 1 is used, to
minimize remote resource usage. For intermediate sites, an appropriate
combination is used.
- dynamic-backlog-low
- This key requires a floating-point value between 0 and 100.
It represents (as a percentage) the low water mark for the host queue. If
the host queue falls below this level while using dynamic-method 2
or 3, and if 2 or more connections are open, innfeed will attempt
to drop connections to the host. An Infinite Impulse Response (IIR) filter
is applied to the value to prevent connection flap (see
dynamic-filter). The default value is 20.0. This value must be
smaller than dynamic-backlog-high.
- dynamic-backlog-high
- This key requires a floating-point value between 0 and 100.
It represents (as a percentage) the high water mark for the host queue. If
the host queue rises above this level while using dynamic-method 2
or 3, and if less than max-connections are open to the host,
innfeed will attempt to open further connections to the host. An
Infinite Impulse Response (IIR) filter is applied to the value to prevent
connection flap (see dynamic-filter). The default value is 50.0.
This value must be larger than dynamic-backlog-low.
- dynamic-backlog-filter
- This key requires a floating-point value between 0 and 1.
It represents the filter coefficient used by the Infinite Impulse Response
(IIR) filter used to implement dynamic-method 2 and 3. The default
value of this filter is 0.7, giving a time constant of 1/(1-0.7) articles.
Higher values will result in slower response to queue fullness changes;
lower values in faster response.
- max-queue-size
- This key requires a positive integer value. The default
value is 20. It defines the maximum number of articles to process at one
time when using streaming to transmit to a peer. Larger numbers mean more
memory consumed as articles usually get pulled into memory (see the
description of use-mmap).
- streaming
- This key requires a boolean value. Its default value is
true. It defines whether streaming commands are used to transmit articles
to the peers.
- no-check-high
- This key requires a floating-point number which must be in
the range [0.0, 100.0]. When running transmitting with the streaming
commands, innfeed attempts an optimization called "no-CHECK
mode". This involves not asking the peer if it wants the
article, but just sending it. This optimization occurs when the percentage
of the articles the peer has accepted gets larger than this number. If
this value is set to 100.0, then this effectively turns off no-CHECK mode,
as the percentage can never get above 100.0. If this value is too small,
then the number of articles the peer rejects will get bigger (and your
bandwidth will be wasted). The default value of 95.0 usually works pretty
well.
- no-check-low
- This key requires a floating-point number which must be in
the range [0.0, 100.0], and it must be smaller that the value for
no-check-high. When running in no-CHECK mode, as described above,
if the percentage of articles the remote server accepts drops below this
number, then the no-CHECK optimization is turned off until the percentage
gets above the no-check-high value again. If there is small
difference between this and the no-check-high value (less than
about 5.0), then innfeed may frequently go in and out of no-CHECK
mode. If the difference is too big, then it will make it harder to get out
of no-CHECK mode when necessary (wasting bandwidth). Keeping this to
between 5.0 and 10.0 less than no-check-high usually works pretty
well. The default value is 90.0.
- no-check-filter
- This is a floating-point value representing the time
constant, in articles, over which the CHECK/no-CHECK calculations are
done. The default value is 50.0, which will implement an Infinite Impulse
Response (IIR) filter of time constant 50. This roughly equates to making
a decision about the mode over the previous 50 articles. A higher number
will result in a slower response to changing percentages of articles
accepted; a lower number will result in a faster response.
- port-number
- This key requires a positive integer value. It defines the
TCP/IP port number to use when connecting to the remote. Usually, port
number 119 is used, which is the default value.
- force-ipv4
- This key requires a boolean value. By default, it is set to
false. Setting it to true is the same as setting bindaddress6 to
"none" and removing bindaddress from "none" if
it was set.
- drop-deferred
- This key requires a boolean value. By default, it is set to
false. When set to true, and a peer replies with code 431 or 436 (try
again later), innfeed just drops the article and does not try to
re-send it. This is useful for some peers that keep on deferring articles
for a long time to prevent innfeed from trying to offer the same
article over and over again.
- min-queue-connection
- This key requires a boolean value. By default, it is set to
false. When set to true, innfeed will attempt to use a connection
with the least queue size (or the first empty connection). If this key is
set to true, it is recommended that dynamic-method be set to 0.
This allows for article propagation with the least delay.
- no-backlog
- This key requires a boolean value. It specifies whether
spooling should be enabled (false, the default) or disabled (true). Note
that when no-backlog is set, articles reported as spooled are
actually silently discarded.
- backlog-limit
- This key requires a non-negative integer value. If the
number is 0 (the default), then backlog files are allowed to grow without
bound when the peer is unable to keep up with the article flow. If this
number is greater than 0, then it specifies the size (in bytes) the
backlog file should get truncated to when the backlog file reaches a
certain limit. The limit depends on whether backlog-factor or
backlog-limit-highwater is used.
This parameter also applies to the debug file when debug-shrinking is
set to true, and has the same effect on this file as the one has on
backlog files.
- backlog-factor
- This key requires a floating-point value, which must be
larger than 1.0. It is used in conjunction with the peer key
backlog-limit. If backlog-limit has a value greater than
zero, then when the backlog file gets larger than the value
backlog-limit * backlog-factor, then the backlog file will
be truncated to the size backlog-limit.
For example, if backlog-limit has a value of 1000000, and
backlog-factor has a value of 2.0, then when the backlog file gets
to be larger than 2000000 bytes in size, it will be truncated to 1000000
bytes. The front portion of the file is removed, and the trimming happens
on line boundaries, so the final size may be a bit less than this number.
If backlog-limit-highwater is defined too, then
backlog-factor takes precedence. The default value of
backlog-factor is 1.1.
This parameter also applies to the debug file when debug-shrinking is
set to true, and has the same effect on this file as the one has on
backlog files.
- backlog-limit-highwater
- This key requires a positive integer value that must be
larger than the value for backlog-limit. The default value is 0.
If the size of the backlog file gets larger than this value (in bytes), then
the backlog file will be shrunk down to the size of backlog-limit.
If both backlog-factor and backlog-limit-highwater are
defined, then the value of backlog-factor is used.
This parameter also applies to the debug file when debug-shrinking is
set to true, and has the same effect on this file as the one has on
backlog files.
- backlog-feed-first
- This key requires a boolean value. By default it is set to
false. When set to true, the backlog is fed before new files. This is
intended to enforce in-order delivery, so setting this to true when
initial-connections or max-connections is more than 1 is
inconsistent.
- bindaddress
- This key requires a string value. It specifies which
outgoing IPv4 address innfeed should bind the local end of its
connection to. It must be an IPv4 address in dotted-quad format
(nnn.nnn.nnn.nnn), "any", or "none". If not set or set
to "any", innfeed defaults to letting the kernel choose
this address. If set to "none", innfeed will not use IPv4
for outgoing connections to peers in this scope (i.e. it forces IPv6).
If not set in innfeed.conf, innfeed defaults to the value of
sourceaddress from inn.conf (which by default is
unset).
- bindaddress6
- This key requires a string value. It behaves like
bindaddress except for outgoing IPv6 connections. It must be in
numeric IPv6 format (note that a value containing colons must be enclosed
in double quotes), "any", or "none". If set to
"none", innfeed will not use IPv6 for outgoing
connections to peers in this scope.
If not set in innfeed.conf, innfeed defaults to the value of
sourceaddress6 from inn.conf (which by default is
unset).
- username
- This key requires a string value. If the value is defined,
then innfeed tries to authenticate by AUTHINFO USER and this value
used for user name. password must also be defined, if this key is
defined.
- password
- This key requires a string value. The value is the password
used for AUTHINFO PASS. username must also be defined, if this key
is defined.
As previously explained, the peer definitions can contain redefinitions of any
of the
key:
value pairs described in the section about global
peer defaults above. There is one
key:
value pair that is
specific to a peer definition.
- ip-name
- This key requires a word value. The word is either one of
the host's FQDNs, or the dotted-quad IP address of the peer for IPv4, or
the colon-separated IP address of the peer for IPv6. If this value is not
specified, then the name of the peer in the enclosing peer block is
taken to also be its ip-name.
If
innfeed gets a SIGHUP signal, then it will reread the configuration
file. All values at global scope except for
backlog-directory can be
changed (although note that
bindaddress and
bindaddress6 changes
will only affect new connections).
Any new peers are added and any missing peers have their connections closed.
The log file is also reopened.
For a comprehensive example, see the sample
innfeed.conf distributed with
INN and installed as a starting point.
Here are examples of how to format values:
eg-string: "New\tconfig\tfile\n"
eg-long-string: "A long string that goes
over multiple lines. The
newline is kept in the
string except when quoted
with a backslash \
as here."
eg-simple-string: A-no-quote-string
eg-integer: 10
eg-boolean: true
eg-char: 'a'
eg-ctrl-g: '\007'
Written by James Brister <
[email protected]> for InterNetNews. Converted to
POD by Julien Elie.
Earlier versions of
innfeed (up to 0.10.1) were shipped separately;
innfeed is now part of INN and shares the same version number. Please
note that the
innfeed.conf format has changed dramatically since
version 0.9.3.
inn.conf(5),
innfeed(8),
newsfeeds(5).