7.3. Managing incoming and outgoing messages with flow-control
This section describes the internal message-processing model of syslog-ng, as
well as the flow-control feature that can prevent message losses. To use
flow-control, the flow-control flag must
be enabled for the particular log path.
The syslog-ng application monitors (polls) the sources defined in its
configuration file, periodically checking each source for messages. When a log
message is found in one of the sources, syslog-ng polls every source and reads
the available messages. These messages are processed and put into the output
buffer of syslog-ng (also called fifo). From the output buffer, the operating
system sends the messages to the appropriate destinations.
In large-traffic environments many messages can arrive during a single poll
loop, therefore syslog-ng reads only a fixed number of messages from each
source. The log_fetch_limit() option
specifies the number of messages read during a poll loop from a single
source.
Note
The log_fetch_limit() parameter can be
set as a global option, or for every source
individually.
Every destination has its own output buffer. The output buffer is needed
because the destination might not be able to accept all messages immediately.
The log_fifo_size() parameter sets the
size of the output buffer. The output buffer must be larger than the log_fetch_limit() of the sources, to ensure
that every message read during the poll loop fits into the output buffer. If the
log path sends messages to a destination from multiple sources, the output
buffer must be large enough to store the incoming messages of every source.
TCP, unix-stream, and unix-dgram sources can receive the logs from several
incoming connections (e.g., many different clients or applications). For such
sources, syslog-ng reads messages from every connection, thus the log_fetch_limit() parameter applies
individually to every connection of the source. E.g., if there are five
connections logging to the source, syslog-ng reads maximum number_of_connections*log_fetch_limit() = 5*log_fetch_limit() messages. The maximum number
of parallel connections that the source accepts is set using the max_connections() parameter.
The flow-control of syslog-ng introduces a control window to the source that
tracks how many messages can syslog-ng accept from the source. Every message
that syslog-ng reads from the source lowers the window size by one; every
message that syslog-ng successfully sends from the output buffer increases the
window size by one. If the window is full (i.e., its size decreases to zero),
syslog-ng stops reading messages from the source. The initial size of the
control window is by default 100: the log_fifo_size() must be larger than this value
in order for flow-control to have any effect.
When flow-control is used, every connection has its own control window. As a
worst-case situation, the output buffer of the destination must be set to
accommodate all messages of every control window, that is, the log_fifo_size() of the destination must be
greater than max_connections()*log_iw_size(). This applies to every source
that sends logs to the particular destination, thus if two sources having
several connections and heavy traffic send logs to the same destination, the
control window of every connection of both sources must fit into the output
buffer of the destination. Otherwise, syslog-ng does not activate the
flow-control, and messages may be lost.
The summary of the main points is as follows:
The syslog-ng application normally reads a maximum of log_fetch_limit() number of messages from a
source.
From TCP, unix-stream, and unix-dgram sources, syslog-ng reads a maximum of
log_fetch_limit() from every connection of
the source. The number of connections to the source is set using the max_connections() parameter.
Every destination has an output buffer (log_fifo_size()).
Flow-control uses a control window to determine if there is free space in the
output buffer for new messages. Every source or connection has its own control
window; log_iw_size() parameter sets the
size of the control window.
The output buffer must be larger than the control window of every connection
that logs to the destination.
If the output buffer is full, syslog-ng stops reading messages from the
source until some messages are successfully sent to the
destination.
Note
If you modify the max_connections() or
the log_fetch_limit() parameter, do not
forget to adjust the log_iw_size() and log_fifo_size()
parameters accordingly.
Example 7.1. Sizing parameters for
flow-control
Suppose that syslog-ng has a source that must accept up to 300 parallel
connections. Such situation can arise when a network source receives connections
from many clients, or if many applications log to the same socket. Therefore,
set the max_connections() parameter of the
source to 300. However, the log_fetch_limit() (default value: 10) and log_iw_size() (default value: 100) parameters
apply to every connection of the source. In a worst-case scenario, the
destination does not accept any messages, while all 300 connections send at
least log_fetch_limit() number of messages
to the source during every poll loop. Therefore the output buffer of the
destination must accommodate at least max_connections()*log_iw_size() messages, that is: 300*100=30000
messages. That way all incoming messages of ten poll loops fit in the output
buffer. If the output buffer is full, syslog-ng does not read any messages from
the source until some messages are successfully sent to the destination.
source s_localhost {
tcp(ip(127.0.0.1) port(1999) max-connections(300)); };
destination d_tcp {
tcp("10.1.2.3" port(1999); localport(999)); log_fifo_size(30000); };
log { source(s_localhost); destination(d_tcp); flags(flow-control); };
If other sources send messages to this destination, than the output buffer
must be further increased. For example, if a network host with maximum 100 connections also logs into the
destination, than increase the log_fifo_size() by 10000.
source s_localhost {
tcp(ip(127.0.0.1) port(1999) max-connections(300)); };
source s_tcp {
tcp(ip(192.168.1.5) port(1999) max-connections(100)); };
destination d_tcp {
tcp("10.1.2.3" port(1999); localport(999)); log_fifo_size(40000); };
log { source(s_localhost); destination(d_tcp); flags(flow-control); }; 