Chinaunix首页 | 论坛 | 博客
  • 博客访问: 2209707
  • 博文数量: 668
  • 博客积分: 10016
  • 博客等级: 上将
  • 技术积分: 8588
  • 用 户 组: 普通用户
  • 注册时间: 2008-05-29 19:22
文章分类

全部博文(668)

文章存档

2011年(1)

2010年(2)

2009年(273)

2008年(392)

分类:

2008-07-14 14:16:33

7.3. Managing incoming and outgoing messages with flow-control
This section describes the internal message-processing model of syslog-ng, as
well as the flow-control feature that can prevent message losses. To use
flow-control, the flow-control flag must
be enabled for the particular log path.
The syslog-ng application monitors (polls) the sources defined in its
configuration file, periodically checking each source for messages. When a log
message is found in one of the sources, syslog-ng polls every source and reads
the available messages. These messages are processed and put into the output
buffer of syslog-ng (also called fifo). From the output buffer, the operating
system sends the messages to the appropriate destinations.
In large-traffic environments many messages can arrive during a single poll
loop, therefore syslog-ng reads only a fixed number of messages from each
source. The log_fetch_limit() option
specifies the number of messages read during a poll loop from a single
source.
Click here to open new window
CTRL+Mouse wheel to zoom in/out
Note
The log_fetch_limit() parameter can be
set as a global option, or for every source
individually.
Every destination has its own output buffer. The output buffer is needed
because the destination might not be able to accept all messages immediately.
The log_fifo_size() parameter sets the
size of the output buffer. The output buffer must be larger than the log_fetch_limit() of the sources, to ensure
that every message read during the poll loop fits into the output buffer. If the
log path sends messages to a destination from multiple sources, the output
buffer must be large enough to store the incoming messages of every source.
TCP, unix-stream, and unix-dgram sources can receive the logs from several
incoming connections (e.g., many different clients or applications). For such
sources, syslog-ng reads messages from every connection, thus the log_fetch_limit() parameter applies
individually to every connection of the source. E.g., if there are five
connections logging to the source, syslog-ng reads maximum number_of_connections*log_fetch_limit() = 5*log_fetch_limit() messages. The maximum number
of parallel connections that the source accepts is set using the max_connections() parameter.
The flow-control of syslog-ng introduces a control window to the source that
tracks how many messages can syslog-ng accept from the source. Every message
that syslog-ng reads from the source lowers the window size by one; every
message that syslog-ng successfully sends from the output buffer increases the
window size by one. If the window is full (i.e., its size decreases to zero),
syslog-ng stops reading messages from the source. The initial size of the
control window is by default 100: the log_fifo_size() must be larger than this value
in order for flow-control to have any effect.
When flow-control is used, every connection has its own control window. As a
worst-case situation, the output buffer of the destination must be set to
accommodate all messages of every control window, that is, the log_fifo_size() of the destination must be
greater than max_connections()*log_iw_size(). This applies to every source
that sends logs to the particular destination, thus if two sources having
several connections and heavy traffic send logs to the same destination, the
control window of every connection of both sources must fit into the output
buffer of the destination. Otherwise, syslog-ng does not activate the
flow-control, and messages may be lost.
The summary of the main points is as follows:

  • The syslog-ng application normally reads a maximum of log_fetch_limit() number of messages from a
    source.

  • From TCP, unix-stream, and unix-dgram sources, syslog-ng reads a maximum of
    log_fetch_limit() from every connection of
    the source. The number of connections to the source is set using the max_connections() parameter.

  • Every destination has an output buffer (log_fifo_size()).

  • Flow-control uses a control window to determine if there is free space in the
    output buffer for new messages. Every source or connection has its own control
    window; log_iw_size() parameter sets the
    size of the control window.

  • The output buffer must be larger than the control window of every connection
    that logs to the destination.

  • If the output buffer is full, syslog-ng stops reading messages from the
    source until some messages are successfully sent to the
    destination.

  • Note
    If you modify the max_connections() or
    the log_fetch_limit() parameter, do not
    forget to adjust the log_iw_size() and log_fifo_size()
    Click here to open new window
CTRL+Mouse wheel to zoom in/out
    parameters accordingly.

      Example 7.1. Sizing parameters for
      flow-control
      Suppose that syslog-ng has a source that must accept up to 300 parallel
      connections. Such situation can arise when a network source receives connections
      from many clients, or if many applications log to the same socket. Therefore,
      set the max_connections() parameter of the
      source to 300. However, the log_fetch_limit() (default value: 10) and log_iw_size() (default value: 100) parameters
      apply to every connection of the source. In a worst-case scenario, the
      destination does not accept any messages, while all 300 connections send at
      least log_fetch_limit() number of messages
      to the source during every poll loop. Therefore the output buffer of the
      destination must accommodate at least max_connections()*log_iw_size() messages, that is: 300*100=30000
      messages. That way all incoming messages of ten poll loops fit in the output
      buffer. If the output buffer is full, syslog-ng does not read any messages from
      the source until some messages are successfully sent to the destination.
      source s_localhost {
                        tcp(ip(127.0.0.1) port(1999) max-connections(300)); };
      destination d_tcp {
                                      tcp("10.1.2.3" port(1999); localport(999)); log_fifo_size(30000); };
      log { source(s_localhost); destination(d_tcp); flags(flow-control); };   
                             
      If other sources send messages to this destination, than the output buffer
      must be further increased. For example, if a network host with maximum 100 connections also logs into the
      destination, than increase the log_fifo_size() by 10000.
      source s_localhost {
                              tcp(ip(127.0.0.1) port(1999) max-connections(300)); };
      source s_tcp {
                              tcp(ip(192.168.1.5) port(1999) max-connections(100)); };
      destination d_tcp {
                              tcp("10.1.2.3" port(1999); localport(999)); log_fifo_size(40000); };
      log { source(s_localhost); destination(d_tcp); flags(flow-control); };   
    • 阅读(1693) | 评论(0) | 转发(0) |
      给主人留下些什么吧!~~