syslog-ng

What is Syslog-ng?

syslog-ng(next generation) is an open source implementation of the syslog protocol for UNIX and UNIX-like systems. It extends the original syslogd model with content-based filtering, rich filtering capabilities, flexible configuration options and adds important features to syslog, like using tcp for transport. The original syslog protocol separates messages based on the priority of the message and the facility sending the message. These two parameters alone are often inadequate to consistently classify messages, as many applications might use the same facility — and the facility itself is not even included in the log message. To make things worse, many log messages contain unimportant information. The syslog-ng application helps us to select only the really interesting messages, and forward them to a central server. It may often require log messages to be archived. Storing the important messages in a central location greatly simplifies this process.

How Syslog-ng works?

The syslog-ng application reads incoming messages and forwards them to the selected destinations. The syslog-ng application can receive messages from files, remote hosts, and other sources. Log messages enter syslog-ng in one of the defined sources, and are sent to one or more destinations. Sources and destinations are independent objects; log paths define what syslog-ng does with a message, connecting the sources to the destinations. A log path consists of one or more sources and one or more destinations; messages arriving to a source are sent to every destination listed in the log path. Log paths can include filters. Filters are rules that select only certain messages, for example, selecting only messages sent by a specific application. If a log path includes filters, syslog-ng sends only the messages satisfying the filter rules to the destinations set in the log path. The following procedure illustrates the route of a log message from its source on the syslog-ng client to its final destination on the central syslog-ng server.

1. A device or application sends a log message to a source on the syslog-ng client. For example, an Apache web server running on Linux enters a message into the /var/log/apache file, or an application running on Windows enters a message into the Eventlog.

2. The syslog-ng client running on the web server reads the message from its /var/log/apache or Eventlog source.

3. The syslog-ng client processes the first log statement that includes the /var/log/apache or the Eventlog source.

4. The syslog-ng client compares the message to the filters of the log statement (if any). If the message complies with all filter rules, syslog-ng sends the message to the destinations set in the log statement, for example, to the remote syslog-ng server.

5. The syslog-ng client processes the next log statement that includes the source of the message, repeating Steps 3-4.

6. The message sent by the syslog-ng client arrives to a source set in the syslog-ng server.

7. The syslog-ng server reads the message from its source and processes the first log statement that includes that source.

8. The syslog-ng server compares the message to the filters of the log statement (if any). If the message complies with all filter rules, syslog-ng sends the message to the destinations set in the log statement.

9. The syslog-ng server processes the next log statement, repeating Steps 7-9.

Procedure for Configuring syslog-ng on client hosts:

    1. Install the syslog-ng application on the host.
    2. Configure the local sources that collect the log messages of the host.
    3. Create a network destination that points directly to the syslog-ng server.
    4. Create a log statement connecting the local sources to the syslog-ng server.
    5. If the logs will also be stored locally on the host, create local file destinations.
    6. Create a log statement connecting the local sources to the file destination.

Procedure for Configuring syslog-ng on server hosts:

    1. Install the syslog-ng application on the host.
    2. Configure the network sources that collect the log messages sent by the clients and relays.
    3. Create local destinations that will store the log messages, e.g., files or programs.
    4. Create a log statement connecting the network sources to the local destinations.
    5. Configure the local sources that collect the log messages of the syslog-ng server.
    6. Create a log statement connecting the local sources to the local destinations.
    7. Set filters and options as necessary.

Installation and configuration of Syslog-ng

We have downloaded the following required packages from:
ftp://ftp.sunfreeware.com/pub/freeware/i386/10

i. syslogng-2.0.9-sol10-x86-local.gz
ii. eventlog-0.2.7-sol10-x86-local.gz
iii.libiconv-1.11-sol10-x86-local.gz
iv. libgcc-3.3-sol10-x86-local.gz
v. libevntlog-1.2c-sol-x86-local.gz
We have installed the above packages on several machines each running Sun Solaris 10 OS.

Installation Procedure of syslog-ng and other dependency packages

We have used the following commands to unpack and install the syslog-ng Open source edition(OSE) and other packages.

# gunzip libiconv-1.11-sol10-x86-local.gz libgcc-3.3-sol10-x86-local.gz syslogng-2.0.9-sol10-x86-local.gz eventlog-0.2.7-sol10-x86-local.gz libevntlog-1.2a-sol-x86-local.gz

# pkgadd -d libiconv-1.11-sol10-x86-local libgcc-3.3-sol10-x86-local syslogng-2.0.9-sol10-x86-local  libevntlog-1.2a-sol10-x86-local eventlog-0.2.7-sol10-x86-local

After installing syslog-ng we have got the sample configuration file syslogng.conf.solaris at /usr/local/doc/syslogng/doc/examples/
We have copied the sample configuration file syslog-ng.conf.solaris to /usr/local/etc and renamed it as syslog-ng.conf:

cp /usr/local/doc/syslogng/doc/examples/syslog-ng.conf.solaris /usr/local/etc/syslog-ng.conf

Disabling the built-in syslog and shell SMF services

We know that every system has its own syslog. As we want to add syslog-ng we disabled the built-in syslog at first. We have done this by using the following commands:

# svcadm disable svc:/system/system-log:default
# svcadm disable svc:/network/shell:default

We have verified that the services have been disabled:

# svcs svc:/system/system-log:default
STATE STIME FMRI disabled 9:28:24 svc:/system/system-log:default
# svcs svc:/network/shell:default
STATE STIME FMRI disabled 18:49:30 svc:/network/shell:default

Writing Syslog-ng startup script

We have created the Syslog-NG SMF /lib/svc/method/svc-syslog-ng method script to start syslog-ng.

#!/bin/sh
. /lib/svc/share/smf_include.sh
# Start processes required for syslog-ng
# Required for certain libgcc and eventlog libraries
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib:/usr/sfw/lib
export LD_LIBRARY_PATH
if [ -x /usr/local/sbin/syslog-ng ]; then
/usr/local/sbin/syslog-ng
else

echo “syslog-ng is missing or not executable.”
exit $SMF_EXIT_ERR_CONFIG
fi
exit $SMF_EXIT_OK

Then we changed the ownership and permissions on the method script.

# chown root:bin /lib/svc/method/svc-syslog-ng
# chmod 755 /lib/svc/method/svc-syslog-ng

Creating the SMF /var/svc/manifest/system/syslog-ng.xml file.

We have created the syslog-ng.xml file.

<?xml version="1.0"?>
<!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1">
<service_bundle type='manifest' name='syslog-ng'>

<service name='system/syslog-ng'    type='service'       version='1'>

        <create_default_instance enabled='false' />
        <single_instance/>

        <dependency
                name='milestone'
                grouping='require_all'
                restart_on='none'
                type='service'>
                <service_fmri value='svc:/milestone/sysconfig' />
        </dependency>
        <dependency
                name='filesystem'
                grouping='require_all'
                restart_on='none'
                type='service'>
                <service_fmri value='svc:/system/filesystem/local' />
        </dependency>
        <dependency
                name='autofs'
                grouping='optional_all'
                restart_on='none'
                type='service'>
                <service_fmri value='svc:/system/filesystem/autofs' />
        </dependency>
        <dependency
                name='name-services'
                grouping='require_all'
                restart_on='none'
                type='service'>
                <service_fmri value='svc:/milestone/name-services' />
        </dependency>
        <dependent
                name='syslog-ng_single-user'
                grouping='optional_all'
                restart_on='none'>
                <service_fmri value='svc:/milestone/multi-user' />
        </dependent>

        <exec_method
                type='method'
                name='start'
                exec='/lib/svc/method/svc-syslog-ng'
                timeout_seconds='60' />

        <exec_method
                type='method'
                name='stop'
                exec=':kill'
                timeout_seconds='60' />

        <exec_method
                type='method'
                name='refresh'
                exec=':kill -HUP'
                timeout_seconds='60' />

        <property_group name='general' type='framework'>
                <!-- to start stop syslog daemon -->
                <propval name='action_authorization' type='astring'
                        value='solaris.smf.manage.syslog-ng' />
        </property_group>
        <stability value='Unstable' />
        <template>
                <common_name>
                        <loctext xml:lang='C'> syslog-ng
                        </loctext>
                </common_name>
                <documentation>
                        <manpage title='syslog-ng' section='1M'
                                manpath='/usr/share/man' />
                </documentation>
        </template>

</service>
</service_bundle>

Then we have changed the permissions and ownership on the manifest.

# chown root:sys /var/svc/manifest/system/syslog-ng.xml
# chmod 444 /var/svc/manifest/system/syslog-ng.xml

Validating the manifest file:

# svccfg validate /var/svc/manifest/system/syslog-ng.xml

Importing the manifest into SMF:

# svccfg import /var/svc/manifest/system/syslog-ng.xml

Verifying the service has been imported into the SMF database.

# svcs syslog-ng
STATE STIME FMRI disabled 9:46:01 svc:/system/syslog-ng:default

Starting the syslog-ng service:

# svcadm enable syslog-ng

Verifying the syslog-ng service is enabled and online:

# svc syslog-ng
STATE STIME FMRI online 9:51:40 svc:/system/syslog-ng:default

Creating the main configuration file (syslog-ng.conf)

A syslog-ng.conf file has the following parts: Source: A named collection of configured source drivers. Destination: A named collection of configured destination drivers. Filter: An expression to select messages. For example, a simple filter can select the messages received from a specific host. Log path: A combination of sources, filters, and destinations: syslog-ng examines all messages arriving to the sources of the log path and sends the messages matching all filters to the defined destinations. Log paths are also called log statements. Template: A template is a user-defined structure that can be used to restructure log messages or automatically generate file names. For example, a template can add the hostname and the date to the beginning of every log message.

Here is a sample client side configuration which we have used in our local server aa.xxx.com

#sample client configuration file

options
{
     use_dns(no);
     #will not use dns for name resolution. it will speed up processing
};

source local { sun-streams("/dev/log" door("/etc/.syslog_door")); internal(); };
# the above source will collect the internal log message generated by syslog-ng
# and other message. In case of linux machines we have to use the unix-streams
# source driver in stead of sun-streams driver.

source th_debug {file("/tmp/prod_debug.log" log-prefix("prod_debug")); };
# we are collecting log from the /tmp/prod_debug.log file and adding a
# prefix with the message so that at server we can filter them.

destination remote_dest { tcp(ip(192.168.6.28) port(1099); };
# the remote destination where all logs will be sent

destination all { file(“/var/log/messages”);};
# this destination file will store the local log messages

log { source(th_debug);    destination(remote_dest); };
log { source(local);    destination(remote_dest); };
# in syslog-ng a log statement bind a source to a destination.In the above case we
# are collecting log message from /tmp/prod_debug.log file as well as the
# internal log message and sending to the central log server located at
# 192.168.6.28.
# End of configuration file

Here is a sample server side configuration which we have used in our central log server
syslog-ng.xxx.com 

#sample server configuration file
options
{
	use_dns(no);
	create_dirs(yes);
};
# there are various options that we can use like defining log buffer size,

source local { sun-streams("/dev/log" door("/etc/.syslog_door")); internal(); };
# will collect internal log message

source remote_src { tcp(ip(192.168.6.28) port(1099) max-connections(800)); };
# will accept log message from remote machine on port 1099.

destination venus_th_deb
{
file("/logs/venus/app-logs/xxx/prod_debug.log.$MONTH.$DAY.$YEAR");
};
# this file will store the log message coming from the prod_debug.log file
# of the host venus. The name of log files will be of the form file_name.DATE
# for example log file name of 1st September will be “prod_debug.log.09.01.08”
# in this way the log files will be rotated automatically

filter venus_filter{host(192.168.1.142) and match("prod_debug:");};
# we are filtering message according to host ip address and content
# in this case we are filtering message those are coming from 192.168.1.31(wt02)
# and having a prefix of “prod_debug” that we have added earlier in the client
# side configuration.

destination all { file("/var/log/messages"); };
# common destination file for local log message

log
{
source(remote_src); filter(venus_filter);
destination(venus_th_deb); flags(final);
};
# the above log statement will bind the source with the specific filter to the
# destination. A message can match multiple filter and thus can be stored in more
# than one destination file. We have used the final flag here to instruct syslog-ng
# not to store this message on any other file.

# end of configuration file

Advertisements

3 Responses to “syslog-ng”

  1. Joe Says:

    Hi,
    Thanks for details about syslog-ng

    Do you know how to set up Service Processor, ALOM, ILOM alerts/issues to come through syslog-ng or qradar?

    Thanks, Joe

    • Sheikh Jafar Tarique Says:

      I didn’t try that yet!! Is there anyway to copy file from iLOM( i mean SP terminal) to remote box?? then you can easily add that on cron and if sylog-ng client configured on that remote box you are done!!!

  2. joe Says:

    Hi Sheikh,
    I’ve set syslog-ng to re be a relay host : server > syslog-ng relay > syslog server
    How to set the config file to pick files from server to go the syslog server

    Thanks

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: