System Monitoring with Xymon/Administration Guide

All things related system administration will be documented here. Design Overview Xymon Protocol There is a version of Xymon protocol in ASCII text format from Xymon author. Architecture of a Xymon System Monitoring Environment TBC Picking an OS for Xymon Server These are some notes and advice from Xymon users. Linux Red Hat Enterprise Linux / CentOS Debian Oracle Solaris 10 Pros Plus 1: Turbocharged TCP/IP. Plus 2: dtrace Plus 3: Self Heal Plus 4: You can configure root and disk to use zfs and have zfs snapshot enabled. Cons Minus 1: Xymon depended on other open source software that doesn't come with Oracle Solaris by default. Following are three sources where you can get the software in binary or source code format. http://www.blastwave.org http://www.sunfreeware.com has lots of open source. http://www.thewrittenword.com List of software required to meet all dependecies and order of installation: common-1.4.5-SunOS5.8-sparc-CSW.pkg.gz pcre-4.5-SunOS5.8-sparc-CSW.pkg.gz fping-2.4,REV=2004.10.12_rev=b2_to_ipv6-SunOS5.8-sparc-CSW.pkg.gz zlib-1.2.3,REV=2007.05.12-SunOS5.8-sparc-CSW.pkg.gz png-1.2.18-SunOS5.8-sparc-CSW.pkg.gz libiconv-1.9.2-SunOS5.8-sparc-CSW.pkg.gz expat-1.95.7-SunOS5.8-sparc-CSW.pkg.gz ggettext-0.14.1,REV=2005.06.29-SunOS5.8-sparc-CSW.pkg.gz libpopt-1.7,REV=2004.05.15-SunOS5.8-sparc-CSW.pkg.gz chkconfig-1.2.24h,REV=2006.12.12-SunOS5.8-sparc-CSW.pkg.gz libpopt-1.7,REV=2004.05.15-SunOS5.8-sparc-CSW.pkg.gz openssl-0.9.8,REV=2007.05.10_rev=e-SunOS5.8-sparc-CSW.pkg.gz imaprt-2004,REV=2006.09.02_rev=g-SunOS5.8-sparc-CSW.pkg.gz freetype2-2.1.10,REV=2005.12.11-SunOS5.8-sparc-CSW.pkg.gz libart-2.3.16-SunOS5.8-sparc-CSW.pkg.gz berkeleydb44-4.4.20,REV=2007.01.27-SunOS5.8-sparc-CSW.pkg.gz ncurses-5.5,REV=2006.02.10-SunOS5.8-sparc-CSW.pkg.gz readline-5.0,REV=2005.06.07-SunOS5.8-sparc-CSW.pkg.gz gbc-1.06-SunOS5.8-sparc-CSW.pkg.gz gdbm-1.8.3,REV=2006.01.01-SunOS5.8-sparc-CSW.pkg.gz perl-5.8.8,REV=2007.03.16-SunOS5.8-sparc-CSW.pkg.gz cvs-1.11.22-sol10-sparc-local.gz rrdtool-1.2.19,REV=2007.02.07-SunOS5.8-sparc-CSW.pkg.gz libnet-1.0.2,REV=2004.04.08_rev=a-SunOS5.8-sparc-CSW.pkg.gz berkeleydb4-4.2.52,REV=2005.04.28_rev=p4-SunOS5.8-sparc-CSW.pkg.gz sasl-2.1.22,REV=2007.06.19-SunOS5.8-sparc-CSW.pkg.gz openldap_rt-2.3.35,REV=2007.04.14-SunOS5.8-sparc-CSW.pkg.gz xymon-4.2.0,REV=2007.04.12-SunOS5.8-sparc-CSW.pkg.gz xymon_client-4.2.0,REV=2007.04.12-SunOS5.8-sparc-CSW.pkg.gz Notes To avoid "xymond status-board not available" error message in bbgen webpage, add "set ip:do_tcp_fusion = 0x0" into /etc/system to disable TCP fusion. References: http://www.hswn.dk/hobbiton/2007/04/msg00187.html Solaris 5.10 kernel patch 120011-14-1, it fix this bug "6449337 kmem exhaustion caused by tcp fusion flow control logic error" . Xymon Server: Solaris Intel 11/06 U3 VMware appliance on a 2GB flash pen drive Following are main procedures for this to-go Xymon server. VMware server 1.0.1 to create Solaris 10 VMware session. Create a 1.9G partition, select custom install. modify the partition table to take out /export/home,only leave /swap and /. decrease default 512M swap size to 300M. select "Core group" (about 573M in size). Install httpd server Install xymon server Xymon Server and Development: Solaris Intel 11/06 U3 VMware appliance on a 4GB flash pen drive VMware server 1.0.1 to create Solaris 10 VMware session. Need to use vmware player 1.0.3 so dhcp will work. Xymon Server Test site Solaris Intel 11/06 U3 VMware appliance on a 4GB flash pen drive Operational difference between Xymon and BB BTF Servers This is a comparison table on how Xymon server is different from BB when performing an administration task. Operation Xymon 4.2.0 and above Big Brother BTF (Better Than Free, 1.9c version above) start/stop server ~/xymon.sh start/stop ~/runbb.sh start/stop Delete a host $XYMONHOME/bin/xymon 127.0.0.1 "drop HOSTNAME [test]" $BBHOME/bin/bbrm add a host 1. add hostnames into hosts.cfg 1. add hostnames into bb-hosts Log data path 1. 1. Clients This is a comparison on how Xymon is different from BB when performing an administration task. Operation Xymon 4.2.0 above Big Brother BTF(Better Than Free, 1.9c version above) addin external module ~xymon/client/etc/xymonclient.cfg $BBHOME/etc/bb-extab References http://www.hswn.dk/hobbit/help/about.html Capacity Planning rule of Thumb is 5MB disk space on Xymon server per machine being monitored Installation Windows Client Run the BBWin 0.13 installer. Under HKEY_LOCAL_MACHINE\SOFTWARE\BBWin (32-bit) or HKLM\SOFTWARE\Wow6432Node\BBWin (64-bit) in the registry set the computer name (as it is in the bbhosts file) Make the top of the config file in C:\Program Files\BBWin\etc (or C:\Program Files (x86)\BBWin\etc on Windows x64 systems) look like this: <setting name="bbdisplay" value="xymon server name" /> <!-- bbwin mode local or central --> <setting name="mode" value="central" /> <setting name="configclass" value="win32" /> Delete or comment out the default lines: <cpu> <setting name="default" warnlevel="85%" paniclevel="95%" delay="3" /> ...snip ... <disk> <setting name="default" warnlevel="85%" paniclevel="95%" /> This causes these thresholds to be set at the server side. Any settings here will override the settings in the server's analysis.cfg file. It is much easier to manage these settings centrally. Start the service at the server. Then edit /home/xymon/server/etc/analysis.cfg and add: #Hostname entries from bbwin clients. # HOST=[[new host name, as it appears in the bbhosts file]] LOAD 65 75 # Load thresholds are in % DISK C 80 90 DISK D 90 95 MEMPHYS 75 101 MEMSWAP 75 85 MEMACT 75 85 PROC BBWin.exe 1 1 Server /xymon/server/etc/client-local.cfg: [win32] eventlog:Security ignore Success eventlog:System ignore Information eventlog:Application ignore Information filtering in: /xymon/server/etc/analysis.cfg CLASS=win32 LOAD 80 90 # Load thresholds are in % PROC BBWin.exe 1 1 PORT STATE=LISTENING MIN=0 TRACK=Listen TEXT=Listen LOG %.* %error -.* COLOR=yellow LOG eventlog:Security %failure.* COLOR=yellow LOG eventlog:Application %warning.* COLOR=yellow LOG eventlog:System %error.* COLOR=yellow Instead you can use the following, but every update to the eventlog is send to the xymon server (instead of local filteret first). CLASS=win32 LOAD 80 90 # Load thresholds are in % PROC BBWin.exe 1 1 PORT STATE=LISTENING MIN=0 TRACK=Listen TEXT=Listen LOG %.* %^error.* COLOR=red #IGNORE=TermServDevices \( LOG %.* %^warning.* COLOR=yellow IGNORE=%.*TermServDevices.* LOG %.* %^failure.* COLOR=yellow Unix-like AIX Debian (Ubuntu) FreeBSD HP-UX IRIX Mandriva (xymon 4.2.3 is available in contrib as of 2009.0, prior to that hobbit was available in contrib) NSLU2 Unslung OS RedHat Linux / RedHat Enterprise Linux / Fedora Core (http://rpm.razorsedge.org/ or http://staff.telkomsa.net/packages/) Solaris As of 20 September 2012, Blastwave has ceased its operation, and its Web site is no longer accessible. Client xymon:/home/xymon/client/bin # ls -lrt total 2020 -rwxr-xr-x 1 xymon 1000 1915 Jul 31 2011 xymonclient-unixware.sh -rwxr-xr-x 1 xymon 1000 3389 Jul 31 2011 xymonclient-sunos.sh -rwxr-xr-x 1 xymon 1000 1849 Jul 31 2011 xymonclient-sco_sv.sh -rwxr-xr-x 1 xymon 1000 1708 Jul 31 2011 xymonclient-osf1.sh -rwxr-xr-x 1 xymon 1000 1914 Jul 31 2011 xymonclient-openbsd.sh -rwxr-xr-x 1 xymon 1000 1917 Jul 31 2011 xymonclient-netbsd.sh -rwxr-xr-x 1 xymon 1000 2821 Jul 31 2011 xymonclient-linux.sh -rwxr-xr-x 1 xymon 1000 1842 Jul 31 2011 xymonclient-irix.sh -rwxr-xr-x 1 xymon 1000 2421 Jul 31 2011 xymonclient-hp-ux.sh -rwxr-xr-x 1 xymon 1000 2092 Jul 31 2011 xymonclient-freebsd.sh -rwxr-xr-x 1 xymon 1000 1550 Jul 31 2011 xymonclient-darwin.sh -rwxr-xr-x 1 xymon 1000 1979 Jul 31 2011 xymonclient-aix.sh -rwxr-xr-x 1 xymon 1000 3252 Dec 12 22:15 xymonclient.sh -rwxr-xr-x 1 xymon root 187072 Feb 8 09:33 xymonlaunch -rwxr-xr-x 1 xymon root 288748 Feb 8 09:33 xymongrep -rwxr-xr-x 1 xymon root 210216 Feb 8 09:33 xymondigest -rwxr-xr-x 1 xymon root 153410 Feb 8 09:33 xymoncmd -rwxr-xr-x 1 xymon root 151751 Feb 8 09:33 xymoncfg -rwxr-xr-x 1 xymon root 180799 Feb 8 09:33 xymon -rwxr-xr-x 1 xymon root 179969 Feb 8 09:33 orcaxymon -rwxr-xr-x 1 xymon root 171691 Feb 8 09:33 msgcache -rwxr-xr-x 1 xymon root 240486 Feb 8 09:33 logfetch -rwxr-xr-x 1 xymon root 188930 Feb 8 09:33 clientupdate xymon:/home/xymon/client/bin # ./xymon Xymon version 4.3.7 Usage: ./xymon [--debug] [--merge] [--proxy=http://ip.of.the.proxy:port/] RECIPIENT DATA RECIPIENT: IP-address, hostname or URL DATA: Message to send, or "-" to read from stdin Server Building from package source using TWW HPMS TWW Hyper Package Management system can help a software developer or system administrator to create different native package formats for different OS. The package source for compiling and packaging hobbit client and server software are in XML format that can be repeated reliably with TWW's sb and pb tools. Hobbit server and Hobbit client package source is GPL licensed on TWW's support ftp server. Building from src RPM Sometimes it's better to build your own RPMs specifically for your environment. If you are using RH Enterprise or CentOS, the Fedora Core or generic RPM may not install correctly. You could also run into this problem if you have versions of dependent libraries that are not compatible with the system that the RPM was built on. In order to build the src RPM, you'll need several packages: openssl-devel, openldap-devel, and pcre-devel from the CentOS CDs. You may also have to make a link from /usr/include/pcre/pcre.h to /usr/include/pcre.h rrdtool-devel I recommend getting this from the DAG repository fping Also available from the DAG repository RPMs from a matching version of RHEL usually work on CentOS with no problem (for example RPMs for EL 4 work fine on CentOS 4) Once you have all the dependencies installed, download the src RPM from SourceForge. Once you have that, just run rpmbuild --rebuild hobbit-xxxx.src.rpm. For example: rpmbuild --rebuild hobbit-4.1.0-1.src.rpm The rpmbuild command should compile and build the RPM for you. You can watch the compiler output for any problems. After it is done, you should have new RPMs in the /usr/src/redhat/RPMS/i386 directory (assuming your architecture is i386). This process will build both server and client RPMs for your system. The server RPM also includes the client, so it is not necessary to install both of them. SUSE Dependencies for installation include apache2, apache2-utils, gcc, libstdc++-devel, net-snmp, pcre, pcre-devel, rrdtool and rrdtool-devel. Download the latest Xymon source from http://sourceforge.net/projects/xymon/files/Xymon/. Ensure that mod_rewrite is enabled in apache2, from YAST -> Network Services -> HTTP Server -> Server Modules. $ useradd -m xymon $ ./configure.server [...] *Where do you want the Xymon installation [/home/xymon] ? [...] What group-ID does your webserver use [nobody] ? www [...] $ make [...] Now run 'make install' as root $ make install [...] Installation complete. cp /home/xymon/server/etc/xymon-apache.conf to /etc/apache2/conf.d/ htpasswd2 -c /home/xymon/server/etc/xymonpasswd <choose an administrative user name> Ensure that fping can be executed by user xymon, either via appropriate sudo permissions, or by chmodding fping to setuid root. Start the apache2 service. /home/xymon/server/bin/xymon.sh start Ubuntu With Synaptic, install the PCRE and RRDtool librarieshttp://www.xymon.com/xymon/help/install.html. Then, download xymon and unpack it. Launch a terminal (CTRL + t) and enter the commands below, in order to install the software in your HTTP directory. Example with Apache: $ adduser xymon $ cd /home/Desktop/xymon $ ./configure.server [...] Where do you want the Xymon installation [/home/xymon] ? /var/www/xymon [...] What group-ID does your webserver use [nobody] ? xymon [...] $ make [...] Now run 'make install' as root $ make install [...] Installation complete. You must configure your webserver for the Xymon webpages and CGI-scripts. A sample Apache configuration is in /var/www/xymon/server/etc/xymon-apache.conf If you have your Administration CGI scripts in a separate directory, then you must also setup the password-file with the htpasswd command. To start Xymon, as the xymon user run '/var/www/xymon/server/bin/xymon.sh start' To view the Xymon webpages, go to http://localhost/xymon If it hasn't already been done, it's necessary to configure Apache to execute the CGI programs: $ vim /etc/apache2/httpd.conf # Add the following lines without the sharps and save: <Directory /var/www/*> Options +ExecCGI AddHandler cgi-script .cgi </Directory> $ /etc/init.d/apache2 restart $ su xymon /home/xymon/server/bin/xymon.sh start Xymon started Finally, test the software: http://localhost/xymon/server/bin/confreport.cgi Hobbit in HA There are two approaches to implement High Availability for Xymon servers,HA-LAN and HA-WAN. Pick one of them according to your network structure. HA-LAN approach This approach is using clustering software to do fail over using a set of Xymon servers. Each OS has their own version of clustering software. We know for Linux we can use Linux-HA plus DRBD. For Solaris, we have Sun Cluster Software. The cons of this approach is the High Availability is at the scale of LAN not WAN level. The server in clustering need to reside at same LAN subnet. If the clustering site went down then we will end up with xymon messages has no place to send message to. HA-LAN using LinuxHA and DRBD HA-LAN using Solaris Sun Cluster software plus TrueCopy HA-WAN approach For networks that span over states or countries, failing over a primary xymon server to standby server over WAN network is not an easy networking task. Following HA-WAN architecture can do fail-over without involve network team to do dns or routing changes. hobbit.test.com hobbit2.test.com | Primary | Standby Xymon server | <----- heart beat -----> | LAN1 | | LAN2 -------------------------- ------------------------- ^ ^ ^ ^ ^ ^ | | | | | | | --------------------------------------- | | | | | ---------------------------- | | | | | |-------------------------- | | | | | | | hobbitc A hobbitc B hobbitc C LAN 3 LAN 4 LAN 5 LAN1: California LAN2: Brazil LAN3: Argentina LAN4: Mexico LAN5: Japan Requirements a script that can detect failing of hobbit.test.com services. Notes hobbit2.test.com pager module is disabled. Hobbit2.test.com and hobbit.test.com reside on different sites connected by WAN. Hobbit clients does not lock on to hobbit.test.com alone. Each hobbit client send messages to both hobbit.test.com and hobbit2.test.com Hobbit2.test.com has every thing hobbit.test.com has and become active as hobbit2.test.com to send out alerts for hobbit.test.com. There is no need to do ip failover of hobbit.test.com to hobbit2.test.com. Pros No need to alter existing network configuration. Cons Increase network bandwidth by sending same message to two different servers. HA-WAN 2 approach From Patrick: we have 3 data centres and each data centre contains a xymon server. All clients in a data centre only report to their local xymon server. However the xymon servers can communicate with each other using BBDISPLAYS (its a little more complicated than that as we utilise a bbproxy in each DC to take the messages and spray them to all 3 xymons). hobbit1.test.com hobbit2.test.com | Primary | Standby Xymon server | <----- bbproxy -----> | LAN1 | | LAN2 -------------------------- ------------------------- ^ ^ ^ ^ | | | | | | | | | | | | | | | | hobbitc A hobbitc B hobbitc C LAN1= has hobbitc A,B LAN2= has hobbitc C HA-WAN3 approach This is a two node hobbit loosely-coupled cluster across WAN. It has following challenge need to be resolved. hobbit.test.com DNS need to failover to hobbit2 from hobbit1 when hobbit1 is down. The web page on hobbit1 and hobbit2 are not in sync. Maintence records are not in sync between two servers. RRD databases on two hobbit servers are not in sync after either one server is down for a while. hobbit.test.com -> hobbitdynamic.test.com (using CISCO DD software). -> hobbit1.test.com -> hobbit2.test.com hobbit1.test.com hobbit2.test.com | Primary | Standby Xymon server | <----- 1985 heart beat -----> | | <----- 1986 history -----> | | <----- 1987 heart beat -----> | LAN1 | | LAN2 -------------------------- ------------------------- ^ ^ ^ ^ ^ ^ | | | | | | | --------------------------------------- | | | | | ---------------------------- | | | | | |-------------------------- | | | | | | | hobbitc A hobbitc B hobbitc C LAN 3 LAN 4 LAN 5 LAN1: California LAN2: Brazil LAN3: Argentina LAN4: Mexico LAN5: Japan Requirements a script that can detect failing of hobbit.test.com services. Notes hobbit2.test.com pager module is disabled. Hobbit2.test.com and hobbit.test.com reside on different sites connected by WAN. Hobbit clients does not lock on to hobbit.test.com alone. Each hobbit client send messages to both hobbit.test.com and hobbit2.test.com Hobbit2.test.com has every thing hobbit.test.com has and become active as hobbit2.test.com to send out alerts for hobbit.test.com. There is no need to do ip failover of hobbit.test.com to hobbit2.test.com. Pros No need to alter existing network configuration. Cons Increase network bandwidth by sending same message to two different servers. Hobbit HA on LAN hobbit.test.com hobbit2.test.com | HA Software | | <- heart beat -> | | | LAN1: 192.168.1.0 ---------------------------------------------------------------- ^ ^ ^ | | | | | --------------------------- | | | | | | | | hobbitc A hobbitc B hobbitc C LAN 2 LAN 3 LAN4 LAN1: California LAN2: Brazil LAN3: Argentina LAN4: Mexico Notes HA Software = Sun Cluster 3.2 + Sun AVS hobbit2.test.com and hobbit.test.com reside on same subnet(same site). Cluster software (Sun Cluster 3.2) is used to do hobbit.test.com fail over. Each hobbit client send messages to hobbit.test.com only. hobbit2.test.com has every thing hobbit.test.com has. hobbit2.test.com is monitoring hobbit.test.com and will assume hobbit.test.com's identity. identity: MAC address and IP address of hobbit.test.com Pros Close to real-time fail-over. Cons Fail over happens only on LAN, not WAN. SunCluster Free and opensourced clustering software from Sun. Commercial technical support is available. Using two sol-nv-b68-x86 VMware sessions with Sun Cluster express 07/07. References http://www.opensolaris.org/os/community/ha-clusters http://www.sun.com/software/solaris/howtoguides/twonodecluster.jsp Analyzing the Application for Suitability Using AVS, not TrueCopy FST HA An opensource Clustering solution specifically for Solaris. FSTha vs other HA Small Text Hobbit Configuration and tuning Encryption and Tunnelling Hobbit(bb)/XyMon port 1984 encryption Using Stunnel References: http://www.stunnel.org/ Plain text bb message will be a bottleneck to make Hobbit a enterprise solution which require high security standard. Following is an attempt to make your CIO smile on hobbit solution. Note: It is possible to use reverse SSH tunnels, using Padraig Lennon's ssh_tunnels.sh script. instead of Stunnel server and client. See more details in Monitor Hobbit clients in a DMZ using reverse SSH tunnels Machine A : has both HB Server and Stunnel server running. Machine B : is a BB client. Machine C : is a hobbit client with stunnel client enabled. hb client will send bb message via encrypted port 1999. Machine D : is a HB client. Note: old bb port is one way, hb's bb protocol's is bi-directional. Machine A (192.168.1.111) --------------------------- HB Server process | <---------port 1984 <--------- BB client (Machine B) | | |1984 | <---------port 1984 ---------> HB client (Machine D) | | Stunnel Server process 1999 | <-------- port 1999 ----------> 1999 Stunnel Client ---------------------------- | (Machine C 192.168.1.141) | --1984 ---HB client Configure stunnel server to run in hobbit server stunnel config file on server to direct 1999 into local 1984 port. accept = 1999, we accept any incoming bb message on port 1999. connect = 127.0.0.1:1984, redirect 1999 to 1984 on hb server itself. bash-3.00# cat /opt/stunnel420/etc/stunnel/stunnel.conf <snip> [hobbit-server] accept = 1999 connect = 1984 <snip> bash-3.00# starting stunnel server on machine A. we can see hobbit-server port redirection is ok. bash-3.00# /etc/init.d/stunnel420 start Starting universal SSL tunnel: stunnel2007.04.29 06:47:50 LOG7[1898:1]: RAND_status claims sufficient entropy for the PRNG 2007.04.29 06:47:50 LOG7[1898:1]: PRNG seeded successfully 2007.04.29 06:47:50 LOG7[1898:1]: Certificate: /opt/stunnel420/etc/stunnel/stunnel.pem 2007.04.29 06:47:50 LOG7[1898:1]: Certificate loaded 2007.04.29 06:47:50 LOG7[1898:1]: Key file: /opt/moto/stunnel420/etc/stunnel/stunnel.pem 2007.04.29 06:47:50 LOG7[1898:1]: Private key loaded 2007.04.29 06:47:50 LOG7[1898:1]: SSL context initialized for service pop3s 2007.04.29 06:47:50 LOG7[1898:1]: Certificate: /opt/stunnel420/etc/stunnel/stunnel.pem 2007.04.29 06:47:50 LOG7[1898:1]: Certificate loaded 2007.04.29 06:47:50 LOG7[1898:1]: Key file: /opt/stunnel420/etc/stunnel/stunnel.pem 2007.04.29 06:47:50 LOG7[1898:1]: Private key loaded 2007.04.29 06:47:50 LOG7[1898:1]: SSL context initialized for service hobbit-server . bash-3.00# make sure stunnel is running. bash-3.00# ps -eaf |grep stunnel nobody 1984 1 0 06:55:00 ? 0:00 /opt/stunnel420/sbin/stunnel root 2133 1811 0 07:04:32 pts/2 0:00 grep stunnel bash-3.00# Testing port 1999 on hb server directly, typing garbage message "asdf" and then control+d to quit. bash-3.00# telnet machineA.test.com 1999 Trying 192.168.1.111... Connected to machineA.test.com. Escape character is '^]'. asdf Connection to machineA.test.com closed by foreign host. bash-3.00# We can see port 1999 has incoming message from 192.168.1.141(machine c)in stunnel log file on machine A. bash-3.00# tail -10f /opt/stunnel420/etc/stunnel/stunnel.log 2007.04.29 06:55:00 LOG5[1983:1]: 125 clients allowed 2007.04.29 06:55:00 LOG7[1983:1]: FD 4 in non-blocking mode 2007.04.29 06:55:00 LOG7[1983:1]: FD 5 in non-blocking mode 2007.04.29 06:55:00 LOG7[1983:1]: FD 6 in non-blocking mode 2007.04.29 06:55:00 LOG7[1983:1]: SO_REUSEADDR option set on accept socket 2007.04.29 06:55:00 LOG7[1983:1]: pop3s bound to 0.0.0.0:995 2007.04.29 06:55:00 LOG7[1983:1]: FD 7 in non-blocking mode 2007.04.29 06:55:00 LOG7[1983:1]: SO_REUSEADDR option set on accept socket 2007.04.29 06:55:00 LOG7[1983:1]: hobbit-server bound to 0.0.0.0:1999 2007.04.29 06:55:00 LOG7[1984:1]: Created pid file /stunnel.pid 2007.04.29 06:55:35 LOG7[1984:1]: hobbit-server accepted FD=0 from 192.168.1.141:38764 2007.04.29 06:55:35 LOG7[1984:2]: hobbit-server started 2007.04.29 06:55:35 LOG7[1984:2]: FD 0 in non-blocking mode 2007.04.29 06:55:35 LOG7[1984:2]: TCP_NODELAY option set on local socket 2007.04.29 06:55:35 LOG5[1984:2]: hobbit-server accepted connection from 192.168.1.141:38764 2007.04.29 06:55:35 LOG7[1984:2]: SSL state (accept): before/accept initialization 2007.04.29 06:55:39 LOG3[1984:2]: SSL_accept: 1408F10B: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number 2007.04.29 06:55:39 LOG5[1984:2]: Connection reset: 0 bytes sent to SSL, 0 bytes sent to socket 2007.04.29 06:55:39 LOG7[1984:2]: hobbit-server finished (0 left) Configuring hb client to use port 1999 add hobbitclientLocalIP into hobbitclient.cfg file. We want hobbit client send bb message to itself. bash-3.00# grep ^BBDISPLAYS /etc/opt/hobbitclient42/hobbitclient.cfg BBDISPLAYS="myotherhobbitserver.my.com hobbitclientLocalIP" # IP of multiple Hobbit servers. BBDISP must be "0.0.0.0". bash-3.00# bash-3.00# egrep -v '^;|^$' /opt/stunnel420/etc/stunnel/stunnel.conf cert = /opt/stunnel420/etc/stunnel/stunnel.pem sslVersion = SSLv3 chroot = /opt/stunnel420/var/lib/stunnel/ setuid = nobody setgid = nogroup pid = /stunnel.pid socket = l:TCP_NODELAY=1 socket = r:TCP_NODELAY=1 debug = 7 output = stunnel.log client = yes [hobbitclient] connect = hbServerRemoteIP:1999 accept = hbLocalIP:1984 bash-3.00# A successful hobbit client stunneling to hobbit server using port 1999. bash-3.00# grep 06:50 stunnel.log 2007.08.19 00:06:50 LOG7[14842:1]: hobbitclient accepted FD=0 from HobbitclientIP:63758 2007.08.19 00:06:50 LOG7[14842:3]: hobbitclient started 2007.08.19 00:06:50 LOG7[14842:3]: FD 0 in non-blocking mode 2007.08.19 00:06:50 LOG7[14842:3]: TCP_NODELAY option set on local socket 2007.08.19 00:06:50 LOG5[14842:3]: hobbitclient accepted connection from HobbitclientIP:63758 2007.08.19 00:06:50 LOG7[14842:3]: FD 1 in non-blocking mode 2007.08.19 00:06:50 LOG7[14842:3]: hobbitclient connecting HobbitServerIP:1999 2007.08.19 00:06:50 LOG7[14842:3]: connect_wait: waiting 10 seconds 2007.08.19 00:06:50 LOG7[14842:3]: connect_wait: connected 2007.08.19 00:06:50 LOG5[14842:3]: hobbitclient connected remote server from HobbitclientIP:63759 2007.08.19 00:06:50 LOG7[14842:3]: Remote FD=1 initialized 2007.08.19 00:06:50 LOG7[14842:3]: TCP_NODELAY option set on remote socket 2007.08.19 00:06:50 LOG7[14842:3]: SSL state (connect): before/connect initialization 2007.08.19 00:06:50 LOG7[14842:3]: SSL state (connect): SSLv3 write client hello A 2007.08.19 00:06:50 LOG7[14842:3]: SSL state (connect): SSLv3 read server hello A 2007.08.19 00:06:50 LOG7[14842:3]: SSL state (connect): SSLv3 read finished A 2007.08.19 00:06:50 LOG7[14842:3]: SSL state (connect): SSLv3 write change cipher spec A 2007.08.19 00:06:50 LOG7[14842:3]: SSL state (connect): SSLv3 write finished A 2007.08.19 00:06:50 LOG7[14842:3]: SSL state (connect): SSLv3 flush data 2007.08.19 00:06:50 LOG7[14842:3]: 1 items in the session cache 2007.08.19 00:06:50 LOG7[14842:3]: 2 client connects (SSL_connect()) 2007.08.19 00:06:50 LOG7[14842:3]: 2 client connects that finished 2007.08.19 00:06:50 LOG7[14842:3]: 0 client renegotiations requested 2007.08.19 00:06:50 LOG7[14842:3]: 0 server connects (SSL_accept()) 2007.08.19 00:06:50 LOG7[14842:3]: 0 server connects that finished 2007.08.19 00:06:50 LOG7[14842:3]: 0 server renegotiations requested 2007.08.19 00:06:50 LOG7[14842:3]: 1 session cache hits 2007.08.19 00:06:50 LOG7[14842:3]: 0 session cache misses 2007.08.19 00:06:50 LOG7[14842:3]: 0 session cache timeouts 2007.08.19 00:06:50 LOG6[14842:3]: SSL connected: previous session reused 2007.08.19 00:06:50 LOG7[14842:3]: Socket closed on read 2007.08.19 00:06:50 LOG7[14842:3]: SSL write shutdown 2007.08.19 00:06:50 LOG7[14842:3]: SSL alert (write): warning: close notify 2007.08.19 00:06:50 LOG6[14842:3]: SSL socket closed on SSL_shutdown 2007.08.19 00:06:50 LOG7[14842:3]: Socket write shutdown 2007.08.19 00:06:50 LOG5[14842:3]: Connection closed: 30068 bytes sent to SSL, 0 bytes sent to socket 2007.08.19 00:06:50 LOG7[14842:3]: hobbitclient finished (0 left) bash-3.00# Using HTTPS Transport A posting at http://lists.xymon.com/archive/2011-October/032866.html describes a technique where Xymon clients can submit client messages using a web connection. It requires a CGI script to be installed on the Xymon server. This method can be used to connect via web proxies, and authentication can be achieved by configuring the web server to enforce client-side certificates or user/password logins. Encryption via Secure-Shell (ssh) Tunnel Xymon can be configured to use the IP address of an ssh tunnel, and thus its traffic will be encrypted. This section describes two ways to establish a tunnel between the Xymon server and Xymon client. Persistent Tunnel This method is essentially creating a kind of VPN between the Xymon server and the client. Once established, the Xymon client is configured with XYMSRV set to 127.0.0.1, and all updates are sent down the tunnel. The simplest way to setup a persistent tunnel is with a tool such as Autossh. There's also a Xymon-specific add-on for establishing tunnels called ssh_tunnel. Ephemeral ssh Tunnel An ephemeral tunnel is a temporary tunnel created only when Xymon data need to be collected. Secure shell tunnels make use of key authentication so that passwords are not required. They can be established by ssh connection made in either direction, depending on requirements. In both cases, XYMSRV is set to 127.0.0.1. Xymon Server to Client For a server-to-client connection, the Xymon server runs an ssh connection to the client with a remote tunnel on port 1984, sets up some variables, and runs the Xymon client scripts. An example is shown here. ssh -R1984:127.0.0.1:1984 -o batchmode=yes xymon@xymon-client '/usr/lib/xymon/client/bin/xymoncmd sh -c "XYMSRV=127.0.0.1 /usr/lib/xymon/client/bin/xymonclient.sh"' This command can be put into tasks.cfg, run every 5 minutes. Xymon Client to Server For a client-to-server connection, the Xymon client establishes a connection to the server with a local tunnel on port 1984, and runs the Xymon client scripts. An example is shown here. ssh -f -L1984:127.0.0.1:1984 xymon@xymon-server sleep 15 && /usr/lib/xymon/client/bin/xymoncmd sh -c "XYMSRV=127.0.0.1 /usr/lib/xymon/client/bin/xymonclient.sh" This command should be run every 5 minutes on the Xymon client, and can be run from cron or from clientlaunch.cfg. 32 bit vs 64 bit binary for hobbit on Solaris This article describe this subject in great detail. Configuration LDAP Authentication Example httpd.conf (Apache 2.0.x with LDAP authenticated against Active Directory): Substitute LDAPSERVER.DOMAIN.COM with your LDAP server <USERNAME>: use account with permission to view LDAP directory <PASSWORD>: password for account (You should limit what this account can do) <Directory "/var/hobbit/cgi-secure"> AllowOverride None Options ExecCGI Includes Order allow,deny Allow from all AuthType Basic AuthName "Hobbit Administration" AuthLDAPEnabled on AuthLDAPURL ldap://LDAPSERVER.DOMAIN.COM:389/dc=DOMAIN,dc=COM?sAMAccountName?sub?(objectClass=person) AuthLDAPBindDN "cn=<USERNAME>,cn=Users,dc=DOMAIN,dc=COM" AuthLDAPBindPassword <PASSWORD> require valid-user </Directory> Same for a Novell-edir ldap server: <Directory "/usr/lib/hobbit/cgi-secure"> AllowOverride None Options ExecCGI Includes Order allow,deny Allow from all AuthName "Hobbit-Admin" AuthType Basic AuthLDAPURL ldap://LDAPSERVER.DOMAIN.COM/o=TREE,ou=Users?cn?sub?(groupMembership=cn=your_group,ou=groups,o=TREE) require valid-user </Directory> Alerts setting Pager Using sms_client [smsclient.org] Create a shell-script (/usr/bin/hobbitsms) like this: #!/bin/bash if [ $RECOVERED != 1 ]; then echo $RCPT \"HOBBIT : $BBHOSTSVC is $BBCOLORLEVEL\" >> /var/log/hobbit/page.log /usr/bin/sms_client $RCPT "HOBBIT : $BBHOSTSVC is $BBCOLORLEVEL" else echo $RCPT \"HOBBIT : $BBHOSTSVC is weer OK\" >> /var/log/hobbit/page.log /usr/bin/sms_client $RCPT "HOBBIT : $BBHOSTSVC is OK" fi Edit hobbit-alerts.cfg and add the lines for the alerts you want to receive: SCRIPT /usr/bin/hobbitsms hobbit DURATION>5 FORMAT=SMS REPEAT=180 COLOR=red TIME=W:0730:1800 RECOVERED Pager. Using snpp sendpage.org Create a shell-script (/usr/bin/hobbitsnpp) like this: #!/bin/bash /usr/bin/snpp -n $RCPT <<SCRIPTEOF $BBALPHAMSG SCRIPTEOF Email. Tuning How to shorten Xymon Server nslook up time ? Xymon server do lots nslookup for every five minutes on the machines that need to be pinged. Install a local dns cache server. I use djbdns for it How to shorten the ping test time ? Hobbit and Remedy Ticket System Overview Remedy ticket system has a web interface for opening up a ticket to a particular ticket queue. The Perl approach is to use the following software to automate the ticket request when an alert occurs. perl LWP trouble_ticket.tgz on http://www.deadcat.net an entrance URL on remedy server web interface. A perl subroutine to open up remedy ticket. Open Remedy ticket on hobbit alerts Open Remedy ticket on demand Migration from BB Cost (efforts) of Migration System and Inventory Monitoring System monitoring and inventory monitoring can achieved by an external module to report a system's inventory's information.(TBC) Trouble Shooting Guide Q. When I click on a status icon I get the message "Status not available". What should I check? A. First make sure that the server is actually running. ps -ef | grep hobbitd You should see several processes similar to: hobbit 32717 32716 0 Nov07 ? 00:01:07 hobbitd --pidfile.... hobbit 32726 32716 0 Nov07 ? 00:00:03 hobbitd_channel --channel=page... hobbit 32727 32716 0 Nov07 ? 00:01:58 hobbitd_channel --channel=status... hobbit 32728 32716 0 Nov07 ? 00:00:01 hobbitd_channel --channel=data... hobbit 32725 32716 0 Nov07 ? 00:00:00 hobbitd_channel --channel=stachg... If the server is failing to start, start looking at the hobbit logs directory. Check here for one location /var/log/hobbit Q. After installing the Hobbit client, my msgs tests are "clear" (sometimes referred to as "white") A. As of the time of this writing, the Hobbit client does NOT have msgs functionality like the BB client does. This can be added by installing the bb-msgs.sh file from the BB client as an external test. Even so, the Hobbit server will turn the test to "clear" instead of the expected status. To correct his issue, you'll have to edit the hobbitlaunch.cfg file (usually found in /etc/hobbit/ or /usr/lib/hobbit/server/etc/) to add --no-clear-msgs to the client channel and restart the server: CMD hobbitd_channel --channel=client hobbitd_client --no-clear-msgs --log=$BBSERVERLOGS/clientdata.log ... References See also /Compiling on CentOS/