Again about the processes on R75 SPLAT Firewall

Posted on August 7, 2012

0



This is a note from a CCSE training when I lost the connection with the training and started my own analysation of the Checkpoint processes.
To understand what goes on in the Checkpoint products its worth not just learn the well known processes but just to take a look on the whole operation.
Take a look at the output of pstree on your firewall to understand who is who in the theather and who triggered what.
pstree arguments I use here (Does anyone knows how to get the help normally on SPLAT??? I have to write a wrong argument to get the help…)

[Expert@mygwy]# pstree -?
pstree: invalid option -- ?
usage: pstree [ -a ] [ -c ] [ -h | -H pid ] [ -l ] [ -n ] [ -p ] [ -u ]
              [ -G | -U ] [ pid | user]
       pstree -V

    -a     show command line arguments
    -c     don't compact identical subtrees
    -h     highlight current process and its ancestors
    -H pid highlight process "pid" and its ancestors
    -G     use VT100 line drawing characters
    -l     don't truncate long lines
    -n     sort output by PID
    -p     show PIDs; implies -c
    -u     show uid transitions
    -U     use UTF-8 (Unicode) line drawing characters
    -V     display version information
    pid    start at pid, default 1 (init)
    user   show only trees rooted at processes of that user

On my test firewall it looks like the following:

[Expert@mygwy]# pstree -hGc
init---acpid
     +-agetty
     +-agetty
     +-agetty
     +-console_agetty---agetty
     +-cp_http_server_---cp_http_server
     +-cprid_wd---cprid
     +-cpwd---avi_del_tmp_fil---sleep
     |      +-ci_http_server
     |      +-cpd
     |      +-fw---dtls
     |      |    +-fwssd
     |      |    +-in.aclientd
     |      |    +-in.ahclientd
     |      |    +-in.asessiond---funcchain---funcchain
     |      |    +-in.aufpd
     |      |    +-pdpd
     |      |    +-pepd
     |      |    +-vpnd
     |      +-mpdaemon---httpd---httpd
     |      |                  +-httpd
     |      |                  +-httpd
     |      |                  +-httpd
     |      |                  +-httpd
     |      |                  +-httpd
     |      |                  +-httpd
     |      |                  +-httpd
     |      |                  +-httpd
     |      +-rtmd
     +-cpwmd_wd---cpwmd
     +-crond
     +-events/0
     +-fw1-wt
     +-fwvt_thread
     +-khelper
     +-klogd
     +-ksoftirqd/0
     +-kthread---aio/0
     |         +-ata/0
     |         +-ata_aux
     |         +-cqueue/0
     |         +-kacpid
     |         +-kblockd/0
     |         +-khubd
     |         +-kjournald
     |         +-kjournald
     |         +-kjournald
     |         +-kjournald
     |         +-kjournald
     |         +-kpsmoused
     |         +-kseriod
     |         +-ksnapd
     |         +-kswapd0
     |         +-pdflush
     |         +-pdflush
     |         +-scsi_eh_0
     +-migration/0
     +-sshd---sshd---bash---pstree
     +-syslogd
     +-udevd
     +-watchdog/0

on the Management looks like this:

[Expert@mymgmt]# pstree -hGc
init---acpid
     +-agetty
     +-agetty
     +-console_agetty---agetty
     +-cp_http_server_---cp_http_server
     +-cpd
     +-cprid_wd---cprid
     +-cpwd---SVRServer
     |      +-cp_http_server
     |      +-cpsead
     |      +-cpstmymonitor
     |      +-cpwmd
     |      +-fw---cpca
     |      +-fwm
     |      +-mpdaemon
     |      +-status_proxy
     +-cpwmd_wd---cpwmd
     +-crond
     +-events/0
     +-fwvt_thread
     +-khelper
     +-klogd
     +-ksoftirqd/0
     +-kthread---aio/0
     |         +-ata/0
     |         +-ata_aux
     |         +-cqueue/0
     |         +-kacpid
     |         +-kblockd/0
     |         +-khubd
     |         +-kjournald
     |         +-kjournald
     |         +-kjournald
     |         +-kjournald
     |         +-kjournald
     |         +-kpsmoused
     |         +-kseriod
     |         +-ksnapd
     |         +-kswapd0
     |         +-pdflush
     |         +-pdflush
     |         +-scsi_eh_0
     +-login---bash
     +-migration/0
     +-sshd---sshd---bash
     |      +-sshd---bash---pstree
     +-syslogd
     +-udevd
     +-watchdog/0

The Red part is the most used part that is good to remember.
I can just reference to the cli reference guide R75 about the cpwd. The cpwd (also known as WatchDog) is a process that invokes and monitors critical processes such as Check Point daemons on the local machine, and attempts to restart them if they fail.
Among the processes monitored by Watchdog are cpd, fwd, fwm.
The process has a log file /opt/CPshrd-R75/log/cpwd.elg that we can check!
On the other hand we can check with a cpwd_admin command what is monitored and

# cpwd_admin list
cpwd_admin:
APP        PID    STAT  #START  START_TIME             COMMAND              MON
CPD        3196   E     1       [20:25:24] 29/7/2012   cpd                  Y
MPDAEMON   3220   E     1       [20:25:30] 29/7/2012   mpdaemon /opt/multiportal/log/mpdaemon.elg /opt/multiportal/conf/mpdaemon.conf N
CI_CLEANUP 3271   E     1       [20:25:45] 29/7/2012   avi_del_tmp_files    N
CIHS       3283   E     1       [20:25:45] 29/7/2012   ci_http_server -j -f /opt/CPsuite-R75/fw1/conf/cihs.conf N
FWD        3285   E     1       [20:25:46] 29/7/2012   fwd                  N
RTMD       10050  E     1       [14:28:59] 30/7/2012   rtmd                 N

Output The status report output includes the following information:
– APP – Application. The name of the process.
– PID – Process Identification Number.
– STAT – Whether the process Exists (E) or has been Terminated (T).
– #START -How many times the process has been started since cpwd took control of the process.
– START TIME – The last time the process was run.
– COMMAND – The command that cpwd used to start the process.
– MON – This is not documented?…It means that the process in monitored continuously.

# cpwd_admin monitor_list
cpwd_admin:
APP       FILE_NAME                     NO_MSG_TIMES  LAST_MSG_TIME
CPD       CPD_3196_6488.mntr            0/10          [13:28:38] 1/8/2012
vpnd      vpnd_16060_13479329.mntr      0/6           [13:28:36] 1/8/2012

The watchdog part but with ps. Interestingly the ps does not see the fwssd process, it is not in the output…

# ps axf
...
3182 ?        Ss     0:02 /opt/CPshrd-R75/bin/cpwd
 3196 ?        Ssl    6:34  \_ cpd
 3220 ?        Ss     0:17  \_ mpdaemon /opt/multiportal/log/mpdaemon.elg /opt/multiportal/conf/mpdaemon.conf
16074 ?        Ss     0:00  |   \_ /opt/CPshrd-R75/web/Apache/2.2.0/bin/httpd -DFOREGROUND -k start -f /opt/multiportal/httpd-conf/nac/httpd.conf -DPORTAL_NAME_
16078 ?        S      0:01  |       \_ /opt/CPshrd-R75/web/Apache/2.2.0/bin/httpd -DFOREGROUND -k start -f /opt/multiportal/httpd-conf/nac/httpd.conf -DPORTAL_N
16079 ?        S      0:00  |       \_ /opt/CPshrd-R75/web/Apache/2.2.0/bin/httpd -DFOREGROUND -k start -f /opt/multiportal/httpd-conf/nac/httpd.conf -DPORTAL_N
16080 ?        S      0:00  |       \_ /opt/CPshrd-R75/web/Apache/2.2.0/bin/httpd -DFOREGROUND -k start -f /opt/multiportal/httpd-conf/nac/httpd.conf -DPORTAL_N
16081 ?        S      0:00  |       \_ /opt/CPshrd-R75/web/Apache/2.2.0/bin/httpd -DFOREGROUND -k start -f /opt/multiportal/httpd-conf/nac/httpd.conf -DPORTAL_N
16082 ?        S      0:00  |       \_ /opt/CPshrd-R75/web/Apache/2.2.0/bin/httpd -DFOREGROUND -k start -f /opt/multiportal/httpd-conf/nac/httpd.conf -DPORTAL_N
16103 ?        S      0:00  |       \_ /opt/CPshrd-R75/web/Apache/2.2.0/bin/httpd -DFOREGROUND -k start -f /opt/multiportal/httpd-conf/nac/httpd.conf -DPORTAL_N
16117 ?        S      0:00  |       \_ /opt/CPshrd-R75/web/Apache/2.2.0/bin/httpd -DFOREGROUND -k start -f /opt/multiportal/httpd-conf/nac/httpd.conf -DPORTAL_N
16118 ?        S      0:00  |       \_ /opt/CPshrd-R75/web/Apache/2.2.0/bin/httpd -DFOREGROUND -k start -f /opt/multiportal/httpd-conf/nac/httpd.conf -DPORTAL_N
16508 ?        S      0:00  |       \_ /opt/CPshrd-R75/web/Apache/2.2.0/bin/httpd -DFOREGROUND -k start -f /opt/multiportal/httpd-conf/nac/httpd.conf -DPORTAL_N
 3271 ?        Ss     0:00  \_ /bin/csh -f /opt/CPsuite-R75/fw1/bin/avi_del_tmp_files
24718 ?        S      0:00  |   \_ sleep 3600
 3283 ?        Ss     0:00  \_ ci_http_server -j -f /opt/CPsuite-R75/fw1/conf/cihs.conf
 3285 ?        Ssl    1:21  \_ fwd
 8023 ?        Sl     0:08  |   \_ in.asessiond 0
15771 ?        S      0:00  |   |   \_ funcchain http 1 0 resolver_list
15772 ?        S      0:00  |   |       \_ funcchain http 2 1 resolver_list
 8024 ?        Sl     0:07  |   \_ in.aufpd 0
 8032 ?        S      1:35  |   \_ dtlsd 0
 8033 ?        Sl     0:07  |   \_ in.geod 0
15715 ?        Sl     0:04  |   \_ in.ahclientd 900
15794 ?        Sl     0:03  |   \_ in.aclientd 259
16060 ?        Sl     1:11  |   \_ vpnd 0
16064 ?        S      0:19  |   \_ pepd 0 -t
16065 ?        Sl     0:29  |   \_ pdpd 0 -t
10050 ?        Ssl    0:04  \_ rtmd
...

The important Checkpoint processes (without the new blades, like DPD):

– cpd = The process is responsible for rulebase installation and SIC init (Check Point Daemon Protocol).
– fwm = The process is responsible for the execution of the database activities of the Security Management server (Policy installation, Management High Availability (HA) Synchronization, saving the Policy, Database Read/Write action, Log Display, etc.)
– fwd = The process is responsible for logging. It is executed in relation to logging, Security Servers and communication with OPSEC applications.
– fwssd = The process is responsible for Security Servers:
– in.aftpd = FTP security server
– in.ahclientd = Client Authentication on port 900 security server
– in.aclientd = Client Authentication on port 259 security server
– in.atelnetd = Telnet security server
– in.ahttpd = HTTP security server
– vpnd = The process is responsible for all encryption activity.
– cpwd = The process is responsible for monitoring other processes and attempts to restart them if they fail. It is a feature of the SVN Foundation
– cvpnd = the Ex-connectra VPN daemon for ssl remote access.
– rtmd = Smartview Monitor daemon, responsible for realtime monitoring

Sources for those deamons:
– Advanced Technical Reference Guide (NGX R60)
– Check Point® Troubleshooting and Debugging Tools for Faster Resolution January 24, 2006

The other way to monitor whats going on in a SPLAT is to install strace and analyse its output. I was not the first one with this idea:
https://www.cpug.org/forums/check-point-secureplatform-splat/15039-installing-strace-rpm-secureplatform.html

Lets check what happens in the background if we install a Policy to the firewall. The process responsible for that is the cpd.

Start it on the management server:

install strace…

[Expert@mymgmt]# scp admin@10.1.1.1:/home/admin/strace-4.5.14-0.EL3.1.i386.rpm .

[Expert@mymgmt]# rpm -ivh strace-4.5.14-0.EL3.1.i386.rpm
warning: strace-4.5.14-0.EL3.1.i386.rpm: V3 DSA signature: NOKEY, key ID 025e513b
Preparing...                ########################################### [100%]
   1:strace                 ########################################### [100%]

[Expert@mymgmt]# ps axf | grep cpd
21127 pts/0    S+     0:00          \_ grep cpd
 3075 ?        Ssl    2:14  \_ cpd
[Expert@mymgmt]# strace -p 3075 -f -o strace_cpd -ff
Process 3150 attached with 5 threads - interrupt to quit

Process 21161 attached
Process 21162 attached
Process 21163 attached
Process 21164 attached
...
Process 21183 detached
Process 21162 detached
Process 21215 detached

--> only exit through another session and kill...

The port used for policy install is the 18191. IF we filter on the output to the port we can see the following

# grep 18191 strace_cpd*
strace_cpd:getsockname(16, {sa_family=AF_INET, sin_port=htons(18191), sin_addr=inet_addr("0.0.0.0")}, [16]) = 0
strace_cpd:getsockname(139, {sa_family=AF_INET, sin_port=htons(18191), sin_addr=inet_addr("10.1.1.101")}, [16]) = 0
strace_cpd:access("/opt/CPshrd-R75.20/database//1819155117.local", F_OK) = -1 ENOENT (No such file or directory)
...

there is a special system call with the name “execve”. LEts filter on it.

# grep execve strace_cpd*
strace_cpd.21162:execve("/opt/CPsuite-R75.20/fw1/bin/fw", ["/opt/CPsuite-R75.20/fw1/bin/fw", "dbloadlocal", "-d", "/opt/CPsuite-R75.20/fw1/state/__"...], [/* 28 vars */]) = 0
strace_cpd.21163:execve("/bin/sh", ["sh", "-c", "/usr/bin/head -c 128 /dev/urando"...], [/* 28 vars */]) = 0
strace_cpd.21164:execve("/usr/bin/head", ["/usr/bin/head", "-c", "128", "/dev/urandom"], [/* 28 vars */]) = 0
...

And on the firewall we can check the same events.

install strace…

[Expert@mygwy]# ps axf | grep cpd
 6291 pts/1    S+     0:00          \_ grep cpd
 5192 ?        Ssl    0:04  \_ cpd
[Expert@mygwy]#
[Expert@mygwy]# strace -p 5192 -f -o strace_cpd -ff
Process 5203 attached with 5 threads - interrupt to quit

Process 6293 attached
...
Process 6537 resumed
Process 6538 detached

–> only exit through another session and kill…

# grep 18191 strace_cpd*
strace_cpd:getsockname(19, {sa_family=AF_INET, sin_port=htons(18191), sin_addr=inet_addr("0.0.0.0")}, [16]) = 0
strace_cpd:getsockname(231, {sa_family=AF_INET, sin_port=htons(18191), sin_addr=inet_addr("172.21.101.1")}, [16]) = 0
...
# grep execve strace_cpd*
strace_cpd.7292:execve("/sbin/cp_logrotate", ["/sbin/cp_logrotate"], [/* 32 vars */]) = 0
strace_cpd.7294:execve("/opt/CPsuite-R75/fw1/bin/fw", ["/opt/CPsuite-R75/fw1/bin/fw", "fetchlocal", "-d", "/opt/CPsuite-R75/fw1/state/__tmp"...], [/* 32 vars */]) = 0
strace_cpd.7295:execve("/bin/sh", ["sh", "-c", "/usr/bin/head -c 100 /dev/urando"...], [/* 32 vars */]) = 0
strace_cpd.7296:execve("/usr/bin/head", ["/usr/bin/head", "-c", "100", "/dev/urandom"], [/* 32 vars */]) = 0
...

and this is something I wanted to check..:

[Expert@mygwy]# cut -f 2 strace_cpd* | cut -d"(" -f 1 | sort | uniq -c | sort -k 2

++++++

Out of scope of this post but maybe interesting:

I have found a command to check the top list of the processes, that have the highest number of opened files. The command is:

lsof | awk ‘{printf(“%s (%s)\n”, $1, $2)}’ | sort | uniq -c | sort -rn | head

Issuing on the gateway:

# lsof | awk ‘{printf(“%s (%s)\n”, $1, $2)}’ | sort | uniq -c | sort -rn | head
419 cpd (3196)
348 fw (3285)
176 vpnd (16060)
169 pdpd (16065)
166 in.ahclie (15715)
156 in.asessi (8023)
154 in.aufpd (8024)
154 in.aclien (15794)
154 fwssd (8033)
142 funcchain (15771)

Issuing on the Management:

# lsof | awk ‘{printf(“%s (%s)\n”, $1, $2)}’ | sort | uniq -c | sort -rn | head
347 cpd (3075)
312 fwm (3143)
263 fw (3140)
131 cplmd (8323)
118 SVRServer (7991)
96 cpwmd (2939)
95 status_pr (3167)
91 cpca (3199)
80 cpsead (3268)
78 cpstat_mo (3432)

Source of the command:
http://blog.ronanfoucher.fr/?p=344

Advertisements