This is a note from a CCSE training when I lost the connection with the training and started my own analysation of the Checkpoint processes.
To understand what goes on in the Checkpoint products its worth not just learn the well known processes but just to take a look on the whole operation.
Take a look at the output of pstree on your firewall to understand who is who in the theather and who triggered what.
pstree arguments I use here (Does anyone knows how to get the help normally on SPLAT??? I have to write a wrong argument to get the help…)
[Expert@mygwy]# pstree -?
pstree: invalid option -- ?
usage: pstree [ -a ] [ -c ] [ -h | -H pid ] [ -l ] [ -n ] [ -p ] [ -u ]
[ -G | -U ] [ pid | user]
pstree -V
-a show command line arguments
-c don't compact identical subtrees
-h highlight current process and its ancestors
-H pid highlight process "pid" and its ancestors
-G use VT100 line drawing characters
-l don't truncate long lines
-n sort output by PID
-p show PIDs; implies -c
-u show uid transitions
-U use UTF-8 (Unicode) line drawing characters
-V display version information
pid start at pid, default 1 (init)
user show only trees rooted at processes of that user
|
On my test firewall it looks like the following:
[Expert@mygwy]# pstree -hGc
init---acpid
+-agetty
+-agetty
+-agetty
+-console_agetty---agetty
+-cp_http_server_---cp_http_server
+-cprid_wd---cprid
+-cpwd---avi_del_tmp_fil---sleep
| +-ci_http_server
| +-cpd
| +-fw---dtls
| | +-fwssd
| | +-in.aclientd
| | +-in.ahclientd
| | +-in.asessiond---funcchain---funcchain
| | +-in.aufpd
| | +-pdpd
| | +-pepd
| | +-vpnd
| +-mpdaemon---httpd---httpd
| | +-httpd
| | +-httpd
| | +-httpd
| | +-httpd
| | +-httpd
| | +-httpd
| | +-httpd
| | +-httpd
| +-rtmd
+-cpwmd_wd---cpwmd
+-crond
+-events/0
+-fw1-wt
+-fwvt_thread
+-khelper
+-klogd
+-ksoftirqd/0
+-kthread---aio/0
| +-ata/0
| +-ata_aux
| +-cqueue/0
| +-kacpid
| +-kblockd/0
| +-khubd
| +-kjournald
| +-kjournald
| +-kjournald
| +-kjournald
| +-kjournald
| +-kpsmoused
| +-kseriod
| +-ksnapd
| +-kswapd0
| +-pdflush
| +-pdflush
| +-scsi_eh_0
+-migration/0
+-sshd---sshd---bash---pstree
+-syslogd
+-udevd
+-watchdog/0
|
on the Management looks like this:
[Expert@mymgmt]# pstree -hGc
init---acpid
+-agetty
+-agetty
+-console_agetty---agetty
+-cp_http_server_---cp_http_server
+-cpd
+-cprid_wd---cprid
+-cpwd---SVRServer
| +-cp_http_server
| +-cpsead
| +-cpstmymonitor
| +-cpwmd
| +-fw---cpca
| +-fwm
| +-mpdaemon
| +-status_proxy
+-cpwmd_wd---cpwmd
+-crond
+-events/0
+-fwvt_thread
+-khelper
+-klogd
+-ksoftirqd/0
+-kthread---aio/0
| +-ata/0
| +-ata_aux
| +-cqueue/0
| +-kacpid
| +-kblockd/0
| +-khubd
| +-kjournald
| +-kjournald
| +-kjournald
| +-kjournald
| +-kjournald
| +-kpsmoused
| +-kseriod
| +-ksnapd
| +-kswapd0
| +-pdflush
| +-pdflush
| +-scsi_eh_0
+-login---bash
+-migration/0
+-sshd---sshd---bash
| +-sshd---bash---pstree
+-syslogd
+-udevd
+-watchdog/0
|
The Red part is the most used part that is good to remember.
I can just reference to the cli reference guide R75 about the cpwd. The cpwd (also known as WatchDog) is a process that invokes and monitors critical processes such as Check Point daemons on the local machine, and attempts to restart them if they fail.
Among the processes monitored by Watchdog are cpd, fwd, fwm.
The process has a log file /opt/CPshrd-R75/log/cpwd.elg that we can check!
On the other hand we can check with a cpwd_admin command what is monitored and
# cpwd_admin list cpwd_admin: APP PID STAT #START START_TIME COMMAND MON CPD 3196 E 1 [20:25:24] 29/7/2012 cpd Y MPDAEMON 3220 E 1 [20:25:30] 29/7/2012 mpdaemon /opt/multiportal/log/mpdaemon.elg /opt/multiportal/conf/mpdaemon.conf N CI_CLEANUP 3271 E 1 [20:25:45] 29/7/2012 avi_del_tmp_files N CIHS 3283 E 1 [20:25:45] 29/7/2012 ci_http_server -j -f /opt/CPsuite-R75/fw1/conf/cihs.conf N FWD 3285 E 1 [20:25:46] 29/7/2012 fwd N RTMD 10050 E 1 [14:28:59] 30/7/2012 rtmd N |
Output The status report output includes the following information:
– APP – Application. The name of the process.
– PID – Process Identification Number.
– STAT – Whether the process Exists (E) or has been Terminated (T).
– #START -How many times the process has been started since cpwd took control of the process.
– START TIME – The last time the process was run.
– COMMAND – The command that cpwd used to start the process.
– MON – This is not documented?…It means that the process in monitored continuously.
# cpwd_admin monitor_list cpwd_admin: APP FILE_NAME NO_MSG_TIMES LAST_MSG_TIME CPD CPD_3196_6488.mntr 0/10 [13:28:38] 1/8/2012 vpnd vpnd_16060_13479329.mntr 0/6 [13:28:36] 1/8/2012 |
The watchdog part but with ps. Interestingly the ps does not see the fwssd process, it is not in the output…
# ps axf ... 3182 ? Ss 0:02 /opt/CPshrd-R75/bin/cpwd 3196 ? Ssl 6:34 \_ cpd 3220 ? Ss 0:17 \_ mpdaemon /opt/multiportal/log/mpdaemon.elg /opt/multiportal/conf/mpdaemon.conf 16074 ? Ss 0:00 | \_ /opt/CPshrd-R75/web/Apache/2.2.0/bin/httpd -DFOREGROUND -k start -f /opt/multiportal/httpd-conf/nac/httpd.conf -DPORTAL_NAME_ 16078 ? S 0:01 | \_ /opt/CPshrd-R75/web/Apache/2.2.0/bin/httpd -DFOREGROUND -k start -f /opt/multiportal/httpd-conf/nac/httpd.conf -DPORTAL_N 16079 ? S 0:00 | \_ /opt/CPshrd-R75/web/Apache/2.2.0/bin/httpd -DFOREGROUND -k start -f /opt/multiportal/httpd-conf/nac/httpd.conf -DPORTAL_N 16080 ? S 0:00 | \_ /opt/CPshrd-R75/web/Apache/2.2.0/bin/httpd -DFOREGROUND -k start -f /opt/multiportal/httpd-conf/nac/httpd.conf -DPORTAL_N 16081 ? S 0:00 | \_ /opt/CPshrd-R75/web/Apache/2.2.0/bin/httpd -DFOREGROUND -k start -f /opt/multiportal/httpd-conf/nac/httpd.conf -DPORTAL_N 16082 ? S 0:00 | \_ /opt/CPshrd-R75/web/Apache/2.2.0/bin/httpd -DFOREGROUND -k start -f /opt/multiportal/httpd-conf/nac/httpd.conf -DPORTAL_N 16103 ? S 0:00 | \_ /opt/CPshrd-R75/web/Apache/2.2.0/bin/httpd -DFOREGROUND -k start -f /opt/multiportal/httpd-conf/nac/httpd.conf -DPORTAL_N 16117 ? S 0:00 | \_ /opt/CPshrd-R75/web/Apache/2.2.0/bin/httpd -DFOREGROUND -k start -f /opt/multiportal/httpd-conf/nac/httpd.conf -DPORTAL_N 16118 ? S 0:00 | \_ /opt/CPshrd-R75/web/Apache/2.2.0/bin/httpd -DFOREGROUND -k start -f /opt/multiportal/httpd-conf/nac/httpd.conf -DPORTAL_N 16508 ? S 0:00 | \_ /opt/CPshrd-R75/web/Apache/2.2.0/bin/httpd -DFOREGROUND -k start -f /opt/multiportal/httpd-conf/nac/httpd.conf -DPORTAL_N 3271 ? Ss 0:00 \_ /bin/csh -f /opt/CPsuite-R75/fw1/bin/avi_del_tmp_files 24718 ? S 0:00 | \_ sleep 3600 3283 ? Ss 0:00 \_ ci_http_server -j -f /opt/CPsuite-R75/fw1/conf/cihs.conf 3285 ? Ssl 1:21 \_ fwd 8023 ? Sl 0:08 | \_ in.asessiond 0 15771 ? S 0:00 | | \_ funcchain http 1 0 resolver_list 15772 ? S 0:00 | | \_ funcchain http 2 1 resolver_list 8024 ? Sl 0:07 | \_ in.aufpd 0 8032 ? S 1:35 | \_ dtlsd 0 8033 ? Sl 0:07 | \_ in.geod 0 15715 ? Sl 0:04 | \_ in.ahclientd 900 15794 ? Sl 0:03 | \_ in.aclientd 259 16060 ? Sl 1:11 | \_ vpnd 0 16064 ? S 0:19 | \_ pepd 0 -t 16065 ? Sl 0:29 | \_ pdpd 0 -t 10050 ? Ssl 0:04 \_ rtmd ... |
The important Checkpoint processes (without the new blades, like DPD):
– cpd = The process is responsible for rulebase installation and SIC init (Check Point Daemon Protocol).
– fwm = The process is responsible for the execution of the database activities of the Security Management server (Policy installation, Management High Availability (HA) Synchronization, saving the Policy, Database Read/Write action, Log Display, etc.)
– fwd = The process is responsible for logging. It is executed in relation to logging, Security Servers and communication with OPSEC applications.
– fwssd = The process is responsible for Security Servers:
– in.aftpd = FTP security server
– in.ahclientd = Client Authentication on port 900 security server
– in.aclientd = Client Authentication on port 259 security server
– in.atelnetd = Telnet security server
– in.ahttpd = HTTP security server
– vpnd = The process is responsible for all encryption activity.
– cpwd = The process is responsible for monitoring other processes and attempts to restart them if they fail. It is a feature of the SVN Foundation
– cvpnd = the Ex-connectra VPN daemon for ssl remote access.
– rtmd = Smartview Monitor daemon, responsible for realtime monitoring
Sources for those deamons:
– Advanced Technical Reference Guide (NGX R60)
– Check Point® Troubleshooting and Debugging Tools for Faster Resolution January 24, 2006
The other way to monitor whats going on in a SPLAT is to install strace and analyse its output. I was not the first one with this idea:
https://www.cpug.org/forums/check-point-secureplatform-splat/15039-installing-strace-rpm-secureplatform.html
Lets check what happens in the background if we install a Policy to the firewall. The process responsible for that is the cpd.
Start it on the management server:
install strace…
[Expert@mymgmt]# scp admin@10.1.1.1:/home/admin/strace-4.5.14-0.EL3.1.i386.rpm . [Expert@mymgmt]# rpm -ivh strace-4.5.14-0.EL3.1.i386.rpm warning: strace-4.5.14-0.EL3.1.i386.rpm: V3 DSA signature: NOKEY, key ID 025e513b Preparing... ########################################### [100%] 1:strace ########################################### [100%] [Expert@mymgmt]# ps axf | grep cpd 21127 pts/0 S+ 0:00 \_ grep cpd 3075 ? Ssl 2:14 \_ cpd [Expert@mymgmt]# strace -p 3075 -f -o strace_cpd -ff Process 3150 attached with 5 threads - interrupt to quit Process 21161 attached Process 21162 attached Process 21163 attached Process 21164 attached ... Process 21183 detached Process 21162 detached Process 21215 detached --> only exit through another session and kill... |
The port used for policy install is the 18191. IF we filter on the output to the port we can see the following
# grep 18191 strace_cpd*
strace_cpd:getsockname(16, {sa_family=AF_INET, sin_port=htons(18191), sin_addr=inet_addr("0.0.0.0")}, [16]) = 0
strace_cpd:getsockname(139, {sa_family=AF_INET, sin_port=htons(18191), sin_addr=inet_addr("10.1.1.101")}, [16]) = 0
strace_cpd:access("/opt/CPshrd-R75.20/database//1819155117.local", F_OK) = -1 ENOENT (No such file or directory)
...
|
there is a special system call with the name “execve”. LEts filter on it.
# grep execve strace_cpd*
strace_cpd.21162:execve("/opt/CPsuite-R75.20/fw1/bin/fw", ["/opt/CPsuite-R75.20/fw1/bin/fw", "dbloadlocal", "-d", "/opt/CPsuite-R75.20/fw1/state/__"...], [/* 28 vars */]) = 0
strace_cpd.21163:execve("/bin/sh", ["sh", "-c", "/usr/bin/head -c 128 /dev/urando"...], [/* 28 vars */]) = 0
strace_cpd.21164:execve("/usr/bin/head", ["/usr/bin/head", "-c", "128", "/dev/urandom"], [/* 28 vars */]) = 0
...
|
And on the firewall we can check the same events.
install strace…
[Expert@mygwy]# ps axf | grep cpd 6291 pts/1 S+ 0:00 \_ grep cpd 5192 ? Ssl 0:04 \_ cpd [Expert@mygwy]# [Expert@mygwy]# strace -p 5192 -f -o strace_cpd -ff Process 5203 attached with 5 threads - interrupt to quit Process 6293 attached ... Process 6537 resumed Process 6538 detached |
–> only exit through another session and kill…
# grep 18191 strace_cpd*
strace_cpd:getsockname(19, {sa_family=AF_INET, sin_port=htons(18191), sin_addr=inet_addr("0.0.0.0")}, [16]) = 0
strace_cpd:getsockname(231, {sa_family=AF_INET, sin_port=htons(18191), sin_addr=inet_addr("172.21.101.1")}, [16]) = 0
...
|
# grep execve strace_cpd*
strace_cpd.7292:execve("/sbin/cp_logrotate", ["/sbin/cp_logrotate"], [/* 32 vars */]) = 0
strace_cpd.7294:execve("/opt/CPsuite-R75/fw1/bin/fw", ["/opt/CPsuite-R75/fw1/bin/fw", "fetchlocal", "-d", "/opt/CPsuite-R75/fw1/state/__tmp"...], [/* 32 vars */]) = 0
strace_cpd.7295:execve("/bin/sh", ["sh", "-c", "/usr/bin/head -c 100 /dev/urando"...], [/* 32 vars */]) = 0
strace_cpd.7296:execve("/usr/bin/head", ["/usr/bin/head", "-c", "100", "/dev/urandom"], [/* 32 vars */]) = 0
...
|
and this is something I wanted to check..:
[Expert@mygwy]# cut -f 2 strace_cpd* | cut -d"(" -f 1 | sort | uniq -c | sort -k 2
|
++++++
Out of scope of this post but maybe interesting:
I have found a command to check the top list of the processes, that have the highest number of opened files. The command is:
| lsof | awk ‘{printf(“%s (%s)\n”, $1, $2)}’ | sort | uniq -c | sort -rn | head |
Issuing on the gateway:
| # lsof | awk ‘{printf(“%s (%s)\n”, $1, $2)}’ | sort | uniq -c | sort -rn | head 419 cpd (3196) 348 fw (3285) 176 vpnd (16060) 169 pdpd (16065) 166 in.ahclie (15715) 156 in.asessi (8023) 154 in.aufpd (8024) 154 in.aclien (15794) 154 fwssd (8033) 142 funcchain (15771) |
Issuing on the Management:
| # lsof | awk ‘{printf(“%s (%s)\n”, $1, $2)}’ | sort | uniq -c | sort -rn | head 347 cpd (3075) 312 fwm (3143) 263 fw (3140) 131 cplmd (8323) 118 SVRServer (7991) 96 cpwmd (2939) 95 status_pr (3167) 91 cpca (3199) 80 cpsead (3268) 78 cpstat_mo (3432) |
Source of the command:
http://blog.ronanfoucher.fr/?p=344
itsecworks 
Posted on August 7, 2012
0