This is the part 2 of the troubleshooting commands that can help you better understand what and how you can troubleshoot on Palo Alto Next Generation Firewall in cli.
The Part 1. can be found here.
1. Capture and logging specific traffic
2. Logging traffic for global counters
3. Updates
3.1 Applications and Threats update
3.2 Software Upgrades
3.3 Licensing
1. Capture and logging specific traffic
Unfortunaltely it is not possible to sniffer simply on an interface but you can on 4 processing stages of a palo alto firewall. It is almost as good as the well known fw monitor command on a Checkpoint Firewall, but Checkpoint made a bit more effort on their tool (it has ~20 “stages”, see fw ctl chain command).
But even on Checkpoint Firewall I mostly used the basic opensource tcpdump on interfaces. This was better, but with Palo Alto we have no chance. :-(On Palo Alto Firewall the tcpdump can sniffer only on the out-of-band management interface..soounds stupid, but we have to live with it.The PA Firewall has 4 process stages for capture, those are the following:
- drop: when packet processing encounters an error and this packet will be discarded
- firewall: packet has a session match, or a first packet with session successfully created. According to the official Packet flow that means on Firewall Fast Path there was a firewall session match or on slowpath a session was succsfull created.
- receive: packet received on dataplane CPU
- transmit: packet is to be transmitted on dataplane CPU
There is 2-3 questions in my mind for this process staging stuff:
- What is with the Content Inspection, CI (SP3=Single Pass Parallel Processing/CTD=Content Thread Detection) and the Application Identification, AI? They are not involved in any stage?
I think everything that does not reach the transmit stage but leaved the firewall stage, must be in the drop stage because of an CI or AI error. If all packets of a session can be seen on transmit stage, then they have survived the CI and AI phases. - What is with IPSec Encryption or Decryption? At which phase can we see it?
- I saw on Checkpoint Firewall that even if the packet has left the outbound firewall VM, we havent seen it on the interface. Unfortunately the fact that, the Data Plane transmitted the packet, does not means that the packet has left the firewall interface.
On a tcp 3 way handshake it is like the following:
Allowed TCP SYN:
Inbound Interface(Eth0) == Receive stage --> Firewall Stage --> Transmit stage == Outbound Interface(Eth1)
TCP SYN+ACK
Inbound Interface(Eth1) == Receive stage --> Firewall Stage --> Transmit stage == Outbound Interface(Eth0)
TCP ACK
Inbound Interface(Eth0) == Receive stage --> Firewall Stage --> Transmit stage == Outbound Interface(Eth1)
Dropped Packet:
TCP Push or Denied TCP SYN:
receive stage == Inbound Interface(Eth0) --> Firewall Stage --> Drop Stage? -- This is your homework to test.
This post is a brief instruction with cli, if you need a GUI how-to goto the official Palo Alto page:
https://live.paloaltonetworks.com/docs/DOC-5853
There is a cli guide from Palo Alto too, but I still not satisfied with this, since it is not as well documented as mine :-) and not up-to-date.
Anyway you can compare this post with it:
https://live.paloaltonetworks.com/docs/DOC-1506
Or this (this is a better version, but no html format):
https://live.paloaltonetworks.com/docs/DOC-2542
In this Post I uses the Software Version 6.0.4, if you have older or newer the commands may be changed!
0. Unmark all sessions in debug (Update)
If you or someone else on the firewall had a filter before and the sessions tagged by the previous filter are still generating traffic you’ll se those packets in the debug. its annoying get rid of it.
# Unmark all sessions in debug, you can check what the command accepts
> debug dataplane packet-diag clear
> all Clear all settings and turn off log/capture
> capture capture setting
> filter Packet filter
> filter-marked-session Unmark session for debug
> log log setting
> debug dataplane packet-diag clear filter-marked-session
> all Unmark all sessions in debug
> id Unmark a specific session in debug
# Unmark all sessions in debug
> debug dataplane packet-diag clear filter-marked-session all
|
1. Filter only what you need
# You can filter the followings:
> debug dataplane packet-diag set filter match ?
+ destination Destination IP address
+ destination-port Destination port
+ ingress-interface Ingress traffic interface name
+ ipv6-only IPv6 packet only
+ non-ip Non-IP packet ------------> good for arp!
+ protocol IP protocol value
+ source Source IP address
+ source-port Source port
| Pipe through a command
Finish input
# Example:
> debug dataplane packet-diag set filter match source 192.168.8.153 ingress-interface ethernet1/1
# Enable your filter:
> debug dataplane packet-diag set filter on
|
2. Activate capture for the 4 stages (every stages in different files)
> debug dataplane packet-diag set capture stage firewall file stage-fw
> debug dataplane packet-diag set capture stage drop file stage-drop
> debug dataplane packet-diag set capture stage receive file stage-rx
> debug dataplane packet-diag set capture stage transmit file stage-tx
|
3. Start to capture
# Packet Length to sniffer, since I don’t know whats the default:
> debug dataplane packet-diag set capture snaplen 2000
# Check your setting before starting the capture
> debug dataplane packet-diag show setting
# Lets go capture
> debug dataplane packet-diag set capture on
|
4. Set the Logging if required apart from capture (Optional)
Every feature or process or protocol whatever it is called can be explicitly choosen from a huge list. They are not documented but from the names it is easy to find out what they for.
> debug dataplane packet-diag set log feature ?
> all all
> appid appid
> cfg cfg
> ctd ctd
> flow flow
> misc misc
> module module
> pow pow
> proxy proxy
> ssl ssl
> tcp tcp
> tunnel tunnel
> url_trie url_trie
> zip zip
# example is here flow and basic
> debug dataplane packet-diag set log feature flow basic
# or for more debugs
> debug dataplane packet-diag set log feature all
> debug dataplane packet-diag clear log log
> debug dataplane packet-diag set log on
|
5. Initiate your test traffic and after that stop the logging and the capture
> debug dataplane packet-diag set log off
> debug dataplane packet-diag set capture off
|
Check and copy all logs and captures (captures on 4 stages) to your ssh server (172.16.5.142).
The log file will be like management-plane_20140915_1217.tar.gz
> debug dataplane packet-diag aggregate-logs
# view capture on cli
> view-pcap follow yes filter-pcap stage-drop
# view logs in cli
> less dp-log pan_packet_diag.log
# use tail on live logs! Dont mess with less...
> tail follow yes dp-log pan_packet_diag.log
#
# or copy the log files
> scp export log-file management-plane to killer@172.16.5.142:/home/killer/temp/
# copy the capture files
> scp export filter-pcap from stage-* to killer@172.16.5.142:/home/killer/temp
killer@172.16.5.142's password:
stage-drop 100% 5660 5.5KB/s 00:00
stage-fw 100% 61MB 830.0KB/s 01:15
stage-rx 100% 9315 9.1KB/s 00:00
stage-tx 100% 84MB 820.0KB/s 01:45
|
This will copy 4 pcap files to your server. Define the extension to .pcap and double click to open.
6. Delete the captures and delete debug and logs
> delete debug-filter file stage-fw
> delete debug-filter file stage-drop
> delete debug-filter file stage-rx
> delete debug-filter file stage-tx
> debug dataplane packet-diag clear all
#check debug state again to see its all cleared now
> debug dataplane packet-diag show setting
|
2. Logging traffic for global counters
If you need GUI Tutorial goto Palo Alto page:
https://live.paloaltonetworks.com/docs/DOC-3199
1. Lets look for a drop or error counter that currently counts bad packets
This is an example with arp protocol. we are looking for some arp issues. Thats a tipical problem on networks…
> show counter global filter value non-zero delta yes | match arp
flow_fwd_l3_noarp 5 0 drop flow forward Packets dropped: no ARP ----- HERE 5 bad packets!!! Look for it.
flow_arp_pkt_rcv 468 47 info flow arp ARP packets received
flow_arp_pkt_xmt 2 0 info flow arp ARP packets transmitted
flow_arp_pkt_replied 175 17 info flow arp ARP requests replied
flow_arp_pkt_learned 2 0 info flow arp ARP entry learned
flow_arp_rcv_gratuitous 10 1 info flow arp Gratuitous ARP packets received
flow_arp_resolve_xmt 2 0 info flow arp ARP resolution packets transmitted
|
2. Activate the log for that specific counter
> debug dataplane packet-diag set log counter flow_fwd_l3_noarp
> debug dataplane packet-diag set filter pre-parse-match yes
# Thats what we did
> debug dataplane packet-diag show setting
--------------------------------------------------------------------------------
Packet diagnosis setting:
--------------------------------------------------------------------------------
Packet filter
Enabled: no
Match pre-parsed packet: yes
--------------------------------------------------------------------------------
Logging
Enabled: no
Log-throttle: no
Sync-log-by-ticks: yes
Features:
Counters:
flow_fwd_l3_noarp drop Packets dropped: no ARP
--------------------------------------------------------------------------------
Packet capture
Enabled: no
Snaplen: 0
--------------------------------------------------------------------------------
|
3. Find the timestamps with the logged drops for the specific counter
You can see in the outpit 4 times in one second at 2015/01/13 16:29:44.
> show log system | match flow_fwd_l3_noarp
2015/01/13 16:29:05 info general general 0 counter flow_fwd_l3_noarp=753176
2015/01/13 16:29:05 info general general 0 counter flow_fwd_l3_noarp=753177
2015/01/13 16:29:06 info general general 0 counter flow_fwd_l3_noarp=753178
2015/01/13 16:29:06 info general general 0 counter flow_fwd_l3_noarp=753178
2015/01/13 16:29:06 info general general 0 counter flow_fwd_l3_noarp=753178
2015/01/13 16:29:06 info general general 0 counter flow_fwd_l3_noarp=753178
2015/01/13 16:29:06 info general general 0 counter flow_fwd_l3_noarp=722221
2015/01/13 16:29:06 info general general 0 counter flow_fwd_l3_noarp=753179
2015/01/13 16:29:23 info general general 0 counter flow_fwd_l3_noarp=753180
2015/01/13 16:29:23 info general general 0 counter flow_fwd_l3_noarp=722222
2015/01/13 16:29:23 info general general 0 counter flow_fwd_l3_noarp=753181
2015/01/13 16:29:24 info general general 0 counter flow_fwd_l3_noarp=753182
2015/01/13 16:29:24 info general general 0 counter flow_fwd_l3_noarp=722223
2015/01/13 16:29:44 info general general 0 counter flow_fwd_l3_noarp=722224 --------4 times in one second!!!
2015/01/13 16:29:44 info general general 0 counter flow_fwd_l3_noarp=753183
2015/01/13 16:29:44 info general general 0 counter flow_fwd_l3_noarp=753184
2015/01/13 16:29:44 info general general 0 counter flow_fwd_l3_noarp=753185
2015/01/13 16:29:45 info general general 0 counter flow_fwd_l3_noarp=722225
2015/01/13 16:29:57 info general general 0 counter flow_fwd_l3_noarp=722226
2015/01/13 16:29:58 info general general 0 counter flow_fwd_l3_noarp=753186
2015/01/13 16:29:58 info general general 0 counter flow_fwd_l3_noarp=722227
2015/01/13 16:29:58 info general general 0 counter flow_fwd_l3_noarp=753187
2015/01/13 16:29:58 info general general 0 counter flow_fwd_l3_noarp=722228
2015/01/13 16:29:59 info general general 0 counter flow_fwd_l3_noarp=753188
...
|
4. We have a timestamp we can start to find the complete entries
Look for the log at 2015/01/13 16:29:44
> show log system start-time equal 2015/01/13@16:29:44
Time Severity Subtype Object EventID ID Description
===============================================================================
2015/01/13 16:29:44 info dhcp lease-e 0 DHCP lease ended ip 192.168.10.107 --> mac 64:76:ba:9e:36:d0, interface ethernet1/4.
108
2015/01/13 16:29:44 info general general 0 counter flow_fwd_l3_noarp=722224
192.168.8.250[29704]-->192.168.10.107[1]
sess id 50530Packet info: len 102 port 19 tag 0 interface 260
IP: 192.168.8.250->192.168.10.107, pro
2015/01/13 16:29:44 info general general 0 counter flow_fwd_l3_noarp=753183
192.168.8.250[29704]-->192.168.10.107[2]
sess id 61728Packet info: len 102 port 19 tag 0 interface 260
IP: 192.168.8.250->192.168.10.107, pro
2015/01/13 16:29:44 info general general 0 counter flow_fwd_l3_noarp=753184
192.168.8.250[29704]-->192.168.10.107[3]
sess id 33643Packet info: len 102 port 19 tag 0 interface 260
IP: 192.168.8.250->192.168.10.107, pro
2015/01/13 16:29:44 info general general 0 counter flow_fwd_l3_noarp=753185
192.168.8.250[29704]-->192.168.10.107[4]
sess id 12976Packet info: len 102 port 19 tag 0 interface 260
IP: 192.168.8.250->192.168.10.107, pro
....
|
There we go. the ARP for 192.168.10.107 has disappeared, the packets cannot be forwarded… That was the reason for the counter value. We have catch that! :-)
5. Delete your filter and set back to default
It is important to have a clear state for the next issue.
> debug dataplane packet-diag clear all
Packet diagnosis setting set to default.
> debug dataplane packet-diag show setting
--------------------------------------------------------------------------------
Packet diagnosis setting:
--------------------------------------------------------------------------------
Packet filter
Enabled: no
Match pre-parsed packet: no
--------------------------------------------------------------------------------
Logging
Enabled: no
Log-throttle: no
Sync-log-by-ticks: yes
Features:
Counters:
--------------------------------------------------------------------------------
Packet capture
Enabled: no
Snaplen: 0
--------------------------------------------------------------------------------
|
And your homework is to learn how to capture the specified counters (the packets that triggered the counter value).
Good luck!
3. Updates
3.1 Content Security and AntiVirus and URL DB and Wildfire Update and its scheduled task
Content Security Update
# Check for Content Security updates
killer@pa-firewall> request content upgrade download latest
Download job enqueued with jobid 32
32
# Install the newest update
killer@pa-firewall> request content upgrade install version latest
...
|
Antivirus Update
# Check for AV updates
killer@pa-firewall> request anti-virus upgrade download latest
...
# Install AV updates
killer@pa-firewall> request anti-virus upgrade install version latest
Content install job enqueued with jobid 12
12
# Or the downloaded one:
killer@pa-firewall> request anti-virus upgrade install file ?
panup-all-antivirus-1465-1938.tgz 2015/01/19 02:15:53 69779.9K
Imported anti-virus package
killer@pa-firewall> request anti-virus upgrade install file panup-all-antivirus-1465-1938.tgz
Content install job enqueued with jobid 13
13
# To check the state of the download or upgrade
killer@pa-firewall> show jobs all
Enqueued ID Type Status Result Completed
--------------------------------------------------------------------------
2015/01/19 12:52:49 13 Install ACT PEND 40%
2015/01/19 12:51:41 12 Downld FIN OK 12:51:57
2015/01/19 12:48:17 11 Content FIN OK 12:49:26
2015/01/19 12:45:09 10 Install FIN OK 12:48:17
2015/01/19 12:41:31 9 Downld FIN OK 12:41:39
2015/01/19 12:39:43 8 Downld FIN OK 12:39:50
2015/01/16 17:08:49 7 FqdnRefresh FIN OK 17:10:07
2015/01/16 17:06:25 6 CommitAll FIN OK 17:08:18
2015/01/16 17:02:45 5 Commit FIN OK 17:04:06
2015/01/16 16:59:48 4 Commit FIN OK 17:01:23
2015/01/16 16:55:18 3 CommitAll FIN FAIL 16:55:21
2015/01/16 14:27:40 2 Commit FIN OK 14:29:02
2015/01/16 11:20:32 1 AutoCom FIN OK 11:22:46
|
# If everything is fine we have a succesfully donwloaded and installed Content and Threath Detection Update and Antivirus Update:
killer@pa-firewall> request content upgrade info
Version Size Released on Downloaded Installed
-------------------------------------------------------------------------
481-2524 22MB 2015/01/14 01:15:34 CET yes current
killer@pa-firewall> request anti-virus upgrade info
Version Size Released on Downloaded Installed
-------------------------------------------------------------------------
1464-1937 67MB 2015/01/16 13:00:01 CET yes current
|
URL database Update
killer@pa-firewall> show url-cloud status
PAN-DB URL Filtering
License : valid
Current cloud server : s0300
Cloud connection : connected
URL database version - device : 2015.01.16.809
URL database version - cloud : 2015.01.16.809 ( last update time 2015/01/19 14:20:48 )
URL database status : good
URL protocol version - device : pan/0.0.2
URL protocol version - cloud : pan/0.0.2
Protocol compatibility status : compatible
killer@pa-firewall> request url-filtering download paloaltonetworks region Europe
PAN-DB update initiated
killer@pa-firewall> request url-filtering download status vendor paloaltonetworks
2015-01-20 12:28:14 PAN-DB download: Finished successfully.
|
Did you know? Order of URL categorisation:
Local Block list -> local Allow list -> Custom Categories -> Downloaded URL DB -> URL Cache (in DataPlane, called “Base db”) -> dynamic lookup (in ManagementPlane, called “Cloud db”)
Source:
https://live.paloaltonetworks.com/docs/DOC-1811
https://live.paloaltonetworks.com/docs/DOC-3454
https://live.paloaltonetworks.com/docs/DOC-3608
https://www.paloaltonetworks.fr/documentation/pan-os/pan-os/section_11/chapter_5.html
Wildfire Update
killer@pa-firewall> request wildfire upgrade check
Version Size Released on Downloaded Installed
-------------------------------------------------------------------------
51173-57875 10MB 2015/01/19 14:18:02 CET no previous
51174-57876 10MB 2015/01/19 14:33:02 CET no current
killer@pa-firewall> request wildfire upgrade info
Version Size Released on Downloaded Installed
-------------------------------------------------------------------------
51175-57877 10MB 2015/01/19 14:47:03 CET no current
51174-57876 10MB 2015/01/19 14:33:02 CET no previous
killer@pa-firewall> request wildfire upgrade download latest
Download job enqueued with jobid 1026
1026
killer@pa-firewall> request wildfire upgrade install version latest
Content install job enqueued with jobid 1027
1027
|
Automatic Updates
Automatic update for URL filter is only possible with BrightCloud, the Paloaltonetworks URL Database will be just in time asked for URLs.
killer@pa-firewall# set deviceconfig system update-schedule ?
> anti-virus Anti virus database
> app-profile Application profile database
> global-protect-datafile GlobalProtect datafile
> statistics-service Participate in Anonymous Statistics Upload Service
> threats Threat detection database
> url-database URL filtering database
> wildfire WildFire database
Finish input
# Example:
killer@pa-firewall# set deviceconfig system update-schedule threats recurring daily at 03:00
killer@pa-firewall# set deviceconfig system update-schedule threats recurring daily action download-and-install
killer@pa-firewall# set deviceconfig system update-schedule anti-virus recurring daily at 19:30
killer@pa-firewall# set deviceconfig system update-schedule anti-virus recurring daily action download-and-install
killer@pa-firewall# set deviceconfig system update-schedule wildfire recurring every-15-mins action download-and-install
# Or Example with Panorama:
killer@pa-panorama# set deviceconfig system deployment-update-schedule pa-av anti-virus recurring daily at 08:15
killer@pa-panorama# set deviceconfig system deployment-update-schedule pa-av anti-virus recurring daily action download-and-install devices [ 001..devices_serials... ]
|
3.2 Software Upgrades
killer@pa-firewall> request system software info
Version Size Released on Downloaded
-------------------------------------------------------------------------
6.1.1 179MB 2014/12/18 00:05:36 no
6.1.0 329MB 2014/10/25 08:34:03 no
6.0.7 237MB 2014/12/08 21:52:10 no
6.0.6 237MB 2014/10/23 20:43:52 no
6.0.5 237MB 2014/09/23 11:04:03 no
6.0.5-h3 237MB 2014/10/08 16:31:06 no
6.0.4 214MB 2014/08/04 19:39:30 yes
6.0.3 207MB 2014/06/11 23:56:10 no
6.0.2 203MB 2014/04/23 22:07:26 no
6.0.1 200MB 2014/03/09 10:40:44 no
6.0.0 353MB 2014/01/19 11:12:47 yes
...
5.0.7 187MB 2013/08/27 14:05:52 no
5.0.6 187MB 2013/07/08 17:04:39 yes
5.0.5 187MB 2013/05/24 22:32:41 no
killer@pa-firewall> request system software check
Version Size Released on Downloaded
-------------------------------------------------------------------------
6.1.1 179MB 2014/12/18 00:05:36 no
6.1.0 329MB 2014/10/25 08:34:03 no
6.0.7 237MB 2014/12/08 21:52:10 no
6.0.6 237MB 2014/10/23 20:43:52 no
6.0.5 237MB 2014/09/23 11:04:03 no
6.0.5-h3 237MB 2014/10/08 16:31:06 no
6.0.4 214MB 2014/08/04 19:39:30 yes
6.0.3 207MB 2014/06/11 23:56:10 no
6.0.2 203MB 2014/04/23 22:07:26 no
6.0.1 200MB 2014/03/09 10:40:44 no
6.0.0 353MB 2014/01/19 11:12:47 yes
...
killer@pa-firewall> request system software install version ?
5.0.0 5.0.0
5.0.6 5.0.6
6.0.0 6.0.0
6.0.4 6.0.4
Upgrade to a software package by version
killer@pa-firewall> request system software install version 6.0.4
killer@pa-firewall> request system software download version 6.0.4
# Dont forget to reboot it is not happened!
killer@pa-firewall> request restart system
|
killer@pa-firewall> debug swm list
Product B Version Built Loaded
--------------------------------------------------------------------------------
panos * 5.0.0 2012-10-26 16:49:01 2014-01-30 14:31:23
panos 5.0.6 2013-07-03 16:49:39 2014-01-30 14:32:18
panos * 6.0.0 2014-01-14 14:02:40 2015-01-15 19:41:42
panos 6.0.4 2014-07-30 12:31:52 2015-01-16 10:36:55
killer@pa-firewall> debug swm status
Partition State Version
--------------------------------------------------------------------------------
sysroot0 RUNNING-ACTIVE 6.0.4
sysroot1 REVERTABLE 5.0.6
maint READY 5.0.6
|
3.3 Licensing
killer@pa-firewall> request license ?
> fetch Get new license(s) using an authentication code
> info Show information about owned license(s)
> install Install a license key
killer@pa-firewall> request license info
Current GMT Date: January 15, 2015
License entry:
Feature: PAN-DB URL Filtering
Description: Palo Alto Networks URL Filtering License
Serial: 00xxx
Issued: January 16, 2015
Expires: January 16, 2020
Expired?: no
License entry:
Feature: Threat Prevention
Description: Antivirus, anti-spyware, vulnerability protection
Serial: 00xxx
Issued: January 16, 2015
Expires: January 16, 2020
Expired?: no
|
Like this:
Like Loading...
Related
itsecworks 
Posted on January 14, 2015
0