Palo Alto troubleshooting commands Part 2.

Posted on January 14, 2015

0



This is the part 2 of the troubleshooting commands that can help you better understand what and how you can troubleshoot on Palo Alto Next Generation Firewall in cli.
The Part 1. can be found here.

1. Capture and logging specific traffic
2. Logging traffic for global counters
3. Updates
3.1 Applications and Threats update
3.2 Software Upgrades
3.3 Licensing

1. Capture and logging specific traffic

Unfortunaltely it is not possible to sniffer simply on an interface but you can on 4 processing stages of a palo alto firewall. It is almost as good as the well known fw monitor command on a Checkpoint Firewall, but Checkpoint made a bit more effort on their tool (it has ~20 “stages”, see fw ctl chain command).
But even on Checkpoint Firewall I mostly used the basic opensource tcpdump on interfaces. This was better, but with Palo Alto we have no chance. :-(On Palo Alto Firewall the tcpdump can sniffer only on the out-of-band management interface..soounds stupid, but we have to live with it.The PA Firewall has 4 process stages for capture, those are the following:

  • drop: when packet processing encounters an error and this packet will be discarded
  • firewall: packet has a session match, or a first packet with session successfully created. According to the official Packet flow that means on Firewall Fast Path there was a firewall session match or on slowpath a session was succsfull created.
  • receive: packet received on dataplane CPU
  • transmit: packet is to be transmitted on dataplane CPU

There is 2-3 questions in my mind for this process staging stuff:

  1. What is with the Content Inspection, CI (SP3=Single Pass Parallel Processing/CTD=Content Thread Detection) and the Application Identification, AI? They are not involved in any stage?
    I think everything that does not reach the transmit stage but leaved the firewall stage, must be in the drop stage because of an CI or AI error. If all packets of a session can be seen on transmit stage, then they have survived the CI and AI phases.
  2. What is with IPSec Encryption or Decryption? At which phase can we see it?
  3. I saw on Checkpoint Firewall that even if the packet has left the outbound firewall VM, we havent seen it on the interface. Unfortunately the fact that, the Data Plane transmitted the packet, does not means that the packet has left the firewall interface.

On a tcp 3 way handshake it is like the following:

Allowed TCP SYN:
Inbound Interface(Eth0) == Receive stage --> Firewall Stage --> Transmit stage == Outbound Interface(Eth1)
TCP SYN+ACK
Inbound Interface(Eth1) == Receive stage --> Firewall Stage --> Transmit stage == Outbound Interface(Eth0)
TCP ACK
Inbound Interface(Eth0) == Receive stage --> Firewall Stage --> Transmit stage == Outbound Interface(Eth1)
Dropped Packet:
TCP Push or Denied TCP SYN:
receive stage == Inbound Interface(Eth0) --> Firewall Stage --> Drop Stage? -- This is your homework to test.

This post is a brief instruction with cli, if you need a GUI how-to goto the official Palo Alto page:
https://live.paloaltonetworks.com/docs/DOC-5853
There is a cli guide from Palo Alto too, but I still not satisfied with this, since it is not as well documented as mine :-) and not up-to-date.
Anyway you can compare this post with it:
https://live.paloaltonetworks.com/docs/DOC-1506
Or this (this is a better version, but no html format):
https://live.paloaltonetworks.com/docs/DOC-2542
In this Post I uses the Software Version 6.0.4, if you have older or newer the commands may be changed!

1. Filter only what you need

# You can filter the followings:
> debug dataplane packet-diag set filter match ?
+ destination         Destination IP address
+ destination-port    Destination port
+ ingress-interface   Ingress traffic interface name
+ ipv6-only           IPv6 packet only
+ non-ip              Non-IP packet ------------> good for arp!
+ protocol            IP protocol value
+ source              Source IP address
+ source-port         Source port
  |                   Pipe through a command
               Finish input

# Example:
> debug dataplane packet-diag set filter match source 192.168.8.153 ingress-interface ethernet1/1
# Enable your filter:
> debug dataplane packet-diag set filter on

2. Activate capture for the 4 stages (every stages in different files)

> debug dataplane packet-diag set capture stage firewall file stage-fw
> debug dataplane packet-diag set capture stage drop file stage-drop
> debug dataplane packet-diag set capture stage receive file stage-rx
> debug dataplane packet-diag set capture stage transmit file stage-tx

3. Start to capture

# Packet Length to sniffer, since I don’t know whats the default:
> debug dataplane packet-diag set capture snaplen 2000
# Check your setting before starting the capture
> debug dataplane packet-diag show setting
# Lets go capture
> debug dataplane packet-diag set capture on

4. Set the Logging if required apart from capture (Optional)

Every feature or process or protocol whatever it is called can be explicitly choosen from a huge list. They are not documented but from the names it is easy to find out what they for.

> debug dataplane packet-diag set log feature ?
> all        all 
> appid      appid 
> cfg        cfg 
> ctd        ctd 
> flow       flow 
> misc       misc 
> module     module 
> pow        pow 
> proxy      proxy 
> ssl        ssl 
> tcp        tcp 
> tunnel     tunnel 
> url_trie   url_trie 
> zip        zip 
# example is here flow and basic
> debug dataplane packet-diag set log feature flow basic
# or for more debugs
> debug dataplane packet-diag set log feature all

> debug dataplane packet-diag clear log log
> debug dataplane packet-diag set log on

5. Initiate your test traffic and after that stop the logging and the capture

> debug dataplane packet-diag set log off
> debug dataplane packet-diag set capture off

Check and copy all logs and captures (captures on 4 stages) to your ssh server (172.16.5.142).
The log file will be like management-plane_20140915_1217.tar.gz

> debug dataplane packet-diag aggregate-logs
# view capture on cli
> view-pcap follow yes filter-pcap stage-drop
# view logs in cli
> less dp-log pan_packet_diag.log
# use tail on live logs! Dont mess with less...
> tail follow yes dp-log pan_packet_diag.log
#
# or copy the log files
> scp export log-file management-plane to killer@172.16.5.142:/home/killer/temp/
# copy the capture files
> scp export filter-pcap from stage-* to killer@172.16.5.142:/home/killer/temp
killer@172.16.5.142's password: 
stage-drop                                                          100% 5660     5.5KB/s   00:00    
stage-fw                                                            100%   61MB 830.0KB/s   01:15    
stage-rx                                                            100% 9315     9.1KB/s   00:00    
stage-tx                                                            100%   84MB 820.0KB/s   01:45

This will copy 4 pcap files to your server. Define the extension to .pcap and double click to open.

6. Delete the captures and delete debug and logs

> delete debug-filter file stage-fw
> delete debug-filter file stage-drop
> delete debug-filter file stage-rx
> delete debug-filter file stage-tx
> debug dataplane packet-diag clear all
#check debug state again to see its all cleared now
> debug dataplane packet-diag show setting

2. Logging traffic for global counters

If you need GUI Tutorial goto Palo Alto page:
https://live.paloaltonetworks.com/docs/DOC-3199

1. Lets look for a drop or error counter that currently counts bad packets

This is an example with arp protocol. we are looking for some arp issues. Thats a tipical problem on networks…

> show counter global filter value non-zero delta yes | match arp

flow_fwd_l3_noarp                          5        0 drop      flow      forward   Packets dropped: no ARP  ----- HERE 5 bad packets!!! Look for it.
flow_arp_pkt_rcv                         468       47 info      flow      arp       ARP packets received
flow_arp_pkt_xmt                           2        0 info      flow      arp       ARP packets transmitted
flow_arp_pkt_replied                     175       17 info      flow      arp       ARP requests replied
flow_arp_pkt_learned                       2        0 info      flow      arp       ARP entry learned
flow_arp_rcv_gratuitous                   10        1 info      flow      arp       Gratuitous ARP packets received
flow_arp_resolve_xmt                       2        0 info      flow      arp       ARP resolution packets transmitted

2. Activate the log for that specific counter

> debug dataplane packet-diag set log counter flow_fwd_l3_noarp
> debug dataplane packet-diag set filter pre-parse-match yes
# Thats what we did
> debug dataplane packet-diag show setting 
--------------------------------------------------------------------------------
Packet diagnosis setting:
--------------------------------------------------------------------------------
Packet filter
  Enabled:                   no
  Match pre-parsed packet:   yes            
--------------------------------------------------------------------------------
Logging
  Enabled:                   no
  Log-throttle:              no
  Sync-log-by-ticks:         yes            
  Features:
  Counters:
    flow_fwd_l3_noarp              drop      Packets dropped: no ARP
--------------------------------------------------------------------------------
Packet capture
  Enabled:                   no
  Snaplen:                   0            
--------------------------------------------------------------------------------

3. Find the timestamps with the logged drops for the specific counter

You can see in the outpit 4 times in one second at 2015/01/13 16:29:44.

> show log system | match flow_fwd_l3_noarp
2015/01/13 16:29:05 info     general        general 0  counter flow_fwd_l3_noarp=753176
2015/01/13 16:29:05 info     general        general 0  counter flow_fwd_l3_noarp=753177
2015/01/13 16:29:06 info     general        general 0  counter flow_fwd_l3_noarp=753178
2015/01/13 16:29:06 info     general        general 0  counter flow_fwd_l3_noarp=753178
2015/01/13 16:29:06 info     general        general 0  counter flow_fwd_l3_noarp=753178
2015/01/13 16:29:06 info     general        general 0  counter flow_fwd_l3_noarp=753178
2015/01/13 16:29:06 info     general        general 0  counter flow_fwd_l3_noarp=722221
2015/01/13 16:29:06 info     general        general 0  counter flow_fwd_l3_noarp=753179
2015/01/13 16:29:23 info     general        general 0  counter flow_fwd_l3_noarp=753180
2015/01/13 16:29:23 info     general        general 0  counter flow_fwd_l3_noarp=722222
2015/01/13 16:29:23 info     general        general 0  counter flow_fwd_l3_noarp=753181
2015/01/13 16:29:24 info     general        general 0  counter flow_fwd_l3_noarp=753182
2015/01/13 16:29:24 info     general        general 0  counter flow_fwd_l3_noarp=722223
2015/01/13 16:29:44 info     general        general 0  counter flow_fwd_l3_noarp=722224 --------4 times in one second!!!
2015/01/13 16:29:44 info     general        general 0  counter flow_fwd_l3_noarp=753183
2015/01/13 16:29:44 info     general        general 0  counter flow_fwd_l3_noarp=753184
2015/01/13 16:29:44 info     general        general 0  counter flow_fwd_l3_noarp=753185
2015/01/13 16:29:45 info     general        general 0  counter flow_fwd_l3_noarp=722225
2015/01/13 16:29:57 info     general        general 0  counter flow_fwd_l3_noarp=722226
2015/01/13 16:29:58 info     general        general 0  counter flow_fwd_l3_noarp=753186
2015/01/13 16:29:58 info     general        general 0  counter flow_fwd_l3_noarp=722227
2015/01/13 16:29:58 info     general        general 0  counter flow_fwd_l3_noarp=753187
2015/01/13 16:29:58 info     general        general 0  counter flow_fwd_l3_noarp=722228
2015/01/13 16:29:59 info     general        general 0  counter flow_fwd_l3_noarp=753188
...

4. We have a timestamp we can start to find the complete entries

Look for the log at 2015/01/13 16:29:44

> show log system start-time equal 2015/01/13@16:29:44
Time                Severity Subtype Object EventID ID Description
===============================================================================
2015/01/13 16:29:44 info     dhcp           lease-e 0  DHCP lease ended ip 192.168.10.107 --> mac 64:76:ba:9e:36:d0, interface ethernet1/4.
108
2015/01/13 16:29:44 info     general        general 0  counter flow_fwd_l3_noarp=722224
192.168.8.250[29704]-->192.168.10.107[1]
sess id 50530Packet info: len 102 port 19 tag 0 interface 260
IP:     192.168.8.250->192.168.10.107, pro
2015/01/13 16:29:44 info     general        general 0  counter flow_fwd_l3_noarp=753183
192.168.8.250[29704]-->192.168.10.107[2]
sess id 61728Packet info: len 102 port 19 tag 0 interface 260
IP:     192.168.8.250->192.168.10.107, pro
2015/01/13 16:29:44 info     general        general 0  counter flow_fwd_l3_noarp=753184
192.168.8.250[29704]-->192.168.10.107[3]
sess id 33643Packet info: len 102 port 19 tag 0 interface 260
IP:     192.168.8.250->192.168.10.107, pro
2015/01/13 16:29:44 info     general        general 0  counter flow_fwd_l3_noarp=753185
192.168.8.250[29704]-->192.168.10.107[4]
sess id 12976Packet info: len 102 port 19 tag 0 interface 260
IP:     192.168.8.250->192.168.10.107, pro
....

There we go. the ARP for 192.168.10.107 has disappeared, the packets cannot be forwarded… That was the reason for the counter value. We have catch that! :-)

5. Delete your filter and set back to default

It is important to have a clear state for the next issue.

> debug dataplane packet-diag clear all
Packet diagnosis setting set to default.

> debug dataplane packet-diag show setting
--------------------------------------------------------------------------------
Packet diagnosis setting:
--------------------------------------------------------------------------------
Packet filter
  Enabled:                   no
  Match pre-parsed packet:   no            
--------------------------------------------------------------------------------
Logging
  Enabled:                   no
  Log-throttle:              no
  Sync-log-by-ticks:         yes            
  Features:
  Counters:
--------------------------------------------------------------------------------
Packet capture
  Enabled:                   no
  Snaplen:                   0            
--------------------------------------------------------------------------------

And your homework is to learn how to capture the specified counters (the packets that triggered the counter value).
Good luck!

3. Updates

3.1 Content Security and AntiVirus and URL DB and Wildfire Update and its scheduled task

Content Security Update

# Check for Content Security updates
killer@pa-firewall> request content upgrade download latest
Download job enqueued with jobid 32
32

# Install the newest update
killer@pa-firewall> request content upgrade install version latest 
...

Antivirus Update

# Check for AV updates
killer@pa-firewall> request anti-virus upgrade download latest  
...

# Install AV updates
killer@pa-firewall> request anti-virus upgrade install version latest 

Content install job enqueued with jobid 12
12

# Or the downloaded one:

killer@pa-firewall> request anti-virus upgrade install file ? 
  panup-all-antivirus-1465-1938.tgz   2015/01/19 02:15:53    69779.9K
                               Imported anti-virus package
killer@pa-firewall> request anti-virus upgrade install file panup-all-antivirus-1465-1938.tgz 

Content install job enqueued with jobid 13
13

# To check the state of the download or upgrade

killer@pa-firewall> show jobs all 

Enqueued                     ID             Type    Status Result Completed 
--------------------------------------------------------------------------
2015/01/19 12:52:49          13          Install       ACT   PEND        40%
2015/01/19 12:51:41          12           Downld       FIN     OK 12:51:57  
2015/01/19 12:48:17          11          Content       FIN     OK 12:49:26  
2015/01/19 12:45:09          10          Install       FIN     OK 12:48:17  
2015/01/19 12:41:31           9           Downld       FIN     OK 12:41:39  
2015/01/19 12:39:43           8           Downld       FIN     OK 12:39:50  
2015/01/16 17:08:49           7      FqdnRefresh       FIN     OK 17:10:07  
2015/01/16 17:06:25           6        CommitAll       FIN     OK 17:08:18  
2015/01/16 17:02:45           5           Commit       FIN     OK 17:04:06  
2015/01/16 16:59:48           4           Commit       FIN     OK 17:01:23  
2015/01/16 16:55:18           3        CommitAll       FIN   FAIL 16:55:21  
2015/01/16 14:27:40           2           Commit       FIN     OK 14:29:02  
2015/01/16 11:20:32           1          AutoCom       FIN     OK 11:22:46  

# If everything is fine we have a succesfully donwloaded and installed Content and Threath Detection Update and Antivirus Update:

killer@pa-firewall> request content upgrade info  

Version               Size              Released on Downloaded  Installed
-------------------------------------------------------------------------
481-2524              22MB  2015/01/14 01:15:34 CET        yes    current

killer@pa-firewall> request anti-virus upgrade info  

Version               Size              Released on Downloaded  Installed
-------------------------------------------------------------------------
1464-1937             67MB  2015/01/16 13:00:01 CET        yes    current

URL database Update

killer@pa-firewall> show url-cloud status

PAN-DB URL Filtering
License :                          valid                                   
Current cloud server :             s0300                                   
Cloud connection :                 connected                               
URL database version - device :    2015.01.16.809                          
URL database version - cloud :     2015.01.16.809  ( last update time 2015/01/19 14:20:48 )
URL database status :              good                                    
URL protocol version - device :    pan/0.0.2                               
URL protocol version - cloud :     pan/0.0.2                               
Protocol compatibility status :    compatible 

killer@pa-firewall> request url-filtering download paloaltonetworks region Europe 

PAN-DB update initiated
killer@pa-firewall> request url-filtering download status vendor paloaltonetworks  

2015-01-20 12:28:14 PAN-DB download: Finished successfully.

Did you know? Order of URL categorisation:
Local Block list -> local Allow list -> Custom Categories -> Downloaded URL DB -> URL Cache (in DataPlane, called “Base db”) -> dynamic lookup (in ManagementPlane, called “Cloud db”)
Source:
https://live.paloaltonetworks.com/docs/DOC-1811
https://live.paloaltonetworks.com/docs/DOC-3454
https://live.paloaltonetworks.com/docs/DOC-3608
https://www.paloaltonetworks.fr/documentation/pan-os/pan-os/section_11/chapter_5.html

Wildfire Update

killer@pa-firewall> request wildfire upgrade check

Version               Size              Released on Downloaded  Installed
-------------------------------------------------------------------------
51173-57875           10MB  2015/01/19 14:18:02 CET         no   previous
51174-57876           10MB  2015/01/19 14:33:02 CET         no    current

killer@pa-firewall> request wildfire upgrade info

Version               Size              Released on Downloaded  Installed
-------------------------------------------------------------------------
51175-57877           10MB  2015/01/19 14:47:03 CET         no    current
51174-57876           10MB  2015/01/19 14:33:02 CET         no   previous

killer@pa-firewall> request wildfire upgrade download latest 

Download job enqueued with jobid 1026
1026

killer@pa-firewall> request wildfire upgrade install version latest 

Content install job enqueued with jobid 1027
1027

Automatic Updates

Automatic update for URL filter is only possible with BrightCloud, the Paloaltonetworks URL Database will be just in time asked for URLs.

killer@pa-firewall# set deviceconfig system update-schedule ?
> anti-virus                Anti virus database
> app-profile               Application profile database
> global-protect-datafile   GlobalProtect datafile
> statistics-service        Participate in Anonymous Statistics Upload Service
> threats                   Threat detection database
> url-database              URL filtering database
> wildfire                  WildFire database
                     Finish input

# Example:
killer@pa-firewall# set deviceconfig system update-schedule threats recurring daily at 03:00
killer@pa-firewall# set deviceconfig system update-schedule threats recurring daily action download-and-install
killer@pa-firewall# set deviceconfig system update-schedule anti-virus recurring daily at 19:30
killer@pa-firewall# set deviceconfig system update-schedule anti-virus recurring daily action download-and-install
killer@pa-firewall# set deviceconfig system update-schedule wildfire recurring every-15-mins action download-and-install

# Or Example with Panorama:

killer@pa-panorama# set deviceconfig system deployment-update-schedule pa-av anti-virus recurring daily at 08:15
killer@pa-panorama# set deviceconfig system deployment-update-schedule pa-av anti-virus recurring daily action download-and-install devices [ 001..devices_serials... ]

3.2 Software Upgrades

killer@pa-firewall> request system software info

Version               Size          Released on Downloaded
-------------------------------------------------------------------------
6.1.1                179MB 2014/12/18  00:05:36         no
6.1.0                329MB 2014/10/25  08:34:03         no
6.0.7                237MB 2014/12/08  21:52:10         no
6.0.6                237MB 2014/10/23  20:43:52         no
6.0.5                237MB 2014/09/23  11:04:03         no
6.0.5-h3             237MB 2014/10/08  16:31:06         no
6.0.4                214MB 2014/08/04  19:39:30        yes
6.0.3                207MB 2014/06/11  23:56:10         no
6.0.2                203MB 2014/04/23  22:07:26         no
6.0.1                200MB 2014/03/09  10:40:44         no
6.0.0                353MB 2014/01/19  11:12:47        yes
...
5.0.7                187MB 2013/08/27  14:05:52         no
5.0.6                187MB 2013/07/08  17:04:39        yes
5.0.5                187MB 2013/05/24  22:32:41         no

killer@pa-firewall> request system software check 

Version               Size          Released on Downloaded
-------------------------------------------------------------------------
6.1.1                179MB 2014/12/18  00:05:36         no
6.1.0                329MB 2014/10/25  08:34:03         no
6.0.7                237MB 2014/12/08  21:52:10         no
6.0.6                237MB 2014/10/23  20:43:52         no
6.0.5                237MB 2014/09/23  11:04:03         no
6.0.5-h3             237MB 2014/10/08  16:31:06         no
6.0.4                214MB 2014/08/04  19:39:30        yes
6.0.3                207MB 2014/06/11  23:56:10         no
6.0.2                203MB 2014/04/23  22:07:26         no
6.0.1                200MB 2014/03/09  10:40:44         no
6.0.0                353MB 2014/01/19  11:12:47        yes
...

killer@pa-firewall> request system software install version ?
  5.0.0    5.0.0
  5.0.6    5.0.6
  6.0.0    6.0.0
  6.0.4    6.0.4
    Upgrade to a software package by version

killer@pa-firewall> request system software install version 6.0.4
killer@pa-firewall> request system software download version 6.0.4

# Dont forget to reboot it is not happened!
killer@pa-firewall> request restart system
killer@pa-firewall> debug swm list

Product     B Version              Built                 Loaded
--------------------------------------------------------------------------------
panos       * 5.0.0                2012-10-26 16:49:01   2014-01-30 14:31:23
panos         5.0.6                2013-07-03 16:49:39   2014-01-30 14:32:18
panos       * 6.0.0                2014-01-14 14:02:40   2015-01-15 19:41:42
panos         6.0.4                2014-07-30 12:31:52   2015-01-16 10:36:55

killer@pa-firewall> debug swm status

Partition         State             Version
--------------------------------------------------------------------------------
sysroot0          RUNNING-ACTIVE    6.0.4
sysroot1          REVERTABLE        5.0.6
maint             READY             5.0.6

3.3 Licensing

killer@pa-firewall> request license ?
> fetch     Get new license(s) using an authentication code
> info      Show information about owned license(s)
> install   Install a license key

killer@pa-firewall> request license info 

Current GMT Date: January 15, 2015

License entry:
Feature: PAN-DB URL Filtering
Description: Palo Alto Networks URL Filtering License
Serial: 00xxx
Issued: January 16, 2015
Expires: January 16, 2020
Expired?: no

License entry:
Feature: Threat Prevention
Description: Antivirus, anti-spyware, vulnerability protection
Serial: 00xxx
Issued: January 16, 2015
Expires: January 16, 2020
Expired?: no
Advertisements