Application shifting gone bad – Palo Alto Firewall

The App sophos-live-protection and the Palo Alto dns-proxy Affair

As I first met with this problem I just thought its my first vulnerability that I have found :-), but it wasnt! It was the administrator who forgot to go to a palo alto school before touching a palo alto firewall. :-)
The firewall I configured worked as a dns-proxy and forwarded every dns request to an outside dns server. Once I just realised that the dns queue of a dns-proxy server keep increasing and will not go back to zero:

mruser@myfirewall> show dns-proxy statistics all | match pending\|Name
Name: dsl-1
Queries pending:0
Name: dsl-2
Queries pending:38

mruser@myfirewall> show dns-proxy statistics all | match pending\|Name
Name: dsl-1
Queries pending:0
Name: dsl-2
Queries pending:64

After restarting the dns proxy service, it worked again. But on the next days or next hours it started to freeze again and had to restart again and so on…
This was a beginning of a really long troubleshooting till we found the problem:

There is a sophos live protection application from the sophos AV company that has already a palo alto application in the firewall (palo alto can identify it and can be used in the policy as an application) . This software from sophos uses that weakness of dns security for legal purposes!
It uses the dns protocol but not for dns service, they use it with the same idea that I wrote in my post about icmp tunneling. Link: https://itsecworks.com/2013/02/15/fire-in-the-hole-of-the-firewall/
It makes data exchange on udp port 53 since it is opened almost everywhere and does not requires proxy setting like it has to have if it would use http or https or ftp protocols in a company network.
Here is the description of sophos live protection:
https://community.sophos.com/kb/en-us/111334
”
When a lookup-enabled detection is triggered by the on-access scanner, on-demand scanner, or runtime HIPS, the SAV service performs a specially crafted DNS query that includes generic information about the file and the detection features, to the sophosxl.net name servers. It then takes action(s) based on the response it gets.

Currently available actions include:

– Ignore the detection, for instance if the file is known to be detected as a false positive
– Treat the detection as malware
– Treat the detection as suspicious
– Request a sample (performed only if allowed by the policy and, please note, only applies to executable files)

If the file is to be sent as a sample and the policy allows automatic sample submission, the SAV service collects information about the file and the detection, packages the file itself and the data gathered into an encrypted package and uploads it via HTTP.Only files smaller than 10MB are uploaded.
”

The problem comes first if you dont allow the application dns bit not this application but your client have sophos software on your internal network and your firewall works as a dns-proxy.
The firewall thinks from the first udp packages that this is a dns traffic, let it through to the dns-proxy but it then realises that this is a sophos-live-protection application and it checks again in the security policy if there is a matching rule and it denies it in case of no match. That way the dns traffic from the firewall to the external dns server will be blocked till the firewall builds up another dns session and that happens after restarting the dns proxy.
In the firewall logs I saw that after those dns queries sent to sophosxl.net the dns proxy fails and does not works anymore. So the application shifting for dns applications can kill your dns-proxy in the firewall!
I sniffered that dns traffic and replayed the packets on another firewalls with the same settings. tcpreplay link: http://tcpreplay.appneta.com/

myfirewall

1. Update the macs in sniffered packets

1.1. MAC 52:54:00:12:35:00 : Virtual Box Hosting MAC. See below:

test@linux-srv01:~/temp$ arp -a
? (192.168.56.1) at 08:00:27:00:68:a9 [ether] on eth1
? (10.0.2.1) at 52:54:00:12:35:00 [ether] on eth0

1.2. MAC 08:00:27:da:e3:15 : Virtual Box Virtual Machine MAC. See below:

test@linux-srv01:~/temp$ ifconfig
eth0      Link encap:Ethernet  HWaddr 08:00:27:da:e3:15  
          inet addr:10.0.2.4  Bcast:10.0.2.255  Mask:255.255.255.0

2. IP 10.0.2.4 : Virtual Box Virtual Machine IP

tcprewrite \
--enet-dmac=52:54:00:12:35:00 \
--enet-smac=08:00:27:da:e3:15 \
--srcipmap=0.0.0.0/0:10.0.2.4/32 \
--enet-vlan=del \
--infile=dns_sophos3_TXT_Type.pcap \
--outfile=dns_sophos3_TXT_Type_MACmod_myfw.pcap -C

tcprewrite \
--enet-dmac=52:54:00:12:35:00 \
--enet-smac=08:00:27:da:e3:15 \
--srcipmap=0.0.0.0/0:10.0.2.4/32 \
--enet-vlan=del \
--infile=dns_sophos2.pcap \
--outfile=dns_sophos2_MACmod_myfw.pcap -C

tcprewrite \
--enet-dmac=52:54:00:12:35:00 \
--enet-smac=08:00:27:da:e3:15 \
--srcipmap=0.0.0.0/0:10.0.2.4/32 \
--enet-vlan=del \
--infile=dns_mcafee5.pcap \
--outfile=dns_mcafee5_MACmod_myfw.pcap -C

sudo tcpreplay -i eth0 dns_sophos3_TXT_Type_MACmod_myfw.pcap
sudo tcpreplay -i eth0 dns_sophos2_MACmod_myfw.pcap
sudo tcpreplay -i eth0 dns_mcafee5_MACmod_myfw.pcap

example:

test@linux-srv01:~/temp$ sudo tcpreplay -i eth0 dns_sophos3_TXT_Type_MACmod_myfw.pcap
sending out eth0 
processing file: dns_sophos3_TXT_Type_MACmod_myfw.pcap
Actual: 1 packets (120 bytes) sent in 0.00 seconds
Rated: inf bps, inf Mbps, inf pps
Statistics for network device: eth0
        Attempted packets:         1
        Successful packets:        1
        Failed packets:            0
        Retried packets (ENOBUFS): 0
        Retried packets (EAGAIN):  0

Before: The dns-proxy is good, no pendings

mruser@myfirewall> show dns-proxy statistics all | match pending\|Name
Name: dsl-1
  Queries pending:0
Name: dsl-2
  Queries pending:0

After: something gone bad,  the dns-proxy has only pendings

mruser@myfirewall> show dns-proxy statistics all | match pending\|Name
Name: dsl-1
  Queries pending:0
Name: dsl-2
  Queries pending:38