Category: Firewall

Troubleshoot strange Internet issue

Sometime March 2016, our Hong Kong office reported that their RDP connection experience to a server in our Head Office in Sydney has progressively slowed to the point where it became unbearable for them. The connection is over IPSEC VPN (HK uses Fortigate 100D over 20Mbps Internet and Syd HO uses Fortigate VM01 over 100Mbps Internet). Below are some observations:

HK Internet browsing - Staff has NOT reported any issue.
SYD HO Internet browsing - Staff has NOT reported any issue
VPN tunnels – all good between Syd HO and ALL sites

HK ping test - Lots timeouts between HK and Syd HO
Other branch ping test – No timeout between other branches and Syd HO

RDP connection of HK staff to Syd HO server is slow (apparently since November last year), keeps freezing, or keep disconnecting.
RDP connection of other branches to Syd HO server is normal

As both Internet connections are OK, it was tempting for us to dismiss this as an issue in between the two LANs (ie ISP or cloud issue) that we just need to wait it out. But since it was apparently happening since Oct 2015, we needed to dig dipper and we asked for a traffic report from the ISP. Here is what we got.

Slow-Internet-Challenge

So the high ‘Inbound’ Internet traffic is a possible cause and it confirms their observation re when it started (around late October).  We now need to know what is causing this anomalous traffic.

Over the years, we have had Internet performance issues caused by among other things

Viruses, worms, Wi-Fi traffic, Windows firewall, Antivirus, some dodgy software installed, actual port speed being throttled down by the ISP, ISP router/line issue, large email attachment being sent to a lot of recipients, Youtube/streaming videos, slow proxy server, firewall, ethernet port speed mismatches, LAN switch issues, faulty NIC (very rare), etc

I was able to rule out a lot of these after checking the LAN switch, firewall traffic activity, server, antivirus logs, system logs, etc.

Whilst the ISP traffic report show huge inbound traffic, our HK Fortigate 100D firewall reports do not show any evidence of this between any IP (inside or outside) and this left me scratching my head. It seems Fortinet is not logging everything. This is when I thought maybe the HK firewall is blocking a lot of attempts of a specific inbound traffic. I enabled data leak prevention logging (since we block some file extensions like .exe, .com, etc from being downloaded) and focused on what is being rejected. There were a lot of the below blocked files throughout a normal day.

Slow-Internet-Challenge3

Checking of the URL link confirmed they’re Microsoft office update files being blocked.

So it seems the cause of the traffic spikes is the firewall blocking Microsoft update. The HK PCs will regularly run an Office/windows update check and request the updates to be installed. However, on the way in, the updates keep getting blocked thus registering on the ISP traffic monitor but not on our Fortigate firewall.

The solution is to exempt the officecdn.microsoft.com.edgesuite.net link in the firewall. It took a day or two for things to normalise.

Internet-Performance-3

Updated traffic from the ISP confirmed the issue has been resolved.

Slow-Internet-Challenge2

Do you recall any weird network-related issue that pushed your wits to the limit?

 

 

 

 

Advertisements

Cursed at 497 days!

One morning back November 2015, we were having connectivity issues with our Asia branch offices which took a good day to resolve.  The offices were using Fortigate 100D firewalls and connect to our Sydney core firewall running on Fortigate VM01 virtualised appliance.

The main symptoms were sluggishness of remote access (vice versa) and application timeouts. Quick check of the firewalls revealed the IPSEC tunnels keep flapping with the system messages below  in the core firewall (similar messages were showing on the branch firewalls).

 Link monitor: Interface SYD-HK-peer1 was turned down
 Link monitor: Interface SYD-SG-peer1 was turned down
 Link monitor: Interface SYD-HK-peer1 was turned up
 Link monitor: Interface SYD-SG-peer1 was turned up

Fortigate-Logs-Up-Down-VPN-flapping

Whilst I personally vouch for Fortigate firewalls’ rock solid stability through the years, we have had some notable and strange issues over the years. A particular one was when emails started bouncing back one morning and the cause was eventually traced to Fortigate’s antivirus update (it’s automatically set to update every hour) rejecting every email coming its way. In this VPN flapping issue case, AV filtering and a few other controls/features (eg IPS) were momentarily turned off but the flapping problem persists.

We have checked time zone settings, NTP server availability, checked IPSEC VPN settings (which were never changed recently), ran vpn debugs, etc but still no joy. Anyone looking after IT systems would know that when you encounter weird technical issues like this -which affects the business and phones are running hot – then part of you wants to suddenly be a yoga instructor or do some job with nothing to do with technology!

Guess what fixed it later that night…a reboot of the core firewall!

We had it running for 497 days non-stop so it probably needed a little break 🙂
Further research revealed this behaviour also affects systems/devices from Windows, Linux, F5 and Cisco!

Why 497 days?
There are plenty of documents online saying this has to do with a limitation during coding where the number of 10 ms ticks since boot can’t fit into a 32 bit unsigned integer beyond, guess what, 497 days!

So when in doubt, the age old recommendation of rebooting the device won’t hurt. Who would have thought eh?