Sometime March 2016, our Hong Kong office reported that their RDP connection experience to a server in our Head Office in Sydney has progressively slowed to the point where it became unbearable for them. The connection is over IPSEC VPN (HK uses Fortigate 100D over 20Mbps Internet and Syd HO uses Fortigate VM01 over 100Mbps Internet). Below are some observations:
HK Internet browsing - Staff has NOT reported any issue. SYD HO Internet browsing - Staff has NOT reported any issue VPN tunnels – all good between Syd HO and ALL sites HK ping test - Lots timeouts between HK and Syd HO Other branch ping test – No timeout between other branches and Syd HO RDP connection of HK staff to Syd HO server is slow (apparently since November last year), keeps freezing, or keep disconnecting. RDP connection of other branches to Syd HO server is normal
As both Internet connections are OK, it was tempting for us to dismiss this as an issue in between the two LANs (ie ISP or cloud issue) that we just need to wait it out. But since it was apparently happening since Oct 2015, we needed to dig dipper and we asked for a traffic report from the ISP. Here is what we got.
So the high ‘Inbound’ Internet traffic is a possible cause and it confirms their observation re when it started (around late October). We now need to know what is causing this anomalous traffic.
Over the years, we have had Internet performance issues caused by among other things
Viruses, worms, Wi-Fi traffic, Windows firewall, Antivirus, some dodgy software installed, actual port speed being throttled down by the ISP, ISP router/line issue, large email attachment being sent to a lot of recipients, Youtube/streaming videos, slow proxy server, firewall, ethernet port speed mismatches, LAN switch issues, faulty NIC (very rare), etc
I was able to rule out a lot of these after checking the LAN switch, firewall traffic activity, server, antivirus logs, system logs, etc.
Whilst the ISP traffic report show huge inbound traffic, our HK Fortigate 100D firewall reports do not show any evidence of this between any IP (inside or outside) and this left me scratching my head. It seems Fortinet is not logging everything. This is when I thought maybe the HK firewall is blocking a lot of attempts of a specific inbound traffic. I enabled data leak prevention logging (since we block some file extensions like .exe, .com, etc from being downloaded) and focused on what is being rejected. There were a lot of the below blocked files throughout a normal day.
Checking of the URL link confirmed they’re Microsoft office update files being blocked.
So it seems the cause of the traffic spikes is the firewall blocking Microsoft update. The HK PCs will regularly run an Office/windows update check and request the updates to be installed. However, on the way in, the updates keep getting blocked thus registering on the ISP traffic monitor but not on our Fortigate firewall.
The solution is to exempt the officecdn.microsoft.com.edgesuite.net link in the firewall. It took a day or two for things to normalise.
Updated traffic from the ISP confirmed the issue has been resolved.
Do you recall any weird network-related issue that pushed your wits to the limit?