I’m working as a Cloud Engineer and one of my daily working routines is to administrate the local network and system,
I have checked for any update released on the Firewall of the branch, Yes I had a version to upgrade the firewall firmware.
Just jumped to the router dashboard, make a safety backup for the firewall then download the firmware from the official portal.
Went to the latest firmware and upgrade the firewall appliance, the first dangerous network issue starting here, the firewall needs to be upgraded of the firmware version by version in a sequence manner like if you have v1.0 and the latest release is v3.0 you must upgrade to v2.0 then to the v3.0 what my bad was is to skip the middle version to the latest version. Then the whole branch becomes offline! out of network and internet, the fighting starts here,
Jumped to the data-center cabinet and pressed the reset factory button for the firewall to format all the user made changes, went back to my desktop and restore the Safety backup file (I thanked my self a lot here), but the firewall is not accessible by the company subnet IP Address like 220.127.116.11/24 “The IP is an example” and the firewall is restored to the default subnet IP address which is 192.168.1.0/24
After several overhead and users start to report and call of internet issue facing them, I changed the IP address to the default subnet (Thanks to Google here)
Also, the firewall default DNS pool starts to assign the end-users PCs with default subnet IP addresses e.g. 192.168.1.0/24 which differs from my subnet settings!
When the restore process is finished my subnet addresses were restored in Firewall only, now the users took IP from the default pool which is 192.168.1.0/24 and they could not access the network due to different IP and default gateway configuration!
Jumped to the offline users and start to troubleshoot the issue by ipconfig /renew command to reset the IP address with no luck!
I tried to disable the network adapter and re-enable it so that it worked!
Discover another router in the company
One of the sales has PCs that took different IP addresses from the default pool and my company pool, I’m seriously shocked! Have I been attacked? after many troubleshoot of tracing IP addresses with no luck I tried to check the default gateway IP address of the subnet from the same network, The result was the TP-Link Router portal! wow,,
Tracing back to the cable to find the hidden router inside the table works as Wi-Fi and router at the same time.
They used this router for Wi-Fi Service, which answers my old question why those PCs couldn’t find the HQ server by local domain name!
I Identified the router assets in my topology sheet branch and secure the default settings.
How to Mitigate Hidden Network?
The above scenario taught me to do Penetration Testing on my local network to find any hidden appliances and weaknesses behind the scene before the attackers do.