One project I was involved in earlier on was the testing of the customer's digital quick reaction force (QRF). This group of security and IT professionals were supposed to be at the ready in cases where the organization's networks or systems were being compromised.
So we set to work. We needed to find a way into their networks. Usually the best way in would be a social engineering attack where we would send the target an email with a weaponized document or a link to a site with code that can exploit different vulnerabilities on their browsers, however this time we also wanted to see how good their security hardening practices were, how their perimeter was set and whether they were monitoring the different network devices at all.
After a short scan of their public facing IP range we found a server with a vulnerable version of IIS (the Microsoft web server software). A little digging around on different hacking forums gave us several exploits we could try.
First we ran an exploit we developed that would set some alarms. It wasn't going to give us access, but it would make IIS dump a lot of weird logs. We randomly inserted messages to be logged hinting to us attacking the server. We even added a contact email address and phone number. After 24 hours of attacking it, we had no one contacting us. We moved to phase 2.
We ran one of the exploits we found online. This one worked as advertised and we had a shell to the server with System privileges. A quick recon of the server (routing tables, shares, users, domain info, etc) gave us the route to continue. Once you set foot on the network, ideally you have to start moving very carefully, you don't want to set alarms. In this case we wanted to test the QRF so we made no efforts in hiding our presence.
We wrote two tools that would allow us to move laterally by trying every computer connected to the domain. This would set a lot of alarms. One tool would query the domain controller for the list of computers and would try to connect to them via the administrative shares (C$, D$, etc) or by performing NULL connections. The other tool was a crude port mapping tools that was set to map the entire internal IP block the server was sitting on (192.168.0.0/16 - these are quite a bit of addresses). If this didn't set alarms I don't know what would.
After a couple of hours of pocking around and letting the tools run, we noticed that some of the IPs being scanned had ports in the 12000 range open. These port is usually reserved to internal controls, custom apps and some Solaris stuff. Both TCP and UDP were being randomly selected on different computers. All these computers seemed to be workstations (the OS was Windows XP with the classic Windows ports open).
Immediately we went into defensive/forensics mode. The QRF was nowhere to be found and we just stumbled upon something very suspicious. Often these high ports are used by backdoors and other forms of malicious code. Could it be that the organization was already compromised? It was possible; we found a way in so could attackers.
We alerted the SOC (where the QRF was located) and the security director of the organization. We stopped the exercise and went into discovery mode. The SOC members and us began checking logs, monitoring tools and some of the affected systems. After 12 hours of hard checking we found a nasty rootkit installed on all the computers we checked. It was using different ports but it was the same malware.
It took us over 30 days to detect all the copies installed across the network, not just the one we initially penetrated, but all the different internal networks the organization had. Servers were the worst, on top of the rootkit, the servers had copies of other binaries on disk that we assumed were stashed there by the attackers for later use or the servers were used as storage.
Even after all that cleanup we helped perform, we still were not sure that their network was completely clean. That's the nature of these kinds of malware, they are very stealthy.
We finished with the recommendation that a full penetration test be performed.
What's the moral of the story?
1- Don't assume that because a computer is behind a firewall or other protection devices then it's safe.
2- Red team exercises can also help uncover system compromises.
Invest on a good security: run penetration tests and vulnerability assessments every quarter. You never know.
Also, train your QRF by hiring a good red team. In this case, they were so confident of their tools that failed to manually monitor logs and domain controller messages.