Again, the weakest link

Some time ago I performed a penetration test that turned out to be more interesting that I originally thought it would.

My customer approached me with a problem. They had an insider leaking information and they wanted to catch him. I've done this before and there is a set of things you can do to start monitoring the network and catch the bad guy: you run network sniffers, you monitor the logs from sensitive servers in real time, you install honeypots with data that looks legit, and many other things.

This one, however, turned out to be a bit more complicated.

After a week of monitoring the security department and I found what appeared to be several bad guys. Confidential files and documents were being copied from a Black server (1), from the main DHCP server, from the main backup server and from several workstations at the HR department. Those systems were located in different parts of the world and on different subnets.

After some deliberation the security department allowed me to place a piece of monitoring software on all the networks affected. I left it running for another week. During that period of time I started reviewing log files from firewalls, routers, mail servers, file servers, domain controllers, etc. It was a very time consuming and boring job and I didn't expect to find anything beyond the information about the files being copied, but since I never assume I did it anyway.

Then I found it.

Hidden, or rather buried, in tons of email messages were four very distinctive email messages. Those emails were directed to key people in the company, people with a high level of access (essentially they can be root or administrator) and were beautifully crafted in order to convince the reader to click on a link. I followed the link and what I found made me grin. It was a masterpiece of malicious code.

Once I had this and with a little bit of reverse-engineering I figured out how to find the code inside the network and clean it. It wasn't an insider, but a well crafted piece of spyware / trojan that arrived via email and exploited the weakest link of all: the human beings.

But I wasn't done yet. I was tasked then by the same customer to figure out who did it. The next six months were really wild. Since this breach affected computers in Europe, Asia and other places I had to work with Interpol and their fantastic cybercrime department. I travelled a lot and was part of a wild chase.

What started a relatively simple project, turned out to be a multi-month, multi-location wild hunt.


(1) A black server is a server connected to a network without internet access, usually referred as a black network. Black networks contain computers used for confidential projects, government projects or simply as a way of separating the internal networks and keeping company-sensitive data away from any potential attacker from the internet. They are usually guarded by an airgap-like system or a one way firewall or router.

(I originally posted this on Oct 25, 2011. Reposting because a lot of people asked me about this)