Roughly ten years ago I had a penetration testing team come in to do an annual assessment of the corporate network. I had a habit of changing the company that did the testing every year. I preferred to have a different set of eyes looking at the network each year with the idea in mind that I would get a different approach. That proved itself to be true in the most peculiar ways. Some teams were absolutely stellar at their job. One thing that I would typically do was have them agree (when possible) to allow me to be an observer to watch the team work. This was a great educational experience for me to watch them attack the site and see how their thought processes flowed. I found this to be hugely beneficial.
In one case I was amazed watching this one tester as he slammed back espresso after espresso until he discovered a problem in one our Internet facing systems. There was no documented vulnerability for this application but, it behaved in such a way that it gave him pause. He sat back in his seat and scratched his chin. Then suddenly hunched over his keyboard and began typing madly. In 15 minutes he had a stable working exploit that he was able to leverage and use this system as a pivot into the network. While this was amazing to watch it helped to set the bar very high.
The next year I had a different team do our annual test and they were a disappointment. I received a report that was visibly a cut & paste from the Nessus vulnerability scanner. Now, to be clear, there is nothing wrong with this tool. The problem that hit me was that this report wasn’t even validated.
This is an interesting article. Not so much for the content but the subject matter. What Dave Lewis writes about is something we experience almost contantly with new customers. Sometimes new customers show us the reports from previous companies that performed a pentest on their network and systems, and you can clearly see sometimes the lack of professionalism.
I'm not saying we are perfect, but we pride ourselves in trying and testing our findings, and going that extra step and finding the undocumented vulnerabilities.