The scenario: Unauthorized content appeared on the company website, and it is possible that the site and other company servers have been compromised. The I.T. department has been hard at work for the last several hours tracking it down and expects to have an answer in the next few.

The question to ask is not how did this happen, but why is it taking so many people so long. In a company with even as many as twenty-five mission critical servers it should not take one individual more than an hour to answer this important question¹.

For this example we will assert that the site in question has the following setup; 2 database servers setup for load balancing and
redundancy, 4 web servers, and an OpenLDAP server for intranet authentication. Let us also say the web servers are using some content management software such as Drupal. The database or web server in use does not matter so long as you patch them regularly. We are only concerned with dealing with a breach and not preventing it (a topic for another day).

Now in this example we have seven servers and are going to reduce the breach discovery time by installing one very important piece of
software and another timesaving one.

The first is intrusion detection software; Tripwire for example has both a commercial and free open source version. Both perform the same basic function we need but the commercial release will provide more features and polish. Tripwire works by scanning the drives of the server and storing a hash of every file and compares them against the *approved* values in its database. Approved hashes are updated by running the command and providing the decryption password for update. This is done after some known system update or some other file change.

The intrusion detection software can generate a report of any files that were added, deleted, changed on the system.

The second piece of software is a distributed/parallel shell utility. Several different applications can be found by doing a quick Google search and the easiest to install on the servers is the best one to use. The distributed shell is used to generate a report from the intrusion detection software on all servers at the same time, and saves a few minutes. The alternative is to simply use different ssh shells to invoke the same command on each server and would be needed if you have different passwords and/or login keys on each server.

Putting this together we are now able to start a report at any time we think we may have a site breach and quickly see from the reports any suspicious behavior, such as a newly installed version of sshd, gcc, etc...

Ideally when the reports run everything checks out and no Trojans have been installed and everything is as it was since the last Tripwire update was run. Otherwise we can see from the report which servers fail the integrity check and know that yes there was a breach and can properly direct the cleanup effort.

However this by no means eliminates the human factor; meaning weak passwords or site updates being accidentally promoted will always defeat the best security.

¹ We assume some flavor of Unix is being used as software mentioned is dependant on GNU software