When something goes wrong, a response is warranted to determine just what went wrong, why it went wrong, and to set it right again. When investigating a security incident, there are a number of best practices to be followed to ensure a thorough investigation and recovery. Basic forensic procedures and swift action can help to keep a “canary in a coalmine” from becoming a catastrophe. This section will cover the following topics:
- Basic forensic procedures
- Damage and loss control
- Chain of custody
- Incident response: First responder
Basic Forensic Procedures
When gathering information to analyze a security response, there are a number of considerations and best practices to implement to ensure all available information can be gathered. The following are basic forensic procedures:
Order of volatility – When gathering information after a security incident, gather it in the order of most to least volatile, which is the order it is likely to have been changed or lost. The most volatile information is held in RAM, and it is constantly changing. Next, there may be information that changes regularly, such as swap files or working files for system processes. Less volatile data would be disk contents. Data unlikely to be changed, and thus the least volatile, is that held on separate systems, such as logs or archived data.
When gathering data, it is not necessarily a good idea to shut down an affected system immediately, as doing so would cause the loss of the most volatile information. If possible, information should be gathered before taking actions that would cause the loss of volatile data.
Capture system image – In order to preserve the non-volatile contents of a system for forensic investigation, standard practice is to create a system image. This image is a snapshot in time of the contents of a disk or array. Upon creating an image, it is advisable to create a hash of the image, such as MD5 or SHA512, to ensure future data integrity. The hash of the file can be compared to a newly generated hash of any future copy of the file to ensure no changes have been made to the file.
It is possible to make disk, partition, or array images. Making an image of a single disk in a striped redundant array of independent disks (RAID) will not yield a usable image. Take care that the type of image you create conforms to the type of data it is necessary to capture.
Network traffic and logs – Investigations of an incident can be greatly aided by analyzing data that is captured constantly and logged. Traffic analysis can only occur if the traffic is already being captured, as network communication is transient in nature. In order to be able to analyze traffic, you must configure in advance logging that may be useful. The same can be said for any transient event. It is difficult to know ahead of time what information may be useful, so an effort should be made to find a balance between monitoring overhead and depth of tracking.
Capture video – Any relevant video should be recorded and kept. This video can be reviewed later as an infallible record of what happened, rather than relying on memory alone.
Record time offset – It is not unusual for systems on the same network to have slight variations in the time of their system clocks. It may be important to know in what order the events occurred when comparing logs from disparate sources. Note the variation of time on a given system’s clock from a trusted benchmark.
Take hashes – Hashes ensure data integrity. A changed file hash is an indicator that the contents of the file are no longer the same as they were when the hash was created. Create and log the hash of any file for which data integrity is important.
Screenshots – In addition to video, screenshots of work in progress can be created to help form a record of the state of a system at any given time.
Witnesses – Like the contents of RAM, firsthand recollections are volatile. Gather information from people soon after the event, as memories do not remain constant over time.
Track man hours and expense – Record the time spent responding to an incident and the work performed for a given time. Having a record of the work that was done and when can be valuable in determining the total cost of a response, in addition to which steps have already been taken and which steps still need to be taken. Tracking the costs of responding to an incident can be useful in determining whether the response is proportionate to the risk and adjusting priorities accordingly.
Damage and Loss Control
When responding to an incident, a major goal is to minimize the severity of the incident, also known as damage and loss control. This is an important consideration that must often be weighed against the other imperatives, such as ensuring system integrity. For some incidents, by the time you become aware of an issue, the threat has passed. At other times, an incident may represent an ongoing threat, such as a virus or a Trojan-horse-infected system. When responding to an incident in progress, steps must be taken quickly to prevent damage and loss that has not already occurred.
Chain of Custody
The list of those who have handled or have accessed information gathered for forensic purposes is known as the chain of custody. Its purpose is to provide documentation as to who has handled the evidence, and to be able to account for the whereabouts of evidence at all times.
Incident Response: First Responder
As a first responder to an incident, you may need to make decisions based on incomplete information. You will need to weigh competing considerations, such as thoroughness of forensic data gathered against the ongoing threat an affected system may pose. This may mean quickly classifying the incident as past or ongoing, and taking immediate action in the case of an ongoing threat. The first priority is to limit the damage while destroying as little evidence as possible. A policy should be put in place that outlines the processes a first responder to an incident should use, such as what constitutes an incident that requires elevation to a team of responders.