Imagine it’s 3 AM and you're eating pizza. You're wearing earmuffs because the data center you're working in is so cold. You're part of an incident response team working round the clock to help a transportation company that’s been hit with a nasty case of ransomware.
While you're running dynamic and static tests on infected endpoints and analyzing the malware variant, others are determining how much access the attackers still have, looking for the best recovery point, and communicating with the national CERT (Computer Emergency Response Team), customers, and the media.
...attackers have told the victim that the price doubles if the ransom is not paid within two days
It’s a noisy madhouse and the clock is ticking. As time goes by, more systems can become infected, and more data stolen. As is common, attackers have told the victim that the price doubles if the ransom is not paid within two days. You must work fast to determine the business impact and the best way to recover.
It's a story worth sharing because it’s likely that at some point in your career you may be in a similar situation. When you recognize that identity-related incidents are becoming ubiquitous, with 80% of organizations experiencing at least one annually, it’s safe to assume that you may well be a victim of an attack.
Incident response is one of the most stressful scenarios an IT or security leader can encounter. It’s like fighting a fire, albeit a digital one. You have to make a lot of decisions quickly, without complete information.
In this blog, you’ll learn a framework for structured incident response, known as the incident response lifecycle, that can make you more resilient, remove some of the stress, and help you get back to business as soon as possible.
What is the incident response lifecycle?
The incident response lifecycle is a six-part framework that covers each stage of response to a cyber incident. It can apply to any type of identity-related attack, such as a data breach, ransomware, extortion, credential theft, lost laptop, etc.
I’ll use a common incident scenario—ransomware—to walk through each stage of the incident response lifecycle. In each stage, I’ll outline recommended security controls, policies, and processes. I’ll also take you back to the night of our hypothetical ransomware attack against the transportation company so you can see how steps within each stage of the incident response lifecycle apply to a real-world attack.
Six stages of the incident response lifecycle
Incident response lifecycle stage 1: Preparation
At the Preparation stage of the incident response lifecycle, the goal is two-fold: 1) lay the groundwork so you can respond quickly and confidently should an incident occur, and 2) practice the incident response lifecycle so you are incident response ready.
Create and plan your incident response lifecycle plan or checklist
One of the most critical aspects of your incident response lifecycle is to have a clear plan for what to expect when an incident occurs. Not all incidents are equal, so you will likely have different workflows depending on the scenario, such as ransomware, a lost laptop, or compromised credentials.
An example of an Incident Response Lifecycle Checklist:
| Incident Response Plan/Checklist | |
| 1. Ownership | 5. In-house capability and 3rd party responsibility |
| 2. Communications | 6. Containment (evidence) |
| 3. Contact list | 7. Press Statement |
| 4. Clear definition of threat | 8. Legal assessment |
| a. Confidentiality - data loss | 9. Eradication |
| b. Integrity - data poisoning | 10. Recovery |
| c. Availability - DDOS | 11. Lessons learned |
For a more detailed checklist, check out our Cyber Incident Response Checklist. Plus, Delinea’s Incident Response Template can help you outline policies, systems, roles and responsibilities to help with the preparation stage of the incident response lifecycle.
Prepare roles for the incident response lifecycle
The incident response lifecycle isn’t an IT or security team-only responsibility. Response to a cyber incident involves many parts of an organization. Having clearly defined roles is essential to ensuring the team knows what they should be doing when the incident response is activated.
Figure out who will be responsible for:
- Project management
- Communications
- Documentation
- Legal issues, including exposure and requirements
- Digital forensics
- Malware analysis
- Data recovery
- Threat intelligence
- Backup and recovery
- Internal communication with the executive team, board, and employees
- External communication with law enforcement, insurance company, customers, and partners
Outside of this list, there are some teams you don’t want to forget.
Include your help desk in the preparations. They can become overwhelmed if an attack goes public and they start getting calls from suppliers, customers, and partners. They need to be prepared and have a process for response, tested and ready.
Decide if you’re going to include any third parties, such as data aggregators, or external incident response teams. Your hosting provider should be involved as part of your extended team. Your cyber insurance provider may require you to work with a third party of their choosing.
Prepare processes for the incident response lifecycle
There are numerous processes and workflows you’ll want to prepare, so you don’t have to make them up on the spot during a live incident. For example:
- Make sure you’re familiar with any legal or compliance requirements you have for incident response. For example, if you need to abide by SEC requirements in the United States, you must report “material” cybersecurity incidents on a Form 8-K within four business days of materiality determination.
- Clarify your risk management decision-making process. Determine your highest value assets and associated risk if an attack breaches them or shuts them down. Know which business processes are tied to them and how that could impact your ability to serve customers or continue revenue-generating operations. For more about risk management and incident response, listen to Virtual CISO Gideon Rasmusson on the 401 Access Denied podcast.
- Maintain up-to-date contact lists. Determine your primary method to communicate, and alternatives as your existing communication solution might be compromised.
- It’s helpful to have pre-prepared press statements that can be easily altered so you can discuss the incident with the public or media. Make sure appropriate people are trained to communicate confidently and transparently about the ongoing incident response efforts.
- Decide on what time format and naming conventions you’ll use for systems and evidence collection.
- Ensure that you have sanitized storage at hand. The space required can run into tens of terabytes or even petabytes. The last thing you want during an incident is to order a bunch of hard disks from Amazon to store your raw images or logs.
- Create incident response workflows for different types of incidents, based on how they apply to the CIA Triad:
Confidentiality – Did they extract data and you’re dealing with loss?
Integrity – Did they encrypt data or change it and you’re dealing with data poisoning?
Availability – Did they conduct a DDoS attack, and critical systems are down? - Make sure you know your in-house and third-party capabilities during an incident. Determine if you can get additional resources to assist during major incidents.
- Have a containment plan to prevent the attackers from moving around the network gaining access to more systems and data.
Reports and data for this stage of the incident response lifecycle
There are numerous resources and sources of information that you’ll rely on during incident response. For example, you’ll need resources such as:
- Asset inventory management
- Network inventory management
- Risk registers and assessments
- Penetration testing reports
- Malware analysis tools to determine malware indicators of compromise and capabilities
- Log analysis
- Audit reports
- Backup systems
- Recovery systems
- Communication and collaboration systems (email, Slack, Teams, wiki, etc.) Systems for documenting and storing evidence (central archive or SIEM)
Now, let's dig into how you use each of these in later stages of the incident response lifecycle.
Ensure access to critical systems
- Make sure you have authorization to access all essential incident response systems, even in case of emergency.
- Provide secure remote access to those systems for any remote workers or third parties that are going to be supporting you.
- Ensure you have access to all service accounts, and know what their dependencies are.
- Resilient secrets are particularly helpful in case of emergency. They enable you to replicate your secrets data to another instance, either on-prem or cloud, with automated syncing across instances, so you can still get into your PAM vault, even if your systems are under attack.
- Set up Incident Response accounts, so they’re already provisioned and can create a separate audit log for incident investigations. You won’t want your team using the same compromised AD account that has already been taken over to test and gather evidence. If they're using the same account to respond, you won’t be able to tell their activity from that of an attacker. Plus, they’re a good chance they’re tampering with evidence.
Identity Detection and Response (ITDR) solutions
For identity and privilege-related attacks, Identity Detection and Response (ITDR) solutions reduce risk with continuous monitoring of all identities, their access, and behaviors. They identify anomalous behavior, understand the most vulnerable identities, determine the potential impact if compromised, and take appropriate action.
Practice so you can be incident response-ready
Many people don’t go through the incident response lifecycle until they experience a live incident. This leads to lots of confusion and stress. That’s why it’s so important to practice as part of the Preparation stage.
When the incident response lifecycle is practiced and simulated, you will typically find things that didn’t get covered in your checklist. Make sure your team practices the incident response lifecycle a few times each year. Simulate different incident types so that when the real incident happens you can respond effectively.
Incident Response Lifecycle Stage 2: Discovery and Confirmation
At this stage of the incident response lifecycle, you find out an attack has occurred and must confirm before moving forward with your response.
Unfortunately, it’s likely that that you won’t discover an attack on your own but will be notified by an outsider. For example:
- Attackers may post on social media, bragging about how they’ve already stolen or encrypted your data.
- Law enforcement may contact you when they’ve detected something untoward.
- Customers or partners trying to access your website or services may find that it’s suddenly unavailable and contact your help desk.
You may find out when employees login to their systems. For example, they may be met with a prompt message that tells them your organization has been attacked:
|
In our transportation ransomware example: Attackers contacted the IT team by email to inform them about the attack. The main reason for contacting IT directly is to speed up the incident response process in an attempt to get the victim to pay the ransom quickly. Being contacted directly can be quite scary as attackers might know your employment history, including personal details or recent performance reviews. |
Red flags and Indicators of Compromise (IoCs) to look for
Ideally, your systems will be able to track signals that indicate an attack is likely underway. Then, you can send alerts automatically to your Security Operations Center (SOC), who can investigate further. For example, you’ll want to keep your eyes open for:
- New users or local accounts being created
- Privileged accounts being used on machines for the first time.
- Suspicious IP ranges atypical for employees’ day-to-day behavior
- Increased bandwidth usage
- Multiple logs in failures
- Systems or applications that can’t connect
Once you determine the threat is credible, you can evaluate its severity. Consider how the attack has impacted the CIA triad. Determine what type of data is impacted.
If the attack has impacted PII, protected customer or employee data, you may be dealing with a data disclosure situation.
You might find you’re dealing with multiple attackers specialized in different aspects of the ransomware attack.
Especially when you’re going back through multiple years of logs, you might find evidence of previous incidents you didn’t know about, unrelated to the active incident you’re currently investigating.
|
In our transportation ransomware example: The investigation found that an employee had installed crypto mining software on company resources and was making money on the side. |
Incident Response Lifecycle Stage 3: Containment and Continuity
The quicker you respond to an incident, the better you’ll be able to contain the damage. At this stage of the incident response lifecycle, you need to determine if attackers are still accessing your systems and determine what you’re dealing with. Then you can respond appropriately to contain the blast radius so you can continue to operate.
Digital forensics is like trying to complete a 1000-piece jigsaw puzzle when you only have 200 pieces.
Ransomware is a very effective tool for attackers to erase puzzle pieces. With the remaining pieces you must put together a full picture. You’re trying to rebuild the attack path and determine exactly what the attacker did and how.
The first rule is “Do No Harm”. You don’t want your responses to further infect systems. Do things with intention. Activities in this stage include:
Activating mitigating security controls
For example, in an identity-related attack, in which attackers are in control of privileged accounts or credentials, you can automatically trigger password rotation via your PAM vault. You can also add layers of identity assurance with MFA and approval requirements. Or you can remove privileges entirely so they can’t be used at all.
Determining extent of the damage
As you conduct digital forensics and root cause analysis, make sure you gather and store evidence.
Look for first evidence of attack:
- How did attackers slip through your controls to gain initial access?
- Which system was “patient zero”?
Find out what the attackers have access to:
- Did they gain access to a domain admin account? If so, it most likely means that your environment is owned (“pwned”).
- Did that result in all system access? All data in in your organization? All applications? On-prem or cloud apps as well? Check your asset inventory to see what systems are infected.
- Can they move laterally? Can you see if they elevated privileges? Check your network inventory, as communications can help you ascertain movements between systems.
Learn what tools and techniques they used:
- Did they use automation to scan your environment and look for sensitive data?
- Did they use staging machines (systems they used to move around or exfiltrate data)?
- Did they create backdoor accounts that they still have access to, allowing them to come back at any time? They might have changed registry keys that would allow them access simply from a command line on the log in prompt.
- Did they disable security? Often that’s the first thing that attackers do, so look for periods when security was disabled for a few minutes to home in on their movements.
Attack path mapping can help you understand the full identity attack path. Reviewing the MITRE ATT&CK Framework and the identity attack chain can help you walk through typical stages of an attack so you can see how attackers commonly enter and move around.
|
In our transportation ransomware example: Attackers had gained domain-level access. Luckily, this incident was limited to on-prem systems. Crucially, the company was still able to use email via Microsoft 0365 for communications. |
Evaluating what you’re dealing with
If your attack involved malware, you could conduct static analysis using tools like IDA Pro or ghidra.
Dynamic analysis is more powerful because it allows you to infect a sandboxed machine with the malware variant and see how it responds.
Piecing together a super timeline
You need visibility into how long the attack has been happening so you can understand and compare it with historical information. It’s common that attackers will delete evidence, remove log files and events, to cover their tracks. You will likely need to review log and event files from numerous systems and correlate them.
Plaso (Plaso Langar Að Safna Öllu), or super timeline all the things, is a Python-based engine used by several tools for automatic creation of timelines. It will allow you to create a history of logs and events across systems so you can put the puzzle together.
Performing detailed digital forensic examinations
The SIFT Workstation is a collection of free and open-source incident response and forensic tools designed to perform detailed digital forensic examinations in a variety of settings.
Reviewing session recordings
For identity or privilege-related attacks you can leverage session recordings and reports.
Delinea’s AI-Driven Auditing (AIDA) transforms privileged session monitoring by seamlessly analyzing recorded sessions to detect and highlight early indicators of potential cyber incidents automatically. This advanced feature identifies suspicious activity, such as authorization and privilege elevation failures and unexpected deletions and downloads. Sessions with anomalous activity are flagged in a dashboard, enabling administrators to prioritize critical events quickly.
With the ability to analyze a full 10-minute session recording in one minute, AIDA significantly reduces the time teams spend on manual reviews, allowing them to focus on critical anomalies. By summarizing activity and alerts, AIDA accelerates investigations, helping teams proactively mitigate risks before they escalate into serious threats.
|
In our transportation ransomware example: The attackers had about 15 days (about 2 weeks) of hands-on keyboard access to corporate systems before they deployed the ransomware. |
At this stage of the incident response lifecycle, you’ll need to decide your go-forward plan. You have several decisions to make:
To pay or not to pay?
Decide how you’ll make the decision. You may have legal or insurance requirements to consider. If you do decide to pay, you may need access to cryptocurrency.
Should you pull the plug?
To answer this question, you need to consider:
- Do the attackers still have access?
- Is sensitive data still at risk?
- What systems are still operational?
- Can you go back to manual processes?
- Should you shut down your systems?
- Should you revert to back up?
- Should you do nothing and rebuild?
|
In our transportation ransomware example: In this case, the internet plug was pulled to regain control of the network and make sure the attacker didn’t have persistent access. Some operations reverted to manual processes. This was crucial to stop the attacker from continuing to exfiltrate sensitive data. |
Incident Response Lifecycle Stage 4: Eradication
At this stage of the incident response lifecycle, the focus is on removing the threat and getting to “all clear” with all systems clean.
For ransomware, it’s rare, but possible that security researchers have encountered the type of threat you’re battling and made a decryption key available. More likely, you’ll need to use the information you gather to figure out the best strategy to eradicate the threat. Many of the systems noted below were used in the transportation ransomware example.
Systems used at this stage of the incident response lifecycle:
Joe Sandbox
Create a sandbox, a separate lab environment where you can test malware without doing any damage. Joe Sandbox is a helpful tool to look at other people’s analysis of the malware itself. You can upload a sample of the Cryptor from the infected system and hopefully find what types of capabilities it has and get some recommendations on things you can do to contain it. It looks at signatures, executables, classifications, a process tree of how it spawns, and much more.
Virus Total
You can upload the malware to Virus Total and find out if other tools out there are detecting it. If the malware you’re dealing with is new, it may not yet have been detected.
Flare from Mandiant (Now part of Google)
With this tool you can transfer Cryptor samples (make sure you use an encrypted, password-protected file) and run process hacker, capa, find hashes, and run against immunity debugger.
Threat intelligence
As part of the Eradication phase of the incident response lifecycle, you want to make sure to close the door so attackers can’t come back—at least not the same way. Look at the Dark Web to see if there’s any chatter. Check if anyone is selling data, credentials, or other sensitive information from your organization.
Incident Response Lifecycle Stage 5: Recovery
At this stage of the incident response lifecycle, the focus is on getting back to business as quickly as possible.
You need to evaluate:
- What systems are recoverable?
- What systems do you need to clean and start fresh?
- What is the point in time you should recover to?
Ideally, you can look at old systems that can help you piece data back together. However, it’s essential to find out if your recovery systems have been compromised as well. For example, if you’ve got backup systems in Active Directory, not segregated, using same credentials as other systems, they are also vulnerable to ransomware and may well have been infected.
|
In our transportation ransomware example: Unfortunately, their backup systems had also been encrypted by the ransomware attack. Luckily, they had a server that was migrated about a year before, and were able to use that as a baseline to restore the environment. It took two to three months to restore one year of lost data, at high cost, (though much less than the ransom demand) and tremendous team effort. |
Incident Response Lifecycle Stage 6: Lessons Learned
At this stage of the incident response lifecycle, you do a retrospective with the goal of continuous improvement.
Ask yourself:
- What security was in place when the attackers gained entry (virus checks, firewall, anti-malware, etc.) and why didn’t it stop the threat as expected?
- Why was the incident not contained? Was the problem a machine or an identity that didn’t have the proper controls? Was a privilege not vaulted? Was an identity misconfigured?
- Looking back, were there indicators of compromise in your environment that you should have recognized? Perhaps there was a signal or alarm, and you didn’t respond to it. There may have been communications on the Dark Web that you missed.
For a retrospective to be effective, make sure you document everything you do as part of the incident response so you can report on it later. Note when your responses occurred, so you can measure how long it took to act at each stage of the incident response lifecycle, from Discovery to Eradication.
Based on your lessons learned, you can refine the lifecycle to suit your needs, execute faster, and ensure seamless transitions from phase to phase for a coordinated response.
How do you avoid cyberattacks that require the incident response lifecycle?
An ounce of prevention is worth a pound of cure.
Do all you can to force attackers to do extra work and make noise, so they attract attention making it difficult for them to stay hidden. Deploy identity security controls and best practices, including:- Implement least privilege access and identity-centric zero trust policies that limit identity and privilege account sprawl. You want to avoid persistent privileges, especially for domain and system admins.
- Don’t share passwords or privileged accounts, and don’t store credentials for applications in code or repositories like GitHub. Instead, securely vault passwords and secrets to limit potential for compromise, ideally using an enterprise-scale Privileged Access Management solution.
- Remove local admin accounts from workstations so users don’t have broad privileges to download applications that could contain malware. Use access control instead, including allow and deny lists, and sandboxing so you can investigate any unknown applications before allowing them to be downloaded or executed by users.
- Implement multi-factor authentication at depth, including log-in and privileged elevation, especially on remote accounts.
- Avoid misconfigured identities, especially common in the cloud. A Cloud Identity and Entitlement Management (CIEM) solution can help you check that identities are set up correctly.
- Build awareness of social engineering tactics, so people can recognize potential threats and know how to report them.
How to test each phase of the incident response lifecycle
This is a lifecycle approach to incident response precisely because what you learn in stage 6 (Lessons Learned) should feed back into stage 1 (Preparation).
It’s essential to be incident response-ready. The last thing you want to be doing is testing your incident response lifecycle in the middle of an incident. You need to know what security processes and solutions you have, where they are, and how to use them.
Run practice drills for a variety of simulated scenarios. Do tabletop exercises to help you plan your response. Involve other teams in practice too, including your executives, legal, communication, compliance, and help desk.
Third-party companies specialize in gamification or simulations of incident response, including ransomware incidents. They can help you include things you might not have thought about in your practice scenarios.
How to ensure seamless transitions between incident response lifecycle phases
As mentioned, having clearly defined roles and responsibilities is essential. So is having a central repository for all your evidence gathering, and a Slack channel or Wiki for rapid communication.
The more context you can provide all parties involved in incident response, the better. For example, in an identity-related incident, you’ll want your detection engineers and SOC to know the history of an identity, expected access, and related and dependent systems for service accounts. An integrated identity security platform can help you make sure data is available, with context, for all parties involved in incident response.
How to avoid common mistakes in the incident response lifecycle
Here are some often overlooked pitfalls that are often overlooked and can create friction or roadblocks.
- Companies often operate across multiple time zones, which can be confusing for evidence gathering. Decide which one you’ll be using and be consistent.
- Create a naming convention when gathering evidence, so you know what machine it was gathered from.
- Don’t run out of disk space.
- Make sure you have alternative means of communication. VoIP phones can be impacted, so you may need to communicate with mobile phones or even radios.
- Create a “Go Bag” so you can response immediately and take care of yourself at the same time. Inside my “Go Bag” you’ll find:
Non perishable food or energy bars
Sanitized disks
Ear plugs for working in loud places
The aforementioned earmuffs
Expect that it will be a long, stressful day and week. Incident response can lead to extremely high stress and burnout. Please take care of your mental health and your teams during an incident response.
Delinea can help you make the incident response lifecycle actionable. Please reach out to talk with our team about how we can help.
