How To Speed Up Insider Threat Investigations with Splunk

As insider threats become more common, cybersecurity experts need tools that make detecting and preventing them easier.

One powerful tool is Splunk, a leading platform for security information and event management (SIEM). In this guide, we’ll explore how to use Splunk to improve your organization’s security against insider threats.

How To Investigate Insider Threats with Splunk

Create Investigation Playbooks

Your security team can dramatically reduce investigation time and ensure that no critical steps are missed by following standardized procedures. You can create playbooks within Splunk to outline step-by-step procedures for investigating different types of insider threats. 

Such steps might include:

1. Initial triage and assessment of an alert
2. Collection and correlation of data across relevant sources
3. Timeline reconstruction of suspicious activities
4. Risk assessment and impact analysis
5. Escalation procedures for confirmed threats

You can run these playbooks using the Incident Review page on the Splunk Enterprise Security app.

Automate Evidence Collection and Analysis

Manual evidence collection and analysis can be time-consuming and error-prone. You can overcome these challenges by using Splunk to automate the process:

1. Use Splunk’s search commands to automatically gather relevant logs and data when an alert is triggered.
2. Develop custom scripts or make use of existing apps to automate the correlation of data from multiple sources.
3. Generate a timeline to quickly visualize the sequence of events relating to a potential insider threat.
4. Use Splunk’s machine learning capabilities to automatically classify and prioritize potential threats based on severity and confidence levels.

Here are some examples of custom searches you can implement:

Detecting unusual file access patterns:

index=file_access user=* | stats count by user, file | where count > 100 | sort – count

This search identifies users who have accessed an unusually high number of files, which could indicate data exfiltration attempts.

Identifying off-hours system access:

index=windows sourcetype=WinEventLog:Security EventCode=4624 | eval hour=strftime(_time, “%H”) | where hour < 6 OR hour > 20 | stats count by user, hour | sort – count

This query helps detect login activities outside of normal business hours, which may be a sign of unauthorized access.

Monitoring privileged account usage:

index=* (sourcetype=linux_secure OR sourcetype=WinEventLog:Security) user IN (“root”, “admin”, “administrator”) | stats count by user, host | sort – count

This search tracks the usage of privileged accounts across your infrastructure, helping to identify potential abuse of administrative privileges.

Integrate with External Threat Intelligence

Applying intelligence about external threats can provide valuable context for your insider threat investigations and help identify potential connections to known threat actors. To integrate external threat intelligence into your Splunk-based insider threat investigations:

1. Configure Splunk to ingest threat intelligence feeds from reputable sources.
2. Use Splunk’s lookup tables to enrich your internal data with external threat intelligence.
3. Develop custom correlation rules that combine internal behavioral anomalies with external threat indicators.
4. Implement automated workflows that cross-reference threat intelligence databases when investigating suspicious activities.

This integration allows your team to quickly determine whether an insider threat may be part of a larger coordinated attack or if external actors have compromised your internal accounts.

Improving Collaboration and Reporting

Implementing Cross-Team Workflows

Effective investigation of insider threats requires the collaboration of key team leaders across your company, including cybersecurity, HR, legal, and IT. It’s important to establish this collaboration early in the development of your plan and to keep the circle small. Insider investigations are highly sensitive, and the insider threat may be accidental. Keeping the circle small helps prevent internal leaks of information about the investigation, ensuring your employees are given the benefit of the doubt and that a thorough investigation is made before conclusions are drawn.

Splunk can facilitate this collaboration by:

1. Creating role-based dashboards that provide relevant information to each team involved in the investigation.
2. Implementing a centralized case management system to track the progress of investigations and assign tasks to different team members.
3. Offering collaboration features, such as annotations and comments, to share insights and findings securely within the platform.
4. Developing custom apps or integrations to allow communication and data sharing with other tools used by your team. Add-ons for this are available in the Splunkbase app store.

One useful add-on is a UEBA tool, which can improve collaboration by providing key contextual data for insider incidents, including:

Baseline behavioral data to help you see when and how employees associated with a potential incident typically work. Employee engagement data to provide an indication of disengagement, an early red flag. Instances of employee disgruntlement, an indicator of increased risk of insider threat. Risk scoring of insider threats, which can help aggregate risk factors and speed up the time it takes to analyze them.

Time-stamped screen recordings, which allow your security analysts to skip right to the time and user associated with an incident. Analysts can view exactly what a user was doing in the time leading up to and immediately following an incident. Individual, role-specific user behavior reporting for HR and legal. Irrefutable evidence that will be essential to the steps legal and HR may need to take with individual employees in very serious situations – or used to help defend an employee’s actions as unintentional .

Generating Comprehensive Investigation Reports

Detailed and accurate reporting is crucial for documenting your insider threat investigations and supporting any potential legal or disciplinary actions. You can make use of Splunk’s reporting capabilities to:

1. Create customizable report templates that capture all the relevant details of an investigation, including timelines, evidence collected, and analysis.
2. Implement automated report generation triggered by specific events or milestones in the investigation process.
3. Develop interactive reports that allow stakeholders to drill down into specific details while maintaining a high-level overview.
4. Create clear visual representations of complex insider threat scenarios.

Continuous Improvement Through Metrics and Feedback

To continue to improve your insider threat investigation process, it’s essential to track key metrics and incorporate feedback from all stakeholders. You can use Splunk to:

1. Monitor and analyze investigation-related metrics such as mean time to detect (MTTD), mean time to respond (MTTR), and false positive rates.
2. Implement feedback loops that allow your investigators to rate the accuracy and usefulness of alerts and automated analysis.
3. Conduct regular reviews of closed cases to identify patterns and areas for improvement in your detection and investigation processes.
4. Develop dashboards that provide a real-time view of the performance of your insider threat program, allowing for data-driven decision-making and allocation of resources.

How to Set Up Splunk for Insider Threat Detection

Identify and Ingest Relevant Data Sources

The first step you need to take to make the best use of Splunk for insider threat detection is to identify the right data sources for it to ingest. These typically include:

1. Authentication logs: Capturing login attempts, successful logins, and password changes across all systems and applications can help identify suspicious access patterns and compromised credentials.
2. File and database access logs: Monitoring who is accessing sensitive files or databases, and when, can reveal potential data exfiltration attempts and unauthorized access to confidential information.
3. Network traffic logs: Analyzing network traffic can help detect unusual data transfers, connections to suspicious IP addresses, and the use of unauthorized communication channels.
4. Email and messaging logs: Monitoring email and messaging platforms can reveal attempts to send sensitive data outside the organization and communications with known threat actors.
5. Endpoint activity logs: Collecting data about user activities on workstations and servers can provide valuable insights into potentially malicious behavior, such as the installation of unauthorized software or attempts to disable security controls.

To ingest these data sources into Splunk, you’ll need to configure the appropriate data inputs. This may involve setting up Splunk forwarders on relevant systems, configuring API integrations with cloud services via the Splunkbase app store, or using Splunk’s built-in data collection capabilities.

Including UEBA data in addition to those listed above will allow you to make better use of your existing teams by speeding up investigations by up to 65%, reducing dwell time, reducing the need for additional staff, and helping you strengthen your response to incidents.

Create Custom Dashboards for Insider Threat Monitoring

Once you have the relevant data ingested into Splunk, the next step is to create custom dashboards that provide an at-a-glance view of potential insider threats. Some key elements to include in your insider threat dashboard are:

1. User activity timeline: A visual representation of user activities over time, highlighting any anomalies or spikes in activity.
2. Data access patterns: Charts showing who is accessing sensitive data, how often, and from where.
3. Authentication anomalies: Visualizations of failed login attempts, successful logins from unusual locations, or out-of-hours access.
4. File transfer metrics: Graphs depicting the volume and frequency of file transfers, especially to external destinations.
5. Top risky users: A list of users exhibiting the most suspicious behavior, based on predefined risk factors. (This element will benefit from the use of UEBA tools.)

To create your dashboards, you’ll need to use Splunk’s Search Processing Language (SPL) to create searches to extract and analyze the relevant data.

Configure Alerts and Notifications

Effective insider threat detection requires timely notifications when suspicious activities occur. Configure Splunk alerts to notify your security team when potential insider threats are detected. Some examples of alerts you might want to set up include:

1. Multiple failed login attempts from a single user or IP address.
2. Abnormal data access patterns, such as a user accessing an unusually high volume of sensitive files. (Optical character recognition (OCR), keystroke, and clipboard monitoring data is very useful here to ensure sensitive data access anomalies don’t slip through the cracks if copied and pasted to alternative, less sensitive locations and files.)
3. Large data transfers to external IP addresses or cloud storage services.
4. Out-of-hours system access by privileged users. (Accurate geolocation tracking will be essential here, preventing users from circumventing location-tracking mechanisms with VPNs.)
5. Installation of unauthorized software on critical systems.
6. Anomalous data access or handling practices. (UEBA data will need to be collected for this type of alert.)
7. Unusual use of data storage or printer devices in conjunction with sensitive data.
8. Unusually long periods of time spent viewing sensitive data. (The OCR capabilities available with UEBA tools are essential for this.)

When configuring your alerts, consider using Splunk’s adaptive thresholding capabilities to reduce false positives and ensure that alerts are triggered only for genuinely suspicious activities.

Teramind: Enhancing Insider Threat Investigations

While Splunk provides powerful data analytics for your insider threat investigations, you can further improve your organization’s ability to detect and respond to insider threats by combining Splunk with Teramind’s comprehensive behavior analysis. Teramind offers a detailed view of user actions, providing the context that—as we have seen—can be crucial in determining the intent behind the potentially suspicious activities flagged by Splunk.

Teramind’s ability to capture on-screen content, keystrokes, and file transfers can provide valuable evidence during investigations, allowing your security team to quickly verify whether an alert generated by Splunk represents a genuine insider threat or a false positive. This integration can significantly reduce investigation times and improve the accuracy of threat assessments.

Teramind’s behavioral monitoring capabilities can significantly enhance Splunk’s insider threat detection capabilities. By integrating Teramind with Splunk, organizations can gain deeper visibility into user behaviors and actions, providing valuable context for potential insider threat indicators.

To integrate Teramind with Splunk:

1. Configure Teramind data ingestion: Set up Splunk to ingest user activity data from Teramind, including screen recordings, keystrokes, file transfers, and application usage.
2. Develop correlation rules: Create Splunk searches that correlate Teramind user activity data with other security logs to identify potential insider threats. For example, correlating unusual file access patterns detected by Splunk with screen recordings from Teramind showing the actual user actions.
3. Enhance user behavior profiling: Use Teramind’s detailed user activity data to create more accurate and granular user behavior profiles within Splunk, improving anomaly detection capabilities.
4. Create integrated dashboards: Develop Splunk dashboards that combine Teramind user activity data with other security metrics for a holistic view of insider threat indicators.
5. Implement automated workflows: Leverage Splunk’s automation capabilities in conjunction with Teramind’s real-time monitoring to trigger immediate responses to high-risk user activities.

Leveraging Teramind’s Data Loss Prevention Capabilities

Teramind’s Data Loss Prevention (DLP) features complement Splunk’s insider threat detection capabilities by providing granular control and visibility over sensitive data handling.

To effectively leverage Teramind’s DLP capabilities in conjunction with Splunk:

1. Define sensitive data patterns: Use Teramind to define and detect sensitive data patterns, such as credit card numbers, social security numbers, or proprietary information.
2. Monitor data movement: Configure Teramind to track the movement of sensitive data across various channels, including email, file transfers, and clipboard actions.
3. Integrate DLP alerts: Set up Splunk to ingest and correlate Teramind’s DLP alerts with other security events for comprehensive insider threat detection.
4. Implement content-aware policies: Utilize Teramind’s content-aware policy engine to create fine-grained rules for handling sensitive data, and monitor policy violations through Splunk.
5. Conduct forensic analysis: Use Teramind’s detailed activity logs and screen recordings in conjunction with Splunk’s analytics capabilities to perform in-depth forensic analysis of potential insider threat incidents.

Enhancing User and Entity Behavior Analytics with Teramind

Teramind’s detailed user activity monitoring can significantly enhance Splunk’s UEBA capabilities, providing a more comprehensive and accurate picture of user behaviors.

To enhance UEBA with Teramind:

1. Expand behavior baselines: Incorporate Teramind’s granular user activity data into Splunk’s UEBA baselines for more accurate anomaly detection.
2. Improve risk scoring: Use Teramind’s insights into user productivity, application usage, and data handling patterns to refine Splunk’s risk scoring models for potential insider threats.
3. Detect subtle behavior changes: Leverage Teramind’s continuous monitoring to identify gradual changes in user behavior that might indicate insider threat activities, even if they don’t trigger immediate alerts.
4. Enhance context for alerts: When Splunk’s UEBA detects a potential insider threat, use Teramind’s detailed activity logs and screen recordings to provide immediate context and visual evidence for faster investigation and response.
5. Implement adaptive policies: Use the combined insights from Splunk and Teramind to create adaptive security policies that automatically adjust based on user behavior patterns and risk levels.

By integrating Teramind with Splunk, organizations can create a powerful, multi-layered approach to insider threat detection and prevention, combining the strengths of both platforms for comprehensive security coverage.