MY WORK

Design / Vision / Framework

Design / Vision / Framework

Design / Vision / Framework

Unlocking Security Insights from a Specialized Data Lake

Unlocking Security Insights from a Specialized Data Lake

Overview

Overview

Overview

Integrate Splunk Federation with a specialized data lake - Amazon Security Lake to access and analyze data for security issues and insights.

My role

Leading the design effort from MVP to iterations, fostering successful rollouts. Helping with roadmap creation by promoting a strategic focus.

Impact

Being highlighted at the product keynote at Splunk's annual conference, receiving high customer interest and strengthening Splunk's leadership in security analytics.

The problem
Why users need to access a specialized data lake?

Data silos and normalization complexities lead to limited data visibility.

Data silos and normalization complexities lead to limited data visibility.

Inefficient threat detection and response delays security data value realization.

Inefficient threat detection and response delays security data value realization.

High costs & low flexibility in data storage and transfer, with limited usage options beyond Splunk.

High costs & low flexibility in data storage and transfer, with limited usage options beyond Splunk.

Lack of control over what data is sent for analytics leads to unnecessary ingestion costs.

Lack of control over what data is sent for analytics leads to unnecessary ingestion costs.

The solution
How Splunk Federation helps unlock the security insights?

Unified Visibility

Unified Visibility

Unified Visibility

Seamless integration with Amazon Security Lake eliminates data silos, enabling direct access to normalized, OCSF-formatted data.

Faster Threat Response

Faster Threat Response

Faster Threat Response

Leverage Splunk's advanced analytics and detection directly within the data lake to streamline threat detection and response.

Cost Efficiency

Cost Efficiency

Cost Efficiency

Optimize costs by analyzing data within the data lake, reducing data transfer costs, and filtering data to ingest only what’s necessary.

Flexible Data Access

Flexible Data Access

Flexible Data Access

Improve accessibility for various use cases, from real-time threat detection and analytics to long-term investigation and machine learning initiatives.

Phase 1: Getting things up and running

Focusing on the basic workflow - onboarding, phase 1 aims to help users onboard their Amazon Security Lake data to use the federated analytics and security detection features. Xinyue is the design lead and also the mentor of a junior designer on conducting his career’s first user research.

The basic workflow - onboarding

The basic workflow - onboarding

The basic workflow - onboarding

Onboarding the Amazon Security Lake data to Splunk and then configure for federated analytics and security detections.

Connect to the data lake

Connect customer’s Amazon Security Lake service account to Splunk Federation.

Configure data access

Configure the data access

Set up indexes for remote data access, analytics and detection purposes.

Set up 2 types of indexes for data ingest and search access


Complete the onboarding

Add the provider and indexes to start streaming data for real time threat detection and analytics.

Persona?User story?

Description

MVP design

Featuring the Amazon Security Lake configuration experience to empower data analytics for users' security use cases.

Seamless integration

Add the Amazon Security Lake as a type of provider to access the customer's the centralized security data in their AWS account.

Wizard design with modern look and feel

The onboarding workflow includes setups on both AWS and Splunk sides, which can be overwhelming. The wizard design
breaks down the entire flow to reduce cognitive load and make it scalable for future iteration.

Guided connection flow

Connect to the data lake with step-by-step guidance. The guided flow helps users understand what and why they are doing it when setting up on both AWS and Splunk sides.

Configure with pre-populated values

Set up indexes for different data access with pre-populate values and mappings to minimize manual effort.
- Add data lake indexes for analytics and detection.
- Add federated indexes for remote data access.

Review before completion

Review the configuration in a one-page summary for scrutiny before saving the changes. Hand off the flow to next steps to create a more connected experience.

The impact
What it does to the business and the users?

The Amazon Security Lake integration makes Splunk more competitive in the market by improving its weakness on searching scan based data and combining with its advantages on data indexing. The product was released at .conf24 and highlighted at the product keynote, 11 customers have signed up for the Private Preview after .conf24, putting the product team on top of Platform and P&T OKR.

Phase 2: Aiming for the bigger picture

The federation experience is not only about onboarding, but also about enabling detections and monitoring the data usage. To create a streamlined experience, the key is to connect with other use cases. In phase 2, Xinyue explores the E2E experience of federation with a holistic approach to support product scoping and get leadership buy-in.

First, let's get feedback from the users

The user research is led by a junior designer under Xinyue's mentoring.

An iterative process

Design unification

Users perceive the configuration as an iterative process rather than a one-time event. This process often requires several iterations to ensure accuracy and efficiency and can include back and forth communications.

Unifying the design patterns with the data management experience is an essential part to get ready for future integration.


Intuitive experience with areas to improve

Design unification

Overall, the onboarding flow is straightforward and easy to navigate. But non-expert users find it difficult to understand the difference between federated index vs. data lake index.

Unifying the design patterns with the data management experience is an essential part to get ready for future integration.


License prediction

Additionally, users have expressed their need to estimate the costs before completing the configuration process, this is particularly important for smaller teams that need to operate within tight budget constraints.

Second, let's look at the connected experience

Second, let's look at the connected experience

Second, let's look at the connected experience

After onboarding, users must enable the detection feature on Splunk Enterprise Security for threat detection use cases. They also need to monitor license usage to stay within limits, thereby optimizing cost savings. While other teams design the connected experience, Xinyue takes the initiative to link all the pieces together and proposes a strategic solution.

Enable threat detection

Configure additional settings to enable the detection feature with pre-built searches.

Validate threat detection

To check or test if the searches are ingesting and scanning data correctly.

Set up 2 types of indexes for data ingest and search access

Monitor license usage

Monitor license usage

Get an overview of the license usage.


Investigate usage peak

Investigate usage peak

Learn about the usage overtime and drill down the top consumer.


Enable and validate threat detections

Text

Monitor and investigate the license usage

Text

Design to connect the jobs
What are users trying to accomplish?

Capturing the E2E experience of data federation with 'Jobs to Be Done' to support roadmap planning with a systematic focus on the entire federated experience, focusing on what users are trying to accomplish instead of the product features.

Initial learning

Initial learning

Building cost guardrail is the key.

Supporting economic buyers to make buying decisions.


Building cost guardrail is the key.

Supporting economic buyers to make buying decisions.



Connect and ingest data

Facilitating the setup process.

Supporting admins on optimizing their data configuration in an iterative manner.

Enable and validate search

To make sure data scan and coming in correctly after configuration.


To make sure data scan and coming in correctly after configuration.





Monitor license usage

Connecting the monitoring to the FA use cases to help economic buyers optimize the purchase plan.

Come up with the design scoping

Enhanced design

Initial learning

Enhanced connect & ingest flow with: clear explanations on the different index setup; better onboarding and hand-off page. 

Include the edit flow to support users on optimizing the configuration.

Building cost guardrail is the key.

Supporting economic buyers to make buying decisions.

Research strategy

A validated JTBD for federation to envision the future experience.

License prediction capability.

Testing capability on providers and indexes.

Validation on the searching results.

To make sure data scan and coming in correctly after configuration.


Design vision

Modernize, simplify, and unify the entire federated experience.

Refactor the architecture.

Parquet indexing (Phase 2) for optimization and improve the single source of true.


The enhanced design

A more user friendly version that improves the UX based on prior research findings and expands the scope beyond just the configuration flow to create a streamlined experience.

Editing flow

Editing flow

Editing flow

Add breakpoints to your blank page, then drop sections to have them responsive out of the box.

Visual aid

Double click the image placeholders to add images. Do the same for any text, then tweak styles and publish.

Better onboarding

Add breakpoints to your blank page, then drop sections to have them responsive out of the box.

Better hand-off

Better hand-off

Better hand-off

Double click the image placeholders to add images. Do the same for any text, then tweak styles and publish.

User feedback

???

What's next

???

Designed by Xinyue

2025