Case Study
2020 – 2021
calculated insights
Project Setup
Business Objectives

One of the fundamental business functions we want to enable for Salesforce CDP (Customer Data Platform) is to help our customers harness their unified customer data for intelligent insights and actions.

We bring this intelligence to Salesforce CDP by building purpose and persona-driven tools for data analysis and business intelligence, empowering customers with the right information at the right time to drive data-driven engagements, decisions, and automated flows. Marketers, sales and service professionals, or anyone in an organization can leverage prescriptive and predictive streaming insights across channels and touch-points to guide customer journeys or power business processes that help build richer relationships.

Role

Lead UX designer responsible for end-to-end user experience from ideation through development for multiple releases.

Primary Stakeholders

Product Managers, Developers, Architects, UX Researchers, Design Systems Engineers and Designers, Content Writers.

Project Background

Marketing, traditionally, has been seen as more of an art than a science. While creative ideas play a critical role in marketing and will continue to do so for the foreseeable future, data is becoming more crucial and even table-stakes to the modern marketer. That’s because data is helping marketers meet customers’ growing desire for more personalized engagements and experiences from the brands they interact with. These experiences can include a movie, product, or news story recommendation that seems exactly tailored to an individual consumer’s experiences or interests.

With Calculated Insights, Salesforce CDP customers can leverage their unified data (see Data Streams & Mapping case-study) to create business and individual-level metrics for enabling key use-cases. This system of insight can help marketers navigate thorny problems like calculating marketing ROI and the potential lifetime value (LTV) of a customer. They can even help uncover new buyer personas and customer affinities within their customer base (e.g., they have more foodies among their customers than in the general population). Gathering these types of insights that inform marketing campaigns will be critical in determining the success of marketing departments.

Value Proposition
Enhance Segmentation & Personalization
  • Use metrics to ease segment definition. *A market segment is a group of people who share one or more common characteristics, lumped together for marketing purposes.
  • Make complex segments easy by combining metrics. e.g., RFM values with engagement rates with CSAT scores.
  • Use metrics to activate personalization attributes. e.g., Send a marketing email to the customer which shows loyalty points and tier.
Enrich the Unified Customer Profile with Insights
  • Use metrics to understand the customer intent and behavior. e.g., lifetime spend, most viewed categories, CSAT, etc.
Deliver BI Insights for Big Data
  • Perform data exploration & visualization with metrics in BI tools like Tableau.
Enable better Artificial Intelligence & Predictions
  • Use metrics and trends to featurize customer behavior.
  • Simplify AI model building with metrics and improve the model accuracy.
Discovery

We kicked off the discovery process by identifying the target personas, jobs to be done, key concepts, and customer use-cases. At the point of project kickoff, the UX research team had come back with a validated list of jobs to be done by customers using CDP. These jobs and supporting research helped us identify the target personas and the expected roles in the calculated insights life-cycle.

Target Personas
1. Data Analyst

Create insights for their organization. Consume insights for visualization, reporting and business intelligence.

2. Marketing Specialist

Consume insights to enable business use-cases.

Key Concepts

In most systems today, Analysts calculate insights by writing SQL queries to define their metrics. An example SQL query used to calculate the sum of the total amount as "Total Spend" for every customer is shown below.

SELECT
   SUM( SALESORDER__dlm.grand_total_amount__c ) as customer_spend__c,
   Individual__dlm.Id__c as custid__c
FROM
   SALESORDER__dlm
JOIN
   Individual__dlm ON SALESORDER__dlm.partyid__c= Individual__dlm.Id__c
GROUP BY custid__c
  • Every report or insight is made up of dimensions and metrics.
  • Metrics are quantitative measurements. e.g., Sum (Total Amount)
  • Dimensions are what you’re measuring or what you’re measuring by. e.g., Customer ID, Month, Product Category

The concept of metrics and dimensions can also be seen in the charts and UI representations below.

Explorations

After cataloging our customer use-cases and business needs, we broke the requirements into two primary flows: Creation and Consumption.

I.

Consumption Flow

Type: Improvement to existing functionality

Primary persona: Marketing Specialist and Business User

Key use-case: Use a calculated insight (e.g. customer life-time value score) to create a segment of people with high LTV score. Activate this segment to send a personalized email.

One of the most significant value propositions of the Calculated Insights feature is to make the segmentation feature in Salesforce CDP better and easy to use. The segmentation feature is used by marketers to identify a group of people (customers) who share one or more common characteristics, lumped together for marketing campaigns.

With Calculated Insights, marketers can simply drag and drop metrics from the list of defined calculated insights onto the canvas. They can apply additional filters on these insights, and define AND/OR logic rules with other data attributes or insights to create an audience segment.

*Segmentation was an existing product functionality. My role was to work with the PM and UX lead on Segmentation to visualize how the metrics would show up in the left-side attribute library and the canvas interactions for filtering on drag-and-drop.

View Prototype ↗
II.

Creation Flow

Type: New Feature

Primary persona: Data Analyst

Key use-case: Define a new calculated insight (e.g. customer life-time value score) by writing a SQL query or using a graphical user interface tool by adding measures, dimensions, and filters.

Based on the identified user flow, we choose to explore two possible UI implementations 1. Writing SQL queries, 2. A graphical interface (No code UI).

1. UI to write and validate SQL Queries

Rationale: According to a Salesforce-wide analyst persona research study, 83% of analysts use code to do their job. The proposal here was to allow users to write SQL code assisted with a library of attributes and supported functions in CDP.

Pros:
  • A SQL editor can accomplish advanced use-cases such as advanced functions (ranking, time windowing, etc.), SQL sub-queries, and complex calculations as demanded by our customers.
  • Lower front-end development effort.
Risks:
  • It could be challenging for customers who don't have the people or resources with the required skillset.
  • SQL queries can get very complex based on the dataset.
  • Business users have to rely on people who can write code to achieve their use-cases. The barrier for entry to define metrics would be very high.
2. Drag & Drop / Visual Interface

Rationale: This concept was inspired by the existing Segmentation feature (see Consumption Flow) in Salesforce CDP. The idea behind this exploration was to re-use the existing drag and drop capabilities and visual styling from the segmentation tool to keep the engineering effort low.

Pros:
  • A familiar graphical UI that could be used by both analyst and business user personas.
  • Re-use of components could potentially reduce the development time.
Risks:
  • The proposal only addressed basic requirements and not all customer needs would be satisfied.
  • More discovery time was required to understand and design the advanced customer use-cases. There was a sense of uncertainty and risk with the future scaling of the UI.
  • The Segmentation tool had some major accessibility bugs and issues that would be inherited if the components were re-used.
Challenge

As we were in the middle of this big discovery project, the product team wanted to ship a UI solution within a single release to meet our demanding pilot customer needs. The ability to generate such insights on the unified data in Salesforce CDP was a key functionality that customers saw as "missing" in the current product capability. This lead to an agreement between all involved stakeholders where the product MVP would focus on writing SQL queries to help speed the feature development.

🎯

Deliver a quick MVP solution (achievable in a single release) for creating Calculated Insights in Salesforce CDP to address key pilot customer use-cases.

MVP Designs
Writing SQL Queries
View Prototype ↗

We shipped the product MVP with a user interface to write valid SQL Queries. We also ran user research sessions with pilot customers for the concept and roadmap validation of this implementation.

Pilot Customer Feedback
Pros:
  • Customers reported that the feature was very powerful and essential as a core CDP capability.
  • As anticipated, most customers felt that writing SQL queries provided them the ability to achieve complex use-cases.
  • Participants saw the experience as simple with the right supporting tools and validation checks in place.
Cons:
  • Many pilot customers asked for a no-code tool to create Insights.
  • Customers asked for extensive documentation and a library of examples, use-cases for reference, and templates.
  • For smaller customer organizations, writing SQL queries required consulting partner engagement which was expensive and time-consuming.

After wrapping up the feature MVP, we kicked off a new design sprint to explore the no-code option for creating calculated insights. It was evident that customers wanted a more user-friendly tool that could work for any Salesforce CDP user. The SQL query experience would continue to co-exist as a more powerful and advanced option for customers.

Requirements Research

We ran additional persona research to understand more about our user's mental models, needs, and expectations for a product UI when building statistical and predictive insights.

Findings:
1.

Defining and creating a calculated insight is an iterative process that is part planned and part organic. Users (Analysts in most cases) are often responding to the business requirements for different parts of their organization such as Sales, Marketing, Service, etc.

1.

Users want a tool where insights are: Easy to build, Easy to edit & maintain, Can scale for future iterations.

1.

Users need to visualize the different steps involved in defining an insight: Select objects, Join objects, Create filters, Group, Rank, Conditional joins, Define scores, and more.

💡

How might we create a graphical UI for defining insights in Salesforce CDP which is approachable for most users yet robust at its core to accomplish key business use-cases?

Designing the "Insight Builder"

Part of our new discovery effort was to identify the common steps between customer use-cases ranging from basic to advanced which needed to be accomplished with the insights builder tool. Once the steps were roughly cataloged, we spend time exploring low-fi UI options for the builder.

Explorations
1.

Step-based UI

Rationale: Our first concept was a step-based UI where the users are presented with a list of common steps. Users would be able to add more steps or re-order them if necessary. Every step/node would consist of a configuration UI with curated user assistance and guidance

Risks:
  • The number of steps can get long for complex use-cases. For one of the pilot customer scenarios, the total steps went up to as many as 20.
  • Steps weren't always linear. Some steps could be repetitive, and there could be multiple sub-steps or branches in a flow.
2.

Visual Flow UI

Rationale: The other approach that we explored was a flow-based user interface. The idea was to create a map or a chart with nodes representing the different steps. More steps could be added on the canvas while supporting branching and supporting scalability.

Risks:
  • Ease of use and branching nodes to be validated.
  • The UI could have a higher learning curve as users would need to know the logical order of steps as they map out their flows.
Leveraging the Salesforce Lightning Design System - "Builder" Framework

Salesforce hosts several content, logic, data, and code builders across its vast portfolio of products. The builder framework is a set of code components and visual design guidelines established as a part of the Salesforce Lightning Design System for designing tools that work with the WYSIWYG declarative elements that can be added with clicks and customized with forms.

We leveraged the SLDS Builder Guidelines to create a high-fidelity version of our concepts. We also drew inspiration from other builder tools in Salesforce like Engagement Studio, App Builder, Journey Builder, etc., to design the canvas interactions, including drag and drop, click-to-add, configuration, expand-collapse frames, and more.

Final Designs
Insight Builder
Research Feedback

We interviewed 7 participants responsible for creating insights, reports and dashboards. Primary user roles - Analyst and Data Systems Architects.

💬

"I really like the direction you are going on with this because every [UX improvement] shouldn't be only about [end user] or about dashboards. What you are trying to do here is improve experience of people responsible for the data." – Optum

Pros:
  • ~70% of participants reported that they would use the builder to create insights over writing SQL queries.
  • The visual flow with nodes grouped by colors and shapes was a strongly liked feature. Participants admired the map view stating that it would improve clarity.
  • Some users reported that the map view could be used as a collaboration space between data users and business users, further requesting additional collaboration tools like commenting and version history.
Cons:
  • Some users reported concerns over the feature parity between SQL expressions and the builder as a potential limiting factor for adoption.
  • Users expressed a strong desire for previewing and visualizing the data at every step (de-scoped from the release).
  • Participants want the ability to switch between SQL expression and the visual graph to improve confidence.
  • Some users reported that a library of templates would ease the onboarding time on this tool.
Development & Release

The Insight Builder expected to be launched in early 2022. Once the tool becomes public, one of the primary success metrics will be feature adoption and customer satisfaction. Some channels for measuring impact will include:

Other Case Studies:
Data Streams & Mapping↗