Adapted from a talk Ken Rudin gave at ThoughtSpot’s CxO event in June 2017.
As a Chief Data Officer or somebody running an analytics organization, what are some of the driving things you can do to really be a force in that business?
I have run the analytics organizations at some familiar places in Silicon Valley, and I’ve had the good fortune to see how data can be used in consumer companies, in B2B companies, and in many different industries. From this experience I’ve found 4 main things that will help any analytics team be a more effective leadership function in their company.
#1 What should you—and your team—be focused on as an analytics leader?
Historically, most of us have been told that our role is to make sure we’re using data to get answers to questions. But the reality is that we should be focusing much more on the questions than the answers.
When you look at what companies typically do and how they think about analytics, you’ll notice that most see it as a service function. These analytic orgs measure their own success by things like how many dashboards or reports they are running, the accuracy of those reports, and the turnaround time for a request.
But this structure is problematic. How do you know that you’re being asked the right questions in the first place? Are you getting the right value out of the data? And much more importantly, this approach positions your team as a service to the company and sets you up to be reactive. When you’re reactive it’s really hard to be seen as—and to have influence as—a leadership organization rather than as a service shop.
So while getting answers is a critical part of your role as an analytics leader, the real value comes from figuring out what the right questions are. I’ve always felt the difference between a good analytics org and a great analytics org is that good orgs get you the right answers, whereas great orgs—and great analysts—help you figure out the right questions.
And by ‘the right questions,’ I mean helping your company figure out metrics that matter and the things that drive those metrics. What is the data telling us about where there are opportunities for us to grow? This is a much more proactive way of looking at the business.
Everyone in a company has a unique perspective—the engineers, the product managers, the marketers, and the analysts. Product managers are helping drive the business by figuring out what products we need to build, what the markets need, and what we have to put in the products themselves. Engineering leadership is focused on how we can use technology or new processes to improve the product and add more value. Analytics leaders should bring the perspective of what the data is telling us about where we should be going with our business and products.
#2 What types of people do you need to hire?
If you’re building an organization that’s about finding the metrics that matter and how to go about moving them, then you’ll have to hire the right analysts. These people are a little different than the typical analysts that have been favored in the past decade or two. The key thing is that hiring people with multiple PhDs in statistics or in technical fields is not nearly as important as hiring people who have business savvy.
“You will find they will give you absolutely brilliant answers to questions that no one cares about.”
Yes, you need people with some technical and academic talent. But if you focus on people who have PhDs in statistics but not general business experience, you will find they will tend to give you absolutely brilliant answers to questions that no one cares about.
You’ve all probably had experience with that. They’re looking for a way to apply the latest and greatest algorithm to some business problem, when in my experience I think the most useful algorithm that I’ve ever seen in analytics is just simple division. For example, revenue per user is often a really valuable metric, and it’s calculated simply by dividing revenue by the number of users. Clicks per impression is also calculated just by dividing two numbers. It doesn’t matter how clever the analysis is—what matters is whether you figure out the right question to ask. Once you have the right question, the analysis to get the answer can often be pretty simple—but that doesn’t make it less valuable.
So how do you figure out if someone has business savvy? This isn’t something that you can always find by reading a resume. But there are questions you can ask in an interview that will shed some light on their business savvy. Instead of jumping right in and evaluating their technical analysis skills, I start analyst interviews with something different. I’ll ask them something like, ‘Here’s what’s going on in our business, we’ve got more people signing up for our product than we’ve had before, but the average length of sessions in our mobile app is decreasing overall. What questions would you ask to figure out what’s going on and what could be done to improve the situation?’
This approach lets me see what kinds of questions they would ask. You’ll find out quickly if they know enough about business in general to ask the right questions. And after that you can then see if your candidate also has the technical skills to back it up.
Even with the right people, if you want to be seen as more than just a service organization, a big part of that is ensuring that everyone in your company knows what your analytics group can do—so you need to help everyone understand what an analyst can and should do for them. Help them understand the tools that your team is using and the kinds of answers you can get from those tools. Also, help people understand how to phrase business questions in a way that can be answered with data. This will help everyone understand what they can and should expect from your team of analysts, and overall make your company more data-driven.
#3 How do you put your analysts into an organizational structure in which they can succeed?
Should you focus on a centralized organization of analysts or should your team be decentralized? The answer is both. Let me explain…
Traditionally the first type of structure you see is a centralized organization. There’s a Chief Data Officer, everyone rolls up into that person, and then there are various business units with business leads. This structure gives you a very well organized data environment. Because everything is in one place, there are standards and definitions that are aligned, a common set of tools, and a common set of processes.
However, the downside of this is that the team isn’t immersed within the business they work with. So it really does instill this notion of analytics being a separate service organization. Organizations set up like this tend to be more reactive than proactive.
On the other hand, in a decentralized organization the analysts are highly aligned with what their business units are focusing on and they share the same goals. But now you have all of these different organizations that are building their own definitions, platforms, and structures. As a result, the data tends to be very messy, and there’s a lot of chaos.
When you’re getting started down the path of becoming a leadership organization, what really works well is this concept of the embedded organization. It’s a hybrid of the centralized and decentralized structures. With an embedded structure, from an organizational structure perspective, the analysts are all in one team, but they are assigned to business units and sit with them. So you get the centralized benefit of having standards, shared processes, shared data warehouses, and so on. Also, because your analysts sit with the business units, they are there for all of the business conversations, they understand deeply what the team cares about, and they know what the business needs are. This allows them to have those proactive “Ah-Ha!” moments and proactively identify ways in which analytics can have a positive impact on the business.
So the embedded structure gives you the best of both worlds. The only downside is that as the CDO it’s kind of lonely because you’re the only one who is sitting where you sit.
From my experience, you need to get to the phase of having an embedded organizational structure if you want your analytics team to deliver the most value. Many companies stay with this structure forever. However, as some companies continue to grow, they’ll want to decentralize the analytics function. And it turns out that’s ok as long as you’ve gone through the phase of having an embedded structure. Decentralizing before you have standards, processes and infrastructure in place is dangerous, because people build their own. But decentralizing after those standards and processes are in place means you can continue to use those standards and processes after you decentralize, so it becomes much less likely that you’ll fall into a state of chaos that’s normally associated with decentralized analytics teams.
#4 How do you measure your success?
You need to focus on the impact your org has, not just the insights. If we look at the evolution of analytics over the past 15 years, it started with reporting. We look at what happened, and we build a report. But people felt that wasn’t enough—they wanted more from their data, so they focused on insights. They want to know more than what happened, they want to know why. In the past 10 years we decided that even insights weren’t enough; we needed actionable insights. Don’t just tell me why it happened, tell me what I should do about it. Today, most analytic organizations will tell their teams that they need to focus on actionable insights.
But I disagree. Coming up with an actionable insight that no one actually acts on is useless. We have to go further. As analytic leaders we have to hold ourselves accountable and focus on what we did to drive impact. Did we help make a positive change somewhere in our business?
If you’re wondering what I mean by impact, generally it’s something you can point to in one of these 3 buckets:
You helped move a metric. For example, your team identified issues with your product’s user registration flow, and suggested fixes which helped increase registration by 5%.
You helped change a product. For example, you identified a feature within your mobile app that is highly engaging but not a lot of people were using that feature. So, you worked with the product team to get them to change the UI to surface that feature more prominently, which improved engagement with your product overall.
You helped change a behavior or process. For example, your analytics team finds a better way to forecast revenue or user growth.
The main idea here is you need to own the outcome. Our job as data leaders is to drive change through data. If nothing changes, you have had no impact. That may be a hard message for people to hear, but it’s simply a fact. It doesn’t matter how beautiful the analysis was, if the business is the same as it was before, then it’s hard to convince anyone that your team had a positive impact.
“Our job as data leaders is to drive change through data. If nothing changes, you have had no impact.”
The push back I often hear is, ‘I don’t control the engineers, the marketers, the product designers, the product managers. You can’t hold me accountable for something actually changing. I can’t force them to make changes.’
But that’s like a sales person saying, ‘It’s not my fault I didn’t make my quota, I can’t force the companies to buy.’ Of course it’s your fault! You picked the wrong companies to talk to, or you didn’t understand what their issues were, or you didn’t successfully convince them that your product would meet their needs.
The same is true for analysts. If the other organizations didn’t implement your recommendations, then it’s likely that you were focusing on something that isn’t important to them right now, or you didn’t find the right way to convince them of the impact that implementing your suggestion would have. To minimize this happening, you need to understand their priorities and interests. (This is one of the reasons why being embedded is so important—it helps ensure that you deeply understand what’s important to the business unit.) And then focus on how you as an analyst can look at the data and figure out how to help this team achieve their goals.
Ultimately, you need to own the outcome—that is your responsibility. If you can focus on the questions more than focusing on the answers, hire analysts with business savvy, put them in the right organizational structure, and focus on impact and own the outcome, then you can have huge impact in your business. The analytic leaders who are doing this are really driving their businesses and are helping redefine what the role of an analyst is in companies overall.
