Flying Blind
HR and the lack of a dependent variable
A recent article in the Financial Times (paywall) reported former Google HR chief Laszlo Bock saying that HR needs to show the return on its investments in AI in terms of human performance, if people are to continue to trust the function. But this will be very hard for the profession to do, for the simple reason that HR has a measurement problem.
This problem is not new; it’s not getting any better in the age of big data and skills implementations and AI; and, at the same time, it’s not exactly what most people in the profession would say it is.
Here’s what I mean. Let’s take learning hours as an example. If you’re not familiar with the concept, learning hours are the total time that people in a given business unit have spent engaging with online courses or in-person training, according to a company’s tracking. They’re a measure of learning activity, if you like. And most HR people I know hate them, exactly because they’re a measure of activity and not a measure of some business outcome. We should stop measuring activities, they will tell you, and instead measure outcomes.
But this is only partly correct. The problem isn’t the measures per se—whether, that is, they are measuring the right thing. The problem is the connection of those measures to anything else that matters—to what is properly called a dependent variable. If I told you that a 15% increase in learning hours led to a 7% increase in revenue you’d be all over it, and measuring learning hours with gusto. What makes the difference is the addition of a dependent variable—a “leads-to” variable—that has some provable value to the enterprise. And HR doesn’t have one.
HR’s measurement problem is not that it lacks measures, it’s that it can’t join the dots, for the most part, from those measures to something an organization cares about. Because of its lack of a dependent variable, HR programs are often the first things cut when cuts need to be made, and the investment case for much of the function’s work relies on business leaders who believe that investing in people is the right thing to do and are willing to keep the money flowing even though the link to positive outcomes is as much faith as fact.
Now, to be fair, coming up with a dependent variable of this sort is hard work, and the leaders who are happy to invest anyway are exactly the sort of leaders we need in our organizations if we’re to make them more human, and the world is an infinitely better place for their faith in doing the right thing by their employees. But having a better yardstick of our collective progress would be a big help.
So, what then might a good dependent variable look like? It will have a few important characteristics (which, to be clear, are characteristics of any good variable, not just a dependent one). What we’re looking for, remember, is a link between two measures such that a pattern of changes in one of them has some identifiable connection to a pattern of changes in the other. One thing goes up, and later so does the other. One thing goes down, and at the same time the other thing goes up. You get the idea.
This means that our variables need to have range. Changes in one thing can’t be connected to changes in another if neither of them changes very much!
Next, our variables will need to be measured frequently, so that cause and effect are close together in time. If they’re not—if, say, we measure one thing in February and look for its impact in a different variable in October—it’s hard to filter out all the other stuff that has happened in the interim and isolate a relationship.
And then, our variables need reliability—which is a way of saying that they change when the thing they’re measuring changes, and they don’t when it doesn’t. A thermometer that wanders about by itself isn’t reliable; neither are many HR measures that involve people rating one another, because our ratings change based on who’s doing the rating, and not on the characteristic of the ratee we’re trying to quantify.
We can hold up various possible measures to these three tests—range, frequency, and reliability—to understand which might be good candidates as a dependent variable for HR.
A company’s annual engagement survey score typically doesn’t have much range—at one organization I looked at, a good year was a 74 and a bad year was a 71—but its real problem is that it’s a once-a-year event so it’s hard to connect to much of anything.
What are called, in the business, “smile sheets”—the evaluation forms you get at the end of a learning experience of some sort—are laughably poor as metrics, because the scores are almost always 4.3, 4.4, or 4.5 on a one-to-five scale, however good or bad the experience has actually been. So, no range.
Performance ratings on a one-to-five scale don’t pass our tests because they only have range when a company uses a forced distribution, requiring a certain proportion of the employee population to be assigned each rating, in which case they’re not a measure, they’re a policy. When you hear that a certain investment or innovation led to an increase in employee performance, you always need to ask how performance was measured, because if the answer is by using traditional performance ratings then it’s not a measure in any conventional sense of the word.
Attrition is promising—it’s reliable, as are any measures that simply count something; it can be measured continuously; and it has range in many companies. But sorting the people who left an organization into valuable employees, on the one hand, and employees who weren’t doing as well, on the other, brings us back to performance again and gets tricky fast.
When I led the people research team at Cisco, we came up with a pretty good dependent variable, and that was team engagement. We measured it quarterly for every team (and teams could measure it themselves whenever they found it useful) so we had plenty of continuous data. Our measure was reliable—there was no forced distribution of scores, and no rating of other people. It had plenty of range from the very best team down to the very worst. And when we started measuring it, we took the time to prove that changes in engagement as we were measuring it led to changes in business metrics such as sales goal attainment and service quality scores—so we knew it had explanatory power.
Because of these characteristics, we could use it to gauge, for example, the value of frequent check-in conversations, or the value of hybrid work, or the value of a career—and we did. We learned what made engagement go up, and what made it go down, and we were able to explain to the company why engagement was important—because our measure was part of a connected system. Our team engagement measure became the yardstick by which we measured many of our programs, and it was a huge help in figuring out what was working and what wasn’t.
There are other possibilities. As strange as it sounds, HR really doesn’t have an agreed measure of individual performance. Marcus Buckingham and I proposed one a decade ago consisting of four simple questions posed to someone’s team leader, and something like this—again, measured frequently—would be a really compelling dependent variable for HR.1 There might be other ways of measuring team contributions or team environment that would fit the bill. We might look at business-unit level measures. And there are plenty of very smart people working in people analytics at the moment who know how to do all this better than I do, and who could come up with countless other suggestions, I’m sure.
But we won’t make much headway until HR and business leaders ask for these sorts of measures, and ask for the evidence that connects them to leadership activity and HR investments, and reject the same-old same-old that they’ve been offered for so many years. Until that happens, when it comes to the people stuff, we’re flying blind.
In addition to writing about work, I advise businesses around the world on leadership and people. If you’d like to explore how I can help your organization, please check out my website here.
The best way to reliably measure something you can’t count is to ask people to rate their own future intent, and the best way we found to measure future intent with respect to performance was to ask this question of someone’s team leader: “Given what I know of this person’s performance, and if it were my money, I would award this person the highest possible compensation increase and bonus.” This, together with questions that gauged teaming, performance risk, and promotion readiness, formed our simple four-question measure.
The team engagment measure at Cisco sounds like a game changer. So many HR functions are stuck trying to justfy their existence with metrics that don't actually connect to anything meaningful. What you're describing about range, frequency, and reliability as core charactaristics makes so much sense. It's frustrating how many organizations keep relying on annual surveys that barely move the needle instead of finding something that actully shows impact.
Brilliant as always.