The Limits of Measurement

Posted on January 28, 2013 · Posted in Business

I had a conversation with a business partner the other day about data, and the importance of being data-driven.  There’s little question that data is important – as we’re fond of saying around the office, it’s critical to avoid “the land of the thinks.”  But to this person, data was everything – as he put it, if something couldn’t be predicted and measured, his business wouldn’t do it.

hammer

This strikes me as a fundamentally limited way to think about decision-making.  While having data is critical, it’s vital to keep in mind that data is a tool – and like all tools, it has its limitations.  How?  Here are the three primary problems I see with being too beholden to data:

  • The “round peg, square hole” problem.  One of the realities of business is that decisions must often be made in low-data environments.  You can test, gather intelligence, do research, but ultimately you may be guessing:  how a market will react, what actions regulators will take, how a counter-party will respond.  It can be comforting in such situations to rely on whatever information is at hand, and elevate its importance.  The problem is that what you may consider “data” may well be little more than “anecdote.”  Or worse yet, modeling based on all sorts of rickety assumptions – “garbage in, garbage out” elevated to the status of solid forecast via the numbing comfort of a financial model.  This can obviously lead to overconfidence and an unwillingness to adapt as reality starts to deviate from the model.  Far better, then, to identify such decisions as what they are: educated guesses.
  • The elevation of irrelevancy.  The benefit of having measurable goals is obvious:  by focusing on those items, and holding people accountable to those goals, you greatly increase the likelihood that they will be met.  Which is great, as long as those are the only goals that matter.  The problem is that there may be other things that matter more.  These may be new opportunities or risks that pop up, or simply more amorphous things that are enormously important but which are hard to measure or set goals for.  The more you get down in the weeds with measurement and goals, the likelier you are to drive grocery clerk behavior: ticking off boxes on the to-do list rather than asking, every day, “what is the best thing I can be doing for the business?”
  • Paralysis.  After nearly 20 years in business, I’m convinced that the ability to embrace ambiguity and make confident decisions in the absence of complete information is the single biggest driver of success.  Yet “analysis paralysis” runs rampant throughout the workplace, as people are deathly afraid of making a decision without having as much data as possible (often defined as “more”).  The problem, of course, is that there is a cost to not acting, of not making a decision.  And you often can’t wait for more data – either because it’s simply not going to be available, or because by waiting on it the window of opportunity will close.  A business that worships at the altar of data is going to find itself constantly at risk of letting hand-wringing and delay keep it from pushing forward.

Ultimately, data can be so comforting, but too often what it provides is a false sense of security: the idea that because we are measuring something, and because we are making progress against that measured goal, we are succeeding.  But businesses – especially those that are new or growing rapidly – operate in an environment of ambiguity and constant change.  Our checklists, goals and to-dos can only offer so much help on that front.