Accelerating Value
Accelerating Value

Episode · 8 months ago

Polishing the Crystal Ball: How to Improve Your Forecasting


How well can you see the future? 

With the tools available today, forecasting has become easier and more precise than ever — but only if you avoid the common pitfalls many organizations fall into. 

To help you steer clear of these potential traps, I invited Bill Schmarzo, Customer Advocate, Data Management Incubation at Dell Technologies, onto the show to share the secrets to forecasting — and I predict it’s going to be a great episode.

Join us as we discuss:

Why humans are natural forecasters

The importance of agility and pushing decision making down in your organization

Why data and technology should supplement human decision making, not replace it

Keep connected with Accelerating Value on Apple Podcasts or Spotify

Listening on a desktop & can’t see the links? Just search for Accelerating Value in your favorite podcast player. 

Today, every budget approval is an investment deal. If you're a marketer, sales or business leader, you had to promise to deliver value and impact. Writing the wave to get there is hard enough. Finding your way through the storm is even harder if you're looking for that path forward so that you don't wipe out. You've come to the right place. Let's get into the show and we're back. Hi Everybody, this is marks Dou's, your host for the weekly addition of accelerating value and, as it happens, this is also our last episode of Two Thousand and twenty one, which I think is is one of those years where if we had had a really accurate prediction of what this year was going to be like back in January, maybe a lot of us would have decided to stay home. So today we have bill Schmarzo back. So bill is the is the dean a big data he is also a really big dog in the space. At Dell. He talks with customers about how to help them them make better decisions today using analytics than they are without analytics or with incomplete analytics. He's very into hey, you know, all the mechanisms really work. I mean that that's important, okay, but the most important thing is the end game, and that is, are you helping people make better decisions and achieve the value that they need to achieve? And if you're not, the whole rest of it kind of loses its raison. Debt right. So, Bill, welcome. It's always a pleasure to add you here. Man. Thanks, Mark. It's a great way to end a very bizarre year, very very bizarre. I in fact, if you had to kind of, I don't know, come down on one side of the other, which would you say was more bizarre or difficult to navigate? Two Thousand and twenty or two thousand and twenty one? Oh Wow, great question. I actually think two thousand and twenty one was more difficult because we have solutions at hand that just have been confused, dirtied with such biased, non factual opinions. and to me two thousand and twenty one is a lot more frustrating because we're we're fighting science, Yep, and so as a culture, you know, I always think of that. Our Culture in America is one that's being very progressive about how we look at bringing in data and new discoveries and use that to improve life. And so two thousand and twenty was frustrating because he just didn't know and we had no data. And so as the year went we captured more data what we thought was a right response to not being not and made to do this and do that. But Two thousand and twenty one we have the solution in hand. But yet it really points to this challenge mark of how do people make informed decisions in a world of incomplete and sometimes even, you know, polluted data, false data? It's really, really hard. So Anyway, two thousand and twenty one for me was by far, by far more frustrating, because the solution was at hand and we just sought to ignore it. It's right. There's a great book by the way. I don't know whether you've read it yet or not, but it's called super forecasting and it gets to the kind of the heart of why forecasting is so important, what is so difficult about it? Why do so many people get it wrong and yet why do a few people have just an uncanny ability to do it well right? So, anyway, well, we'll talk about the good. It's explore. That sounds fun now, no, it's it's actually I started reading it yesterday and I tend to kind of paste myself with a book like that. You know, I'll read it for like an hour and then I'll put it down and I'll go do something else, and I don't do that with a good novel, right, I'll stick with the novel for like several hours, and I went like three hours with this book. So it's very well written, pretty if you're a lay person, it's it speaks to you right, and you won't be sitting there going look, I can barely understand right. And also I think it walks a really fine line here, because if you are bill Schmarz, oh right, you're not going to sit there and say, okay, this is like so overly simplified it's ridiculous. Right. So it really kind of hits that little ground pretty well. I was like a good book to read over the next week. Yeah, I am going back to rereading the undoing project. I...

...thought that was such a great, great book and I want to hit it again with, you know, more fresh eyes and spend more time really studying and taking it apart. Do you do? You make a lot of notes yourself in the margins or yeah, which is why I really do preferred hard copy books. Maybe I know on books that you can you can highlight, but I write might know they're all thumbnailed and they've got all kinds of comments and I'm you can kind of see that. You know they've got there's things inside the books to tell me where I want to go. And so, you know what's a good book? You don't read a good book is study, and I always trying to find out, you know, what are the when I read it book, what are the two or three nuggets that I can pull out that I can apply to me? And so I've always in search of those nuggets to help make myself better. So that's a great way of putting it right, really true. Okay, so we are going to talk about today something that is, you know, obviously very top of mind, particularly in the last say two years and moving into two thousand and twenty two, which I think a lot of people are also very anxious about. There's a lot of you know, I think the speed and volatility of change has definitely made people think about predictive or think about prediction in a very conscious way, that maybe they kind of thought it was more nice to have in the past and that, you know, past was always going to be sort of prolog right, and and all this kind of stuff. And then two thousand and twenty and two thousand and twenty one came along and basically said, guess what, if you think that past is prolog I have a ive, some ocean front property in Arizona that you need to look at. So prediction is also fraught, right. It's fraught with all kinds of stuff, right. We think about it in terms of magic, we think about it in terms of maybe religious prophecy, we think about all kinds of stuff like that, and it makes us even more anxious and we're some people get anxious about the fact that are we even supposed to be trying to be predicted? Right, that's sort of a metaphysical idea. It really really is important to say when we talk about being predictive in this context, we are not talking about any of that stuff. We are talking about what we can learn from data, not only in its raw form or, you know what, nonanalyzed form, or right, more specifically, what are we going to learn when we start to analyze this, particularly in the form of different kinds of aggression analytics, and then also machine learning for pattern matching and things like that right. What can we say? Well, okay, you know what, given what we know about historical patterns and all this kind of stuff, how do we use predictive, the whole idea of being predictive today? So that is what bill and I are really going to talk about now and we're going to we're going to try and talk about this in a really credible way, in a way that is understandable. And this is sort of a part of the nut of the problem when I talk to business leaders, and I'm pretty sure that this is something that bill and counters as well. If you simplify it too much and you make it to business the right then then you're in danger sometimes of leaving behind some really important facts about all this, and if you get overly technical, you lose people really fast. Right. This is this is very much about kind of creating a bridge of understanding. So Bill, you ready, you know, yeah, I've got all kinds of thoughts. All Right, okay, so question number one. What does it mean to be predictive today, particularly given the fact that data, all data, by the way, this is not a negative statement, but it's a statement of fact. All data is about the past. Yeah, so let's let's set the frame here. First off, predictions are what humans do all the time. Let's be really honest. It's a human it's a natural human tendency. You know, the Caveman had to try to decide what route to take to get home, was so not to get eaten by the Saber to a tiger. Right. So they would predict where the tiger might be. They would predict the safest route. They might look at whether the predictive. How fast could I get there? We we live with predictions all the time. When when your wife says what times we leave to go to the movie theater, you're going to go back and predict. Well, you're going to think about traffic and the time of day and with a weather condition is and how far you the how far you buy the part, and you make a predictions. Well, we need to leave in fifteen minutes or twenty minutes or ten minutes. We make predictions all the time. It's not like it's something unusual. It's something that that humans have done forever, and so what we're trying to do... to put more of a systematic approach around it and understand it, because humans are actually prediction machines. We may not be very good because we have some natural flaws, confirmation bias and, you know, the ability to sut costs and the things like that, but we are by nature prediction machines, and so I think what we learn and prediction. You said they know predictions are made on his on historical data, which is true right. You look at and historical data to try to find Trans Patterns and relationships that you can use to make predictions. But you're also overwegh current data versus historical data. I mean looking at trying to go to the movie theater from data that twenty years ago, miss is the factor. Even a year ago, it's the fact that there's construction work going on right and you're going to have to leave five minutes earlier, or there's a concert that's going to cause traffic to backups, you need to leave ten minutes earlier. And so yeah, we look at historical data to make predictions because it's in that historical data that we find those variables or features that really help us to make more effective decisions. But then we have to tune it for what's going on in today's environment. So it's important historical data is important. Without it we don't know what those important features and variables are. But we have to be able to update those models based on current data in order to figure out, you know, and make a more accurate, more informed prediction. Yeah, totally agree with that. Right. I mean, I think a lot of times people mistake the number of observations that you have to have for to have a credible model as a statement about the value of the past. Right that, and one is not necessarily the other. You know, this is a this is a good point here, mark, is that I've gotten into some some social media, you know, wars over the fact that all you know, some data, you know it's old and just throw it away. It's not valuable. Right dated decays. And I say, well, it depends on the latency of the decision you're trying to make. Right IT base. For example, looking at the ramifications all the tornadoes that hit Kentucky this past week, we can tap back into historical views of tornadoes and what they did to other communities. I may fifteen one thousand nine hundred and sixty eight, I lived in Charles City. I with those hit by an mfive Tornado, killed fifteen people, destroyed the city. Lot of people can learn from what worked and what didn't work. We try to rebuild the city. Right. There's so historical data doesn't necessarily have to decay. It all depends on the time frame of the decision you're trying to make in the latency of that decision. So again, I think historical data is got legs. I, like you, you know, maybe don't keep it on hot storage. It's really expensive, but I want to be able to go and get it. Maybe got to pull it off tape somewhere. I got to pull it off tape to find out what happened, to see what were the economic ramifications for the top hundred tornadoes that hit over the past twenty years and how that impact of the economics of that particular community. That's useful. Absolutely. No, I mean as a as somebody who has in really pursued historical study, not professionally, but I would say I'm a very serious advocational historian. You know the you know I'm working primarily in the fifteen century most of the time, right, and so you you have to your numerator denominator. Relationships alter right in that situation and you're not comparing. This is actually what? So a cornerstone of history, right, historical study, is that you cannot judge the actions of someone in the fourteen century based on twenty one century values. Right, Yep, and and that's really what we're talking about here in a more general sense, right, and that is you can't. You know you'd be if you were looking at hurricane or plague data from one, three hundred and fifty and comparing that with today. There might be a few things that you you could find that we're in commonalities, but you there are so many differences between the two situations that you would have to be extremely careful with that mark. And I can I riff on something here. You said a very important word, the single most important word for a data scientist, and I want to argue that a single most important word for a business that wants to survive the world of change, and that is the word might there might be a few things in that historical data that might be important. Right. So what is data science? As I describe data science? Two executives, I said, I think a sixteen words. Data Science is about identifying those variables and metrics that might be better predictors of performance period. That's all. It is. We're going to go through a lot of processes. You're going to use fancy words like feature engineering to describe it. We're all trying to as find those variables that might be better predictors of performance, and so again, I think they're. There's this thing...

...where we've got historical data. You know, we could certainly use the swine, pig ands and the stars data that came out to understand sort of how how these things, these virus has spread, how these pandemic spread. Looking back at the Spanish plague probably doesn't help from an economics perspective, but you can see the you could probably get a feel for, if you had that kind of data, sort of the economic ramifications for those communities around those hot spots. And again, there might be a few variables in there are just a couple that could be useful, because it's sometimes it's just a couple of variables that distinguish a good, very model from a great model. Right, it's just a couple of those sinkle variables that you might find in the data. It's so true. So if you are, let me, let me ask this question, given, given what we've just been talking about here. If you're a business leader and you're talking to Bill Schmarrozoh, and they are used to, and I'm going to kind of use some data science lingo here that that a business leader would not use. Okay, so they are. They're used to making decisions based upon twenty thirty percent confidence levels, right, and the data scientists, all he has or she has is stuff with forty five percent confidence levels, which they've been trained from school to say this is trash, right, and and they basically anything below eighty is highly questionable and the ninety five percent is is the goal. Right. So the business leader gets extremely frustrated with this for two reasons. One, Hey, you're telling me you've got something, but you don't have confidence in it. So now I don't have confidence in it because I don't know enough about data sciences science to make the judgment. And also, you know, like what's the speed of the RECALC? So like, how do we test this right and how do we test that? I was in fact able to make a better decision with your insights than without it. We talked about that a little bit, sure, because I think the only way that a true data scientists can know if a model is good enough for not is to understand the cost of the false positives and fault to negatives. Let me give you an example. So I was at Yahoo, I was the vice president advertiser analytics, and if I had a sixty percent confidence level that somebody into my site was interested in digital cameras, man, that's that's better than flipping a coin, right, right, pretty damn good. Right. Yeah, particular when human behavior is is the issue. Yeah, sixty percent conference level is really good. And the reason why it's really good is because the cost of showing you the wrong ad was less than a penny. There's a fraction of a penny to show you the wrong add so the cost of being wrong wasn't very expensive. Now another example. Let's say that you're looking to take the COVID vaccination and you've got a six percent confidence that it's not going to kill you. You might want to not you might want to go to a different hospital. Right. So, because the cost of being wrong with the covid vaccination could be you know, you could die right or other sort ramifications. And so part of the conversation that must need it, that must take place, is have a conversation with the business stakeholder's not the data scientists, the business taker, to say, well, what is the cost of this being wrong, because then I can get the data scientists a frame it. You know, forty five percent versus twenty percent maybe absolutely friggin great. That's great's been on the cost. But you know, if you're making a decision today based on twenty percent, which is less than you know, flipping a coin, then you know again it has to get back to a conversation with the cost of being wrong. And then I like your point about the refresh and testing. How am I? How A my learning? So I make it a decision based on this. The decision executes. Can I measure the false positive, fautse and make it is in the model? Can I? Can I measure, for example, when we hire somebody we thought was going to be a good employee but wasn't, that's a false positive kind of measure that effected us and feed it back in the model? The answer is, of course yes. But can I also measure the false negatives? A person I didn't hire, that I shouldn't so I should have hired right. How do you figure out the false negative? By the way, there's a way to do that. It takes work, it takes commitment, but if you are looking at driving models, it continuously learn and improve and can improve that confidence level, then it's well worth your time to also instrument your processes. Far The faults, negatives. Yeah, I so agree with that. Right, people a lot of times will say, well, you know, can't prove a negative, right, you don't know what didn't happen, and I think we have found that that is to your point. I mean it's requires a lot more work, right, and you're essentially running simulations of what might have happened, but you can learn a ton from that. Yeah, yeah, yeah, and I...

...think it may be part of what is hurting people here on this whole issue that we're discussing is they are seeking validation as opposed to learning. Let me treat it a bit. You have people who want to have one hundred percent confidence at the decision you're going to give them is going to be right, that it never going to happen right. Never, right. It did, it didn't. Never happened before and because you have models, it's not going to be a hundred percent. I mean, we couldn't have guessed that this pandemic was going to hit. I think a lot of us were surprised by the omicron some of us weren't. Some people were right that this saying another wave is hitting. Right. So the fact that you're never one hundred percent accurate is not an excuse for not using models to make decisions, because you're already not one hundred percent accurate. Now you're you say to tell your wife we're good, to go to the movie. Theater is going to take fifteen minutes, but it takes thirty because there's an accident, you know, and then she yells at you and you missed the first ten fifteen minutes of the movie and now you don't know what's going on and now you're to blame. But how? But you? But again, we are making predictions, and so I think the key point here is it. We need to be comfortable. We need to teach probability literacy, for lack of a better term, but what it really means to make decisions, because you today are making decisions based on probability. All we're going to do is try to make them not only more accurate, but create models that can stinuously learn from the decisions that are made in the wrong ones that are made. That's that's right. I mean, I think that the other part of this that's so important is to realize how little we controls. Right. In fact, it was when I started working with as the beneficiary of analytics, as a business leader, it was actually sort of traumatic, say fifteen years ago, to realize how little of the overall equation was under my control and thus how important it was that the bit that I did have control over was done really well, right and in a very timely way. And it kind of brought me back to a to a lesson here that I learned as a teenager because I was the I was the navigator on a racing sailboat that the man's my family did a lot of that kind of stuff and that was my duty station. I was the navigator. And you know, you have a chart and it says, okay, you know, here's where you are and this is the end of the race over here, and you take a wax pencil and a slide rule and you're basically drawing a optimized course across that and your chart is noted for prevailing winds and prevailing sea currents and all this kind of stuff, and so you know that you know you're not going to be riding that line at all, but you know you're going to get there. The problem is, you know, not so much today, but when I very first started, I was still shooting the sun with a sextant to fix our position on the chart. And so there were days when the weather made it impossible to shoot the sun and we would go for sometimes two and three days not really knowing exactly where we were anymore. And all of a sudden we get the window, we shoot the sun and we were like, holy crap, we have way over here. Right, we are way over here. And what you started to realize was is that the test of sailing in this case was can I mitigate all of these factors that I am having to deal with and still win the race? Right? That's that's really the essence of it here. And so that kind of today, the commonplace, everyday thing that we all deal with, right, is the GPS on our phone, right, going. And so let me ask you this bill. It seems to me that so we have a prospect right now that is a very, very mature user of marketing, mixed modeling, which is the use of multi variable linear and not your regression to understand marketing impact, and they're using one of the really, really huge consulting firms to do this, and I know this firm. It's a great firm and they have great data scientists. So this is not about the math. It is, though, about the ability to operationalize the math in an effective way, particularly around prediction. So one of the things that we discovered not too long ago was that this prospect had just received all of the analytic read doubts for the back half of two thousand and twenty and they received... in like October of two thousand and twenty one, okay, Yep, and and it had all kinds of predictions in it, except the reality of time had overrun those predictions and they were pointless. No one could do anything with them. Right, and this is sort of goes to the heart of where a lot of business people get really, really torked, I don't think that's an understatement, with the data scientists that they work with, because the combination of human powered data science, latency and the cult of precision combined to create something that they feel like never actually gives them any practical benefit. The words, the predictions are not actionable in the real world by the time they get them. Where do you think this is going right now? I mean one of the things that you're really known for is peering into the future and having your own predictive capabilities about where all this is headed. What are you thinking? So, as you told your story might be of the general patent quote and won't have the quote exactly right, but I think he said something like an average plan wax well executed is better than a perfect plan never executed. And the problem, I think it gets back to the comment we were talking about earlier. You said this cult of precision, which I it's a great term by the way, which is is this idea of probability literacy? Is that? You know? Maybe we talked about data literacy. Well, I think it's really more relevant talk about predictive literacy and helping people understand that sometimes you make decisions on imperfect data in an imperfect sit we do it all the time. I mean you talked about, you know, being a navigator. People on Sports teams are always, you know, making, you know, spot decisions based based on what data they currently have. They have some historical perspective on the players in the game, but they're so you're always making less than perfect decisions. But the real key, and you said it's really well, is the ability to cycle and iterate on a quickly. You know, I have a phrase that the economies of learning are more powerful economies of scale, that the ability to learn more quickly than your competition is a critical sometimes you throw a model out there that means did earlier. Maybe it's only forty five percent confidence level. Let's get that out there and let's start building on it. But so the Big Butt you have to make sure your business stakeholders understand the process as well, that we're going to start with forty five percent and we're going to learn. Eventually we'll get the sixty five and seventy and seventy five right when. Eventually it will get better. But but oh by the way, it also might do this if there's another major trauma, there's a another pandemic or an economic collasse, whatever. Right, these models are always constantly trying to learn and stay afresh and solic idea. This fast refresh cycle becomes critical that you you can't if you wait for the perfect situation, it ain't ever going to happen. So I again I think. I think athletes kind of know that, especially good coaches know that they're making decisions, imperfect decisions, based on the current data they have, with some historical perspective. But they're very the goods are very, very quick to learn and pivot and say, Whoa, that's not working. Right, we're not going to we're not going to run a zone against Steph curry. Trust me, that's a bad idea. Right. So, so we're going to sliderly good on paper, but what do you do it practically? You know practicality. Just hit five state three, five straight three's on you. You might want to go back to Aman to man right. So again I think that your point here is that this cult of precision means that, I guess the old term for it was analysis paralysis, right, where we just try to analyze things until we get it to some level of confidence for the end user to actually act on it, which, by the way, that opportunities may have already passed and now it's irrelevant anyway. Right. So how do you see combining kind of traditional regression based prediction with pattern based machine learning? How do you see that happening going forward? Well, I think the beauty of machine learning is. It can sometimes find things you don't see on your own that you then can feed into your multi regression models. It's the best data scientists don't use one technique. They orchestrate a whole series of techniques trying to figure out you know, you'll go through a process. I know our data sciency and they've start off with unstructured machine learning trying to find variables that were have certain characteristics. Then they pushed into a structured machine learning variable to find out which ones were most development than they put into, like you said, a regression model that they actually get used to predict. But they put that's doesn't they don't do that once. Who never done right? Right constantly visiting that. They're constantly saying, well, as other things have changed, they're always trying to find those x next variable that might be better particular performance. So I think the good data science teams have this Nat's natural ability to bring in...

...different tools of different times to help them figure out which variables might be most valuable. And regression analysis is still one of the best tools for actually making predictions because you know what variables you're working with. It's it has the advantage of transparency right in front of it. So one of the things that I do when I when I prepare for these PODCASTS, as I go out and I talk to people and get their questions. So this actually comes from the CFO of one of the largest management consulting firms in the world. He just wants to know how far ahead in time should they be able to investigate with some degree of confidence? How far out? And I would say that first thing that drives that response is how agile are they? Because if you're really agile organization you don't need to be way out in front. You don't need to have an eighteen or twenty four month window. You can get by with a three to four month window if your organization is Agile. Now that doesn't mean your business leaders are agile, I means the entire organization has been empowered. Your frontline employees at people who touch customers, who run the operations, have been empowered to basically be able to take new data, make new predictions and make changes accordingly. So I get I don't think you need a long window if you're a real agile organization, but if you're not an agile organization you probably need a big window which, by the way, you make a twenty four months prediction, who the heck knows what's going to look like twenty four months from now. We could have not a year ago. We would have been we would have been surprised, and where we are today. So it really points to this idea. So that call organizational improvisation, which is a create an organization that is so much and and locks up of each other. It can flow back and forth because you've pushed decisionmaking down to lowest part in the organization and you've clarified the KPIS and metrics against which the organization and measure success. If you ask the average person in organization, know somebody WHO's not at the senior most level, but somebody who says, you know a manager or you know maybe even a director in order the Kpis against which the organization measures its success, they look at you like you got lobsters crawled out of your ears. Profitability, increase in sales right. Well, no, I always say that those are all lagging indicators, they're all lighting right, and by the way, there I always say that the KPIS that an organization uses to measure itself says more about its culture than any artificially defined mission statement. So what is your how did your organization if you say you're that you know environmentalism is important to you. What are the KPIS metrics against what you're you're measuring that? Are you trying to increase the reusability of your products? Are you trying to reduce carbon footman, what are the metrics you're trying to use? If they're not spelled out to everybody, organization knows that. The person at the frontline who is the briefe to talk, and a customer knows that I want US sell. You know, I'm not going to give you a plastic Straw because that goes against our creative trying to reduce. You know what, it's got to be driven down to the organization. So this idea if you can create an Addi all organization, which, by the way, is why I'm a big believer in design thinking and democratizing ideation, and push it down the organization. If you can do that, then you only need to have a shorter window in front of you because you can pivot very quickly. You know, I like the joke about the two guys throughout hiking and they've been waiting in the water and all sadden the bear comes out of the woods and it is chasing them and one guy starts running barefoot through the woods. Like I said to us, I was put on his shoes and the guy yells back at him, Hey, putting on those shoes the thing to help you out run that bear. And he says, yeah, I just need to run you, which is a right. And I think that's the key here, is is that we spend so much time talking about, well, you know, the models and a latency and the accuracy. Know, how do the people were going to use the models? How have you empower them so that they can use an interact and flag things and you create this culture of continuous learning and adapting? The AI and ML will do that, but humans are also really important part of that equation, which we tend to think we tend to skip over. Augmented intelligence is the future, not not more than artificial yeah, so, okay. So actually this is sort of surreal. So here's questionable to from from the outside and the way it did. Did I answer that question right? Yeah, no, you did. Oh, yeah, I absolutely. I mean we may find out from our audience. We just ended my semester Thursday with my students. We had to give final exams and I feel like I'm I can get my own final exam here. Holy Coww did I do, teacher? So and guys, I did not share any of these questions with bill ahead of time. So I hope that in me clarifying that, you're going to see that this is actually a very, very mainstream set of questions that that are are probably similar to...

...what you're you and your leadership are grappling with. So the second one was went something like this. So revenue, margin, cash flow, bunch of things are lagging indicators, right, that we're trying to get a handle on and and find leading indicators, or we might use the word predictive indicators. Right. So this person, pretty pretty senior in this company, Software Company, believes that most of that selection of leading indicators is based upon tribal knowledge only and wants to know if that is in fact valid and what kind of improvement could analytics bringing to the identification of better predictive or early indicate leading indicators. So I love that question because there's a lot about that. It's that a spot on. First off, leading indicators is really the heart of feature engineering. You think about it, right, we just give it a fancy term, call it feature engineering, but it's really about identifying those, those leading or predictive indicators, and I think this person is spot on that the tribal knowledge is where you want to start these conversation to find those heuristics, are rules of thumbs that people make decisions on. We're doing a project for a large manufacturing concern and then one of the operators would say, well, when that red light blinks three times within a fifteen second span, I know been the next two and a half minutes, this machine is going to go down like really. Well, you know what, let's try to validate that, right. What are the variables that are actually called that to happen? Because if that's true, then we can actually have the basis for starting to predict things. So I think it's in the in the heart of these tribal though, these two heroistics, these rules of thumbs that people used. Another great one is we're doing a project your your if I'm understanding correctly, Bill, you're you're essentially saying start with the tribal knowledge, because that is a hypothesis. Yes, yes, yeah, it's like when I when I work at clients, the first thing I want to say as well, so many spreadsheets you've built to make that decision, because those spreadsheets are full of gold. Now they're all isolated and disconnected and etc. Etc. But you'll find you spreadsheets will be like, well, I'm going to use these variables to make this decision and and I've got weights in there. I've done some you know, some Calculi's like wow, that's that's where you start. So we like to start our data science engagements by understanding the heuristics are rule of thumbs that they use. The data science team then tries to prove them, tries to value those are actually real and what are the variables in inside those heuristics that are actually the leading indicators or the features we want to go after. So I think it's spot on, which is why you data side is can't do feature engineering by themselves. You want to find those business stakeholders, of people the frontline of the organization who can tell you that, oh, when I see this kind of behavior, I know the custers getting ready to leave. Really well, let's validate that. Or again, the case of the blinking lights, saying this problem is this particular machines going to go down? So I think it's. This is why the the data sience process that I advocate for so much as a collaborative process where you bring the business stakeholders in, you you teach them to think like a data scientists, right, and then the data side is has the basis around which do not only build the right products, the right sort of models, but now has a established a communication channel with the stakeholders who are going to be the front lines trying these models. So, yeah, we try the model really didn't work well in these situations. Oh, let's describe those situation the where details. Explain those to me. Can it's all about creating these economies of learning. Right, absolutely. So I guess the the kind of the last question about being predictive that that I got from the outside is how do you know when your model is representative of the situation, understanding that you'll never know everything that might be important? And yet you can't, I mean to your earlier point, right, you can't just say, well, it's impossible for me to know everything that might be important, therefore I'm not going to a model. Right, you got to proceed forward. But how do you? How do you know that your more stable than not on on creating a predictive model? I think the starting point is, I can go back old schools AB testing. Right, you just test, you test. The model said do this and you sometimes you don't do that. When I was at Yahoo, we built a set of analytics. The delivered recommendations to the media planners and buyers and campaign managers and now we said, you know, these are these the adds you should show to these audiences this time of day. These the site you want to you know by your site. And we always gave them for these recommendation. We gave him three options. They could accept the option. They could accept that recommendation and we would implement it, they could reject it and we...

...wouldn't implement it, or they could change it. They say now I said, I do that. I want to buy this particular side, I want to target this audience. Each one of those three situation we try to measure how effective that decision was. Right. The MODEL said do this. Right, they did it. That it work as planned. The model said do this, they didn't do it. What were the results of it? Or you in power them? To say no, I actually want to do this and you got okay, cool, let's see if that works better. So I do think that you have to put in place some really basic concepts around a be testing, to allow people to test things. And along with that line, I also think that you need to allow humans to be able to override the the recommendations from decisions from the models. I heard this great story. It was from a financial services organization was making loans and, you know, they got up an AI model that basically cranks out a number that says whether you should get a loan or not. And so they're telling about the story. This person to come up and they had a really spotty his history of work record and, you know, sir, education level was, you know, they dropped out of high school. They had was not a very good not a good risk. And the person was asking for a loan to buy a car and the model said, no way, as that's never going to repay the low back. The load officers did away a second. Can you tell me more about what you're going to do with the car? And the person said, well, my life has been sort of dependent upon others. I've never really taken control of my own life. I want to buy a car, become an uber driver, I want to take control of my life, I want to basically be in charge. And a loan officer said okay, approved and but of course they measured. The person repaid the loan. The person actually applied for a second loan, they bought a second car that they actually leased out of other people. The guy is this person that started a little company around Uber. And so the human in this process you talked about, you know, augmented intelligence. That's exactly what it is. It shouldn't replace to him in the human should take a look and say, okay, the model says no, what are the risk factors around why it says no? Can I explore on those risk factors so that I can make a better decision? And, Oh, by the way, the model can then learn from that decision if it works out well, Yep, all right. So here's my last question for you. So this is a collaborative effort. Prediction is a collaborative effort from between a teen of data analyst, Data Scientists, mathematicians who know their stuff really, really well and in in many cases, though, have very little subject matter expertise, very little business context understanding right, and on the other hand you have all these business leaders who most of the time have a lot of contextual awareness and understanding of how things work and all this kind of stuff, and they don't typically have a data science knowledge and, making it worse, sometimes they have very distorted ideas about the reality of what data science really is. How do we fix that? Like you have a story where you've seen that really work really well. Yeah, so, I mean it's just probably a bad question ask me, because I've written a book called the art of thinking like a data scientists that I use in my student classes, mostly NBA students, but it's also been adopted by a lot of organizations on both the technology side in the business side. The the charter here and the art of thinking like a data side is methodology. is a design thinking centric process for how you drive collaboration, how you democratize ideation, how do you take an culture where all ideas are worthy of consideration, where you are trying to find those variables and metrics that might be better, particular for formants, where you are pulling in those that tribal knowledge to drakes. So it's of all about establishing a common language between the business stakeholders who have them domain expertise, and the data scientists, who I don't really need to have demain expertise. But they should understand which neural network technique, which machine learning technique, which regression technique. They should understand how to orchestrate in a series of different analytic techniques is sort of come up in validate the variables of metrics that these suchet matter experts come up. So, yeah, I've seen this work time and time again. I mean, I've got, I said, I got a methodology. So it is really bad of me to say this. What I really also hear you saying there is that it's up to the it's up to the same the the business leader, to say, Hey, this is the kind of stuff we really want to know. The answer to yeah, and and it's how do you demistify data science? So you make it something that they go, Oh, I understand that. So they understand. So they subject matter experts to business stakeholders understand not only the role that they play but the critical value of the role that they play. You know, I mentioned earlier understanding the cost of false positive faults, negatives. Right. How do you know if a model is good enough. Right's problem based around the cost of the false positive false negatives. That's...

...not a data science exercise. The cost of the false positive false negative is a suchect matter, extra business exercise. And so you need to have this common language. And so we've developed a methodology. It's full of design temple. It uses all the design thinking techniques to try to, like a term, you know, democratize ideation so that they feel empowered. And, by the way, the people who need to be empowered in this process isn't the executives who sit on Mahogany row who don't know anything anyway. Right is a people the frontline. They're the ones who are constantly interacting with the customer or the operations and learning what works and what doesn't work. They see trends much more quickly than anybody in the fifteen or twenty floor of a building. And so again it's about how do you push power down? How do you create this this organizational improvisation like a great soccer team where everybody's empowered around a common set of KPIS and metrics? Yeah, want actually, one of the ways that we did that at honeywell aerospace when I was seeing mother. There was we were dealing we were working in so many different countries. We realize that number one, the the best approach in country a, was not going to be the same best thing for country B or C or D. Right, and we and so we had to have this way of saying, okay, we have these goals at the center that are kind of really super important and we can't leave them behind. And so there's that. That's the accountability piece. But we need to have an analytical lifeline and umbilical cord that runs all the way out to the edge that gives them the freedom to do what they know they need to do and gives us the measurement in the analytics coming back to validate that it worked for us. And Mark, here's the hardest part about what you just said. Is Business leaders willing to give up the decisionmaking authority and push it down to the front lines of where it happens. So you give the front lines the authority, the responsibility, any authority to make decisions. It's not the front lines problem in most cases it's the executives who just don't trust anybody but themselves. Now I guess I was fortunate I didn't have that much self confidence. But but yeah, I know you're right. I totally agree. All right. So, guys, another really terrific conversation with Bill Schmartzoh, if you haven't read as books, highly recommended. Right. They're all over Amazon. And I'll tell you this. There are a lot of books in the space that are very readable, right, and I and I a lot of times I talked about that. There are not as many that are also entertaining, right, and I would say that that one of Bill's gifts is that he knows the really important message that he's trying to get across and he understands that being provocative is part of the way that he gets you to think and to open up your head. Right. But he also he's funny, right, really is, and I don't mean like you know super nerd funny, I mean he's just funny, right. So I just really encourage you to pick up any one of his books that looks interesting. I have almost read the entire anthology, at least the parts, the ones that I'm aware of. So if you have any if you listen to this and you have any questions and you're maybe a little nervous to ask bill, you send me a note now and I'll tell you where to which one I thought is the best place to start, because I didn't read them in order right. So I now kind of have it different, maybe a different view than maybe someone else does about the right order to read them. They're not. They're not like Lord of the Rings. They don't really they don't really build you a Fresndo, ha ha. Now they're spokes on a wheel, right, you go there. Yeah, so very much. Yeah, which, by the way, I so this is with the cool thing about all that, you multimedia interaction, right, because bill and I are doing this and in the previous twenty four hours there's been hot and heavy conversations on linkedin around. You know, is data like water? and is it an ocean versus a lake, and you know, and all this kind of stuff, right, and one of my friends that I be am has a has a somewhat low opinion of data lakes and all this kind of stuff, and and he refers to it as a data goo log right, or occasionally, and most people won't understand this reference, but he calls it a data oubly it right, which is a French word that means to forget. But in the Middle Ages they would create oubliets were which were like dungeons underneath dungeons,...

...and basically they throw somebody in there and forget them, right, and that that's what happens to a lot of data in some of these situations. Right. So, so, anyway, this is all you know. If you want to have fun on this, right, and this is something that really interests you, there's so much going on on Linkedin that involves bill that I would just encourage you to follow him. You know he is prolific, and so if there is something that doesn't really you know, meet you where you are, that's fine, because guess what, two days later he'll get you right. Thanks Bill, thanks mark. That was a great endorsement. Yeah, and people get out, get involved in social media, on the good part of social media. Share your opinion, share your thoughts, have conversations, because we're literally we're all standing on each other's shoulders. That's right. Need want to. I'm doing a so I have kind of a separate thing that I do around history at this time and and I've been add us to do a series on my ten biggest lessons from history, from studying history, and the first one is all going to be around the idea that war and peace are both team sports, right, and they the idea that there is no great man or great woman on horseback that somehow does it all exactly exactly, so, exactly to your point, Bill, right, we all stand on each other shoulders and season. Thanks mark. The sooner you can optimize your marketing spend, the quicker you can start delivering clear, measurable value to Your Business. That's exactly where business GPS from. Proof analytics can help. Learn more at proof analytics DOT AI. You've been listening to accelerating value, where raw conversations about the journey to business impact help you weather the storm ahead. To make sure you never miss an episode, subscribe to the show in your favorite podcast player. Until next time,.

In-Stream Audio Search


Search across all episodes within this podcast

Episodes (35)