Human Focused System Design - Justin Warren

Machine-generated from the episode audio and not hand-corrected, so names and technical terms may be imperfect. The audio is authoritative.

All right, listening to another episode of the CTO Advisor podcast, I looked up on the podcast, RSS1. We don't have all of our historical podcasts on the new website, the new-ish website. The website's been out for a little bit over a year now. And we brought over the podcast, published the podcast where we interviewed the CTO of the New York Times, at least the former CTO. And I looked up, and the old podcast isn't there. We will fix that gradually over the coming weeks.

But with that, I was looking for the last podcast I did with our current guest, Justin Warren. Justin, welcome back to the show. Thank you. Thank you for having me. So for people who can't go back to the last podcast we did together, Justin is the founder and principal of Pivot Nine Consulting. That's right. That's still the current title, right? Yeah. I call myself Chief Analyst these days because I do a lot of analyst work. That is very appropriate because you are very good at taking systems and governance and breaking it down to the most basic components, which is the topic of today's podcast.

What executives do at a very high level is very difficult to train, Justin. It's very difficult to say, okay, we're going to create policies, and we're going to create a governance structure, and this is the certification test you have to take to get there. As we transition into these roles where we're leading and we're creating frameworks, et cetera, I get this question very often, like, how do I train for that? And we're going to have a very interesting conversation. Over the past few years, I've been watching your passion around some of the governance issues you folks in Australia are having.

If people haven't noticed, Justin has a slight accent that's different than mine. He's from the other side. I don't even know if you can say the other side of the pond. I don't share a pond with you. The Pacific is an extremely big pond. It is a very big pond. A little bit bigger than the Atlantic. A tiny bit. So Justin, the other day I was trying to figure out how to talk about IT infrastructure, and I had this brilliant idea that's not a new idea of looking up traditional public infrastructure and academic papers on how do you describe, how do you sell, how do you formulate new infrastructure.

And I can't help but wonder, are there parallels for IT governance, IT policy, et cetera, and how do you teach it and correlate it to traditional systems that's non-IT? Well, the key there is that the one thing that's the same about them is the humans. The technology is actually not really that important as it turns out. And we see this a lot in the consulting work I do working with senior executives is everything becomes systems, particularly if you want to make it repeatable.

And the systems are not, they're socio-technical systems, academic type stuff. So it's the combination of both the, call it the technology parts, education is a technology on taking the sort of broad view of things. So how do you train people? What tools do you buy them? How do you teach them to use those tools? And how do you organise people? How do you collect them together, motivate them to do the things that we want to do together? All of that forms a system.

In fact, it's generally interlocking systems. And when you're working as an executive, that's basically what you do all day long is look at the systems that you have in place and how can you improve them or change them? So I can give a very tangible example of this. I was with the Department of Housing and Urban Development over here in the US. It is the department of the federal government that's supposed to encourage and support home ownership as well as meet the needs of the under housed, let's call it, or homeless.

The project that I was working on was the migration from, I'll date myself a little bit, maybe not too much, from Windows XP to Windows 7. Technically, the bits between Windows XP and Windows 7 isn't that difficult. But even the migration itself technically isn't very difficult. We start the rollout and then it's a halt, halt, halt, stop, stop, stop, stop, multi-million dollar project, 15,000 users. Seems that we forgot about the user part. And the union stopped us and said, wait, our people need to be trained on how to use the new system.

It was from a technology technologist perspective. I mean, I'm a tech, especially back then I was a younger technologist. I mean, what do you mean is Windows go to the start menu and you use the system just like you did before? But it's not that's not how things work in the real world. You know, it's you have to consider people in process as part of the technology. But what's the training ground for that? Like how could I have had empathy?

Actually, it wasn't my fault, but I'll own it. I kind of didn't own it just then. It's funny. It was how do I have empathy for that people in process part of it when I'm not really exposed to it from a day to day as a technologist? I understand the bits and pieces of how to do the migration, but I really don't understand people. Like, how do I get the empathy part for people? Well, I think that someone else is famously quoted that I don't know how to tell you that you need to care about other people.

We kind of expect that as social organisms, we as humans are brought up in some way that society matters and that you are not an individual that is a sacrosanct God that is able to do whatever you want, whenever you want, with no constraints. We live in a society. So I don't know how to teach that part. I think that there are many pathways to this and like traditional education, I did an undergrad as an engineer. I started engineering and it was engineering has a rich history of, well, failure, the one thing and learning from that about and that the humans are important.

The reason we do engineering is actually about society and people. So there is a lot of, there's education built into an engineering degree, or at least it was in mine, to consider the people that will be on the receiving end of the thing that you build. And it's a profession. So there is an expectation that, for example, civil engineering, if you build a bridge and it falls down and kills people, it's your fault. So you need to be careful. You need to be, you need to make sure that you pick the right materials.

The reason we care about that is because in the past people didn't and bridges fell down and killed people. Regulations are written first in blood and then in ink, is the old saying. So I think in technology, in IT, we are transitioning from the idea of it essentially functioning as a trade where people learn it on the job from other people. And it's, I think it's really growing up and becoming more professional, which is that you need to consider more than just, hey, I'm putting a pipe in the ground here.

You know, the senior trades people understand the bigger perspective of the job. It's like, no, no, if you do that, it's going to cause maintenance problems. And then the homeowner is going to cause, they're going to ring me when the pipe bursts because you built it wrong. Or, no, we have to do this in a particular way because regulations say that you must put, you know, you have to use this grade of electrical cabling because otherwise it will catch fire and burn the house down and kill people.

That's why we are mandated to use this particular kind of material in building a house. Those sorts of ideas are now coming into computer systems more than they used to. I would say even if we go back 30, 40 years ago, there was a greater appreciation of that, partly because things were much more expensive. So failure cost you more. I think this period since around sort of the dot-com boom where everything went, you know, move fast and break things territory. I've never liked that.

As an engineer, my job is to build things, not break things. So that kind of callous disregard for why things exist and we should just break stuff willy-nilly because who cares, I think we're starting to re-evaluate that. And that's part of that training of the, I guess, IT, particularly the online kind of IT, it's training it as an organism to care about this stuff. And then that will flow through to educating people in a very calm and collected way that we see in other professions.

So I completely agree with you. This is something that is probably out of the scope of our responsibility to teach. You can't teach empathy. You should have empathy. I go back to my university. I got a BA. So if I, a Bachelor's of Arts. So if anyone can attest to that, I went to a Catholic school that was built on service. So it was the attitude of service first, technology, and even process secondly. But this is where we're going to get into kind of the meat of a lot of your frustration online when it looks at some of the systems you critique, some of the people you hold to power.

You would think from a government services perspective, when you're serving the public, that that's the job, to serve the public. But you've been dealing with this issue. I think we actually talked about it the last time you were on a podcast, which was probably about three or four years ago. You've been keeping track long term of a failed government system. Give us an overview of what that failed government system is and what learning can we take from it for those of us who are responsible for building technology systems?

Oh, okay. And this is quite topical because it relates to automated decision making. It's kind of at the core of what this system is. I'll have to give the quick version. It's very complicated. It's been running for many, many years. So there was a system in Australia which was colloquially called Robodebt. The premise behind it was that the government would provide welfare payments to either the unemployed or just additional payments to people who have families. You get a thing called Family Tax Benefit B.

So if you have kids, you get tax breaks. So it's all within one big department. It's one of the biggest departments the government runs because it pays out a lot of money. As with all systems, particularly government systems, it didn't work flawlessly. So sometimes it would give people more money than they were entitled to as strictly interpreted by the law. So the premise behind this was, well, we've overpaid people, so we need to get them to give us the money back.

But we won't ask nicely. We will demand it with menaces in some cases. So we'll threaten to send debt collectors to you if you don't pay the money back. That was the kind of underpinning. So there's a core of the idea of system design of what is the purpose of this system. There's a principle in cybernetics, which is the study of sociotechnical systems. I highly recommend looking into it. There was a term coined by a guy named Stafford Beer, who was one of the founders of cybernetics, which is that the purpose of a system is what it does.

So what your intent is doesn't really matter so much as when the system goes into operation and starts behaving, you can just observe what it's doing. That tells you what its purpose was, regardless of what your original plan might have been, because purpose changes over time. So with this system, the idea was that rather than designing a system to not pay people too much money in the first place, we will create a system that goes and demands that they give us the money back.

One of the problems there is that there are lots and lots of people. So there was a system in place before that, which was very manual. It only looked at around 20,000 cases a year of suspected overpayment and trying to partly detect, okay, how many of these are through deliberate fraud, vanishingly small amounts, like less than a tenth of a percent, I believe it was. And then the rest is just honest mistakes, because it's a very complicated system. It's really hard to make sure that you fill in the paperwork in the correct way.

Rather than making it easier, though, this was historical stuff. So we will figure out a way that we can get people to pay the money back. But 20,000 a year is not enough, and we have budget pressures. We want to make sure, so there was larger political machinations going on where we want to save the government more money, or we want to reclaim this money, which we can then put on the budget and say, okay, that's income, it's not an expense. How do we convert a manual system into a more automated system and do this at scale?

So that's what they built. They built a system. They apparently ran a pilot project that said, okay, well, we'll test out this algorithm. And it's showing us, oh, okay, well, we'll be able to reclaim 10 times the number of debts. And then they put it into operation. But they failed to understand a few things about when you automate manual systems, which is if you scale something and you have an error rate of, say, 1 in 100, like 1%, it doesn't sound like a lot.

But if instead of it dealing with 20,000 people, it's now dealing with 2 million people, an error rate of 1%, now I'm going to have to do some maths in my head now, which is going to hurt. What's that? About 20,000. About 20,000. So if your error, so by changing the amount of it that you do, if you don't reduce the error rate, you end up with a lot more of the things that you then need to, a lot more errors.

And those errors need to be fixed. So if you haven't implemented a system to deal with the fixes, which tend to be much more manual, it's not an automated fixing system, I've now got to hire a whole bunch of staff. I've got people who are ringing the call center, and I haven't hired enough people to answer the phone. So it has all of these cascading failure effects. There's all of these other systems that you impact because you just scaled an existing system without reducing the amount of error it had.

And in this case, not only did it not reduce the error, there was a whole bunch of additional error in it which they hadn't taken into account. And as a consequence of that, it was, you know, governments tend to do this and executives as well. We made a decision, it's not going terribly well, we'll dig our heels in and pretend that everything's fine. And they did that for many, many years. And it ended up being a class action that went to court, and the entire system was found to be unlawful.

So it was, so the government stole people's money is the upshot. That is, I am drawing so many parallels to many of the failed projects over my long career that it's not even funny. But one of the things that I wanted to click into as we look at this, so I think most people listening to this podcast can kind of grok that you need to design a system for its intended purpose. And that may sound simple, but it's not. You have to think about it.

And I think one of the outcomes, and believe it or not, I follow as much as any other Australian your coverage on this topic. So one of the things that I wanted to hit into is observability. And how do you sample the system to make sure that the system is doing what you intended it to do versus what it's actually doing? Yeah. And there's a bunch of research for many years on design of metrics, particularly around management of systems. So people know all about OKRs and KPIs and a bunch of acronyms.

So metrics that, what is it? Why can't I remember what OKR stands for? Observability, was it? Objectives and key results? Something like that. It's KPIs with a different name. Thanks, Google. These are, what are we measuring? So what gets measured gets managed. So if you start to count things, people start to care about the things that you count. So what you pick to measure is important. Because if you measure one thing and not another, the thing that you're measuring is what will get people's attention.

Something else that you're not measuring will get ignored. But then if you start a measure that becomes an outcome, or what is it, a measure that starts as a measure where you're just measuring something to see what it is, if that becomes an objective, it tends to stop being a very good measure because people will start to game the system and change the way that the measure is interpreted. So, for example, if my measure is I want to make sure that I don't spend too much money, so if my measure is money, and I'm worried about my budget within my organisation, I will start to look for things that I can do to cut costs or to reduce spending, even if it hurts the organisation more broadly.

Because I'm measured on my little sub-organisation's budget. I'm not measured on the results of the overall organisation. So I will do things that make perfect sense for me to do in isolation that actively harm the greater organisation and harm the company. And we see this all the time. So that's one of the challenges of designing an observability system is, well, what are you going to measure? And then what happens when people start to game that system? And there's not a lot of thought put into that, what could possibly go wrong?

And I spend a lot of my time thinking about what could possibly go wrong? How can we design a system that is robust and resilient against those sorts of things? Yeah, and I spend a lot of time, we both lead our organisation, so I spend a lot of time thinking about this exact problem. If I reward people who get certifications, then people will just get certifications. When the desire is to somehow better serve the client, and again, comparing government policy to some of these same topics, you look at bounties for something like rodents.

And then what people tend to do is that they tend to breed rodents because you get paid $25 per rodent. So I'll just breed rodents instead of capturing the rodents in the wild. I know which paper you've read because I know exactly that case. I'm still in the case that I can't remember the paper, but too many academic papers in our history. But I think these academic papers, these government policies, the systems and the things we use to observe the systems we build in IT are based basically on real world things.

They are avatars for real life. And sometimes we can forget that they're avatars for real life. I talked to an AI ethics expert on the podcast about roughly a year ago, I'll bring up the podcast. And she's laser focused on making sure data scientists don't repeat a lot of the same mistakes that we have regulations there to protect folks from, which is just because you have the data doesn't mean that you can mine the data without considering people. This is actually people's data.

So when you say, oh, I put in this postal code and when I compare this postal code to defaults, I get a correlation. Therefore, when I see an application, I'll put in AI ML algorithm that says devalue application based on location of where they applied from, you're now redlining. That is illegal in the US. So because more than likely, the correlation that you're not seeing is that those people are normally minorities. So it's a system goes back to your point, a system does what it's designed to do.

Or not a system is doing, a system's design is what it does. Like if it's- Yeah, the purpose of the system is what it does. If it's denying colored people loans, that's what the system does. That's what it's designed to do. As I said, the intent is the difference between murder and manslaughter. The victim is still dead. It just changes the consequences for the person that killed them. And so there's two parts there that I think you've touched on that I think are important.

One is that one of my favorite authors ever, Terry Pratchett, says in one of his books, evil begins when we begin to treat people as things. And when we're dealing with data and computer systems and so on, they're quite abstract. And when someone, when people, when we forget about the people behind this and we're just focusing on cells in a spreadsheet and saying, oh, we'll fire seven people or 70,000 people this quarter because that's just how we run things. Those are real humans who have mortgages and kids and other things.

So it actually affects real people. Keeping that in mind, I think is really important. But the second part is that so much of this has already happened before. Another great quote is that people don't learn very much from history is one of the greatest lessons that history has to teach. So many of these things have already been done before. And in technology, I think there's this temptation and I've suffered from this myself, certainly more earlier in my career. It's a lot of fun to build it yourself, to do it from first principles.

But it takes a lot longer and you make all of the same mistakes. You can learn about how to do all of this by going to the library and borrowing a book. And so much of the stuff that we're dealing with here in technology has been done before in other fields. And there is a rich history and research and so much that you can learn just by picking up a book and reading it that will save you a lot of time reinventing the same wheels and making all of the same rookie mistakes as the people did the first three.

Yeah, I'll close out with we're creating an application modernization framework. And it's basically how do you get from prototype to factory mode for modernizing your application landscape. And I'm gonna tell you right off the bat, I'm just gonna borrow a lot of it from manufacturing. That's where I borrow that's where I borrow when we did P2Vs or V2Vs data center migrations a lot of the terminology is from manufacturing. You do prototyping and you move from prototyping to engineering sample and from engineering sample to setting up your machinery, etc.

etc. I don't need to recreate that. The tooling looks different. The tools may be different but our manufacturing friends have already solved the problem. How do you set up a factory? Yes, it's nice that IT has discovered the Toyota production system that is certainly helping us to take a giant leap forward to the 1970s or earlier. And we're following the same sorts of parallels. That's another thing that you can learn from history is that if you can see the parallels of what's happened before it's really easy to predict the future because all you're doing is repeating the past.

So Justin, if people want to learn more about what Pivot9 does, what Justin the I'm gonna call you the social champion does you've been quite the thorn in some of your government leaders side and I appreciate it. How can they find your musings and even engage your company? com so you can find us on the intertubes as with all people everywhere we will happily consult with people in a variety of fashion. We mostly help enterprise technology companies understand why they exist and what they're for and explaining that to their potential customers.

For me personally, you can find me on Twitter. I hang out there a bunch. My handle is jpwarren where you will see me both talk about the social technology governance side of things which I enjoy quite a lot and the more prosaic stuff of being a technology analyst. com is the website I'm at ctoadvisor you want to talk to me about any range of topics that we talked about today or anytime DM's are open at ctoadvisor. Talk to you next CTO Advisor Podcast.