Australian Census Fail - CTO042

Machine-generated from the episode audio and not hand-corrected, so names and technical terms may be imperfect. The audio is authoritative.

All right, you're listening to episode 42 of the CTO advisor check. We're going to go a little bit off script this time, unless you've been like in a Twitter dungeon or something. You may have noticed the census fail hashtag. And that is actually a very not American hashtag. No, we are not in some type of emergency census. This is for our proper English speaking friends on the other side of the world. It's not even the other side of the pond over in Australia.

And I have special guest Justin Warren with me. Justin, go ahead and introduce yourself. Hi there. Yeah, I'm Justin Warren. I am chief. What am I these days? Chief analyst and managing director of Pivot Nine consultancy. T. T. stuff. So no surprise. We've had Justin on before, I think at least a couple of times on this version of the podcast, talking on how his Pivot Nine consultancy helps companies make really big decisions. I think was the last podcast we had.

And I think judging by your tweets, you think the Australian government could probably have used your help. What was going on with the. Not just me. T. Twitter has been trying in vain to get them to pay attention to some of our advice for at least a week. And in some cases, several months. So by way of a quick background, like most modern nations, Australia runs a census periodically. Ours is legislated to run every five years. And it is compulsory.

There are fines. That's one hundred ninety dollar fine per day. If you absolutely refuse to fill in the census and there are penalties for lying on the census as well. So it's it's a pretty big deal. Those penalties are almost never enforced, though, because up until this year, people have actually really enjoyed the census. The census has been one of the or the Australian Bureau of Statistics is one of the most trusted organisations in Australia or has been very well regarded internationally for its ability to both collect and analyse statistical data.

And it's used for everything from understanding the economy of the nation. So things like its GDP, poverty rates, how people are earning, where they're spending their money, what religions we have, all of that good gear. It's a sister organisation to the same Census Bureau concept in the United States and census bureaus throughout the world. So knowing you, you're a big maths guy, you like stats. So it has to be one of your favourite has to be one of your favourite outcomes to take this data and peel through it.

It is. And there are many nerdy types just like me where we actually kind of really like the census. The last one that was run in 2011 had a particularly amusing Twitter personality running the Twitter account for the census that actually made it really, really fun. But yeah, the data that's available and some of the stuff that's there, a lot of us use it quite frequently. You can go in and pick out data around economic conditions and just understanding things like employment rates, all of this sort of stuff.

It's it's vitally important data. And there are lots and lots of people use it. And, you know, when you like numbers and stuff, it's it's fun to do. You you take pride in this civic duty that you have. So for a modern world government, it's impossible to run the government without census data. So it is not I would say it's a necessary evil. It's not even a necessary evil. I think it's mostly good. So what what went wrong with this year's census?

Well, that's yeah, there's a couple of issues. So the census has a long history of actually being people not liking it and having issues with it because it is it's quite intrusive. So the information it collects is very personal. It asks you about things like your religion, how much you earn, where you live, whether you speak English and how well when you speak other languages. What's your ethnic background? Who are you related to? This is all very sensitive data.

So it needs to be protected and kept private and well secured. Now, most organisations that collect this data have spent a lot of time making sure that data is very well protected. We have specific acts that have been passed in parliament that legally protect the data that is retained. So the the Australian Bureau of Statistics is not allowed on penalty of all sorts of legal consequences. They're not allowed to provide personally identifiable information to anybody. The government can't request it.

ASIO can't request it. You know, NSA, all of the TLA agencies, they're not allowed to look at this data. However, this time around and around December of last year, the census or the the Australian statistician, the guy who's at the head of the census, decided to retain Australians names and addresses information. Now, that's that's not super unusual. Well, the retention is unusual. It is collected. So names and addresses is collected so that they know that people have filled it in.

And B, you do need to know where people are so that you can understand how many people live in a particular area. So things like it's in a representative democracy. You need to know how many people live there so that you can draw the boundaries of who votes by when you're represented by a certain number of population. So House of Representatives, Congress in the United States, similar principle. But after processing, so there's a period of about 15 to 18 months where they need to scan the forms and collect it and do a lot of data cleansing and making sure the data is in good shape.

Anyone who's had to deal with large amounts of data would understand what an interesting problem that is. Through that process, the names and so on are kept for that initial processing. But once that's done and the statistics are calculated, it's all aggregate data. You don't need names and addresses. The personally identifiable stuff gets thrown away. The forms would be shredded. Any of the data would get deleted. Those columns don't exist anymore. This time around, however, the Bureau has kind of decided somewhat unilaterally that no, actually, we're going to hang on to that information and we're going to use it to link to other data sets.

Data sets like medical records, tax information, and other information that's held by other parts of the government. The ABS can't send the information across to them, so they can't say, well, we'll send census data to the tax office, for example, or the IRS. Right. But we can ask the tax office to send us their information and then combine it with our information to create what they call new products. So new statistical products, which is basically just a new collection of data cubes that people can then use for doing either research or marketing information.

So let me understand this, because, you know, stats and anonymized data, because most of this data, they collect so much of it that you can, if not make direct correlations, you can make rough correlations between anonymized IRS data, in our case, IRS data and census data to, if not get exact, you know, person to person, person to tax it, taxes records. You can you can get enough to make it useful for services information. So you can say, you know what, based on the correlation between census data and tax data for this particular area within a community, we can we can understand that the the financial demographics of a certain race.

You don't need PII to do that, I don't think. Kind of. So the the issue here is that the process of linking between two different data sets and it's called linking is done by matching up identifies in one set of data with identifies in another so that you can make sure that you so the characteristics of either groups need to match because otherwise you're linking the wrong things. It's much easier to do if you have a common key. People have done databases will know it's the it's kind of a foreign key concept.

So what the statistic, what the Bureau was planning to do was create what's called statistical linking keys. Now, previously, they've done it using what you described, which is probably sorry, probabilistic methods where they can say, look, we've linked these records using a system of probability and using some complicated mathematics to be able to say, well, look, that person in this set of data is probably the same person over here in the other data where 70% confident and based on how your confidence interval you can get a better or worse match.

If you're if you set the confidence level too low, your data doesn't really match that well. So you can't say as much about what's coming out. And if you set it too high, you don't get any matches. If you've got name and address, though, people's name and address tends to be the same in different data sets. So it gives you a way of creating a primary key or a primary and secondary key to be able to link those two data sets together.

So they're called statistical linkage keys. And what they do is they take part of your name, part of your first name, last name, date of birth, sex, and where you live, things like postcode and even down to street address. That gives you much better idea that the record in data set A is the same person as in data set B. So the census data in set A is the same as this tax record in set B. So now I can link information that's in the census that that sorry, information that isn't in the census with information that is in the tax records.

And I can say, well, OK, well, that's your income. Well, now I know how much tax you paid over different years and this sort of information. Now, that's fine, but that's not what the information was originally collected for. And this is one area that people have issues with, particularly when it comes to things like Medicare. So we have a Medicare system. We have socialised medicine, a bit like what has been recently introduced into the US. And there's a lot of services.

So when you go and buy prescription medicines and when you go and see a doctor, that information is stored against your Medicare account. That's different to the census. But now the Bureau is proposing that, well, actually what we're going to do is we're going to take that Medicare information that you've provided to us and we're going to link it with the census. We didn't actually ask your permission to do that. And the census is compulsory. You don't get a choice.

You have to do this. And that's one one aspect of things that people have a real concern with. And any university ethics committee will tell you that informed consent is vital when you're doing any kind of research on humans. So this is statistical research and consent is not being sought. So that's one of the issues that people have raised. So that I think here in the US we would be having a problem with census data tied to what we call HIPAA or medical records for the very same reasons.

One, we don't trust anyone here anymore. Whether it's nefarious use by third parties, by the government itself, by insurers, they can't deny us insurance anymore. But they can definitely make us pay more than what we would like to. So those are the social challenges. What have been the technology challenges? And that's really a vital issue. So there are these specific issues that people were raising concerns about. And the retention of names is one. The security of that data also came up.

It's like, well, you're keeping this data for longer. We've all seen news of breaches. There was the Office of Personnel Management in the US that was breached. There have been people who have misused Bureau of Statistics data here in Australia. It wasn't census data, but it was other data they had. We had one of the ex-analysts was actually sent to prison for supplying unpublished economic data as part of an insider trading scam that he was running with a friend at a bank.

So there is a risk that if the information is kept, it may be exposed. And that's true of any system. So information security is like, well, if I don't collect the data, then it can't be exposed. But the fact that I've now got it, and it's got name and address, and it's personally identifiable down to the individual, well, that makes it a very attractive target for someone to try to break into and steal. So that was a concern that was raised.

But it really does come down to trust, as you mentioned. The validity of the data requires people to trust that it will be looked after. Because if you didn't trust the census, if you didn't trust the people looking after this information, you would either not complete it, or you'd leave stuff out, or you'd lie. And if enough people do that, then that breaks the statistics, because now you can't trust the information that you've received. So trust is a very, very important part of any country running a census.

I note that the United States Statistical Bureau supplied information to the Department of Homeland Security in, I think it was about 2010, might have been 2004, not too long ago. It actually supplied census information, including names and addresses of Muslims, to the Department of Homeland Security. That's really bad. And that's one of the kinds of concerns that people might have. If you do that kind of thing, you break trust. So the issue that we've now got with the Bureau and the way that they've handled the census is that it has eroded that trust.

And from the way the technical performance of the online version of it worked last night and yesterday, that trust has, well, I don't know if it's been completely destroyed, but it's certainly taken a very, very major blow. So basically logic would tell us, we're tech guys at heart, is that if a system doesn't scale to be able to handle requests, then we start to question the overall architecture of the underlying system, including the security. Is that basically the concern? Yeah, pretty much.

And so to summarise, basically the site fell over under load yesterday. Now, the reason for that is not entirely clear because we're getting some... Again, the way the PR and comms has been handled for this whole census thing has been a debacle from start to finish. It's going to be a great case study for anyone doing PR and comms in all sorts of places. So there are questions about the credibility of the information we're being fed. But the story that they've come out with is that they suffered a denial of service attack.

On the one hand, they're saying that it was coming from the US. And then on the other side, they're saying that actually we blocked a whole bunch of traffic from the US. They're also saying that it was a confluence of different failures. So there was a denial of service and they had a router overload and fail because of some false positives that were being triggered. Now, that's kind of worse because a denial of service attack is not something that shouldn't be anticipated with some sort of system of this scale.

It was being implemented by IBM and at least part of it was being handled in SoftLayer, their cloud stuff. You would assume that they are familiar with denial of service attacks and would have measures in place to defend against them. So even if they did, they weren't effective. That's not good. So there's all of these credibility issues around, well, if you can't do that, if you can't defend against what is a pretty standard thing and the system fails under load, how can we trust you to keep our information secure in the computers that you've said, no, no, don't worry about it.

It's totally secure. So while they are different issues, it goes to credibility. And again, it comes back to trust. Yeah, so I can see why there's problems or at least PR problems, at least. Let's bring this back to kind of the podcast theme. What's what's the I think big lessons learned from a enterprise perspective as you look at this? S. government projects, you change the use case a little bit. This will look like a lot of enterprise IT projects, some of the same feelings.

What's I think what's some big lessons learned for any executive looking to implement some new? And basically, I think the government had good intentions, but I think they missed a couple of key points. And what are those kind of highlights for you? Yeah, so there's there's kind of three three parts to this. So one is the purely technical side of things, which is that you should plan for failure. It's easy to say if you cheap out on stuff, which I suspect is one of the reasons why this had issues.

If you go for the lowest possible price, you may compromise the ability to provide service. So in today's world, you can't just take a system offline if it's not performing well. If it's a major public facing service like this, you need to have contingencies in place for how you degrade gracefully. If there are problems, because there will be problems, you need to be do something like a premortem, which is to go through. OK, what are all the ways in which this could break?

How are we going to mitigate against it? And don't allow if you're an executive and you have IT people talking to you, don't allow them to just hand wave it away with, you know, I'll don't worry about that or give you some complex jargon. They should be able to very clearly and concisely articulate how they guard against a lot of different failure modes. And when they're discussing risk, they should be speaking both about likelihood and impact. It's just something saying something that is very unlikely to occur.

Well, that's not good enough. The, you know, the global financial crisis. I'm not sure what you call it in the United States. That was very unlikely. It still happened and the consequences were massive. So not putting any effort into something which is unlikely but has outsized risk, that's just stupid. So that's one thread of things. Second one is around how you actually manage public perception. PR and comms is important and how you manage things and how you communicate to people is really, really important.

That's one of the major failings in the way the ABS has handled this is they were arrogant and dismissive and that can be quite common with government agencies. If you treat people in that way and then there's any kind of failure, then people will jump all over you gleefully. You don't build a lot of goodwill that you can then draw down on later if you have issues. If you're changing things, particularly when it comes to people's privacy or anything security and IT related, people don't know much about it.

So if there's an information vacuum there, people will fill it with fear and doubt. So you need to be proactive about communicating, again, clearly and concisely but in detail about exactly what you're doing to keep people safe and how it works rather than just being condescending and talking down to people and just saying, oh, don't you worry, you're pretty ahead about that. That's kind of the second thing. And I guess the third part is really understanding that the whole world is different now.

You can't just plow through assuming all will be well. There are a lot of complexities involved here and you need to be across them. IT is a vital business. It's a vital part of the functional part of the business now. So you need to be at least familiar with how it works because it will come and bite you. There will be a failure at some point and you may need to front the media to explain what's going on with that.

If you don't sound credible, then it might cause a whole bunch of issues for you. So one of the things I'm most definitely going to provide as a note in the – or at least a link in the show notes is that you wrote – this is maybe about six months ago. You wrote a pretty good post on designing for failure, if I remember correctly, maybe about six months ago. I'll make sure to provide that in the link. I refer to that an awful lot.

So with that, I've gotten a quick education on the Australian census process and how it has been historically and kind of the challenges that's going on now. I think I'll follow it a bit closely now because I'm extremely intrigued on how the government pulls this out. Any parting thoughts? Yeah, I guess one thing I was recommending to people on Twitter is there's a book that was recently released by folks at Google called Site Reliability Engineering at Google, which basically goes through some of the principles for how they design their internal services to scale, basically.

It's surprisingly detailed about the internal functions within Google, and I have some contacts within Google, and we were all laughing privately and predicting when exactly this whole site would fall over last night, and I won. m. It's a great book. It really does go through a lot of very principle-based things rather than just talking about specific technologies. There's lots of great stuff in there about how systems fail and the ways in which you can mitigate that, and it's quite pragmatic. So it doesn't say thou shalt do X and this is the one true way.

It is really all about tradeoffs and understanding what you're doing. So if you are in any way responsible for running large-scale systems, I highly recommend reading the book. It really does give you a good grounding in how you design systems to manage scale and how to degrade gracefully. That will most definitely be in the show notes. Well, Justin, I appreciate you popping on again, especially at last minute. Where can folks find you online to stalk you, basically? Yeah, sure.

If you want to have a chat with me about this or any other issues, I can be found on Twitter most of the time. com. com and some other publications around the place. All right. Happy to get in touch. All right. And for me, again, you can find me on Twitter at CTO Advisor. The website is the CTO Advisor dot com. And please leave a comment on the Facebook page. The CTO Advisor like us on iTunes.

If you don't like us, don't say anything. That would be great. You can send you can send send me a note on Twitter and let me know how I can improve. Thanks a lot. And we'll talk to you guys later.