Replay-Ethical AI with Swathi Young

Machine-generated from the episode audio and not hand-corrected, so names and technical terms may be imperfect. The audio is authoritative.

Hey, you're listening to another episode of the CTO Advisor. My voice is sounding a little raspy. It's because I have a CTO on the healthcare service provider, Swarthee Young, and she's got me up super early in the morning, Swarthee, but I think it's going to be worth it. How you doing? Doing great, Keith. How are you today? I'm doing pretty good. So Swarthee, I met you at a book signing for Ray Wayne's book, Digital Giants. It was a ...

I bought the book, I've read some of the book, I'm notorious about buying books and reading, not the whole book. It's not that Ray's book isn't good, it's very good, it's just, it's me, it's not him. Have you read the book yet? I'm just like you, I have. You can see the collection of books at the back, and I have so many books next to me, so I think it's a very impressive book. I've been sort of browsing through, but I have it on my list of holiday book reading to finish.

And if you ever met Ray Wayne, he's kind of a very active personality, and if you see him present in person, you're like, oh, I just got the whole book right there at the presentation of the book signing, so I feel like I don't need it. But I will read it, I'm looking, I think the audio version is out, I'll put a link in the show notes, and I think I'll listen to it on the treadmill. But Swarthee, you're an author in your own right, and you're an ethical AI expert.

You have, you've co-authored a book already, and you have a book in the works. Tell me about the two pieces of work. Absolutely. I would say a minor edit. C. gave me the honor and opportunity to co-author the ethical AI framework that any stakeholders in federal, state, or local governments can use. So my journey into this started because a couple of years back, I started implementing AI ML solutions, and one of the things that I was noticing what was missing is that whole human-centric design of sitting with the users and trying to understand what is the current process, and what are some, you know, what if scenarios, what if this recommendation engine goes wrong, you know, who is liable, and so on.

So as part of co-authoring the ethical AI framework, I very much focused on the bias section, which as a data scientist and leading a team of data scientists, I'm very much aware of because there is nothing called unbiased data. So that led to me getting very involved in this AI framework, ethical AI framework for the government that was published end of 2020, actually. It was a good one-year effort by, like I said, 30-plus industry, academia, and government leaders involved in it, but I was crucial and a leading author of it, especially in the bias section.

And that further led me to opportunities to speak about the various aspects of ethical AI, whether it is bias, responsibility, social trust, and transparency. These are all the crucial considerations when it comes to ethical evaluation of AI or even machine learning algorithms. So that made me early on come up with the concept called NODES, which is an acronym to various factors that go into considering the ethical AI. So NODES, in a nutshell, is what are the needs and objectives and data, ethics, and social trust.

So I had the good fortune of presenting this framework to Manning Publications. So I now have a book contract with them, and I'm in the process of writing it and hopefully will release the book with Manning in the summer of 2022. So NODES, I love the acronym. Very good. It's much easier to remember than the specifics. And I think this is one of those areas in technology that we look at and we say, OK, we need to understand ethics and AI, but I also need to get my job done.

And there's a bunch of things that kind of scare us as a society when we think about ethics and AI and machine learning. I think it's helpful to have a framework. So let's structure the rest of the conversation around just some of those concerns that CTOs, architects should consider when they don't have a dedicated ethics AI team, like some of the big cloud providers, et cetera. So I was at AWS reInvent. Were you at reInvent? I unfortunately was not there in person.

I attended some sessions virtually. So I was at this session and or actually it was just a conversation, hallway track conversation. And someone from Amazon was talking and saying that they have their Amazon show thing that can see what you have on and then make recommendations or whatever about clothing, et cetera. And one of the engineers, they were relaying the story of how one of the engineers said, oh, and then we can see what's in their closet and then recommend other things that's in their closet as they're showing something on.

And the guy relaying the story was like, we all in the room had to hit the brakes and say, no, that's not ethical. So let's talk about, you know, kind of. Engineers want to be engineers and they want to solve problems, but where where do we go wrong to begin with? And that's a great question, and the reason that I started my framework, if you notice with nodes, you start with N and O, the needs and objectives. So as any engineer, what they're solved like I am an engineer, a career technologist, I'm very excited about bright, shiny objects and tools like you must be.

And we want to get our heads down, write the code. But what I saw missing when I worked on projects like, for example, using deep learning for lung cancer research was one of the projects I advised and led the data science team is not spending enough time up front about the needs and objectives. We are spending more time getting quickly into the data analysis and modeling and discovering data instead of having the conversations that engineers are not the best in making those decisions or even understanding things like what are the ill effects if the decision is wrong?

What are some of the existing rules and regulations? What compliance issues can arise, especially if you're talking about health care, finance and other regulated sectors? So I want to emphasize a lot on the needs and objectives, which is a conversation with multiple disciplinary teams and this multidisciplinary team, if it's a small sized or mid sized organization and don't have an ethicist, they still can have a conversation with their legal, with their HR, with some of the other executives who know the compliance officers and try to understand what if scenarios, the focus on the what if scenario.

So let me give two simple examples. One is a person is applying for a business loan and the financial institution has has discovered that the traditional credit risk models they were using was not enough. They still had a lot of defaults on the loans. So they had a hypothesis that if we use machine learning, we can use the applicant's social media data, their business news data, their location data, and we can use this machine learning to make a recommendation about the risk of this applicant defaulting the loan.

So on the onset, OK, you have the social media data with all the API. So as a technologist, you are ready to go. But wait a second, have a conversation with your credit risk officers, financial institutions have compliance officers have a conversation and say, OK, if if the machine learning algorithm gives a recommendation that this applicant, based on the zip code they are living in, has high chances of defaulting the loan. So we reject the loan and you send a letter. So but there is rules and regulations around rejecting about the reasons for rejecting.

Now, when the data scientists get in the room with the compliance officers, when they will understand, oh, there is a rules and regulations that we need to follow. So it's very important for our machine learning algorithm to be transparent to the extent to understand what are the factors that went into decision making, what is the weightage of those factors. So that's just a simple example. But what I'm emphasizing here is the need to have that multidisciplinary conversation to have the scenarios. And ultimately, maybe the algorithm is recommending a certain zip code because of the demography there, because of the population who reside in the zip code might have lower incomes and thereby a large sector of the population is adversely impacted.

Right. So I would say upfront that conversation and discuss in depth before you write a single piece of your code. So the one of the good pieces of advice that I was given when I worked in supporting the government sector was I had a manager who told me two things. One, Keith, if ethics ever call if the if ethics if you ever get a call from ethics lawyer up, that's one. I'm like, OK, that's an interesting piece of advice. But I have stories and for people who want to know, we'll have those stories offline.

But the other piece of advice that they gave, it was pretty common amongst the organization was. If you do something, if you create something or do something and it landed on the front page of The Wall Street Journal or the first front page of The New York Times, is it something you would be embarrassed about or it was something that would embarrass the company? How transparent do you as you're giving me the example, how should you think about transparency when you're thinking about the the approaches you're taking to AI and email?

No, that's a great question, and just one comment I would make is like, yes, I think that's a phrase we've all heard. We don't want to get in the newspaper for the wrong reasons. But the observation as a data scientist and seeing all the all the machine learning algorithm in practice, they're impacting us in a smaller way. It's like the cycle is churning already in small sectors without the public knowing or getting into newspapers. So that is what is scary, even scarier than, OK, a big incident happened and it got to the newspaper.

But what's happening is smaller things are happening all around. And and, you know, policy is not caught up. Policy has not caught up. We are in a world where a giant social media CEO has asked, what is your business model in the Capitol Hill? Right. So so the policymakers haven't caught up. So we, I think, as ethicists and all the data science community who are talking about ethics are coming together and trying to say, hey, there is a risk in not evaluating the ethical considerations and trying to highlight this.

So I just want to put it out there. Specific. OK, go ahead. No. So I think you're naturally going to a place I would love to talk about next, which is, OK, as an engineer, there's, you know, an engineering lifecycle. And as an engineer, you know, I put this into the requirements gathering of what are the boundaries of the system that I'm going to create. But I think, you know, and you've said this in so many words, but AI and ML is kind of squishy from a system design perspective.

You're trying to make sense of data in ways that we've never been able to do before. R. and legal and, you know, I'm ready to go and I start the project and I'm getting great information. What are some of the the the safety checks I can supply to make sure that I don't get that float? Right. No, that's a great question. And I'll address your transparency question as part of that as well. So having come from software engineering background, I understand traditional software development lifecycle.

Now, data science lifecycle is very different. It starts with data and most of data science projects are predictive. Right. Based on historical data, how can you predict the future? Whether you're talking about autonomous vehicles have imbibing lots of images and recognizing an image when it comes across or whether you're talking about historical financial data of a loan applicant. Right. Either ways. So data is the starting point of the of the data science project. And there are multiple steps.

So you have the majority of data science projects to write a machine learning algorithm and 60 percent of data analysis, what you call feature engineering, data imputation, et cetera, et cetera. And then you get into your machine learning modeling. And the uniqueness about data science projects is there are a lot of assumptions that go into making when the data analysis is being done. So a simple example, if you have seven million records and five million records had gender data and two million records did not have some approaches is that the two million records might be dropped depending on is the gender data element crucial to your outcome or not.

Or sometimes they they imputate the data, which means they assume based on statistical analysis, what is that gender type and they will assign a value. Right. So so a lot of assumptions are being made during the data analysis phase, which directly will impact your model output and outcome. So that's one. So every assumption that you're making, you need to go back and ask the question. Right. So if it is gender and you're coming up with imputating the values for the two million missing records, is it is it actually statistically accurate to the demographic of the population?

Right. So you need to think deep and not just do that decision. And most data scientists are in the hurry to get to the algorithm. So they make some assumptions and every step of the way they have to go back to the domain expert. So, for example, when I was working in the lung cancer research project, we had to ask the lung cancer surgeon what the data element value means and why is it relevant to the outcome? Because we are data scientists.

We don't know. And even though the surgeon had very little time, we had to make sure he gives us that input. And so I would recommend every step of the way when you make that data analysis and assumptions and imputations, you need to go back and think through and speak with your subject matter experts. The second thing is coming to algorithms. You asked a question about transparency. Right. So in the world of machine learning, the algorithms deciding which algorithm to pick, whether it is a decision tree or the TensorFlow or the XGBoost, depends on multiple factors like your false negatives, false positives, your precision and accuracy.

But apart from that, the transparency should be one of the considerations. And sometimes it's a tradeoff between if the algorithms are more transparent, maybe random forest or a decision tree is a better one, although we might have lesser accuracy. So those are the type of conversations we need to have more. I know there are some set of data scientists who are very, very eager to increase the accuracy. Right. But these are the factors if you want to have the transparency.

And and then there are a whole lot of black box and white box modeling and more research papers have come out about transparency techniques. But end of the day, to your point about lawyer, if you're taken to a court of law, can a data scientist stand in the court of law and say with a high degree of confidence, what are the weightage of the factors that went into this decision making? This need not be jargon. It could be these 10 factors were considered by the algorithm and these five factors were the most important ones, maybe geographical location, maybe a demographic.

So if they could, to that extent, offer an explanation with high degree of confidence, that will increase your transparency. Right. Of of saying, OK, based on these weightage factors, this recommendation seems to be appropriate. Well, this is so complicated from a just. Ethics perspective, like there's there from a technology perspective, it's pretty complicated from a data perspective, you know, their causation versus correlation, you know, the you know, we look at the demographics and you look at zip code as one factor.

So zip zip code isn't the causation. Zip code is just, you know, correlation. What how do I get to the issue of the data? Then there's kind of the politics of when you come out of the you've gotten what could be pretty accurate things. It could be it could be very accurate that there's a correlation between the number of people who default in their race. Doesn't mean that that's the cause, but that's a correlation and it is definitely a red line.

So from but there's all kinds of things in data and data analysis that have similar impacts. And whether you rate it, you know, point ten percent of a tenth of a percent in your algorithm or however you weighted, the fact that it's there can cause political harm to your company, et cetera. So if people want to stay out of these types of harms, how do they follow you on social media to get kind of your your the gist of what you say? How do they find your content?

So most of my content is curated on LinkedIn and I do cross post on LinkedIn, any medium blogs that I do. I also blog on Forbes, but I would say if you have to follow one place, it would be my LinkedIn. I do have my YouTube channel, which is on a hiatus and I I hope to revive it in 2022. But everything is cross posted on LinkedIn. So LinkedIn would be a great place where you can follow my blogs, my video content and any updates.

I also update about new tools, new research papers that are coming up and so on and so forth. So that's one stop shop I would recommend. So Swarthee, thank you so much for sharing your knowledge. I know that when you're down in the weeds and you're solving these problems, it's really hard to come up for air. Gather your thoughts, write them down and share them so that we all learn from them, and I so much appreciate your contribution to the community around this extremely important concept that basically, you know, it is more or less brand new for most of us in the entire industry at large.

If you want to learn more about the CTO Advisor, you can follow us on the web. com is the website, the in the front, CTO Advisor. com website that I cannot get the rights to. And if you want to follow me on Twitter is at CTO Advisor. And I'm pretty prolific on LinkedIn, too, so follow me on LinkedIn. It's another great place to have long form conversations. Make sure to subscribe and rate the podcast. Until then, we'll talk to you next podcast.

If this is reaching you before the holidays, have a great holidays. If it's reaching you after the holidays, have a great new year. See you soon. Thank you so much, Keith.