Primer on Reinforcement Learning for General AI - Dr. GP Pulipaka

Machine-generated from the episode audio and not hand-corrected, so names and technical terms may be imperfect. The audio is authoritative.

Hey, welcome again to the CTO advisors coverage of SAP Sapphire 2019. We have a treat for you. We're going to go on to a topic that we don't normally go deeply into, ML machine learning with GP. GP, you're a data scientist at Accenture, right? Yes. Really appreciate you coming on to the show. You have an upcoming book on ML AI. Give us the overall impression of the show, kind of what you what your research has been and how it relates to the conference.

Sure, deep reinforcement learning is the most advanced machine learning technique on the landscape of artificial intelligence and it has been revolutionizing several corporations. Basically, they're all looking to gear towards AGI, the artificial general intelligence. I think that's where I'm writing up a book on this to cover all the algorithms on reinforcement learning and how it benefits the corporations. Like, corporations are investing into robotics, Internet of Things, cloud computing, industrial Internet of Things. All of these technologies are fueled and powered by reinforcement learning.

In the recent times, MIT Technology Review has performed a research to see what are the latest bleeding-edge machine learning trends are looking like. And what they found was that there are around 16,625 research papers trending from 1993 to 2018. And the trend shows that actually the reinforcement learning technique is completely off the charts. It's on the top of the technology right now. The Brain Institute in the past decade has invested more than a billion dollars to understand how the neural networks operate in the brain, how the consciousness powers the coordination of all the neurons in the human brain.

Because there are around 87 billion neurons in the human brain, when a decision emerges, it's so synchronous and it's so simultaneous, and it just goes through all the neural paths and goes through the synopsis and sends the signals out and then connects that and it provides a decision. It could be so many decisions, but in the end, the person chooses a particular decision for going forward with implementation of an action. The reinforcement learning is definitely based on the same architecture. Basically, what it does, for a robot to basically, for example, to come out of a maze, there could be several paths and there could be several states, and the robot has to perform a specific action to come out of the maze.

And the reinforcement learning does that, basically. It goes through multiple episodic training of the algorithms and it produces the action that is required in a most optimal path to come out of a maze. There could be several environments like that, like OpenAI Gym. I'm covering a book on this topic with a hands-on demo of all of these algorithms, how they can be utilized. It can be adopted by several corporations where they would like to basically build a data science team. They can just tap into this book and go through the recipes, train their data scientists, and come up with an organizational structure, how it can be implemented for their corporations.

The book is explained, is actually in ten chapters. Basically, it has like a trusted region optimization, trusted region policy optimization algorithm, cross-entropy algorithm, policy gradients, deep Q-learning. Several algorithms like this is on the reinforcement learning, mainly on the deep neural networks. As I was mentioning earlier, that's where the AGI is geared towards because once we understand how all of these neurons operate together, compute that into a neural network, then we can find out the paths for these neurons and how does that decision emerge from the neural paths.

And that's where this leads to the AGI. Other than the OpenAI Gym I was talking about, I leverage a lot of OpenAI Gym environments for the implementation of this reinforcement learning algorithms. Also, DeepMind has come up with their own lab environments for reinforcement learning. There are a few other companies who are also implementing this reinforcement learning for the artificial intelligence. So let's talk about like the current state of the technology. So there's neural networks and leveraging neural networks and this reinforcement learning technique.

How can companies apply that to AI ML today and some of their decision processes? So most of the artificial intelligence machine learning techniques that the corporations apply are based on the historical data sets. So what they do is they take a lot of historical data and then the machine learning algorithms simply learn from the data and then they produce the outcomes in terms of like predictive analytics or advanced financial forecasting. Those type of applications corporations are building. So today, you know, Intel gave us that a couple of weeks ago, a few weeks ago, at their data centric event and said that only 2% of the world's data has been analyzed that has been generated in the past five years.

So that's a tremendous amount of data. But the way that we're doing ML and AI today, we take that data, see patterns in that data, right? And then we can make decisions or predictions based on that data today. That's a pretty common use of AI. However, what you're sounds like what you're talking about for reinforcement reinforcement AI is a little bit more free world. Exactly. Yeah. So because it does not require any data at all, right? It's just like a human decision.

Let's say a car has to go onto a mountain. And then it's challenging because of the uphill task. Either the person has to turtle or go in the right direction or left direction to balance out the car and reach to the top of the mountain. And that doesn't have any... You've not seen the mountain before. Exactly. You're kind of as a human, I kind of intuitively know exactly how to navigate the mountain. Exactly. So teaching neural networks how to do that same thing.

Right. Is the goal. What's the key indicators that we're starting to know that we're making ground? We have seen some significant improvements and achievement, like on the chess, where like the machine has totally has beaten the humans in terms of the chess. We also have seen a few other, that was AlphaGo, and a few other instances where some of the gaming environments also have shown significant improvements in terms of operating it through artificial intelligence, applying the reinforcement learning. Mojako environment is a gaming environment.

So there are a few other environments where we have seen that kind of a pattern where the reinforcement learning completely operates on its own, does not have any historical data set. It basically goes through episodic iterations. For each iteration, it gets some rewards for doing it right. Right. And then there will be a discount factor on the reinforcement learning, how much has been the reward versus if there is any loss. There are also value approximation functions that can calculate the reward.

And then finally, when we get the output, it's basically an action to be performed. And that's what is needed to accomplish the AGI, because most of the time we see like autonomous vehicles operating on the road. And then again, there are pedestrians crossing the road, and it does not know whether to save the passenger or to save the pedestrian. Right. That is an action that it has to make instantly, right on the spot. It's not like something it has to have some data for it.

It's a dynamic decision. And humans struggle with some of these decisions ourselves. Exactly. The ideal of concept of rewarding the algorithm for making certain decisions, you know, I have to imagine a lot of the difficulty is in, to your point, is it is AI making the decision based on the reward? Or because some other factors? Exactly. Kind of getting those knobs, right? Yes. So that's what the reinforcement learning does. It doesn't have any data, it just has to perform a dynamic action in a number of states.

And there could be so many states of it. Like, for example, when it's navigating through a mountain, or it's navigating through a road, there are several states for it. And again, some of the objectives have been fulfilled through like sensors like LIDAR, for example, laser light range detection technique, or radar sound technique, where it senses the movement of people around the car, and then basically finds that these objects are there, and then kind of navigates through that. And this was actually implemented on Mars by NASA before, for the Mars rover vehicles, and the rover, the way they calculated is because of finding the motion of an object and is detected by the vehicle, and then it can apply some SLAM algorithm, and then it can navigate through that.

So these type of decisions are like, for example, for any shuttle to land on Mars, it doesn't have a prior, it's Mars, it's different environment altogether. I mean, you cannot predict like how it's going to land there, or how it is going to navigate there. And we don't have a specific navigational map, or how does the whole thing functions in terms of like, we don't have a GPS or anything. And because of latency, we can't put a human remotely to navigate. So there has to be some type of AI to navigate and make decisions.

Exactly, exactly. So that's where reinforcement learning performs all of these dynamic actions in the combination of state and action. It exactly figures out for what state what action needs to be performed. That way, it can just operate completely independently, navigating on the Mars or any other planets. This will propel into a lot of other applications NASA is building, for example, like they're building solar sails, they're building laser sails, which is the future of data science, that actually can push us to the next level of artificial general intelligence.

So that's where the reinforcement learning can be significantly be applied for some radical advancements in terms of future of data science. And that's what I'm trying to do in this book. Basically, taking multiple environments, it could be a gaming environment, it could like a mountain car, it could be acrobat, different types of gaming environments, and or even if it could be a simple traffic problem. We're like, basically, we are, there are cars that are going on the road, and then there's always problem in the traffic.

And but there is so much of space, it provides the insights for the cars, find the most optimal path to go through those routes and spaces and kind of come out of the traffic situation, which currently we cannot, we cannot envision in terms of like, oh, how the traffic is going to look like for next few miles. MIT Institute has applied this reinforcement learning algorithm on this to resolve the traffic problem. And that's pretty fascinating in terms of finding the optimal path is the most difficult and complex and challenging task for AI.

So these type of applications is where reinforcement learning can be applied in everyday life. And that could help us solve several environments. So as I was explaining about different algorithms for reinforcement learning, like trust policy, region, optimization algorithm, and policy gradients. So there are several recipes that I'm developing for the book. 0, deep reinforcement learning, and it will cover it will be there will be like 100 recipes in it. And each chapter will explain a snippet of code, how to implement these algorithms and methodology for most optimal path and optimal strategy to resolve and solve the environment.

And when do you expect the book to be released? Well, we were actually supposed it was supposed to be released in May. That's how books go. Yeah, right. I could only complete like three chapters. So far, we are expected to complete like 10 chapters. Okay. And so I mean, I haven't completed all the rest of the seven chapters yet. That's okay. So are you do you participate on social media, Twitter? Yeah. What's your Twitter handle?

My Twitter handle is a GP underscore PUL IP aka. And I share lots and lots of content on reinforcement learning and different machine learning techniques, how these have been deployed by different companies on the production landscapes, how it benefits and what are the most optimized ways of doing it, some resources for people to pick up and commercial project that they would like to implement, or it could be just, you know, data science enthusiasts, who just want to do some research, research work, perform some research on it.

So I share all of these articles and also get feedback from different different people on these topics. And which is pretty good, because it helps me to contribute back to the society. Yeah, exactly. So GP, I really appreciate you coming on to the CTO dose. com. You can follow me on Twitter at CTO advisor, talk to you next CTO dose.