Why can’t a computer be more like a brain? (July 27, 2009)
Jeff Hawkins, author of On Intelligence with Sandra Blakeslee, presented the Oppenheimer lecture in Los Alamos, NM, on July 27, 2009. Read an edited transcript below, or watch clips from the lecture here. Visit the J. Robert Oppenheimer Memorial Committee’s website, and Jeff Hawkins’ Numenta.com site. Posted with permission of the J. Robert Oppenheimer Memorial Committee.
First of all, I want to thank the J. Robert Oppenheimer Memorial Committee for inviting me, and Gar [Garrett Kenyon, Chair of the J. Robert Oppenheimer Memorial Committee], who invited me over a year ago. My family and I had been visiting Las Vegas and we had just visited the atomic testing museum, which is really cool if you haven’t been there. I was in the middle of reading this biography about Oppenheimer when Gar contacted me and invited me to give this Oppenheimer talk. So thank you, Gar, for bringing me here, it’s been a great experience.
It’s an honor to help remember someone so important as Robert Oppenheimer. I’m not sure I’m up to the task, but I’ll do my best. There’s an obvious problem, which is that Oppenheimer was born in 1904, I believe, and I was born 53 years later. So we weren’t peers, in any sense of the word. I can’t tell you any personal experiences I had with Robert Oppenheimer. So I’m going to try to do two things. One is, I’m going to try to relate my work in theoretical neuroscience, which is what Gar does as well, to what was going on in theoretical physics back about a hundred years ago.
But I also want to try something else, too, so I actually have here … here’s my title slide on this talk, “Why can’t a computer be more like a brain?” I went back and looked into my past and determined that my life overlapped for ten years with Oppenheimer’s. I was ten years old when he died. So I thought, since we were on the earth doing something at the same time, I wonder if there was any kind of connection there. So I went back in my family archives and I found a couple of very interesting things I’m going to show you.
But before I do I just want to point out this famous picture of Robert Oppenheimer. I love this picture. I don’t know if it really looks like him or not, but what I like about it is that he looks like a cross between Indiana Jones and James Dean. And for a scientist that’s pretty good. He’s looking you straight in the eye, as if saying, “Boy, I know what I’m going to do and if you’re in my way you better get out of it.” I heard that he was not a particularly open and emotional person. I read that, maybe that’s not an accurate portrayal. But think about that when you see the next picture.
Now I want to show you two pictures from my past, when I was about seven or eight years old. That’s me on the right and my two brothers, and we’re obviously doing some sort of physics experiment. I want you to look at our facial expressions. We were definitely emulating that guy Oppenheimer back then. I mean, we were in this for serious work there. My older brother Jim is kind of hidden behind the veil, we’re not sure what he’s doing. I’ve tried to find some records and recall what this experiment was about since I can’t remember. I couldn’t find any records so it must remain classified I assume. But we’ll see.
The next picture I’m going to show you just occurred a short time after that, and I think it might have been the outcome of that experiment. Here I am again with my brother and we’re designing some sort of advanced weapon system. That’s me in the carriage. I’m not sure what my operational role there is-some kind of monitor or something, maybe I’m a fire control system, I don’t know. Somehow it’s all been lost in the past.
Now let me talk a little bit about what was going on in theoretical physics a hundred years ago and then relate that to what is going on today in theoretical neuroscience.
Back around 1900, all physicists were experimentalists. There were no theoretical physicists. If you were a physicist, you worked in a lab, you built equipment, you tested it, and if you did any theory it was based on your experiments. In fact Robert Oppenheimer did that, too. He started out trying to be an experimental physicist, but he was no good at it, self-admittedly, and he didn’t enjoy it at all. Fortunately for him, it was just around the time when people could earn a basic living and be respected as purely theoretical physicists. Nothing was happening in the United States when he came of age, at that time all interesting physics was happening in Europe. At the time he was just finishing his studies overseas, the United States made a concerted effort to bring physics to this country. Theoretical physics was becoming really important, and he was the right guy in the right place at the right time.
Today there’s a very healthy balance between theoretical physicists and experimental physicists. Neuroscience today is in a similar quandary. I’m going to explain that with my own little story here.
My interest in neuroscience started in 1979. I read the September issue of Scientific American. The September issue of that journal is always dedicated to a single topic. In 1979, that issue was on the brain and they had stories about the neuron and about neurogenesis, about development and diseases, and so on. The last story was written by Francis Crick, of DNA fame. He wrote that all this stuff that people talk about regarding how the brain is built … don’t believe it. They really have no idea how it works. And he was a proper British gentleman so the way he phrased it was – and I remember his words exactly – he said, “What is conspicuously lacking is a theoretical framework for understanding and interpreting these ideas.” What’s conspicuously lacking is a framework. He was saying, there’s no theory here. And I said to myself, then a young man of twenty-two, “What a great thing to work on! We have all this data. We have no theory about how the brain really works. Let’s go do this. This is what I’m going to do for my life.”
I tried to do that at twenty-two. However, I was unable to do it because you couldn’t be a theoretical neuroscientist back then. I’ll tell you a related story … in the mid-80s I was a graduate student at Berkeley, UC-Berkeley. I went there to basically work on brains, and I wrote a thesis proposal for the chairman of the graduate group in neurobiology. And in this thesis proposal I laid out a whole bunch of ideas, how I wanted to attack this problem and work on it. The chairman reviewed it, as well as several other faculty. They got back to me and said, “This is really interesting and really good. It’s a great problem, you have good ideas. You can’t do it.” I said, “What do you mean I can’t do it?” And he said, “You can’t do what you want to do. You have to work for some other professor as a graduate student and there’s nobody doing this at Berkeley. And by the way, there’s nobody doing it anywhere. You have to go work in an experimentalist’s lab.”
I considered that very carefully and decided I wouldn’t be able to do what I wanted to do because taking the experimentalist’s route would take me down the wrong path. So it was a very difficult decision for me then. And even to this day it’s very difficult for people to be theoretical neuroscientists. Now Gar is one of them, which was a surprising discovery on my part. Most people can’t get funding for doing pure theory.
This will change, and this is why, as Gar mentioned, eight years ago, I was sitting around talking with some neuroscience friends of mine. One of them said, “You, know, the best thing you can do, Jeff, is to start a dedicated institute just for theoretical neuroscience.” I said, well that sounds like a crazy idea. Who’s going to be the first one to show up? What, do you just hang a shingle on the street, “Wanted: Neuroscientist”? But they helped me, and it turned out to be very successful. Out of that came the theory I’m going to tell you about, and then we wrote a book, and then I started a company because I want to promote these ideas and get more people working on them in interesting ways.
Now I’m going to get into my talk. The first thing I’m going to do is hang up my little prop here, which is going to be harder to do with one hand on the microphone. I brought a couple of things with me. I have a brain here. This is a plastic brain, not a real brain, and it’s about the size of a young adult brain. When we think about the human brain there’s one part that I’m particularly interested in, and many people are, and it’s the neocortex. In the human brain the neocortex is this wrinkly thing around the outside. It’s about 85% of the volume of the brain. It’s everything that you think about, everything that’s important. The rest of the brain is shoved up inside, like a post, almost like this microphone, and the neocortex is wrapped around it. Now I say it’s the most important thing because it’s where all high level vision occurs, all language occurs, all high-level motor behavior, planning occurs there. Almost anything you can tell me verbally, that you can express and describe about things you need to know about the world is stored in the neocortex. It’s really what makes us unique as a species.
We have a very large neocortex. Other animals don’t. In fact, only mammals have a neocortex and in humans it’s very large. Now I could remove the neocortex from your head, which wouldn’t hurt because there are no nerve endings in your brain. And if I were to iron out the wrinkles in it, it would look like this dinner napkin. It’s about two to three millimeters thick and about a thousand square centimeters. Just like this. It contains roughly thirty billion neurons, and all their connections. And this sheet, which is inside of your head right now, is you. My sheet of cells, my dinner napkin, is speaking right now and yours is listening. This sheet contains just about all of your knowledge of the world. And we want to know, how does this thing work? What does it do, and how does it work? And if we know how it works, I figured a long time ago that we could build machines that work on similar principles and then that would be a really good thing for society. So that’s what I’ve been working on and I’m going to tell you what we’ve learned so far in this process.
Now first I’d like to describe the problem a little bit better. What does a cortex do? In this picture, the box on the left depicts your world, or what you think about your world. It contains things, things like people, cars, buildings, and so on. And on the right we have your brain, the neocortex. And in between we have your senses. We tend to think of the three major senses, the eyes, the ears, and the skin, but we actually have more than that but let’s just think of those three for the moment.
Here’s what your senses do: they don’t actually perceive the objects in the world, they just take these slices of sensory things like light, and sound, and pressure, and they convert them to patterns on firings of nerve fibers, nerve cells. There are about a million nerves on your optic nerve, about a hundred thousand on your auditory nerve, and about a million coming from your spinal cord. Those neurons, those nerve cells, are nearly identical. There’s no difference between the patterns that represent light and the patterns that represent sound. In fact there is no light or sound or touch entering your brain. It is purely patterns, spatial and temporal patterns.
And this is one of the most amazing things: your perception of the world is not really based on light. It’s not really based on and sound and touch and so on … it’s actually just built up from these patterns, pure abstract patterns. This is a fact, this is not speculation. Most neuroscientists have come to realize this and it’s been written about quite a bit. Now we can ask ourselves, what does the brain actually do? The simplest way to say it is that it builds a model of the world. When you’re born, your neocortex is there but it doesn’t know much. It doesn’t know about buildings and cars and people and Oppenheimer and Los Alamos and Santa Fe and all these things. It has to build a model of the world. We tend to use the term causes for structures in the world, such as what is the ultimate cause of the patterns on my retina? A cause is a recurring pattern in the world. And in the right, we want our brain to learn to model those causes, to form representations. We have cells that actually fire and become active when those causes in the world are actually happening out there. So we have a model of causes. This is the language we use when we talk about this.
Your brain, your neocortex does four things. All are very important. The first thing it does is, it discovers the causes in the world. You don’t know them when you’re born, you have to discover them. No one programs your neocortex for you, you have to discover the causes through your interaction with the world. This in itself is a very valuable thing. This is what we as scientists or what everyone does as people, trying to understand the world. Our job is to look at the data and figure out what are the underlying causes-what made this happen?
The second thing that a brain does, once it has a model of the world, is that it can do inference, which is just a fancy word for pattern recognition, so you can see new patterns coming in. In fact, you always have new patterns coming in because you’ll never have the same pattern coming in twice in your life. You can look at me a million times and it will be a different pattern every time because things are moving around and changing, however slightly. So you have these new patterns coming in and you want to recognize that they are based on the model.
The third thing that the brain does is it makes predictions. It has a model, and the model recognizes how things behave. Therefore when I’m observing something, I’m also predicting what’s going to happen next. Now you’re not generally aware of these predictions, but your brain is making them all the time, every moment of your waking life. So because you understand English and you’re listening to me speak, your brain will automatically predict what word is at the end of this … [pause]… sentence. You’re doing this every moment of your waking life. When I put my hand down on this lectern, I’ve never been here and never done exactly the same thing, I have expectations of what it’s going to feel like, even without thinking about it. And if it felt like it was liquid or wet or jello or if my hand passed through it, I would be very surprised. Your brain is constantly making these predictions.
The fourth thing your cortex does is to direct the high level motor control system. So it’s creating behavior. My speech, for example, is coming completely from my neocortex.
So that’s what the brain does. We want to know how it does that. The way we approach this problem is that we use neuroscience, anatomy, and the physiology of the brain as a very strong set of constraints. We’re not just trying to invent intelligent computers, we’re trying to understand how the brain does this. How does the brain do this? So we have to look at the constraints that the brain provides, and then we add theoretical constraints on top of that. So let me just go back to my model here and look at the sheet of cells.
We know a lot about this sheet of cells. It turns out that everywhere you look, the cells look very similar, there’s the same sort of structure everywhere in the neocortical sheet. There are six layers of cells, and they have the same cell types, and so on. But the most notable thing about the neocortical sheet in your head is that it’s divided into regions that do different things. There are areas dealing with vision and language and hearing and touch and so on. These regions are connected together through bundles of nerve fibers. So if Region A connects to B, we know also that B will connect back to A. They’re reciprocal, but they’re asymmetric, meaning that we can tell which is the feed-forward direction and which is the feedback direction. You can, over many years, map out all the regions of a particular mammal’s neocortex, and map out all the region connections between regions. This very difficult work has been done for a series of animals.
Let’s look at the map for the macaque monkey. Before you say, oh my god, what’s this all about, first recognize that this is a famous diagram for neuroscientists. Second, it’s really not that bad, and third, you don’t really need to understand all the details. I’m going to give you some highlights of it. Now you can pretend this is your brain because humans aren’t that different from a macaque monkey. We can’t do these experiments on humans, but this is pretty close to you.
These little rectangles represent the regions of the neocortex, the different regions of that sheet. The lines show how they’re connected together. Every line is a feed-forward and a feedback connection. In this particular drawing, sensory patterns are coming from the bottom, so feed-forward is going up and feedback patterns are coming down. On the right we have regions associated with vision, on the left we have regions associated with motor and sensory behavior, and they’re connected together at the top. So this is the map of the macaque monkey’s neocortex, or at least a good portion of it.
Now already we can say some very interesting things about this. First of all, remember this is a memory structure in some sense. It’s a model of the world, about how the world looks and behaves and so on. So we can say some things about how information is stored in this model. The first thing we can say is that information is stored, or knowledge is stored, in a distributed fashion. If I wanted to ask you, where do you know what a dog looks like, or what President Obama is doing, or whatever … you’re not going to find it in a particular region. It’s going to be distributed throughout, because information has to flow through many, many regions to get anywhere.
If you follow modern neuroscience, you’ll see studies now where they show which brain region lights up when you do different things. These can be very misleading in that they’re only showing the regions that are lighting up the most. When you do anything, many of these regions become active all at once. So information is distributed in a hierarchical fashion. That’s not like we do it in computers.
The second thing we can do is we can say the system is self-training. When you’re born, this structure is there but the knowledge of the individual connections is not. So your brain has to learn, each of these regions has to learn to store something about the world. We can’t program your brain, it has to learn purely through sensory patterns. Even when your mother or your father or your teacher was teaching you things, their teachings still had to go through your sensory organs, there was no direct input into the brain. So it’s a self-training system.
The third thing here is very, very surprising. It was known for many years that these regions of the brain look nearly identical. There are some differences, but these regions have a tremendous amount of commonality in terms of the cell structure and how they’re connected together. So if you looked at one region at the top of the visual hierarchy and another at the bottom of the auditory hierarchy, if you just looked at the details they look almost identical. Thirty years ago, a physiologist named Vernon Mountcastle proposed that, not only did the regions look the same, but they were actually doing the same thing. Each region does the same type of activity. What makes a visual region do visual things is a function of what it’s connected to. An auditory region does auditory because of what it’s connected to.
At first, scientists couldn’t believe this, it seemed impossible. How could the brain solve language the same way it solves vision, the same way it solves somatosensory or touch? But it’s true. There’s an overwhelming preponderance of evidence that suggests this is true. That makes our jobs as theoreticians a lot easier. Now we have a system with a whole bunch of these things that are doing the same thing, arranged in a hierarchy, learning on their own, and so on.
I want to give you a few more details. I’ve highlighted some regions in yellow. These regions are part of the primary visual pathway, called the ventral pathway. These are the most important ones in recognizing what things looks like.
Now I’m going to show you the same visual regions but I’m going to show you how big they are. And again, the yellow ones are the primary visual pathway. So here you see that the first regions, the ones closest to the eye, are huge. In the macaque monkey, they represent 25% of the volume, or the area, of the neocortex. That is, 25% of what the brain knows about the world is low-level vision information. And you can see that it gets narrower as it gets to the top. You might think, oh, we have all these high level concepts, we need big regions at the top. No, it turns out we have a lot of big regions at the bottom, near the sensory input.
Now, finally you might say, this looks like a flow chart in which information comes in, it gets processed, and so on. That’s misleading. So now I’m going to show you another picture that’s more reflective of what’s going on. Here we have the main visual regions of the brain. Again, the eye input is coming in from the bottom and what you see here looks more like the bottom regions of a lot of little regions. It’s more like a tree, an inverted tree with the branches converging as they go up the hierarchy and diverging as they come down the hierarchy.
Now, as a computer scientist, this says something to me. This is a memory structure. This is the design of a data structure, if you will, for how the brain stores information. This happens to be the visual information in your cortex, but you have similar hierarchies like this in your auditory, in your somatosensory or touch regions. In fact, all information stored in the neocortex is stored in a hierarchical, tree-shaped structure like this. They vary in size, how many neurons, and so on. There’s a lot variation. But the basic idea is the same. So now we have to ask ourselves: if we can figure out what each of these regions is doing, and we know how they work together, then we have a good theory, don’t we? We have a theory about how the brain learns and how it forms representations about the world.
We call this theory Hierarchical Temporal Memory (HTM). By the way, this theory didn’t just come out of the blue. A lot of other people have been working on similar ideas. This is the term that the people who work in my lab and I use, Hierarchical Temporal Memory.
The first thing we can ask is, what is going on in each node, each little region of the neocortical sheet, each part of that napkin depicted in each of those little boxes? Well, it’s pretty simple. The first thing these nodes do is they learn common spatial patterns from things that happen at the same time. If one input and another input happen at the same time, more than chance, we can say they will likely have a common cause in the world. A dog barks, you see the dog, you hear the sound, you see the dog … you can assume it’s probably the same dog.
The second thing the brain does is it learns sequences of those patterns. It’s like learning melodies. The idea here is that if you see things that occur together in time, one after another, often or more often than chance, there’s a good chance that they have a common cause in the world as well. And you can think of it like learning a melody. I hear that series of notes over and over again and I learn to represent that melody.
Then, each node passes the name of the melody, the name of the sequence to its parent in the hierarchy. And then the parent does the same thing. It looks for spatial patterns of sequences, and then sequences of sequences. So you have a very fast-changing pattern, like my speech, which is occurring on the order of milliseconds, coming into the bottom of your auditory hierarchy. As these patterns go up the hierarchy, your brain is recognizing things … that’s a phoneme, that’s a word, that’s a phrase, that’s an idea. As the top of the hierarchy is reach, the neurons respond to slow changes and that’s what we ultimately see, hear, feel, and so on.
The other thing that each node can do is it can pass down a prediction. It passes down the remembered sequences to predict what’s going to happen next, in a probabilistic fashion, not necessarily exactly. These are the things that are likely to happen next, and it passes that to the child node. And if you take a slow-changing pattern at the top of the hierarchy, and you unfold its sequence, and each of those elements unfolds into a sequence and so on, you can generate a pattern like my speech, or a very complex motor behavior. That’s the basic idea of what each node is doing.
Now, how do the nodes work together as a whole? Well if you put them together in this way and you do all the proper training and so on, your brain creates a hierarchical model of causes in the world, with little details at the bottom, big concepts on top, and so on.
We use a technique called a Bayesian Belief Propagation. Some of the scientists and other people in the room will know what that is, but many of you will not. It’s a simple but beautiful mathematical technique invented a while ago by Judea Pearl. This technique shows how you can have a network of nodes that don’t know what each other is doing, but they all collaborate to very quickly come to common agreement about things.
This means that I can put a force or belief or pattern on some set of nodes, and in a single pass of the system, all the other nodes will form a sort of copasetic set of beliefs. And that’s what happens when we look at the world and see the world. We basically take this noisy input-and the technique works with very noisy inputs- and we say, what is the most likely cause of all this? So at the top of your brain you perceive the world very cleanly-you see me very clearly, here on the stage, there’s no doubt about that. But if you actually looked at the data coming into your brain it’s very messy. The Bayesian Belief Propagation technique makes it very clean and clear by the time the system gets to the top of the hierarchy.
So that’s the basic theory. Now I’m going to show you a few technical slides. I don’t expect you to understand them, but I want to give you a sense for what theoretical neuroscientists like me, and my colleagues, do. Here is a picture of a single node, a single little area of your neocortex expressing what it does in the language of belief propagation. Essentially you have these messages that come in and go out and there are these four different boxes here. Two of them are labeled Markov chains. That’s just a fancy name for sequences. And the other two are labeled coincidence patterns, or spatial patterns. You can look at a standard belief propagation node this way. You can define the equations for what these boxes are doing, which I won’t talk about here. But there are a bunch of little equations up here, and you can map those equations back onto the neuroscience. You can ask, well, these equations must be being implemented somehow in the brain, so can we map them onto the actual neurons that we know exist?
This is sort of a prototypical or archetypical diagram of a section of the cortex in six layers of cells with the different cell types and so on. This is straight neuroanatomy. So with these mapped equations, a theoretician can make predictions about what experimental neuroscientists, should see. And we’re doing that.
Now, we’ve taken this theory and we’ve begun to implement it into a technology, at least we’re trying. Gar might have given you the impression that we’re really farther along than we are. In fact, we’re at the beginning of this, the very beginning of it. But we’re looking at how to build this in software, which we’ve done, and then how to apply it to different problems.
These are different problem areas that people have come to us with. We’ve worked on several of these, including voice recognition, digital pathology, we’ve worked with some automakers, and we’re starting to do some fraud detection in banking. We’re trying to figure out what the technology is good for and where it’s going to work. It’s a very difficult problem, it’s not simple. This class of problems tends to have the same things in common-they have a lot of data for which we’re trying to discover the underlying causes. Can we solve the underlying causes? Can we detect patterns of fraud in banking situations? Can we detect patterns of attacks on networks, and things like this, and can we recognize the typical patterns these things exhibit?
Now we’ve spent most of our time doing vision, computer vision, and that’s because everyone knows computer vision. Gar works in computer vision and it’s a well-known area so it gives us a lot of benchmarks to test ourselves, and so on.
I have a computer science friend of mine who was a professor at Cornell, Dan Huttenlocher. A number of years ago, he proposed that we create a grand prize in computer vision. For a large prize, like a million dollars, we’d propose a challenge that might not be solved in ten years. But if somebody successfully met that challenge, they would get the prize. So I asked, “Dan, what should the challenge be?” Now you might find this hard to believe because you would find this so simple, but this tells you the state of computer vision. He said, “We should make it a computer that can recognize cats from dogs. That’s worth a million dollars.”
Now here are some pictures of cats and dogs. I assume that you have no problem in telling me which are the cats and which are the dogs. A five year-old can do it. How hard can that be? A million dollars for that? But the computer’s got to do it, not a child.
So you might ask, if you didn’t know any better, “How would I go about solving this problem?” You might say, “Well, we’ll come up with a definition for a cat. You know, cats have short noses and they’ve got pointy ears and whatever.” And another definition for what a dog is. But that doesn’t work. It doesn’t even come close to working. There are exceptions to every descriptor. For example, you can’t even determine if there’s an ear here in this image. Some of these images are impossible to understand. And then you might say, “Well, I’ll just take an image of a cat and an image of a dog and I’ll try to figure out a new image which is closer to the image of the car or the dog.” You can’t do that. There’s no transformation you can do to turn one of these dogs into another dog, at least no regular transformation.
There’s no mathematical equivalence between these dogs and cats, and so on. You might just say, “Well, I’ll just remember every dog and every cat I’ve ever seen.” That could theoretically work, but there are two problems. One is, you don’t have enough time, because you’d have to see a gazillion billion dogs and cats, and there will always be more you haven’t seen. And secondly, you don’t have enough memory. So you don’t have enough time and don’t have enough memory.
But in a sense, that is the answer. The way these hierarchical temporal memories work is they store a lot of information but they do it in a very efficient way. They do it in an efficient way that says: I’m going to remember almost everything I see. I remember a lot about the world, but I’m going to be able to generalize when I can, and I will do it in an intelligent way. That’s a solution to this problem.
As good engineers (and I’m an engineer, as well as a neuroscientist) we want to build things. So let me tell you how we went about this four years ago. We started with a very simple vision problem. This particular work belongs to a colleague of mine, Dilipe George, who’s a co-founder of Numenta. He suggested we make a very simple vision system. This vision system has a “retina,” if you will, or an image space of only 32 by 32 pixels, and we have three layers to our hierarchy. We have 64 nodes at the bottom layer, each node is looking at a 4 by 4 pixel patch, then we have 16 nodes above that, and one node above that. This is our miniature, little micro-cortex.
And then we trained it on these little patterns. These are little line patterns. We made movies of them. So we would take these little patterns, 48 of them, and we’d make movies and we’d expose the movies to this memory system.
And we learned by the principle I just explained. Once the system was trained, we could show it novel patterns and ask it to identify the pattern. And these are all novel patterns in the sense that they’re distorted, they’re noisy, they’re different sizes. You can see on the left there’s something we call a dog that’s facing left, then it’s facing right. Now this is a toy problem, I understand that. But it’s a very difficult toy problem.
This system is almost unbeatable. You can sit down and play with it-it’s really good- but it is a toy problem. Maybe you can see in the fifth column over there’s something called the helicopter. It’s a little line drawing of a helicopter, you’ll see that again in a second.
Remember, this is a system about time and we store sequences of patterns. In fact, most of what you recognize in the world is based on time sequences, not spatial patterns. Knowing this, we decided we should be able to do the same thing here with our system.
So we first took a pattern and stuck in a noisy field. Now here is a helicopter in this little picture, on the bottom if you can see it, and it’s surrounded by a field of random noise. If we thread that into our HTM, we found that it didn’t do very well. Actually, it got about 20% right, which is still pretty good. That’s what we call static inference, meaning there’s no motion-it’s static- and it’s with noise.
The next thing we did, as you can see in this movie, we moved the helicopter through the noise. You can watch it here and you’ll see it, it sort of pops out. Now when we show that to the HTM it does a lot better, much better just like you did. And finally, we can do the same thing but we can have the noise changing on every frame. So this is a movie where every pixel is changing in every frame, and you can still see the helicopter in there – it pops out for you, not as easy as before. Just like in the brain, we don’t do as well but we do a lot better than before. If I did these experiments today we’d get a hundred percent on this, remember this was four years ago. We were encouraged by these results-maybe we’re on to something, maybe this is going to work.
Since then, we’ve been spending a lot of time trying to make this system realistic. I’m going to show you a couple of little demos here. Unfortunately I can’t make it any bigger, I apologize for that. We’ve now scaled this up to pictures. We have a much bigger HTM, more complex and a lot of other stuff that went on there. And we can now feed it in gray scale images.
So here’s a picture of a sailboat. On the right you’ll see this system only knows four things: sailboats, cell phones, rubber ducks, and cows. So the system is going to think that every picture is one of those four things. (We have to start someplace.)
This is actually very, very hard to do. I’ve got to keep reminding you of that in case you think this is easy.
Here’s a cow I drew on my white board. I fed it into the model and it was recognized as a cow. But one of the things I can do with this system is, I can distort the image.
Now every time I change this image, every time, it’s sending it through this HTM again. Notice this is all live, this isn’t a canned demo.
As I zoom in on the cow, then parts of the cow, the system recognizes it as a cow until a certain point,
when it’s going to get confused. Is that a cow, or maybe a cell phone? A sailboat? It doesn’t know.
We can also manipulate the image and do things like rotate it. We can put noise on it, so we have use this test bed for a variety of things.
We’ve just shipped this software tool that make it real easy for anybody to do this.
You can also create networks like this. Here’s a little flower network I have. I’m not going to demonstrate this tool completely, but you can take pictures of things, here are pictures of lilies and daisies, and you just feed these pictures into the tool – you don’t need to know any computer science, you don’t need to know anything about brains. You train the tool on all your different images. You can actually make computer vision things.
Now this isn’t human-like at all, it’s very far from that. But this is the easiest it’s ever been to make a computer vision application. I think I can state that pretty clearly. We’re working with a company doing video processing, just like we did with the little movies of the helicopter moving through noise. We can basically look at video scenes and try to detect which objects are people, which are cars – things that we haven’t really been able to do well up to now.
Here’s another example of what we’re currently doing with a company in the area of digital pathology. In digital pathology, physicians take high resolution images of cell tissue – this is prostate gland tissue here. Trained medical doctors, MDs, have to look at these images to recognize certain types of cell structures. They can spend up to forty minutes looking at a single slide of very big, high resolution images. It’s costly and difficult to do. So we asked, can we automate this process?
On the left here you see four pictures of glands in a prostate. And on the right there are four images of things that are not glands. What makes a gland? Well, a gland has these black cell types, and they surround white vascular tissue area. But you can see they’re quite varied. On the right we have pictures that have black cells, we have white vascular areas, but they aren’t quite the right thing.
We ran it through our vision toolkit and trained it on a couple hundred images in just one evening. The toolkit did really well with an accuracy rate of about 95%. And the errors were reasonable errors, that is, they were occurred in images where humans might disagree as well.
So now we’re working with a company and trying to bring this type of application to market. It’s a lot of work to do that, much more involved than just this simple test. To make it really valuable you have to do a fair amount of work. But there’s a lot of hope for these technologies.
I want to remind you again this is not just about vision. We think this can be applied to lots of different things. In fact, it’s sort of like a whole new way of doing computing. There are a whole series of problems that people haven’t been able to solve with computers, that humans can solve, and we think we can address some of them.
I want to conclude with a few comments regarding the question, is this dangerous? Now I mention this because when Oppenheimer started the Manhattan Project, they knew it was dangerous. There’s no question about that. They knew what they were doing, and they thought long and hard about the ethics of it and whether this was the right thing to do. Later in his life, Oppenheimer encountered a lot of difficulties when he opposed the hydrogen bomb development when other people like Hans Bethe were not. So there’s a long history in physics, especially nuclear physics, about the dangers involved.
Now there is some controversy here about the work we’re doing – is this dangerous? Are we going to create monster robots? And are they going to take over the world? And are we going to make monster brains that we can’t control and things like that? There are some serious people who are really worried about this. I want to characterize some of the problems that some people have raised, perhaps with a little tongue in cheek.
There is a group of people who worry about what they call the singularity, as described in a book that came out recently called The Singularity is Near, by Ray Kurzweil. Ray Kurzweil is a very smart guy but I think he’s wrong about this one. The idea here is essentially that if we can create machines that are intelligent, and more intelligent than humans, then those machines can create machines that are more intelligent than them, and then those machines can create even more intelligent machines, and then you have this sort of runaway chain reaction. Instead of protons hitting each other, we have computers designing new brains and things like that.
This is not going to happen. I’m just telling you, it’s not going to happen.
It’s not as though, what if I had a computer that could design even faster computers? Would we all of a sudden have infinitely powerful computers? No! There are limits to this stuff . There are limits to actually building intelligent machines, and then training them. There are certain capacity limits and it takes a long time to train them, as well as a whole series of others issues here.
I do believe we can build brains-computer brains, artificial brains, or intelligent machines-that are faster and higher capacity than humans. But they are nothing at all like a human. These are just boring boxes like my laptop here. These are not emotional machines that experience lust and sex and want to control the world and reproduce and so on.
When the steam engine was invented, some people worried at the time about technology taking over the world. The steam engine – that was amazing, it was miraculous! And this was when people first started worrying about the technology taking over the world. The same thing happened when the computer was invented. It’s not going to happen.
Now maybe some day, somebody will decide they want to try to mimic the rest of the brain, all the emotional centers of the brain, which we’re not talking about here tonight. And maybe they’ll want to try to build a humanoid robot like this thing down here. This is really science fiction. It’s not going to be like that, maybe three hundred years from now, I don’t know. But it’s not something we’re talking about here. We’re just talking about building memory systems that can model different types of data streams. That’s all it is.
There’s another concern that I call “the Matrix” or “download your brain.” In this little picture here someone has some ports installed behind their ear. This idea that you might become immortal … like, we’re going to figure out how brains work, then we’re going to suck all the information out of your brain and stick it in some computer and then you’re going to be living multiple lives, and so on … this is not going to happen either. I’m sorry if that disappoints some of you, but it’s just not going to happen.
You know, you have trillions of synapses in your brain. It’s not as though there’s a binary code you can read out. Those synapses are complex, bio-chemical things and each one is located on a particular distance on a dendrite. The transmission delays on those dendrites are critical to how your particular brain works. So in terms of “downloading a brain” … this is not going to happen. And maybe it would be nice, but it’s not going to happen, at least not for a hundred years or more. Who can know after that?
Our world today has benefited greatly from the fundamental physics that was done eighty to a hundred years ago. And I feel the same type of benefits are going to happen with neuroscience. We’re on the verge, we’re just starting to get started here. And I think in twenty, thirty, forty years, this is going to be a huge industry and a huge area of scientific endeavor. But it’s also going to be really beneficial for humanity to have the ability to come to grips with who we are, how our brains work, and having machines that are really smart at understanding patterns and helping us as scientists and helping us as people discover how the world works.
So, although there will always be some danger to something, I think the danger here is really, really minimal.
If you’re interested in learning more about this, there are several things you can do. You can read my book. I don’t care if you buy it, borrow it, it doesn’t matter, just read the book. You can go to our website, www.numenta.com, and you can sign up for our newsletter. You can download demos like the ones I just showed you. You can download the vision toolkit and make your own vision computer application. We have people starting to make iPhone and pre-apps now. We have a very detailed product called NuPic, which is really technical and not for most people.
And if you’re a young person in the audience I have two things for you. One is, think about making this a career opportunity for you, especially these scholarship winners here. I believe this is going to be one hell of an exciting place to be working over the course of your lifetime. The second is, we have a lot of interns at Numenta so you’re welcome to come do that. And you can always contact me at this email address.
So here I am, at the end of our time here. I found this nice picture on the web, that you may recognize as an old guard shack here at Los Alamos. I don’t know if this shack still exists or where it was. But I thought it was fitting as a closing picture. As a person who’s never really been involved in what you do here in your laboratories and in this community, I just want to say I think the work that’s been done here over the years, and continues today, is really wonderful. I want to thank you for what you do, as well as thank you for having me here tonight. Thank you very much.