Bay Area AI: Milton Huang, "How the Brain Learns"
Recording: Bay Area AI: Milton Huang, "How the Brain Learns"
To give you my background, as Alexi said, I worked as a psychiatrist, so I did that whole training. It means I went through medical school, put people in involuntary hospitalizations, did electroshocks on people, prescribed medications, did all these kinds of things that you do when you learn to be a psychiatrist. You try to understand people, seeing them in all sorts of conditions, psychotic to depressed, and try to use whatever tools, now it's almost always medications, but psychotherapy and everything else that we could possibly think of to see how do we get people to a new state, how do we get them to be better. I actually stopped all that, and for the last seven years I've been using actually a different type of modality. I've been doing neurofeedback, and I've spent much of my time learning more about brain waves, and electrophysiology, and doing EEG analysis, and statistical analysis of EEGs, and using neurofeedback, which is basically brainwave biofeedback. So the way that works is you put electrodes on the scalp, you read the brain waves that are coming in, and you actually use biofeedback training to help the person create particular frequencies in particular locations. And if you can do that, you actually end up creating different types of effects. So I have to say it's a much more targeted method than using medications
I saw effects there that I'd never seen with anything else. For example, with autistic kids, I've never seen medications really do anything, so maybe slow them down, sedate them. But I actually saw kids become more social, want to engage verbally. And that again was by slowing down certain types of processing that they have issues with. We use it for anxiety. We use it for, ADD is perhaps the best research area for that. And you can change people's ability to focus, concentrate. Although we can do that with drugs
In fact, caffeine is probably one of the best known drugs for helping people focus at least for a short period of time. Anyway, what this path is leading me to now, because I want to leave that too. And I'm here in part because I'm interested in becoming more of a technologist. Actually before even medical school, back in high school. Actually my undergraduate degree was in electrical engineering. And I was a programmer and I wanted to do all that before I started getting, finding out about people and learning how they tick. And I'm guessing coming full circle. Because what I'm interested in, and maybe what a lot of you are interested in, is how do you create something that can help people in a bigger scale in changing what we do
How we live our lives. I mean, one of the interesting things about all the, you know, all the technology we're creating is it gets into our lives in very subtle ways. It gets into the automatic things that we do. I mean, everybody now has their little cell phone. We've changed how we have our postures. How we, how skilled our thumbs are. We're remapping parts of our brain. These behavioral things that exist in every design of the technology we create
And specifically as we create new technologies. Here people are all interested in artificial intelligence. And how do we create systems that learn? How do we create systems that sort of mirror how people learn? And anticipate how people learn. What was interesting in Tatiana's talk, she had put up that list of all the problems that came up with the data collection. How people were biased. How they would use the same term and everyone would mean different things. And all these different problems that came up with the data collection. And those are all basic things about how the brain operates
And I can go into that a little bit too. Brain learning, assumptions we make. But those are all important factors that we need to be thinking about when we make the technology, when we interact with people. And you know, know that they're going to make assumptions about what a computer can do. What does it mean to be smart? What does it mean to be learning? People just sort of assume that your software, your technology knows what a computer can do. And you have to sort of fill in those gaps. Anyway, before I start just going off too far, I'm all, like I said, if anyone has questions, feel free to interrupt me. If you have questions about even drugs and stuff, I'm happy to answer those
I'm going to give a little overview of brain learning and some of the things I think are important to think about in terms of how people learn as well. So starting with brain learning, I'm going to assume that everyone here is familiar with McCulloch-Pitts models and perceptrons and all that kind of stuff. And I think I'll just sort of say, you know, some of the main differences when you go to the human brain as opposed to some of the simple models that we have, because it's hard to make these models, is the level of complexity you have. And if you have a PITS model you have, basically, you know, integrate FIRE. And we take all these weights in our neural, current neural networks and we sum them up and, you know, put on a sigmoid or a, or, you know, or RELU or whatever it is and set it out. And, you know, in the brain, we have about 20 billion neurons in the cortex, which is this area, sort of on the top, on the outside, which is the most recently evolved, what I call neocortex, the most recently evolved area of the brain. And we think about that as being maybe the most advanced because all the mammals and advanced, you know, more complex behavioral creatures seem to have more of this neocortex. Well, those 20 billion, you know, if you take those, most of them are, well, the ones that we tend to be interested in, in which the PITS model is based on our, what are, are pyramidal neurons
They have a long axon, so there's one single output, and they have a huge net of inputs, called the dendrites. And dendritic nets can have between 5,000, 15,000 inputs for one neuron. And so, not only is that a lot of weights to be thinking about for one neuron, for one cell, but for each of those connections, each connection that's made onto those dendrites, those all are biochemical, you know, they have the synapse that probably you're all familiar with. And again, feel free to interrupt me if I start going off into something that's, you know, too esoteric. But, you know, the synapse basically is the connection between one neuron, so you take the axon of one, going to the dendrites of the other, and you pass the electrical signal from one to the other. And so you can have, again, 10,000 neurons summing their data up for one output. But the release of that information to transfer from one to the other is biochemical. It goes by neurotransmitters
We're familiar with a lot of them, like dopamine, serotonin, norepinephrine. These are all classic ones. But by using specific types of neurotransmitters, what you find is neurons automatically organize themselves into clusters, into circuits, into groups, so that certain neurons can have, turn on their effects within the dendritic knit and have turn others off. And what makes this even more complicated is our other neurons that are involved. They're called interneurons that connect in between the neurons that are constantly remodulating the connections that the two, like if we just have, think of two neurons, that every one of those connections is going to be modulated. And then even on top of that, there's another layer. There are support cells called glia cells. And there are many kinds of these glia cells
Astrocytes are probably the most, one that's talked about most. And they provide, we think of those as nutrition cells, but more recent research we know actually, they also provide cellular information learning mechanisms. And they can recognize different neurotransmitters. They also modulate the effects of neurotransmitters actually at a synapse level. So one astrocyte cell in one particular connection can decide to turn on all of a particular neurotransmitters connections or turn off another one. Yes? Sorry. I have a question. Some people are taking drugs, for example, some of us, in the brain
Has this stress come in kind of from your observation on these patients learning energy memory? Oh, yes. I'm actually interested in the question for the report. Okay, so she was wondering about drugs that are used for things like schizophrenia. Do they affect learning? Do they affect memory? And that's unequivocally so. In fact, some of these neurotransmitters, for schizophrenia, for example, the most commonly used, well, the original schizophrenia drugs were all dopamine blockers. And, as opposed to say, amphetamines, which are dopamine enhancers, things like cocaine, which cause dopamine release, and therefore increase. And so, you know, by using drugs, we learn to increase the flow of our, you know, of the neural flow. In fact, you know, some connections are inhibitory, some connections are excitatory, some neurotransmitters are excitatory
Glutamate's the most common one there. Others, like GABA are inhibitory, and things like caffeine work, in part because they can increase some of that, you know, certain, the effectiveness of, like, glutamate receptors by binding NMDA. Anyway, what will happen, and actually there's studies that use these low dose neuroleptic dopamine blockers in order to impair memory. And so, I mean, we, you know, psychologists have, scientists have actually used some of these drugs in order to have some of those blocking effects. And to see what happens, because, you know, in some studies, you know, with rats and so on, you know, we're trying to separate, you know, where does the learning occur, which parts of the brain, which types of neurotransmitters, and, you know, you can actually, you know, you want to try to block out certain areas. It's clear, and this is one of the problems with drugs, in my opinion, that, you know, they have a very global effect. If you hit every, you block every dopamine receptor, I mean, you're going to hit certain groups of neurons, you're going to hit certain sets of activations, and it's going to have, and in some ways, one of the challenges is that, that set of responses is going to be different from individual to individual. And, in part, this is all genetic, because if we look at, say, the number of, say you look at a synapse that has dopamine receptors on it, we know that the number of dopamine receptors that are going to be present, how fast they get generated, how fast they get broken down, also, how fast does dopamine get created, how fast does it get broken down, all those variables affect that single synapse, and they're all, those are all controlled, in part, by genetic factors
And so, some people are just going to be more sensitive to dopamine, some people are going to be less, everyone's going to have different effects. And, I hope I'm answering your question, but, you know, in some ways, this gets, again, to the level of complexity, and how, you know, I mean, for me, drugs can push people into a whole different state. But, usually people will fall back, or, you know, to their old state, after the drug effect is gone. You know, what is really fascinating, and again, I don't know if you're going to classify this as learning, but there are cases, like with cocaine addiction, where you see someone gets that initial blast of dopamine in a way they've never had it before. And what is missing in many of the models of learning is, you know, there are people who, on that one shot of learning, from that first blast of cocaine, never, their brains never get back to where they were before. Instead of retreating back to the previous settings that they had in terms of their dopamine sensitivity and so on, they, you know, they never respond again to dopamine the same way. And what's interesting, too, with the MAP models and so on, people studied this, you know, and we know you can actually get, in some circuits, the same amount of dopamine response in the, there are particular circuits in the lower limbic system. And, just to get back to a brain model, you know, you have, we talked about all these neocortical new cells, you know, underneath it, there are sort of old, well, I don't want to say just older, there was a model called the triune brain, which had, you know, the neocortex, which they called the million brain, they had a, the midbrain, they had a, you know, and you had this brain stem, you had lower different areas
There are subcortical areas that are wired pretty differently. Limb, limbic areas are included that, and we usually think, associate those with emotion. But a lot of things like memory go there. So things like your amygdala, which exist in your, which are buried in your temporal lobes, and associated with memory. Hippocampus, I mean, well, hippocampus isn't, sorry, hippocampus in temporal lobes, amygdala is nearby. That's associated with fear, but also with assessing importance. And there are all these different circuits, again, basal ganglia, all these different connections in the lower brain that help us automatically, very quickly assess, you know, because we assess everything. How important is this event? Do I want to pay attention to this situation? Do I want to pay attention to this person? Do, you know, what do I, you know, am I in danger? Is this something that I'm going to be getting a payoff for? And a lot of these systems work in parallel, automatically, and very quickly, and, and, although they're wired very differently
And then I didn't even talk about the cerebellum, which has got like 100 billion neurons in it, and almost no astrocytes. Again, the ratios, they're different, it's almost like there's different computational units in all of these different areas. And what you see as you, again, you apply a drug, if you try to figure out the learning, sometimes learning can persist in one section. So, for example, when we look at where dopamine has the effect and the reward circuits, this came back from research in the 70s. I don't know if people are familiar with putting electrical stimulation into the brains of animals, where they would, if you stimulate a part of the hypothalamus electrically, and hook it up to a lever, a rat will push that lever for that electrical reward. And it will do so thousands of times an hour. And they will stop eating in order to just keep pushing. And they will, I mean, they'll die from starvation, because they just get, again, there's so much stimulation that they get from that type of behavior, that they would continue to do that
And, you know, people will try to figure out, well, is this a pleasure center? Why does this happen? I mean, in the decades it's gone since then, there's been a lot of research about, and that's a dopamine circuit as well. And we understand a lot more that some of our systems that try to figure out, is the, is an action I'm going to take going to help me? You know, what's predicting the outcome? You know, because we need, you know, that's one of the main things most of our circuits are designed to do. We're trying to anticipate what's going to happen in one minute, in an hour, in a day, in a week, whatever. And even, even visual systems are trying, your muscles in your eyes, when, if, the eye is moving constantly in, in motions called saccades, that are just shots of neuronal volleys that cause muscles around the eye to move in different directions. And all of these things get integrated in the brain, so that you do not, you're not aware of all these shakes that the eye does. But the brain takes all the information, integrates it, and stabilizes the picture for you automatically. In fact, you know, that could be one of the themes of learning. The brain stabilizes so many things, it goes back to creating a stable picture that we just assume is always present, even though when you go look at what's actually happening in the eye, it's having all these little jiggly movements in it
And I guess I'm going to, I'm going to make a few just sort of basic points that you see at every level of the brain, whether it's at the small circuit levels I've been talking about, or even bigger levels, like behavior. What you see is that the brain fills in the gaps. People make up stories. If you, and it, and I'm not just meaning, you know, inventing things, although people do that too. I mean, if you go back and, you know, when we look at people's ability to recall memory, and when they do tests, and how, not only, there's some great studies where they, you know, they put two people of different genders, and they try to have a conversation, and they try to figure out, ask them afterwards, how good impression did you make, and how people completely misjudge what the other person thinks they're, how they came across. And I mean, we fill in, we create models in our head, and we fill in the data to get those models. And, and it even happens at a lower level in the eye, where, you know, if some data is missing, we fill it in, in the visual space. And, and it happens in listening to information
There's studies where we, you know, where people, when you ask them what they recall and what were said, you know, actually, you know, you can, you bias people towards certain perceptions, certain things. Yes, you have a question over here. Yeah. I think that's a question over here. And it's used by the systems, you know, in part, in part learning. And so efficiency comes in part because by going, by having an actual rhythmic structure, the amount of energy you need to sustain the rhythm is not as much as, you know, taking a completely off system to on or off. So all of these systems move, work together in concert to create, you know, an ongoing balance. And this makes the, I mean, the brain's an incredibly dynamic system in terms of, you know, what turns on, what turns off
And it's a hard thing to model. I'm going to have a question over here in a sec. Let me just get back to what, you know, what you don't realize with the alpha rhythm. When I, if you close your eyes, your alpha in your visual cortex is going to shoot up tens of magnitudes. And you can see, you know, that created this idea that, oh, maybe it's because you're closing your eyes, and the visual cortex doesn't have anything to do. And we know actually alpha correlates with, you know, decreased metabolism, you know. So there may be decreased blood flow on, on fMRI and so on. And we can, by, I mean, this is part of why the neurofeedback thing is such an interesting tactic
Because, you know, you can, if you have the sensors, you could actually try to train different areas to have more blood flow or less blood flow. Getting back to the thalamic thing, because the thalamus is actually gating things on and off, and at least that's one of the theories here. So it's actually, if you have a 10 hertz gate, you know, there's 10 cycles per second that you're letting data through to the cortex. You aren't letting it through constantly. And so your brain is, you know, it's not getting data constantly in, it's getting it in these little spurts. But it adds it all up. And again, there's summation functions perhaps, or something else that's going on. That's another interesting question
But this happens with every one of our sensory systems. So, you know, auditory also goes to auditory thalamic area, then it goes to, you know, auditory cortex. There's actually somatosensory, so touch would go to the thalamus before it goes to your sensory cortex. And processes, oh, someone touched me and I, it was located here and it was, you know, how intense it was and those types of things. So thalamus really important and it's involved in generating a lot of these rhythms as well. And I didn't know if you wanted to ask a question or if I should just keep rolling. I wanted to say, first of all, thank you for, you know, pointing out that if you have like a rhythm, it actually makes it much more efficient than it's just like by the least. So, so one of the, um, sort of the manifold issues that we're looking at in AI is that intelligence is defined as information processing that reduces probabilistic risk
Um, the limits of that risk essentially between the zero and one states. Um, and, you know, philosophically the argument is that, um, that means that, you know, AI systems are very efficient, right? So if you're saying that actually it has to have a rhythm, right? Then that fundamentally changes what AI models should be, right? Um, so not based on these ways between zero and one and the political ratings of, you know, um, of like a perceptron system. Um, but my question is actually due with, um, you talked about like sensors in the scale. Um, I wanted to know what your view is, um, about neural laces. So, so there's this thing where, um, yeah, so, so, so, um, what, you know, the kind of like, the kind of like, the kind of thinking at the moment is that because the AI is limited, right? Given that it is, you know, mostly dealing in a kind of like, like, one-minded or a true situation of these are one states, regardless of if it's like, you know, a cubic structure or just, you know, a simple kind of like 2D chip structure. Um, the neural lace is supposed to be an intrusive, uh, sort of like, aesthetic inside. So it's like, like, you know, like, you know, like literally a set of electrodes, right? Microelectrode that you put inside to actually connect with the neurons. And, and through that, um, somehow, uh, it might be able to pick up the biofeedback, which is currently missing from like, um, you know, AI
Okay. So this is a question about neural lace. I'm not as familiar with it as I probably need to to make an intelligent comment on it. Uh, but it does sound, as from what you were describing, that it definitely gets inside the brain. So it would be like, you know, the scalp sensors, albeit that it's like- But it's inside the brain. It would actually, and it would be able to read and stimulate neurons. And so it would have to have a sense of its own spatial mapping. Okay
Obviously. Uh, and how it relates. Because, as I was saying, each of these different areas in the brain specialize in different things. I mean, I already described- actually, you know, just to give you a general brain background, most, most of the stuff in the back of your brain, including your spinal cord, this is all sensory. And things move forward. So all your motor control tends to be up front, as well as a lot of our executive cognitive systems for processing information. So, you know, if I- you flash a light in my eyes, you know, in 10 milliseconds you're gonna get hit here after it gets through the thalamus. And it's gonna move forward by, you know, and by 200, 400 milliseconds, you know, you've already bounced front to come up with a motor plan
Do I need to respond to that? Back to the sensory, do I need to start ignoring? You know, and so on. Now, you know, these- those are rhythms. And, you know, again, I don't know how you're gonna program this lace. And I'm not sure, you know, I- I mean, once you're inside the brain, it's amazing how much more specific you can get than using electrodes on the scalp. I mean, we use electrodes on the scalp because it's easy. I- you know, when I do a neurofeedback session, I can put, like, you know, if I just wanna train a specific area, I mean, I can set that up in a couple minutes. But, you know, the- the problem is if you put one electrode here, I mean, it's picking up the effects of neurons from, you know, multiple centimeters away. And not only that, there's an element called volume conduction
So that neurons completely on the other side of the brain through changes in electrical field throughout the entire cerebral spinal system are going to have- that's also gonna be picked up by the same- So, I mean, and in fact, you know, one of the biggest problems is you're- you're even picking up electrical activity of the heart, of the muscles, of the jaw. I mean, those are actually much stronger than the brain signals because the brain's increased by the scalp. I mean, the skull, bleh, sorry. And, you know, so it's working in microvolts that you're trying to pull out. If you're inside, it's amazing how much specificity you can get. And, you know, I- you know, this is interesting. One of my main frustrations actually with the whole neurofeedback thing is- and actually with a lot of psychotherapy and other things in general- is, you know, sometimes you can push people to a new place. You can make someone feel happy or, you know, by making- pushing- giving them the right drug, giving them the right, you know, therapeutic intervention
They start- they can feel better, they can feel less scared. Yet, to sustain that is a whole different question. And most of the, you know, techniques that we have out there, we don't try to figure out how are we going to create a structure that sustains. We just sort of assume, oh, well, you know, we'll just keep pushing you. We'll use the same drug every day to push you back to the same place. And, eventually, it may start wearing off. Or, you know, same with a lot of psychotherapeutic. I mean, and this happens with lots of learning in the body as well
If you do massage, you'll feel, okay, wow, that felt a lot better, but then, you know, it comes back. And, I mean, in part, this could be due to the fact that, you know, we're complicated- I mean, we need to maybe treat multiple systems. It's not just the brain. The brain exists in a body. And if you eat crap, then, you know, it's gonna keep- and you keep eating the crap, you know, that that's gonna have an effect on what- how effective the medication or the psychotherapy can be. If you don't get sleep, I could give a whole lecture on sleep, because that's really critical for learning and so on. But, you know, that affects your ability to have energy, to have motivation. And, you know, in fact, motivation's a really important thing for learning
When you think about- you know, this is something that, again, we don't have in our circuits. But, you know, because our circuits aren't energy dependent, like our body is, if your body starts losing energy, you know, you got- I mean, people- we had pizza today. And, you know, knowing the glycemic index of that food, I mean, most of you probably have had a crash and are- there's gonna be- unless you did caffeine to counteract that, there's gonna be a really hard push, especially this late at night, unless you- maybe you've trained yourself through just being, you know, hardcore programmers, I don't know. You know, to be used to like- or training your adrenaline circuits, actually. That's probably the other way to do it. I mean, if you stay up late enough, you know, you'll get that little adrenaline surge. But, you know, it's hard- I- okay, getting distracted. The bottom line was that I wanted to say, you know, it's hard to create- we don't know what the systems are that we need to shape in order to keep somebody in some place
If you- you know, if there is- if someone has a pain here, you know, you don't know, is the pain because- you know, it- I mean, maybe it's the way they're using their arm, maybe it's, you know, a weakness in another area- location. And I should keep track of my timing too. I think we actually have only a few minutes left. You know, we can- we basically have a horoscope at 9, right? Okay. It's about a quarter of a 9, so maybe we'll take, you know, a few more minutes and we'll have some more time to chat. Okay, but, you know- I'll have only one more question. Okay, one more question. Wow, I just sort of flew through a lot of different stuff
You had a question back there? Yeah, I have a question. It's like- so my understanding is that the neocortex is divided over broadening areas. Mm-hm. And they're separated by side-arm architecture. What is the exact nature of the side-arm architecture difference? Okay, so this is a question about broadening areas and sort of how some of that works and the way the brains are divided up. Yeah, because one of the- I think- one of the, I guess, phenomenon is like when people lose sight or lose a point of a sense. Another, I guess, part of the report that's going on. Yes, and so there's a point about- What are the areas of this? So there's an issue of what he's talking about neuroplasticity
So, first question was about Broadman areas and how the brain is divided up. Again, this is another area, way the brain's divided up that I didn't talk about, but it has to- Broadman was a neuroanatomist who went through and identified specific cell- neuron cell types. Because there are lots of them. I just mentioned the pyramidal cells, but there are a lot of other cell types. And even within a particular cell type, they're located in- they have different distributions, they have different ways that they shape themselves, even if they're the same general kind of cell. And so, he classified- what he did is he found that the same cell types would group themselves in different areas of the cortex. And so, a lot of these areas were called Broadman areas, are basically groups of the same type of neuron. And what we found is, subsequently in trying to do testing and try to find out, oh, this is somatosensory region, it corresponds- oh, here's a motor cortex
These correspond to the areas like Broadman 3, Broadman 2, Broadman 1, and he numbered all these areas. And so, when you look at someone's brain, you know, in general, these Broadman areas can correspond to the same functions. You know, you go to 41, you're gonna be getting vision. If you get to, you know, some of these other areas, they all respond- you know, you're gonna have auditory, you're going to have, you know, sense of self-processing, you're going to have all these different things in different parts of the brain. One of the things that happens, though- does happen, though, is that there is flexibility. And that was the second part of your question, is known as neuroplasticity. Now, you know, our brains aren't endlessly plastic. It would be nice if they- well, maybe not
That's a whole other area about when we're plastic and when we're not, there's actually a developmental cycle where we tend to be very plastic in the beginning, having lots of pruning, and then as we progress, we get more fixated in the solutions that we've come up with. And we just sort of keep the same patterns. But, you know, especially in the hippocampus, where we actually have new generation of neurons, and new connections made all the time. We know that certain areas of the brain are plastic. And if someone in development, especially when they're young, is required because of, say, blindness, there is- Oh, there's actually a great story in Doidge's book, The Brain That Heals Itself. He- or I can't- something like that. I can't remember the exact title. But it- he describes the case of a person who was born with half a brain
And because it was right hemisphere only, you know, had problems with- they're missing the left hemisphere they couldn't- they- they had problems with learning speech, they- all their milestones were delayed, they had problems moving their right side of their body. You know, the doctors thought the child had had an infant-headed stroke when it came out. But, you know, it wasn't until years- years later when MRI technology was- I mean CT technology was invented that they actually found out this person was missing half their brain. But this girl actually developed because of the plasticity. You know, this half of the brain was able to form all the things that needed to control the others- the left hemisphere of the brain would have formed. So, you know, especially in early development, there- where we have a lot of these- and what's really interesting too is now that we're getting into all the genetics, you know, there- there are new techniques that people are coming up with, especially with this- the embryonic stem cells and people are trying to understand more about how this differentiation and fixation in particular patterns happens. You know, we may be moving to some very interesting places and even with- with all of our technology including this lace stuff. I don't know where it's going
But, you know, there is an ability of the brain to reformulate itself, do new things. And, you know, like I said, I'm trying to reformulate myself because I want to be in this. I guess what- I guess what- what I wanted to ask the question was that- Mm-hm. I'm trying to reconcile the fact that different- that I guess any- there's like a homogenous structure in terms of like how a neuron is organized in the vortex. I guess like it's- but right in telling that with the- the fact that there are- there are different structures to- physical structures in terms of the- You know, I see the brain as a pretty diverse- in- with unique structures in every location. I mean, you know, you can- there are cortical column models that we have to, you know, model cortex. But, you know, that really varies a lot. There are times when, you know, some areas of cortex are missing
They don't all have six layers. And, of course, then there are the areas of the brain that only have four layers. And the primitive brain- then you have your cerebellum, which like I said has a hundred- you know, five times more neurons. And it- its processing is very- even though it's pretty well organized, it- it does a lot of interesting things. I mean, they're important in things like learning music, learning- anyway. I think I'm- I've run over. But, uh- hopefully- We basically have a couple minutes. We need to be out of here at nine, as Dan told us
So I guess- we have about ten minutes. Yeah. We actually- our crew has to be out of here at nine. It means we have to clear the space. Yes. So we're- we need to wrap. Okay. So I'm- But can we- I appreciate all- you guys' attention and- you know- Thank you, man
Ready?