Agents By the Bay: Thor Whalen
Recording: Agents By the Bay: Thor Whalen
No. All right, now let me see. Okay, I think it's going. Okay, very good. So, all right. So, Thor Wallin, how do you say it? Thor Wallin. Thor Wallin. >> Wallin and like >> Thor Wallin
How do you Thor? Thor, yeah. Different different countries say it differently. So, Thor is good. >> Thor Wallin. Thor Wallin. Yeah. What's your official title now? Right now, sabbatical in Sabbatical. On sabbatical
So, founder on sabbatical now. Yeah. Okay. But that's do it all again. No, no, that's good. That's good. That's good. I I didn't start yet
So, I'm just asking. So, you're here. Okay. Now we're starting. So, now I'll say, "Hello everybody. I'm Alex Kravets, the AI community architect at Neo4j. Here we're on location at Pi Bay for the 10th anniversary edition of Pi Bay with Thor Wallin >> Yeah. who is the founder on sabbatical and a member of the community
Welcome, Thor. Hey, welcome. We just had a conversation over lunch. You turn out to be a mathematician, a founder, a graph person, and a graph theory person, and obviously Neo4j is a graph company. So, we love to meet people. Tell us a little bit about how you combine all these things. Where did you use them? How you come you, you know, you know at Pi Bay? Uh well, you know, I might graph theory was my thesis in for my PhD in math. Uh but how did I use them? In in various ways as soon as I needed to connect things
For example, in my earlier years, uh recommender systems and uh yeah, all kinds of interactions and marketing of people with products and pages. Um later on, I used it as a way to uh represent the connections between um um feature vectors of sounds uh that we collected for the startup that I made, which is in sound recognition AI. Right. Um and now now now uh I founded a company with a partner of mine uh that is using both uh point data and graph data to be able to offer very beautiful interface to interact with big amounts of data and zoom in and uh you know, get more information on it and and basically something that's more human in the loop for the enormous amounts of data uh that we get in AI both producing and using. Which is all we need because we are drowning in it. But, you know, uh let me kind of zoom in uh on the sounds because when you described the sound startup, I thought like, I wish I could do that startup. That sounds so sexy. That sounds so fun
>> Yeah. Right? So, you collected a bunch of sounds and you convert them to feature vectors. And then you start to think, you know, how to connect them. Right? So, can us can you tell a little bit more like what's the use case? What are the applications? Who are the customers? Why did somebody buy it? Like, tell us the story. Well, you know, that that that's the old startup. I should talk about the new startup you see. >> us first about the old startup. But, uh well, new and old there there's something that are in common, which is which is you're dealing as a data scientist, as an AI worker, you're dealing with objects that can have various complexities
Yes. >> Right? And there is two extremes uh in the trick to be able to uh represent these objects of interest. Mhm. Uh on one side are feature vectors. >> Yes. Right? Which is a flat set of uh of numbers. And on the other side is the discrete side where you basically connect uh that object to various other objects through various relationships or properties of them. Um and so in sound for example, we got the sound
Obviously, it's a it's already a feature vector of of samples. Yes. But it's not one that is ready to be analyzed as is. So we transform it into a different set of of numbers. Mhm. Usually smaller, but sometimes it can be bigger. >> spectral transform or what kind of Yeah, yeah, you Usually FFT is is is involved for some set and sometimes not, but uh But and then you you you boil it down to to represent um this as a feature vector, right? >> Right. Um those feature vectors then you can put it in the standard sort of most data science machine learning engines, models deal with those fixed-size uh feature vectors, right? >> Mhm
There are some others as well outside the feature vector space that that deal with more structural uh information. Mhm. And then you need to transform your data into that. So for example, uh tagging the sounds, you know, with all kinds of metadata. >> Geolocation. Geolocation, etc. Origin. Actually, to tell you the the truth, there's something interesting that I did back then that actually produced the the you know, the best result was from the feature vector themselves, which is a in a big large space, right? They're all over the place and sound itself actually moves
So you're you're really moving in the feature space in some kind of manner. Mhm. And it was obvious that it was not only the location of it that counted, but the sort of curves that it was creating in that space. Mhm. So I actually took those curves and transformed it, long story short, into symbols, >> Mhm. which today I called back then phones and then I called them snips Mhm. for sound nips. Today you would call them tokens
Right. So I created basically I projected the uh sound recognition problem into a language. Yes. uh model problem because there were so many NLP things out there that I could use. I could use techniques for search instead to to sound to find similar sounds. >> Right. I just had to find similar text written in a language that I didn't invent but that the sound sort of invented for itself in the movements that it was making. >> Interesting
>> And then and then the rest is history, yeah. Interesting. So that's that story. Was it human voices or any kind of noises? >> In fact, we we very purposely uh avoided the parts of the market that were already quite crammed. So not voices Mhm. not um uh music. >> Right. Though I wish it was music but Mhm
uh No, no. Everything else. So we we started for for uh deaf people. Right. Um uh and then we moved on to many other things because the health care quite hard to penetrate. Right. So we did things like a lot of machines at some point but also some very other sexy problems such as >> Like predictive analytics or machine noises? >> Yes, yes. It was exactly we ended up in predictive uh maintenance in fact
>> Predictive maintenance, that's right. On the way to finding a place that would, you know, create the revenue and create the interest Yep. there were much sexier things like for example, some company who I will not name that said can can we recognize in a phone uh recording the background sound what city in the world it is. Uh-huh. Nice. >> I won't say who it is. >> it. Yeah
And I proposed to do that but also uh maybe to try to guess what people are typing Uh-huh. from the sound that the that the keyboard makes >> it. I love it. >> etc. This is great. This is great. >> Those are sexy problems. >> like sound signatures of places? >> Yes, but why are we talking about my old startup? I sold that
I don't >> it. This [laughter] No, this sounds great because, you know, what what I like about it >> Yeah. uh it really caught my attention because like you know, everybody that brought their talking about startups and they'll like, you know, run a lot of meetups and it sounds almost like everything is done. But once you describe this it's an enormous source of data. And now we know and everybody who started doing AI they come to realization. Your AI is only as good as data. So, you cannot just say I want to do an AI startup in this in the next like I want to do an AI startup in biology, in medicine. The next question is do you have a data set? Do you have the data? And if you have the data almost your startup becomes almost trivial
The main problem is solved by collecting the data set. Right? I mean, I like your new startup because you have a beautiful thing. It's true, right? But so the the the kind of most AI startups in the main, right? We do AI for X, AI for pets, right? AI for commerce, AI for this, AI for that. Like the first question do you have the data? And if you have the data almost like data is almost like dictating what should should be done usually. So, but you just described like walk around, we hear sounds all around us. We're very attuned to the sound and like we can make so much sense just by collecting the sounds, connecting them together. So, like I don't think like having one startup that precludes other people from thinking of this. You can collect sounds like from your iPhone right now
Put them like this is this could like I can write code. Like this is I mean, I'm just inspired. >> Good luck. I'm inspired. [laughter] I'm inspired by this, right? So, and another thing you described which really caught my attention as a graph person. So, you did a PhD in math in graph theory. Can you talk a little bit about it? I mean, there's not I think much interest besides for graph theorists in what I did, but it was extremal graph theory, which means you try to see the properties that happen when you add usually density of edges or of usually it's density of edges in some kind of way. And my particular specialty was connectivity
So what kind of connectivity structures emerge as you you know say a minimum number of degree of of a node for example or a minimum degree of two nodes together and they can share between so those kind of things. And some things emerged. Precisely what I what I actually the only part that I was and then that's proud of was the part where I subsumed a bunch of desperate sort of results into one unifying sort of formula where you just ask for a few properties of of the the structure you were looking for and then the formula spit out the minimum degree condition that you needed to be able to get this thing which then as a corollary had several dozen previous papers that looked at the structures one by one and so that was that was the big part of it. Interesting. So when you talk about structures emerging from degree of a node, what So are you talking about subgraphs? Are you talking about properties of the graph as a whole? What structures? What patterns? >> Yeah, so the the main structure of interest uh to to to me for connectivity was uh was what we called emergence. Uh essentially it's paths and the the sort of inherent shape of the path structure. So is it is it for example two nodes with three paths between them and do the path cover everything or not? Or is it, you know, four nodes maybe with, you know, a path between A B C D and D E, and then one in between between A and C, right? So, those kind of things when do they happen in the graph, where do they happen, and how much can you extend them? >> basically, subgraphs. Yeah, that's all, yeah
Yeah. This is I mean, to me this is all very practical and sexy. Like, I don't think it's smash. >> Practical, I don't know. >> [laughter] >> I mean, because a lot of stuff, so I can tell you as an inside story about my little life. >> the side. Yes, my little, because, you know, I'm not tooting my horn, but like this there is very direct connection. So, so I did a paper in 2008 when I was making my PhD called Language of Life
So, we had the reality data set from MIT, which was the first study of, you know, remember Nokia phones? Yeah. So, Nokia phones were not smartphones. So, they instrumented them to record uh you passing through cell towers, because it's a phone knows which cell tower it's connected to. So, they had a granularity of cell towers. So, people go through hexagons in Cambridge in Boston. And so, the data set was basically like several hundred graduate students who were given these phones. And so, I asked the question, how many how many hops do I need to distinguish all of the people in this data? So, I called the paper Language of Life, and translated basically the tokens are IDs of the towers, and I ran a language model, which was unfortunately not called large language model then, all right? And so, I found that in four hops Yeah. You can define you >> the data set, because somebody wants to walk around the river, somebody goes to the coffee shop
So, and that was just IDs, but I'm sure that, you know, if I map it on the map, I could get some patterns as well. All right? So, so probably, you know, it would be very interesting, because like I don't I I probably could recreate the graph of the towers, and see where they actually going through a line, or did they do a little kind of loop, but it just strikes me. >> No, it's interesting because what what what you I'm going to make a prediction because podcasts love that, right? >> Uh-huh. Uh this tokenization uh is is a trick. I think right now we're applying it to language. >> Yes. But it's a trick, you know, ta- take financial, you know, uh data and tokenize those things. Take uh sound, tokenize it as done
Take the the other things, tokenize it. >> Yes. What is that tokenization trick, right? >> Mhm. And and and why does it have potential that I think we're only starting to tap tap into? >> Mhm. To me, it it ties back to the thing we were talking over lunch, right? >> Mhm. The sort of continuous sort of multi-dimensional space of vector spaces, right? >> Mhm. And on the other side discrete structures such as such as a graph, right? >> Yes. What we were talking about over lunch is that there's actually a spectrum between both, right? And both sort of representations, along with the operations for those representations, Mhm
uh have their advantages, Mhm. right? And so and exploring the space in between really does. Now, let's think of this tokenization, right? It's uh why would one tokenize sound or even your data was continuous first, right? >> But then So I can't talk about your data so much, but let's say for example sound or patterns in in financial market, right? >> Mhm. It's It's It's You already have the feature vector. You already have the you know, >> Numbers. >> the numbers, right? >> Yeah. Uh and numbers models deal with numbers much better, right? Right. But here's the problem is how models deal with the numbers is with uh I I don't know what to call it with I don't want to insult operations, right? But it's it's it's through these fixed operations that may not be appropriate, right? Multiplication times Okay, you filter them through, you know, a logic function or a thing like that to to warp the space a little bit
And that's what you're doing. You're warping the space, right? To do this thing. >> Right. Um that's fine, right? But those functions, and they work well because they're analytical, they they they're they're they're producing something, this analytical warping of a of a continuous space. >> Right. But sometimes that's a waste. Sometimes you have a connection between a cluster of numbers of feature vectors over here and over there, right? >> Mhm. And the numbers almost could introduce some uh some noise, right? In that there's a variation, but really what matters is they're over here
>> Mhm. And then they're over here, and then they end up over here, right? If what what really matters is the general space between those and maybe the movement or the order Right. of those things, or the fact that they were here at some point and then here and then there, >> Mhm. now the discrete world is a lot more appropriate. Mhm. So, if you if you map these continuous uh spaces, especially if you're in movement, right? Into something that just says, "I'm going to simplify this." >> Right. There's eight things happening, there's eight regions of this big space that are important, >> Right. right? That carry the information, right? >> Mhm
And by just getting rid of the details, you're only able then to focus on the composition, the structural composition, maybe this, that, and that, or maybe the order counts. For example, when we did this with with like um uh What was it? With the with the machines, right? At some point in the recording, there was a a voice that came, right? >> Mhm. And you you could have this I I wish I could show it right here, but I did this I think it was with uh with cars. And we were tracking four different uh problems, right? The braking pads that were worn down. Right. Uh the >> sounds. The something, yeah. The four basically problems that they like to do that
And in fact, with an alphabet of 18 tokens. Right? So, I call I think I call them snips back then, but phones, right? >> Yep. 18, right? Not 26, not however many LLMs use, right? 18 of those and not even a dynamic sort of Markov, you know, latent space and definitely not a deep learning, you know, neural network. No, no, no. Just the snips. Mhm. Just those ABCs, let's say. And Bayesian factors on the snips
Mhm. Right? So, again, the order didn't even matter. What mattered was huh. It's like sort of author recognition, right? >> Mhm. That author tends to use these words a lot. Correct. And four times more than anyone else. >> Correct
>> Right? So, the Bayesian factor would say given that that word is there Well, that's that's all right. >> What's the probability that that word is And just with that Mhm. just with that. So, you take sound, which enormous amounts of like it's it's 44 thousand, right? >> Right. numbers Right. >> That's right. Coming in, right? >> CD quality. You take you get rid of that
You We looked at We looked at just uh smaller chunks of that, right? In that vector space, you know, FFT it, etc. Project it to a space with 18 Yes. Interesting. >> Imagine how how many bits you need for 18, right? And you have a stream of that. Right. And then the model is just you know, counting oh, I got a lot of A's. Yep. You know, there is a 80% now probability because I never hear A's when it's something else
And it's funny because one of those I'm going to call them tokens because people call them Right. One of those tokens only appeared when there was voice. Uh-huh. Right? When in the car, you know, someone starts talking >> Right. that that thing sort of came up. So, it sort of took all the voices and boom. Right. There's the voice right there
Right. Which also means if you want if you have privacy issues Yeah. and you want to say, I want to compress things in such a way that voices are going to you just have to remove that sort of >> Remove those letters. remove that one token, right? Yes, yes, yes. And then you have it. Anyway, very interesting stuff. >> is great. I mean, I love all of this, right? It's a very interesting but it sounds to me like right like like like very interesting and promising kind of direction
Uh and I mean, also you talked about graph topology and it just struck me that nobody in computer science I know in industry talks about topology. Like we we talk of maybe of graph algorithms at most, right? But we use them as graph databases. And so when you you talk about mapping multi-dimensional spaces to graph topology, I understand there is you know, mathematics involved and people in math treat graphs this way. It's almost like in practical computer applied computer science, nobody thinks this way. So, when you mentioned this, I'm immediately like it got me going. And so, like I think definitely there is you know, stuff in there. But like let's kind of you know, go back to your visualization, your new thing, right? So, the world >> Thanks. My my current [laughter] my current startup that we should talk about
>> That's right. So, you know, again, right? Like visualization is extremely hard. And so, so like do you think there will be a magic trick like you you basically through visualization you want to let people find something new. So what they're looking for, right? So is it this going to be based on the similar idea like language of life, finding the language simply because you need to simplify the space make make whatever it is they're looking for noticeable, visible, right? Is this kind of where you want to take it? Like find things. >> Well, you know, uh you know as as anyone who's done a startup with partners knows, you know, there is where the partner wants to bring it or is bringing it and where do you want to bring it, right? >> Right, right. So my partner who I think is also going to interview here is a is an absolute genius data visualizer. He does beautiful beautiful work and that really draws you in, right? Which is something very important I realized I hired him actually for the previous startup to be able to give people some view of the data that they were dealing with, right? Now, where I hope, you know, we'll be able to bring this is saying, "Look, AI is all over the place, right? And ML was all over the place as well. There's a sort of continuum right there
What might be stable here? Well, right now definitely connections so on the graph side which I think there's not very much of as much as the feature space side, right? But both of those things they're they're they're still there and they will still be there probably for a while. So we have another problem which is we have a lot of data. Mhm. Uh how do we, you know, being able to see, get a feel for that data, explore it, look at it under different angles, explore the connections is very useful in many parts of the of the way, all the way to when you have your model, right? If you have a lot of data and you're trying to let's say construct a context >> Mhm. to be able to inject in your data, you may want to be able to annotate these amounts of data that you have to choose for now something that you're going to put in or to annotate for later automatically being able to re-link into the proper context to inject into your LLM. Mhm. But even on the other side, once when you have all the training data itself, right? How how do you see through and and decide what's good? Now Right. of course, we all need to do it at some point, you know, with our fingers, with rules, with, you know, filters and try models to actually decide what to put in or or out of the training and how to deduplicate and etc
etc. Mhm. But one thing that really helps is if you can see all the data all at once and look at it in different perspectives. >> Right. So, what one of the things that we're putting forward is saying, "Look, uh whether it be points in space