Machine Learning Startups

ai.bythebay.io: Bradford Cross, Machine Learning Startups

Recording: ai.bythebay.io: Bradford Cross, Machine Learning Startups

[Music] thanks uh hi everyone uh today I'm going to talk about vertical AI startups um which I'll Define in a second um but first i'm going to talk about some lessons learned so my background around uh I've been doing applied machine learning since 2002 when I left school uh I was first at Virginia Tech and then I started doing hedge funds um at that time I went to Google in uh the mid 2000s I worked on a system there called Borg which is like what misos is based on if you know misos it's a distributed operating system kernel um and then I left and started doing machine learning startups in 2009 first one was flight Caster that was a YC startup we sold that after couple of years and then uh I did some graduate work at Berkeley um and then I I started doing more you know machine learning startups uh I also started dcvc at that time that was 2011 uh which is now grown into being kind of the leading AI Focus VC uh globally uh so our our timing was really good on that and the point of showing this whole thing is you know that I've been doing this kind of stuff for a while so I'm trying to share some lessons learned uh from from having a a fairly large sample size of my own experience my friend's experience and my experience with dcbc in building AI startups uh for nearly 20 years now um so first uh a few Lessons Learned so a model is is not a product you know working backward from a market instead of forward from Tech is really important uh I see a lot of people now that are diving into the AI startup game that are making exactly the same errors that we were you know 20 to 10 years ago um with these sort of hammers looking for nails and and it's definitely a huge cautionary thing like to you really don't want to do this you you want to focus on the market need and find something that matches up with like a core need for AI by all means so you don't end up building like a simple crud app which you don't want to work on but really start from the need first don't start from the tech first um another one is data being more valuable than models and the fact that there's many ways to capture dependencies between things um so there's all kinds of different model families you can use and approaches you can use for different problems and it really doesn't matter as much as being able to get the right data for the problems that you're tackling um so I see a lot of people that are sort of founding on this premise that they have some Superior new like deep net architecture and it's just not really a defensible thing it doesn't really matter very much um because in the end of the day you want to turn hard problems into easy ones anyways and a lot of the times it's about these clever data acquisition hacks and stuff like that more so than than whatever particular kind of models you're using um and also just like you know the way that research is done like in a modern setting in my view is sort of having the researchers working directly with the engineers only having research Engineers that are writing production code not having any of this more scientist or data scientist type role that does some stuff that's not really proper code and then somebody else has to rewrite it separately on the engineering team don't really believe in doing stuff that way and I think in particular for startups it's a real killer because you just don't have the time for that type of development cycle um so now specifically let's get into vertical AI startups so the rest of this talk I'm not sure how interested people are going to be in it honestly it's very Financial in for a little while here I'm basically trying to explain what the motivation financially for why you should want to start these vertically oriented AI startups instead of horizontally oriented ones like machine learning as a service or other kinds of things like this um so first um actually the so what what are what are vertical AI startup so they're full stack uh products they require subject matter expertise from a particular domain they have some type of proprietary data that they gather from that domain um and then they they use AI to deliver the core value proposition of the product it's not a bolt-on thing that you're doing for marketing or whatever you actually needed to deliver the core value of the product some kind of computer VIs or NLP or ranking or what have you is like really core to the whole thing um so why would you do this why would you go vertical what are the motivations for it number one is don't get ripped off right so when you solve the business problem directly you have you're in a position of Leverage to capture much more value in the stack when you're sort of an SDK or an API or something like that and then somebody else is satiating the end need of the user um on top of that they kind of are in the position of power right because they're the one who's interacting with the end user and they're the one who's accepting the real money from the end person who's paying and then they're paying down the stack to you you're in the weak Market position there it's really not a great business and we see a lot of things that are doing this and you really need to bump up the stack if you don't want to get ripped off um so the other one is software is eating the world so every company now needs to be an awesome tech company and of course they can't even really do that if you look in different sectors in financial services for example or all kinds of other older School domains they just can't really become tech companies effectively so either beat them at what they're doing or sell them stuff that makes them into tech companies but this is another reason for going vertical right if you look at all the other tech companies they're all F small of f uh full of smart tech people right and so you don't want to compete against them you want to compete against all these other verticals where there aren't a lot of smart tech people but they have to basically become tech companies right it's much more white space there um another one is that when you do vertical product Focus you get this proprietary data effect where you're capturing more and more data that's specific to the types of problems that you have this is really the sort of money-making machine right you end up optimizing this really esoteric problems that are very specific to a particular domain and are extremely valuable and you can charge a lot for and in many situations you can get your business into a into a position where you're the only only one that has that data right which is incredibly valuable I mean obviously you don't want to become abusive or whatever but you basically have sort of monopolistic pricing power and so on there it's the old Peter teal thing of like wanting to get into a monopoly position um and then most importantly vertical cohorts win this is what I'm just going to show right now and winning is awesome you actually want to win so Enterprise cohorts what's up with these things so this is this cool uh couple of posts that the sapphire Ventures guys put out so if anyone I'll pause here for a second if anyone wants to like jot down these headlines or whatever as a quick note to look for them it's pretty cool they did a bunch of research but I'm going to summarize it all here um so the point is that Enterprise exits come in cohorts and uh consumer exits come in these huge outliers right like Facebook or SnapChat as we just saw or Whatsapp or whatever um but let's dissect the numbers and think about what that means for us that are building a AI startups so in Enterprise we've actually in the past uh this is since I think uh ' 95 yeah ' 95 um there's actually been more total exits right um but there's been less less than half of those exits has come from the top five as compared to Consumer so if you look at this consumer actually has much larger amount of cash that's been returned from the top five exits but if you sum the whole thing up it's actually still smaller than the than the rest of the the exits that are not in the top five from the Enterprise world right so you think about it we're actually going after a pretty sweet and large pie if we just completely ignore consumer stuff and just look at Enterprise and not even in the top five right that's a big pot um and this is just further showing you know how the basically 36% of the overall returns are from the top five things in consumer right and so in reality I think you look at AI startups you know frankly I don't think you should be imagining yourself as building another Facebook or another Google where you're going to build a $300 billion company right um but the idea of building a multi-billion dollar company and having a good shot at doing so within a broad set of cohorts in different Industries I think is actually very interesting outcome and kind of what we should be aiming for so these cohorts come from waves of venture investment when when people sort of get excited about going after one of these little vertical niches or whatever and think that the timing is right for doing so so keep calm and be a cohort winner like that's your kind of goal right is like pick one of these kind of cohorts that you want to go after that you think is going to be have a material impact in an industry and just go after that one right so let's look at the Unicorn cohort so this is coming from a current list that I just got off of the CB insights guys site um and I took the the 150 or so um unicorn startups and I clustered them by their uh their Focus area right um so just taking a look here as an example right of and these you know a lot of these things are are going to flop but quite a lot of them are actually pretty legit businesses so I think we'll see a bunch of IPOs from a lot of these companies that are on this list so if you look at some of these like you have a pretty nice cohort in fintech um pretty nice one in healthcare some of these other ones are techy kind of niches like the whole movement and Ecom and On Demand uh right but you you kind of get a sense of you know also cyber security there you're going to get a sense of some of the cohorts that are up and coming where where multi-billion dollar companies already exist in the private Market um and then so zooming in specifically to fintech right um You can see that actually within all of financial services the vast majority of the uh investment dollars and traction is just in lending and payments right so I've taken to doing a bunch of stuff in fintech because I'm intrigued by it because it's one of the largest sectors in the world and there hasn't really been that much Venture investment or startup activity and it been confined almost entirely to lending and payments so if you look at like real estate and insurance for example which are just monstrously large markets right there just isn't very much activity at all you know as compared to lending and payments um so now let's look specifically at AI so now I looked at the data for the top 120 companies um in AI that currently have raised $30 million or more um so let's look at where where we see some kind of developments here again here you look at fintech and at uh Healthcare and they're pretty well represented at the top of the Heap there in terms of the dollars that have invested in the number of companies that you see um and then again like security is up there marketing is up there right so these are the kind of things where you have actual you know traction this is kind of our best proxy for traction some of this stuff might be Vapor Weare or whatever but all these guys have raised over 30 million bucks which isn't easy to do and presumably a lot of them are doing so on the backs of real business traction because these are B2B companies right so I I think it's interesting it's showing you some of the emerging um you know like sort of Industry clusters within AI startups um now zooming in a little bit on what are the problems that they're kind of solving right what are the classes of different uh sort of statistical or machine learning problems that these startups are tackling within those industry verticals so it's interesting that the top one is underwriting actually because the lending stuff has has blown up so so big right and those guys are all counted among these AI startups because they're mostly all using machine learning for their underwriting models this thing's actually the top category even though there's only six companies there there's there's just pretty big successful set of companies it's also interesting to acknowledge that that doesn't even include the ones that have gone public yet some of them have already exited so it the real number is even bigger there if you look at the cohort that's formed over the past decade because some of them are already public um but you see also just like prediction problems optimization problems Vision NLP is up there and almost at the same level um so you start to see you know a really kind of high value clusters of problems that people are attacking that are behind a lot of these industry problems too so I think that's also worthy to think about is like you know what are the types of Industry problems that you can attack with the types of things where you're strong like if you're a really strong NLP person or you you're coming out of a computer vision group or whatever you know you can kind of align those skills with particular industry verticals where they're highly valuable at the moment as opposed to thinking about you know let's just make an API for tagging photos um so and and also interesting within the AI startup world you can see that um the vast majority of the investment has been in Enterprise right um so it's about 10x okay so uh diving into finance and healthc care a little bit these are the two biggest sectors really in terms of verticals that we should think about outside of tech right because obviously you know if tech this is by market cap in the US right so Tech is the biggest obviously we're Tech so that's not the vertical we're trying to go after um so Finance that's one consumer services we're not really trying to go after that that's not a specific like subject matter uh oriented vertical that's not the goal of this whole vertical AI thing I'm talking about so we want to go after Healthcare that's the next biggest one and you can see an order there you know energy is also a very interesting one but for now I'm just going to zoom in on finance and Healthcare the biggest twoo um here notice that there's only four of these sectors that have a market cap in aggregate that's above 5 trillion us right um and and again healthc care and finance are in that group and notice it's a pretty significant drop off after that so this is really like the these world's largest sectors are they're pretty large right um and if you look at by margin also they're represented here so again if you take an aggregate this is everything every company that you have financials on in all these sectors um those that are above 8% margin in aggregate only include the four that are at the top and again you see financial services and Healthcare represented there so great uh Industries with margin at scale so the point of these slides is to show you know keep your markets big go after big markets that's an awesome starting point um and then look for wides space and timing so this wides space and timing thing is really pretty important for me um it's a for me something that I have come to appreciate over time is incredibly important and probably more important than almost anything else that you can do because number one you don't want to be part of a of a of a group where there's way too many people and too few opportunity and you see that a lot of times in startups where people back a big cohort like a theme and there's not enough there's not really a market there and then you have a ton of startups with a bunch of smart people all going after and they're all overfunded and it's sort of like this horrible death march um so you don't want this situation where there's no white space you want like open wides space you don't want to see a lot of other people doing it like for example me personally right now I wouldn't do anything in self driving cars right there's too many smart people there and maybe there's going to be another one of these billion doll exits that's started this year that some giant car company buys that could totally be the case but for me personally it looks like there's not that much white space still in in the in the self-driving car thing because there's just too much uh smart people working in that area for me um too many rather uh and so the other one is timing like why is this going to happen now a lot of times people will pitch a startup especially like a deep Tech startup and they'll start with like something is broken like X is broken um and the fact that X is broken kind of doesn't mean that much because there's a lot of x's that are broken in the world and and the question is like why are people going to fix this one now right because the default state of most stuff is for it to be you know a little bit uh messed up let's say right and so why is this going to be the the thing that I'm going to focus on this year of all the things that I be trying to fix um and and there has to be a lot of reasons like a stack of reasons I call them catalysts right but you want a stack of reasons as to why um this should be done right now and it should have very little do to with your opinion or the fact that you worked on this stuff in grad school or whatever and it should have a lot to do with why people want to buy it immediately because they really have this as a top priority for this year so really focus on the timing thing there's a cool talk about this if you want to check it out that bill gross did the guy from idea lab as like a little tedex talk it's like a little 10-minute talk or whatever I highly recommend watching that one it's it's pretty cool it really it was after one of my past failures and I was like picking myself up and and I was watching this talk and it really resonated with me and and that's what really changed my focus to be much more on timing you know and he was just like it gives all these examples of how it doesn't matter if you even had the right idea and the right team and everything else but the wrong time like even a difference of two years sometimes can be make or break and he gives the example of YouTube and how there were other people right before YouTube and they just actually didn't have the uh bandwidth you know in general in the US yet to make the streaming video work and just like two years or three years later it started to barely be possible and I think that's when there was finally the door open for somebody like YouTube to be successful um so again it's this timing thing where there's so many factors and you really have to think about it a lot um and don't be the hammer looking for a nail right um this is the the thing that you see all the time in deep Tech and in Ai and data startups and we've backed a lot of them and I've I've been guilty of this myself multiple times like you you have an idea and the world really doesn't give a like just only F focus on what people actually want seriously it's going to be better off for all of us um so future cohorts that I think are interesting uh like I mentioned I think um that insurance and real estate are awesome because these are you know the two of the biggest areas inside financial services and two of the biggest areas in the world and as you just saw when I showed earlier the sort of fintech breakdown um you just have not seen the amount of traction or investment dollars going into these areas yet but you start to see some exciting glimmers like um one of my friends and a guy who used to work with me at Prismatic is one of the founders of uh open door which has this whole model of sort of buying on on spot using their their machine learning model somebody's house and then setting it up in a certain way for people to come and look at I don't know if you guys have read about them or whatever and checked out their model but that's really cool that's an example of somebody doing something very very Innovative in real estate and I know they've been growing fast doing large financings and they're like pretty richly valued and I think a pretty healthy looking company um and I you know I'm I'm kind of looking at and doing a bunch of stuff in insurance that I can't really talk about but I think that's a very exciting area obviously um but like lending and payments for example I think they already happen so like you know again doing some analysis you you might look at that area and decide maybe it's not the right time to do anything there um I think in terms of Pharma you know like the unsexy tail end of it all the clinical trials and Manufacturing and everything the more we've analyzed that the more that looks exciting The Upfront part of it where a lot more startups Focus that are doing for drug Discovery and so on can also be a good model you can figure out a compound and just flip it to Big Pharma but like the massive amount of money and spend and big issues and bottlenecks in it are actually on that tail so that's pretty exciting for me um and then aerial imagery is another area that I think is really exciting um there's big new data sets there coming online um that are really you know useful in a huge host of different domain problems and what's exciting for me here is that you know when you take all the new stuff and deep learning and all these giant new data sets from aerial imagery and you think about all the features you can extract from that and the data sets that you can join it with you know based on Laton time um there's a lot of possibilities for predicting different things automating different tasks Etc um but when I see what the folks have done in the space so far it's mostly very very obvious stuff which is like Agriculture and defense for the most part with a bit of Finance hedge Fundy type stuff um but we did an analysis sort of looking again like I was showing at every different industry vertical what are the use cases in each vertical um what are the reasons why that should happen now you can find business opportunities where the business catalysts intersect with these technical catalysts and that becomes very interesting because now you have this massive new reservoir of data that that clearly has a ton of value in it and we have the tools to be able to extract the value now um but you have a bunch of people that are only focusing on the very obvious problems and meanwhile every other vertical is sort of untouched thus far so I think that's an exciting opportunity um so that's it that's vertical AI startups um that's that's kind of the all I want to talk about and then I was hoping to have a bit more time for a [Applause] Q&A right here so on timing how do you evaluate timing I mean it's not easy to say the timing is right yeah it's definitely not I think if it were easy then there would be a lot more giant companies um yeah I mean I think it's it requires a lot of analysis and being honest with yourself I think a lot of people could easily go and have the conversations and do the market research to do it but one of the biggest blockers is this cognitive bias of trying to validate your own ideas right you get excited about something and trying to validate your own stuff so taking more of a scientific mind early on being a little bit more kind of mind like water in terms of what you're willing to do in the beginning instead of being really attached to an idea I think is a really big deal because people face this upfront psychological block which is a real killer um and then the other one is just being really methodical and like you know looking at doing Market research going out and talking to people you know figuring out how to synthesize all the data together in a way that gives you a real picture of how exciting this is looking at other options so you look at what your opportunity cost is and like or is this really look good compared to these other things or not um so and I think the more you do it the more you develop your own sort of toolbox and techniques uh for doing that um but I suggest to people that they spend more time on that upfront than than otherwise you might and you know the idea that you just go go go go go and iterate is sort of like if you think about it from an optimization standpoint right like you're just starting at some place that you landed in the space and just sort of you know like hill climbing around right and it's like how do you know you're in the right area at all right you don't really so how you you you don't you can't efficiently explore the entire space in this little iterative like execution way so need some analytical framework to to explore more broadly first and then narrow so you know what that analytical framework is depends on the thing but it's got to involv Market Research talking to customers and like really spending some time on it before I think you just start randomly building stuff so yeah MH hey I got do you want to go first all right well the microphone's here so oh sorry okay sorry sorry no problem so uh first of all as a um DCB founder I want to say you guys are great and would recommend anyone looking for funding uh talk to you guys it's been a good relationship second of all um really appreciate the talk I like what you said about focusing on the Enterprise about it being kind of less about long taale and much more likely to have an exit but one of the things you said is something I've been struggling with which is you should focus on kind of the quick hacky say quick but hacks around data acquisition finding data and reconciling that with Enterprise B2B plays like in the consumer space you you can think of making a Facebook game or an app or something like uh the comma AI guys have their dash cam app what approaches do you take to that data Gathering to bootstrap an Enterprise B2B startup when you know companies aren't going to just hand over their their data bases to you have anything great question so I think there's two classes of approaches here one of them is the approach where you know uh I have this thing give me some data we plug it in and you get instant value out of it that's one approach this the tit fortat strategy let's call it um the other strategy is um this sort of outside in strategy where it's like I'm going to go and use a bunch of information extraction and crawling and snarfing variety of data sets from all over the place and I'm going to go and do some like weird deal with my friend startup and get data from them or you know just anything that you have to do this like hustle and ground kind of like acquiring data aggregating something together so that you have something from the outside and then you come in and you're like listen you know I already have this like relatively canonical understanding of thing X you know um I can I can tell you this about your business now and I can tell you this other deeper thing if we like plug your data together with it so that's another maneuver that can work but I think it's you're kind of doing one of those two things and there's always this argument that this is this bootstrapping problem that you have in the B2B space with startups there's this argument that the companies have it's like you're you know 3 months old how could you know you don't have anything you don't know you don't know about right like you don't know anything you know why am I going to do give you data or do anything with you um and you you got to have an answer for that which has to be that you've done some clever thing in terms of being able to build something from the outside that you can immediately show value with if data is plugged into it or that you've been able to gather your own data from the outside and and show value even into dependently of doing anything with them and then you can show them how like there's even more value if they plug their data into that um so I do think you need to watch out for um you know things where you don't have either one of those value propositions and you have like a dubious cell where you like actually don't really understand much of anything and you're just trying to get them to give you their data and present you with the business problems that need to be solved and stuff like that right and there's like these super dubious early stage startup prop positions like that you got to make sure you have something you know when you come in but I would run either one of those two playbooks those are the ones that I see like work consistently yeah okay over here now yes uh thank you Bradford um if you're looking at areas where the validation is going to take longer than a few months it's going to take perhaps even a few years yeah and you're also looking for that same kind of quasi monopolistic lock on the data become the The Source doesn't that argue to move down your list a little bit further than your top five and be a little bit more anticipatory yeah yeah yeah I'm not necessarily saying that you should need to focus all your energies in those ones that are already always the that are already the biggest ones right like I was pointing out in fch you know that you saw already there's been some some number of modom of winds with the with real estate and insurance but not nearly to the effect of lending and payments and I was advocating for that as an example of why you might focus on those two areas instead so yes I definitely agree um but at the same time you want to be careful about there's also this other phenomena where a lot of the times the reason that nobody's working in an area is because actually nobody giv cares about it right and that's also a trap too so you got to be careful about these like the white space that's like that for a reason because there's nothing over there right so yeah that and that happens a lot too so yeah it's actually a black hole of energy right exactly that exactly over here I think um great talk I really enjoyed it Jack Porter with Razer think and and we do vertical but instead of being industry vertical we're kind of like segment vertical so we do churn upsell um campaign Effectiveness yeah but we do it in finance with American Express and Discover card but we also do it in telecommunications with with um Reliance you compan how do you see that play into this kind of yeah it's it's interesting it's there's sort of like this like tools level of stuff in B2B which also includes like you just like a lot of the sales software and marketing software and stuff right so I I would almost say that it it is in a like I showing up there um you know that there's like the sort of marketing like cohort right within the AI startups and and so I think in a way you kind of are you know specific yeah vertical within that sort of like area that sort of like task based area but you might have like multiple actual industries that you serve um same as like a Salesforce does right um yeah I mean I think you get a lot of the similar types of effects y you get the same kind of thing with cohorts happening and it's just sort of like it's a layer down the stack from like the heavily domain specific like super vertically oriented uh in terms of Industry vertically oriented uh startup but I think that there's there are going to be successes there and there already have been you know um and when you talk about like marketing and sales and things like that or customer engagement um and and especially uh you know again focusing on the key verticals like Financial Services where there's a lot of you know dollars for that type of software um I think those companies will be also very successful and we'll get some similar effects in terms of potentially even like Network effects of data and things like this as they can get people multiple customers to op into sharing either data directly or if you can't share data sometimes you can share like you know what are called derivative Works where you can share like learned model parameters across customers where you can get the similar effect yeah so I I definitely I definitely think that that stuff is uh interesting and you you've still moved way up the stack to like a direct business problem right and the the the thing is that like the function of sales and marketing across different verticals is is very very similar so you get a lot of reuse whereas you know something like uh working in a back office compliance problem in a bank might be very different from anything else outside of you know banking or fintech um so you're not going to repurpose across verticals so where yeah uh as you know technology investment in finance is probably one of the most active uh among all the sectors out there you know ever since the 70s banks have been deploying you know kind like models that utilize some form of um you know like learning or or uh and now today they're doing machine learning so when you look at Finance as overall industry in terms of how competitive it is uh how do you assess kind of like competition from startups versus the incumbents like the banks and hedge funds that have created AI teams and things like that um so yeah so I I actually have a a really uh lengthy post about this that I that uh is on my blog that talks about like quants versus machine learning and like the era of quants and finance versus the era of machine learning so the way I think about it is actually pretty simple it's like the the Quant era is like still very smart people of course but they're more trained in math stats physics stuff like that and they sort of of they hack stuff together in like a systems building sense and the Machine learning era is like these are people who are computer scientists that are also statisticians and can do all this other you know fancy math and optimization stuff um and because they're computer scientists and Engineers you know foundationally the stuff that they've been able to attack has just frankly been like larger scale the results are superior and they've just sort of like dramatically eclipsed what the Quant world was able to do um and I I think that those companies actually frankly like have struggled to adjust when I left the hedge fund world I I was looking at you know the top uh the top shops like dsha and all these kinds of folks and when I looked at what they were doing it was honestly pretty crappy you know when I when I looked at it it was worse than some of the stuff I was already doing and I didn't think I knew what I was doing and when I looked at what my friends at places like Google were doing out in Silicon Valley I was like this is 50 15 years ahead and this was around the 2005 to 2007 time frame when I was seeing that um and I think we it the the new approaches have eclipsed what was happening in Banks and hedge funds for sure they're they're really trying to struggle with how to figure out how to catch up right now but fundamentally they're just not like Computing organizations that were ba built on like CSN engineering like in a deep way and like a company like Google or or even Facebook now is so you they're trying to hire those people over there like really desperately um you know you'll see this at the big banks at the i banks at the hedge funds they're trying to hire in the Google and Facebook people they're paying huge amounts of money to try to bring them in and help them to like retool for this era and some of them are more successful than others at doing that but it's a big transformation and that I think the scale of these systems and the sophistication and just the results of them and the problems that you can attack are all like a level up from what was happening in the Quant era okay yeah so we do quite a bit in compliance and fraud and there's a lot of privacy implications and bureaucracy around that so even just obtaining data can take three to six months and so what do you what have you seen in the startup space when when you when you when you have to deal with those kinds of problems because I agree data is the most valuable thing out there but when it's difficult to obtain you know you you you spend a lot of time in this kind of this white space right absolutely yeah it's a awesome question and I deal with that a lot because a lot of these that you know I told you I'm in interested in fintech stuff and in particular like Financial Risk problems like this um because it's a little bit underserved I think in general um and there's a bunch of other reasons but the point here is like you know that's sensitive because you're dealing with risk and things that that people don't just share easily so it you can have a big weight cycle for getting like the actual data that you need and you have this sort of like Catch 22 of why they would give you that data in the first place um so there's a number of approaches the first one is definitely just as rigorously as you can search to see what papers people have done in Academia and who has might have some sort of anonymized data sets that you can get your hands on that are going around in academic circles often times you can find stuff like that um second one is uh simulations right I think that's one of the one of the sort of like newer areas that you'll start to see is these these you know these sort of Big Data machine learning problems where you're just doing a vanilla classification or prediction problem at with a massive amount of data like we know how to do that there's tons of stuff you can do it's really a solved problem this this issue where you're talking about where you're trying to tackle a problem and you have like either sparse data or no data and you have to bootstrap there's a lot of problems like this and there's much less work what I think you're going to start to see is more clever ways of simulating this stuff to generate a lot of data just in a fully bootstrapped fashion um or other like these sort of like creative hacks like a distance supervision or weak Supervision in NLP where you start from you know something and bootstrap your way in with heris STS to a noisy data set but it's like actually good enough to train with no human in the loop so I think a lot of those types of tech techniques for like self bootstrapping uh tricks with simulations and other things um and then and then I think the last one is like I I was saying earlier this sort of like this sort of like these two tit fortad strategies where you have you you have something that you have built that you can give them immediately like you can tell them you know we've we've identified these Complicated new patterns based on things that are happening in the dark web right and we've got this dark web data and the we can run it against all your transactions and anything that hits we're going to tell you and it's going to you know this is an experiment and we can like run it for free and if we get a bunch of stuff and some of it turns out to be true then you know you would want to work with us right um so you can come up with some some of these sorts of Tricks where you basically do like a they immediately want to opt into sharing the data with you um and then another one is more of a BD uh customer development type of a of of a hack which is like you know go around go around broadly talk to people find out if somebody already has like a little data a quote unquote data lake or whatever set up with some data for doing this because they have some strategic initiative where it's like we need to kill like cross Channel fraud it's like really big issue for us this year and so therefore we've set up this data set and we have some data scientist guy working on it but if you guys are going to tackle this too you know maybe we can get you in on a p around this data set so you can also you know maybe nine people tell you to buger off but one already has this setup and sometimes you can find that so it's a bit of a crapshoot but if you can find it that's awesome because that's just real data yeah this last one yeah yeah and more and more people are doing that and they're really excited for startups to work on it often times if you play the social engineering part of it right also the data science person on the inside will be excited to work with folks if you have good folks because often times they're like the lone wolf in there who are trying to learn how to do this stuff and they only kind of know what they're doing and it's exciting for them to work with like an awesome Silicon Valley team so absolutely that can be a lure right all right we're going to have one last question back here um cool all right here you go great so going back to the hedge funds thing for a minute uh there's been like a cottage industry of data startups UPS around alternative data so aerial imagery credit cards it seems like a lot of these startups are basically their premise is while we can sell to a new wave of hedge funds a data set that nobody else has and hedge funds are in turn hiring like the types of folks at Netflix and Google I'm just curious where you think that's going to go and what that's going to look like in five years um yeah I mean I I mean it's probably not going to surprise you that I think it's a lousy model right um it's just like the idea is you're just selling these Alphas right you're just like basically you're doing feature extraction and flipping these Alphas to hedge funds it's it's exactly the naive approach that I'm trying to speak out against with this whole vertical AI thing which is like all you've done is a very very simple machine learning problem it's like a very low-level task-based thing and now you're turning around and saying I'm just going to sell the data from this right well what you've done will be commoditizing very quickly right A lot of people tell themselves no we're so smart and other you know other people aren't going to figure this out or we've you know we've got the best team in the world for x or whatever and it's just all not true right it's just like you you've done you've done a classifier and you you're trying to make a business out of it and you don't really have anything so you can you know in the very beginning you can make money off of these Alphas right and there's people that are doing this actually exploiting it and making a ton of cash um but it's sort of like this highfrequency Trading type stuff except a much smaller Market where you sort of like can you can get some Alpha on it for a while and then the hedge funds will if they if it's valuable enough and you're charging them a lot they're just going to either do it themselves or whatever they're just Outsourcing to people right the process of generating these Alphas so if you want to know my thoughts about it it's sort of like uh there's a there's a company called World Quant right which is this crazy evil master plan that this uh brilliant Russian guy came up with to like Farm out the generation of these Alphas to a huge distributed network of people who have no idea how their Alphas are being used in the overall trading system that's basically What's Happening Here is the hedge fund industry is outsourcing to startups here go and like dig around in the sand and if you find any clams like come back with them right and then and then gradually once we know where the good spots are then we just cut you out right so I don't think it's a good long-term model maybe you can make some money in the very short term but I think the more interesting thing is if you can actually price something better based on this data then why don't you go out and price it or even better like what's sitting on top of pricing it is it like lending or Ender writing or doing something in debt uh whatever it is try to like move up the stack and don't like play in that space would be my advice great thank you cool [Applause] [Music]