Inventory History as Pure Functions

scale.bythebay.io: Sky Yin, Inventory History as Pure Functions

Recording: scale.bythebay.io: Sky Yin, Inventory History as Pure Functions

you alright thanks and France stitch fix I guess majority of the people in the room probably are already heard about this new company in Silicon Valley just a little bit has up i I think I originally thought this talk was for the for the data track somehow maybe that that was my mistake and this is putting the in the functional track so I guess like our folks more like a high level design and instead of the real implementation so I will go over a little bit about the contacts for this internal project we are doing a stitch fix and especially with so you so I guess you guys audience can understand why we want to do this why we cannot use existing time savers database to do this then I'll go over some of the high-level design and implementation of tracer so I guess how many people already familiar with and what our community alright so it's basically personalized the starting service I guess majority of the fashion company us they have had caught on in New Yorker but we are probably only maybe maybe two exception neck on the here we have Corey in San Francisco to majority of the our user we feel like they they think us as a man magic black box with certain kind of certain kind of artificial intelligence like they're actually some user they they are tried to like they try to like educate or all the other pairs like there's a certain tricks to to cheat our system like a you were potty or a like a to two years or three years old three years younger we may recommend something better there has kind of funny but what happened internally is we are combine the recommendation algorithm with our human studies the curation so there's a long history why we do this we actually internally test if we only rely on recommendation algorithm and only rely on human they are both worse than the combination of of these two so that that's what we do here there's a link here we have our query at like a pretty fancy of the date data visualization - like a world cursor order process we do internally in our data team in in order to like and about enable this kind of work so why does this talk is focus on our internal database tracer are like per which is a inventory service so why do we care about inventory so when you think about this as a recommendation our problem we want to recommend to each client what they may like on the other hand we need to have stuff in the inventory in order to make recommendation you can't really make any method recognition if you don't have good content just like in the case of nav nav effects when they first discovered the power of recommendation then they quickly realize they need a good content so on the other hand recommendation algorithm actually work in two ways we we we normally call this the way like a reverse algorithm in the concept that you if you know this client you know that's kind of kind of what kind of image we shall repair should we prepare for this kind of client so that's also in recommendation at a task that we recommend to the merchant I think we need that kind of human so we are not only providing in order to provide a recommendation for each plan do we really need a personalized entry to serve good recommendation here is just a flow of the thing for each item in our warehouse they have to they have to go through these different stage and in the eventually they will end up in the in the in the box we sent to the final client so an item in the warehouse the first first they have to be available and they have to be little then they have to be qualified qualified basically ideas are the client where like a specified they don't want certain material for for example like a polymer okay we were never sent and in polymer that has great impact or the imagery actually then then that less of imagery will go through the our recommendation algorithm and then the algorithm will only recommend a small set of thing we think who will has a highest possibility that Klein will buy so studies that will only see a very small set so in in the sense that they can finish this task as soon as possible otherwise I mean the otherwise one of the better proposition of fixed services we say time for our client if you're going to spend like an hour online shopping and our studies to only spend 15 minutes then that's the value so then status was select five item in the in the box and and finally they were they were check check check it's kind of like like online shopping but they are doing online shopping for client not for themselves so in this context we need a time series database so we can understand what's going on in the inventory at any point of time then when stylist was when start is the walking on that's this kind of venturi for this given other the disk wine so then I'm gonna go over a little bit the design of the tracer this is a little bit abstract i apologizing and england events i see the time so tracers basically it consumed III events to in order to like a continuously consumed I don't advance in order to build a time series database but when we think about what kind of I at any events what what kind of pansies we want to build their a more multiple solution actually probably because of the the determent times nervous is kind of ill the fine this is the one example let's say we want to have a time series of skill counts skill basic means as some minimal kind of stuff you can have multiple you have have some certain quantity in your warehouse so this time seems like a the the counts for skill one at time at time 1 so we get events then we get another events in time to dance and go on so question is how can we build something that can tell us the count for this given skill at any given point of time because the this time t1 t2 t3 are they are dead squid so one of the quick response you can say oh this is are asking too much I mean if we are you if you want to wear it this state at any given point in time I mean even whizzing a second that's probably too much let's just do it like I say let's do it well let's generate this time series at a predefined interval say 10 minutes so we basically we snapshot the transaction database from engineering team every 10 minutes every time is we do a count we do a group I count from the engineering database because engineering the database they walk walk on the i-80 level every item is a row they have to work on that level but we we are carrying we in in the data team we care more about the the quantity of the thing not for each individual item so this is actually when the this actually that this is a solution what we had like a beer for week we build tracer we're constantly calling send the query to the engineering database and to the point that they they really hate us I guess it's you're sending a creative constant prayer prayer prayer pressure to their production database and majority of time and this query is kind of a kind of work weighted waste of lots of computing power on their database and also in during the during the busy time during the peak hours like invoicing a day it's not very reliable sometimes this query take more a couple minutes to finish so we cannot guarantee the intervals always ten minutes another problem is there during the very busy during the busiest hour of the day like a we can have too many things even a couple hundred events like that kind of event I can't change like happening every second and then we really want to understand what's going on and we can't really tell from these 10 minutes naps like for example like there are four especially some very popular item you see there the the quantity in these snapshots is oh-oh-oh-oh-oh-oh or always zero which means they're not available but if you look at transaction log their management actually happen between these ten minute they were within these ten minutes so which means like at it as as soon as something like all available in the imagery for this given popular item status were we're just we're just grab grab it and then it becomes there again so that's like candle that's what happened in the in the imagery dynamics so we really want to know exactly what status saw when they actually start when the actually in in our internal starting to snap feminist Naturals is just not enough so another solution we can go to another extreme okay left every second of course that's not gonna be realistic for the engineer inside I mean we already create enough like a enough enough load on their database and even if we only query they're like they have a red rabbit replicates of that of the main trend trend transaction database or we're only query we only snapshot against the the slave but still that slowed down the master because we hope we slow down the synchronization between these between the mass and the slave and and also they are as I said engineering database walk on the ia to Napa which means we they always have to continuously do that through by pre and premise is a waste of the computation they of course they can maintain account table but I guess they don't want another issue is like a during during midnight which is where my daughter has died so we only have market country in the US so majority of the studies that work u.s. I probably should say all you can imagine your midnight there's no me anything like I'm moving in out in the inventory so if we're doing this every second that's basically pretty much pointless so another idea is like okay fixed interval if we're on a constructs interval like a time series is not going to work in in in in these above cases so let's do it in another way so we only generate event back and when something happened to the to tutor to the skill only the only the skill like only the count when the count change happened we were generating advance and sand and create this string in this way this is actually an interesting idea because probably uh probably you you are remember in the morning the during the keynote the speaker Hollander mentioned time see risk you can in one way you think about this it's about time but also time can be defined as events as as we flow through all the events that that's the definition of time so in in this sense ago so we have a very variable this had t1 t2 t3 can happen at any time we we don't guarantee we don't have any guarantee they're gonna have happen every second or every ten minutes again they're still constrained on the engineering they have to add still have to do aggregation another problem in this solution is a t1 and t2 can be far away to each other in the case that the item is very unpopular there's no events happening like to change the quantity of that one and if we want to know that quantity of that I didn't at certain point of time that between the t1 and t2 we have we have to go back to very long long to in order to find the t1 then we can tell okay the state at account for this thing is account one at t1 because that's the only thing can't have it can happen so imagine in this stream this is just the industry and there are multiple skills so if the for this given skill one we want to have to go back that far we have which means we have to go back through all kinds of other unrelated skills that happen in between in order to decide that's count for this for this skill of course there's a there's a solution from computer science like we can index we can index that time so it would be quickly we can quickly locate the that that that put that point of time in in industry but we have to employ index for each skill that's too much we have a tens of thousands skill in the in a warehouse so another way we're going to think about is to we only record the we only record the state transition in in this string the advantage that we defer the process to translate differ the process how we gonna use this transition we can say in this case we only care about account so all only these transition that affect the quantity of the skill we were later turn that into another string in other cases that we care about our okay we only have our things got damaged our god may be missing or things got returned there are many kinds of events so that give us some flexibility we can turn that into a change in this case I call them Delta Delta basically means the change in the in the quantity with that we can construct the state basic construct account for any skill as long as we have an initial state just to start with this actually what we actually do and inside the tray so that this is the final solution we we just have to construct in the streaming in input data so tracer cam can use this so in summary the whole design can be summarized as two pure functions one is inventory state function we only need two the only parameter is time T the other thing is the transition it can be including many state transition you can think about and we can reverse Li apply this transition in order to compute the the change between two state in the inventory so the implementation we consume the item events in the in the in the previous describe the way actually as I said we need an initial state but in interest they only have a one state is not enough so which means every point we wonder we won't have to go back to the very beginning for in then calculate the back so we created these snapshot every hour then I if we query any time in between we just need to find the previous our snapshot and apply the difference in between to speed up of course we can create index for this snapshot we only need 24 you only need index for 24 snapshot in a day this is all built on spark and with the scholar and the other result all the internal data store has sparked a deferment so the advantages that all the other ETL stuff can be integrated with with with the with the result we press Scala and Python API Python API are actually important like a majority of data sense using Python or are inside of stitch fix that that's a little bit rough [Applause]