DevReal: Structured Output with Ollama, Parth Sareen
Recording: DevReal: Structured Output with Ollama, Parth Sareen
keeping the energy up nice all right so uh we already did a who uses olama but I I kind of want to see it again just so I know how deep to go into um who knows AMA and uses olama okay we we have we have some people um so I'll still run through like the generic you know who are we and what do we do um and so we'll get started there so AMA is pretty simple we run large language models very simply it is super super easy to run them um and often people will run them on their laptops um so even earlier we had the great Granite talk um that Granite model is available on AMA as well um and you can search for various different models through here I hope it is visible and you can kind of see and use whatever fits your model and so um just to give a quick run through of what AMA run looks like um we can just do AMA run deeps R1 so this is actually a dis old version of Deep seek I believe on quen um just to give you guys an example of what it looks like so I can just say hi and a thanks right we can ask it 2 + 2 but we also don't want to be here forever so I will tell it don't overthink and so clearly thinking a lot I I like to think of it as a gen Z who you know is very anxious about what it's thinking um and there we go it has put out four as expected all right so one of the things I'm talking about today is well you know this is cool and all but how do we actually make use of this uh one of my favorite uses for large language models is going from data which makes no sense you know where you can write rejects for hours and you cannot get the data you want um to data that is clean and outputed nicely so that's kind of what we're going to be covering um we can dive right into the code so over here very simple um if you haven't used pentech before or our old llama python uh package we're use making use of both over here and I'm doing a couple things the first thing I'm doing is defining a friend info class in which I have name age and if that person is available and over here we have friend list so this is just a list of friends so in this case I am passing this content which is I have two friends is it uh visible for Everyone by the way good sick um I have two friends the first is AMA 22 years old busy saving the world and the second is Alonzo 23 years old and wants to hang out return a list of friends in Json format so usually you'd actually want to pass in the schema as well for better performance but in this case I have not done that so let's actually run this and see what the llm spits out great so that was pretty quick I can check what model I was using so it was just the llama 318b and you can actually see that it returns this very well structured object from just a string I decided to pass in where it says is able to pull out that friend information that I was looking for and so you can extend this to whatever use case that you can think of whether it's going from random receipts random data to something a structure in a Json or within your python or typescript as objects and the one of my favorite things about this is that it actually works for images as well so over here we have um defined another couple objects uh for images instead so over here I have a class object and I also have image description so essentially I'm just showing that it can be nested as well and so over here we can have objects and I've just passed in couple images or one image actually so so I have this beautiful sunset which I believe is from Paul Alto obviously and I'm using llama 32 Vision in this case and so all I've said is analyze this image and return a detailed Json with a little bit of this information and then we also set temperature to zero to have a bit more deterministic output and all I'm doing later on is putting that Json back into a pantic object so let's run this this one takes a little bit more time obviously because it's a bigger model doing the entire image portion of the model as well and I will try and keep talking while this runs so it's not awkward at all and we are still waiting it is running locally on my Mac so that is pretty cool but we are still waiting all right taking a bit today okay there we go so uh actually I forgot to save it and we got cow instead so we actually pass in this image and what we got was a red race car racetrack red afternoon and the setting is outdoor and all of those things are pretty accurate and this is really cool considering it's a pretty small model and let's actually do it with the sunset now I'm going to run the same thing I just saved my file it's going to take another second few of those seconds doing is I've only taken like a very small portion of the entire workflow but the first thing that the model is really doing is a forward pass this is where it's taking in all the information that it has and then coming up with a set of logits what are logits they're essentially probabilities or unnormalized probabilities which map to different tokens in the like in the model's vocabulary so this can be things like a b c but it can also be full words like name banana I don't know like anything really that the model's vocabulary has trained out to be and so what really happens is the model will go through this forward pass we'll give you these logits and then there's a sampling process where it's looking at distribution at the probability distribution from those logits and then picking a token so in this case it picked token two now in structured outputs this entire process looks a little bit different so we have our models forward pass we get those same logits but while we're also doing these things there's something we also need to do which is we pass in a Json defining the structure of the output that we need back and so in this case just very simple name age and if they're available very similar to the python example that I showed and this is what is essentially creating okay let me look at the Json and let me see how we can use that to work with the llm so what state-of-the-art uh sampling methods we'll use is essentially a state machine uh we're currently using a rules engine and one of the new ones that I'm working on is going to be using push down automa which are also State machines and that's kind of more on par with suit of the-art performance in terms of both accuracy as well as uh speed but essentially what the rules and or the state machine are doing is you have like your entire list of probabilities but not all of them fit your schema right like if I if I have a Json and I don't see this open curly brace first I don't want any other token to be coming back so essentially what's happening is we do something called masking of the logits and so after it's gone through the state machine we come up with another essentially a mask to go over and say which tokens are valid or not so this is barring probability just tell me what is good and bad and so what essentially ends up happening is you get a bunch of invalids so in this case only token one has like a positive probability or like a non uh as has a probability in this case and then we again do sampling and then we get the token one back in this case so you can kind of think in that similar fashion the way that ama is doing structured outputs is also the same where we are Computing this mask and figuring out which tokens are valid and which tokens are not and then we kind of go through that entire process of generating a Json which is what's being sent back and forth so there we go our Sunset which was over here goodbye lightning mqueen so beautiful sunset and we can see that it's described pretty well by our 11b model in total which is you know it says it's a sunset orange sky with yellow Hues Hills trees and you know it's even describing the scene very very nice um and yeah that's kind of the main use of structured outputs um if you're interested in olama ama.com is where you go and you can get up and running really really easily be there in a second um and essentially there's a few different things you can also check out so we have our main repository if you would like to contribute or just kind of follow along and then for more so using uh AMA we have a JavaScript library we have a python library and we're also opening eye compatible so you can just change up and say that you want to point to olama and it'll just work and if none of those things work you can add me on GitHub and we will try and fix it cool all right we can go for questions now thank you bar I think someone over there had a question hi so um I did a project literally yesterday on a hackaton on githu because uh I knew AMA but the very basic functionality that I saw like on the um uh quick start part and uh I used it uh it went well I actually did a nice project but the thing is uh it worked really slowly for me and I didn't realize I could do structor out s um and all of this looks really good but I couldn't find the docks for some reasons so I'll just point you to it so if you're using the python package um you just go under examples and then we have structur outs should right there yeah so it's yeah um and the other thing how do you make it go so fast because I'm using really light models in a this computer this computer is a little bit bigger than that it's yeah this is just a really beefy uh Mac but in general what I recommend like having gone through this a bit is just one get a cloud instance and deploy what get a cloud instance and deploy with like a GPU or you just have to use a smaller model um or a different kind of quantization so there's varying I mean I'm using actually uh the 1.5 million all right so it's very small right and this is not a a a light computer this is Beefy so um yeah it might be going way too slow even for like yeah that is that is interesting so maybe it's a GPU thing this is not using GPU we can check that all thank you yeah other questions beside tech support it's all good that's like half my job did you try restarting computer um hi um for the structured output is that something that's dependent on the model or do is it always work does it vary in performance yeah uh the performance is a bit uh dependent on the model size so bigger models perform much better just because of how much how many neurons essentially exist in the model um and they're able to make better pred and that's really what's happening at the background anyways like the model's still picking token by token what we're doing is saying what tokens can they pick or not um and for AMA you can use any model with structured output so as I showed you like we use a tax based model as well as a um as well as a vision model now uh with reasoning models it's a little bit different because they have to go through this like think process um the hack around it for now is you do one request where you get all the thinking tokens you grab the thinking tokens you inject them back and then you get it to do structured outputs cannot guarantee any good mileage but that's something you can play around with let's do one more question Bart I noticed that you're putting the schema within the structured output definition is there an opportunity to go directly to like schema. Json to get the schema of a particular domain for tools instead of structured outputs yeah uh actually we can do kind of that already um if you're using the python repository I'll just do a quick example where are tools all right tool up P um yeah actually you can just pass in the python function itself and I have oh let's do small I have parsing which will kind of take care of that for you so over here we have two functions one is add two numbers and subtract two numbers very usual examples and over here you can see that I just passed in the function itself into tools so the python U library deals with that for you I reference that structed so this is for tools and structured outputs is a little bit different um in structured outputs what is happening is you can Define these pantic objects and you're just dumping the schema you see the schema line that you have there my question was oh this yeah scha line can it be a web U that I can reference a schema oh I've not tried that I'm not sure you could just go to the web load that schema and put it no because sometime the doain yeah so if it is really big um I usually say that it impacts the performance just because of how much uh how many tokens you're stuffing into the context and it really just varies on how much context size you've allocated all right thank you very much bar [Applause]