Enterprise-ready document processing for AI | Kerim Satirli & Tu Nguyen AI By the Bay25
Recording: Enterprise-ready document processing for AI | Kerim Satirli & Tu Nguyen AI By the Bay25
So, good morning everyone and thank you for joining us right before lunch to learn about enterprise ready docs. My name is Tu. I'm a product line lead at Clerk where we help apps manage their users and orgs so you can focus on building your apps instead of building arbback team invitations enterprise SSO and more. >> Beautiful. And my name is Karin Satiri. I'm a um senior developer advocate at Hashi Corp, an IBM company uh where I focus on infrastructure and orchestration workflows, telling people how to implement best practices with our tools, but don't get paid when stuff breaks. Perfect job. So, um our room manager introduced this as it's going to be an amazing experience
And that's not how it started out for us. Um we had this problem uh a year ago at this point one and a half years ago uh you know like it's Friday you're bored you you've got a weekly hackathon and somehow those always end up either being a front-end project or something with robotics for us uh we ended up building those little robots um this little tank you're seeing here uh six wheels only four that actually can be driven a lot of terminology and jargon that you will forget the second you're reading about and you'll find yourself in a situation where somebody in the company sees that and goes like, "Well, you know what? We should have that for a conference." That's a great idea, right? Um, now all of a sudden you're trying to shoehorn an HTP API into the robot because I mean, when has it ever gone wrong to give connectivity to robots? Uh, I think Schwarzenegger had some thoughts about that. Um, but it was great except if you've ever done any IT project, you know that there's a lot of spec sheets that are badly described, usually badly designed, um, multicolumn, which of course is actual killer for um, AI models. Um, so there's a lot of stuff that can go wrong. And to give you an idea, what we had to deal with was uh Wi-Fi spec sheets um and the actual RFC's surrounding it. Uh what else do we have? Um SDKs and formulas to calculate distance, which of course is not as simple as giving it feet or inches or meters or yards. No, it's it's LED pulses because each wheel is considered and treated in the hardware as an LED. Yes
light emmitting diode. Um, no, that does not make any sense, but there is some learning there. I'm not a light expert. Uh, so having an AI that could help us figure out an assistant, sorry. Uh, what we could do with that was definitely useful. I mean, as software engineers, there's nothing more that we want to do. This is intentionally left uh dark. Um there's nothing that we want to do more than write good maintainable code, right? That's I mean that's why we use AI assistance
That's why we spend our weekends and evenings improving code, improving documentation, improving everything. In our case, we wanted robots that wouldn't drive off the table and kill themselves because things get hard when you have to constantly replace them. Uh our first iteration we brought I think 12 devices and we brought home one that was still functioning after three days. Uh the rest helped us improve the software uh by virtually killing themselves. Um so build safe code, right? I mean stuff that fails safely and and is easy to deal with. But even those simple robots have so much domain knowledge that we depend on. All the stuff you don't see, all the stuff you immediately forget, the stuff that will affect how you work with your code. And you have to figure out how to deal with that
And that's a hard part. So a question that we like to focus on is how do we access the domain knowledge that we need in a way that's userfriendly and doesn't break the bank. Um I think that's that's not actually the actual question. The the reality of this question is it's not so much about the knowledge itself but rather about accessing the knowledge that we need to have that our system knows or should know about things that I have memorized. You might know some parts of the Wi-Fi spec. might know some parts of the former we're using, but you're never going to memorize all of it because it's not relevant to you. And this translates 100% to every other software engineering project you're doing, right? There's there's never a need nor a time to memorize everything because I mean, break fast and um uh sorry, >> ship fast and break >> ship fast and break um things is yeah, I'm I'm just in the break fast area. Um but thank you to likes to ship I break
Um we're we're specialists like that. And so you know the the easiest thing here is we're all data hoarders, right? Um your default smartphone these days has what like 128 gigs of storage. There's no reason to not store all the PDFs that you get uh from the vendors of your hardware. And then what do we do with that? Um well in the past this was easy right? You spin up a knowledge base confluence at Lashian will sell it to you for 10 bucks get your first uh nine friends in and before you know it you're paying a lot more. Um the value is definitely there if you use it but now somebody on your team has to become a librarian and make sure all the documents that get added are treated the right way. They are handled the right way. They are versioned the right way. um you know it's it's a thankless job and most teams couldn't run without it once this starts becoming a big thing but you know um fast forward to that you could spin up media wiki or whatever um what else notion pages I think could work all of these just have that one thing in common it's very much you know pulling information in a traditional way so of course we make life easy right AI chat app doesn't have to be chat GP PT can be anything that your company hosts
You always get this beautiful question like how can I help? It's like well the thing I need you to do is the thing you can't do not even with MCP because there's way too much stuff involved here. But the bigger problem is you might have an admin interface, right? And normally that means upload the right file and then query your model based on that file. It's great if you have 10 uh 10 files you deal with and you know exactly in what file your answer is. Except the thing is most of the time if I know in what file to look for the answer, I'm still going to be faster with my keyboard than most AI models. Also a lot cheaper because my GPU can be running Minecraft in the background. Um and I don't have to fall back to that. Doesn't matter if you use OpenAI or Anthropic. It's it's usually the same interface
um upload a file, ask questions on that file and then of course very little memory. So um if two asks the same question tomorrow and doesn't have access to the file, somehow it's all forgotten and that sucks. But when you find yourself in a situation where you don't have just one file, not just your um uh what was it? Admin PDF and API MD and maybe a few other files. Um, this is the overview for our robots, which as much as it looks like a micros service is not, it's just really badly designed software and hardware. Um, knowing which file has what is not worth it, right? So, um, before we move on to the solution, I just wanted to take a peek under the hood of what Kareem just showed you. So, at its core, this is how most chat apps work. The user uses a chat interface. Uh so Kareem was using open web UI but this could be uh open AI uh Google AI studio or anthropic uh and the user uses this chat interface to request something from the model itself
Um [clears throat] and then after the l model finishes thinking it returns a response and that's what it looks like. So if you have a question or a request about a specific document, you can technically just copy and paste the entire document into the prompt. And remember, this is only if you remember uh and you know which document it is. Or in uh Kareem's case, if you're using a knowledge base like Confluence or notion, you can uh reference the document with a hashtag. Um however, you run into uh a couple problems pretty quickly. If you copy the entire document into the prompt, you might hit the limits of the model's context window. When this happens, the model isn't as responsive. It's super slow
It might return incorrect data. In addition to this, um, O Lama has a default context window of 4K. Um, and then most models have a default window context window of 128K. Um, to put this in perspective, 128K is about 50 pages worth of content. That might sound like a lot, but then every single time you're going through the slow, it's taking the entire conversation and uploading it, right? So that 50 pages of usable content is much much less, especially if you're having a uh full-on conversation with the document. Uh, also this workflow assumes that you know exactly which document will answer your question and then only one document would help. This is a pretty big assumption because you don't know what you don't know. Next up, the document is easily called be a pasteable
Uh, this is again great if you have like something like HTML or markdown, but this isn't true if you have images or PDFs that are like scanned in or you have videos or um like audio files. Last but not least, your documents might uh contain sensitive uh data. Yesterday's talk there was a healthcare talk, right? So, if you really trust to uh Anthropic or Open AI or Microsoft and upload all your sensitive health data to them, go ahead. Uh I'm I'm not sure that's the best idea though. Um so that leads us into our solution. What did we build? So the first two problems are centered around how do we get the relevant data from our documents, especially if we don't personally know what's in each document. as you saw earlier uh with that uh that mosaic of documents uh I sure did not read through like all 30 documents I don't know what's in them so like how do I determine which one is relevant um and this is also useful for like if I spent like a month reading through all of it I know it but then now Kareem has onboarded to the project or someone else has onboarded to the project how do I share that knowledge itself Um, for AI, uh, there's this tool called, uh, or there's this method called rag or retrieval argumented generation, uh, that kind of solves this problem. >> Wait, so if you haven't read through all the files, how did you approve my pull requests? >> I mean, I just threw it into uh curs to me was like very quick
>> Yeah. Uh so I definitely do on my job and it's definitely not cloud code in the background or cursor. Uh so um this is what rag is um because I know a lot of people throw around rag but this is actually what rag is u first you collect all the documents that you want your chat app to be aware of. uh then you use an embedding model to um convert all these documents into numerical representation and store them into a vector database. So this is what it looks like. So all the documents you use an embedding model and you store them into the vector database. The cool thing about this is you can um if you embed your own documents you can define how big the chunks are. Uh so for example, if you have a 10-page document, you can specify the chunk strategy to be a page, a couple paragraphs, a paragraph, or something based on context
So what this means is if you are interested in just uh a specific street artist in Oakland, instead of um uploading the entire 10page document and having the LOM go through and analyze the entire document, you can have rag uh chunk through the data. So uh it will be three paragraph chunks and it will only pull out uh paragraphs that um that has raw one on it. Uh so when it comes time to put a request to the chat interface what it does it is it will convert your request into a vector embedding. it will uh use that to query the vector database for relevant information and then >> sorry >> my bad >> and then uh [clears throat] and then the chat interface passes the original request with the retrieved documents to the lol model so it can create a response. So this is an highle overview of the current pipeline and this is what like rag is at a very high level. Uh IBM has a open source tool called Dockling that helps with um converting all sorts of data because as um we mentioned earlier uh if it's just markdown super super simple but uh what doling helps you do is convert things like PDFs uh Microsoft Word documents, powerpoints, even images and videos too. So it's really really cool. it will convert them into something that looms can understand
So in most cases this is markdown or even lossless JSON. >> The point that >> is really relevant about document processing is we joke about you know columns in a PDF are hard to deal with. We know this because if you ever try to copy a multicolumn layout in a PDF, it will look like crap in any other document, right? And surprisingly, the same way that happens to your clipboard is what happens to an LLM. Uh there's I I forgot the exact example, but there was this medical paper that was um cited which had a two column layout and the left side started something electromagnet microscope. Sorry, the left column was the word electromagnet dash and on the second column which a human would identify is a completely different paragraph but the raga process for them did not was the word microscope at the same line. So now 100 plus papers cite this technology that does not exist simply because it was ingressed the wrong way. And that's a real problem, right? We talk a lot about reliable AI. We want a system that gives us answers that we can trust
Um at least I do. Um because the more research I need to do after I get the answer, the more time I lose. At which point it's like if you're not helping with the research, I might as well do my own research. Garbage in, garbage out. So um it's important to have a good pipeline around that. >> Yeah. So the best thing about this is for the user nothing really changes right uh the workflow is completely the same but if you adopt dockling you can just support more file formats and then uh what to what Kareem said you can also uh have more reliable data too. Uh so the best thing about adopting something like dockling for part of your document processing workflow is it's more knowledge that you can ingest and more knowledge that the LOM model can play with and work with
Uh but then under the hood uh you have to start a document pipeline. So you have this data uh it's been processed through dock lane but what happens if there's sensitive data in your documents? uh you can integrate a tool like vault to mask and tokenize the data. Uh so this ensures that the documents in your database doesn't have sensitive data. Uh we worked with a customer once where um they were an insurance company and they had uh the their customers addresses and like social security numbers in it. That's information that you definitely don't want as part of your rag database. Right? So what you can do with hash court vault is um identify those sensitive datas. So if it is in the format of a social security number or a date of birth or a credit card, it will automatically mask or tokenize that uh for you uh the resulting clean data. You will convert this into a vector database and then store it in
Um and this is a very high level pipeline or high level overview of our pipeline right now and this is how we actually implemented it. So instead of lo model uh we use O Lama and IBM granite uh for the chat interface itself we use open web UI um and then we also use open web UI for their built-in knowledge base for a vector data base. It's good to mention here that we use the models that we have access to, but if you want to run this through DeepSeek, we've tried it. It works the same way. Uh yeah, the model you pick is the model that makes the most sense for you. Uh the ingestion process is no different for anything. Uh it it's really a part that happens beforehand. Sounds good
And then for our uh processing pipeline itself, we roll a uh custom web app that will allow users to upload their files. It will go to the uploads blob storage. Uh from there, our uh document processor will run on a cron job. Whenever whenever it detects a file in the blob container, it will start running dockling. Uh after documenting finishes it will run hash court vault to just uh clean out that data. Uh and then at the very end of it it will both uh use open web UI to vectorize that database and or vectorize that um information and store it into open web UI but we also are uploading the final document to the process. The reason why we need to uh also store the completed or or like the verified document is in uh open web UI if it's in a vector database it's just numbers right there's no observability if it is like referencing that file we don't know exactly what it's referencing so um just having a blob storage just for the process files would be nice for us to go back and see like What is the source of truth there? >> The interesting part there is of course also that once we've processed something, we can ingest it into any different model into any different uh knowledge base. The whole document processing operation is cheap computationally and and hardware wise, but it's not free
So if we don't have to constantly repeat it, I mean there's always a benefit. Um, and then similarly, let's say you're upload v2 of a spec sheet because you're now using both types of hardware. It's good to be able to also diff the two files. And if you've ever diff PDFs, it's not impossible. Uh there's a couple tools that will do it, but it's so much easier to just diff um two dockling JSON rendered files and have an AI actually and then a model actually tell you like what the difference between those two is because when you're looking at like 125 uh pages I think was the one for the robot we had same vendor same hardware but one was revision one was revision two because a chip uh got replaced And I think 60 of the pages actually got touched and had like little updates. It's like okay so now we need to maintain two branches in our code as well to make sure we can deal with that. Um >> you want to deop >> see if we um so this is going to be really interesting. Um, if we can get this to work
Let me see. And I'll pop it over to the screen in a second. So, right. So, um, we're going to start with Yeah, I'm gonna shift the jobs over. So, just to give you an idea, um, this is the Nomad interface. Um and while the white might be a little bit searing right now as you look at it um these are all the jobs that are running. So we've got our web load web upload app um our file processor which is dockling of course open web UI or lama on a GPU instance and vault for sanitizing our data but keeping the relevant parts available to us. Um in the case of the insurance company, their goal was to be able to have marketing material around easily produced marketing material where you can say, "Oh, we've approved so many of these types of um uh operations." Um, of course, without leaking the data because the person who wants to, you know, upsell you on a service shouldn't necessarily know um how that happens
So, let me u move over the next one. Um, all right. So, then going to break my neck here. So, uh, simple web UI for uploading a couple files and I think that should be move on plan. All right. So while I'm talking this because it's a small file it gets uh easily um um uploaded and processed and let me pop this one over >> we should have opened the um storage account to just show that the files are actually being processed. So this is not movie magic um which is very relevant. Uh so if I upload the right file
So to give you an idea um the project we have is called Pathfinder because I mean in tech either you call your project something like what Naza calls it or um you give it the name like Atlas uh because every company seems to have a project like that. So what we see here is the default answer um that Pathfinder is a tabletop role playing uh game. Um, pretty sure that's not by Wizards, but um, that's cool. Uh, I thought theirs is more like Dungeons and Dragons. In any case, it's great. Uh, now um, what is a pathfinder movement plan? Let me um Oh, yes. >> Might want to share it. >> Yes
Wait, time because quickly switch to my phone. Yeah, >> I think we should just >> Yeah, give me one second while we're switching um networks to my phone's connection because surprisingly we do need internet for this. Not happening. Okay, why don't you make some elevator music while I reconnect to the u other network. >> Thank you. >> So, as soon as we're back, um I'll just keep talking about this. Um as soon as we're back, you'll see that when we now ask about what the movement plan is, if that uh ever decides to load, um then it will talk about the API or the the interface we use for a pathfinder movement plan. um which is um that is a very long uh answer and check if the file has been ingressed
The problem with live demos is always that you know like you're depending on a couple of different race conditions all working out the exact way you need them. >> I also think >> and most of the time when we upload those files we have a little bit more uh time. While that answer is not incorrect, it is ah yes, it would help to actually select the right knowledge base. Thank you sir. Uh you'll get a relevant answer provided the wifi works for us. The important part is all this code is available uh for you to download. Why don't you pop back the slides? Um, and we'll go back to the answer as soon as it's loaded or when it loads so we can go to the next one. Oh yeah, there you go
>> Go ahead. >> Ah, >> yeah. This is you. >> This is me. Uh, so we'll get back to the answer in uh or to the answer that we get from the LLM in a second, but the question when we showed this internally was like, so why not just fine-tuning it? And in our case, it doesn't actually solve the problem because we want to retrieve the right information at the right time, right? Um, for us, I think the recommendation was fine-tuning makes sense when you have a very specialized uh task that requires deep understanding. You need a specific tone of voice um as an output or consistently generate the same type of response for the same query. uh which in in a particular style of data. So you know um um a service uh desk agent wants to get the same output in the same way every time
Um and this mostly works when your data sets your inputs and outputs are very standardized in that case. Um I think for us if we wanted to fine-tune a model the cost of it versus just using rag does not make sense and for many companies it might not make sense uh especially if your data set keeps changing for robots or any hardware project you know you get different revisions not a massive change but now imagine looking at your infrastructure looking at log files those change all the time uh even though the structure is the same it's it gets expensive to um retrain all the time. Uh small dynamic data set not worth it and just simply through prompt engineering we could still get the output format that we wanted. Uh I think we'll switch back to our demo in a second but let me uh talk a bit about this. There's three things that we learned that are crucial. Uh, as engineers we know garbage in means garbage out. Um, inconsistent data also can be a problem. We had some spec sheets that were partially in English, partially in Mandarin
Uh, which nobody notices because you you open up the files, you know, you look at the first 10 pages, looks good, ingress the rest of the document and now it turns out that your model can't actually answer or it answers in a way where you need to ask another question which is could you translate this for me? um and the tools required. While this industry is moving fast and we're making huge leaps every six to 10 months, we're not there yet. Uh all of this leads to a bad experience, which is a little bit of a problem. Uh but you know, download the code, try it out for yourself. Uh it's all open source, of course. Bring your own models, throw your own models at this uh you know, make your home lab. have some fun with uh you know stress testing your GPUs. And I think that's it for that part
Let's uh switch back and see if we finally got an answer. >> Um >> yes, >> I think we need to check if the file got processed correctly. >> Yeah, >> because otherwise that's going to be >> that would be a good reason. >> Uh so the file we processed >> the file we processed might be too small to be considered right away. That's um >> so we have a movement plan which is a spec of how robots should move. It is a very small document because movement is generally very easily described. So if that doesn't outweigh the other stuff known around the terms movement plan uh you sometimes get crappy answers like this. With that being said, we'll um figure out why the demo bearing or uh angle should work
Uh we'll figure out why this didn't work. But if you have any questions, let us know. >> Okay, go ahead. >> A has a mic coming. >> Yeah, thank you for your talk. So my question would be about this thing called dock link. Yeah, you mentioned that you can upload your PDF into your vector database and just wondering how it works with like things like nonext information like pictures uh graphs whatever dockling uh uses a couple of do we need to repeat the question for the recording or >> no it's good >> perfect uh then so dockling is a CLI app it it is not a hosted service unless you want to turn it into one uh which means you decide what kind of u libraries you give it access to. It's a Python application
So if you give it access to um anything in the VLM space, uh so to do vision stuff um there's a couple of different processors. So the audio gets transcribed to text before it gets processed and then because it's text information it gets uh translated. With video it gets split into audio and video uh sorry audio and images. um and the images get analyzed. Audio processing obviously computationally cheap and much cheaper than processing a video file. Uh for what it's worth, um when we test with video, there's very negligible difference between uh 480p versus 4K because the general information stored in the image is still very much visible unless you're looking at scientific data. If you're comparing I think um is it the university the University of Madrid with number three in the name uh I forgot whose king it was but um they're doing a bio engineering project where they are counting um fish population for that because you know different patterns on the skin tell you a different story about what fish you're seeing. It's relevant to have higher quality data, but in general, look for the smallest possible source that you can feed in um and that gets processed
It's actually it is super cool to see it work. Thanks for your explanation of rag. That's probably like the clearest definition I've I've heard uh since. Um my question is building off of that for like using a tool like dogling. Do you just trust the embedding or is there any like testing you do to it? Initially like with any document uh we'll have a set of sample questions. Um IBM has this model called an instruct model where you can say this is my data set generate questions off of that. So it's a it's a little bit about like that Spider-Man meme you know like three Spider-Man pointing at each other. So we have AI processing the document we're throwing in or sorry dockling using a granite model to process the information we're throwing in
We're also throwing that same information at an instructional model which then generates questions. So let's say we have a document um that has a schedule for this. One of the questions could be at what point is the dockland talk. So that's a question that gets generated. That question then gets fed to the process document which then will tell us was the data pro uh correctly processed. uh that is computationally expensive and it doesn't make sense to do that 100% of the time. I think one thing I would like to add in our pipeline is uh kind of like a a flag of like 0.1 or 10% of the questions of of the data that gets in should be processed that way just to have like a sample size of like okay we can see that this is working and obviously like that number in the beginning will always be higher. So in the beginning might maybe you'll do 100% of things and then down the road you're going to do 50 20 whatever percent
>> Yeah. And then part of this is like writing uh evals right. So just like making sure that like like for example uh with our demo uh it obviously didn't process. So what we could have done is like write an eval like hey we know the movement plan has to have like information about bearing stuff. So uh as soon as you do that you can just hit the API um and ask it uh okay so I expect the bearing to have this answer you can use loom as a judge to also uh like complete your evals and then we found out like a semi-automatic flow usually works the best where like some of them are like al as a judge but there's always like a human person in there checking to see like oh is this actually accurate. It also depends on how you classify the data, you know, like if it's something like a manual on how do I set up a new iPhone if your company has those and you know that gets fed into a service desk agent, totally different story than how do I set up um I think one of the keynotes had that example yesterday. How do I set up where my money for my pay stop goes? Uh you definitely want to have different safety guards for that. I think you had a question
Uh you guys kind of just answered it. I was going to ask how do you handle conflicting uh information if uh on the docking level or is that just all processed afterwards? >> Wait, can you repeat the question? >> Yeah, sorry the audio cut out a little bit. >> How do you guys uh process conflicting information? But I think you just answered it just now. >> Conflicting in um in the sense that the user is trying to do something that they shouldn't be doing or conflicting as in we have the same document processed twice. or like uh I upload one document that says uh water is blue and another document that says water's red um but it's processed all at the same time. So what's the output? >> So the this is a fun one because this comes up and in our case it came up because we had two documents referring to the same part of a hardware but giving different answers. Uh one part is garbage in garbage out right. Um and and this might be unintentional because you don't know all the content of the documents
Uh so context is relevant um in terms of figuring out why is water blue or red in this case. Is it because let's say we're uploading two pictures. One has blue water, one has red water. Um what's the context of the the red water picture? Is it that it was, you know, ran through a Snapchat filter? was it taken at the end of the day and the sun's reflection is different. Uh so the metadata you provide with each file can be incredibly uh relevant. Now with pictures super easy, right? We've got this excit format that we can write to and put anything in there and it will be picked up because it should be because it is part of the composition. If I take a beautiful picture of the um Golden Gate Bridge at night, it's going to look very different than during the day. uh with text information that's slightly harder because you don't necessarily have that meta information at which point um I think you mentioned the the semi-automatic approach has to come in when we do the testing you know is what what what's the color of the water and we get the answer redback give me the source of which of my documents told you that story because I then have to figure out what's happening there you could probably automate this and figure out It's read
This document says that that is the absolute truth. Why is that document in there? How did that get into our data set in the first place? Uh we don't have an automated solution for that. It's a cool problem. >> Thank you. One more round of applause for our presenters.