Eclipse LMOS&AGC: Agentic Computing for Real-World Enterprise Systems | Arun Joseph, AI By the Bay25
[Music] the picture at least at least Silicon Valley is where you're building the jet zeros, you know, the the new advanced flights and things like that. But in enterprises, it's all it's all not so shiny, right? We have to put up with what is available, the old cars, the trucks, and put this new AI engine to make it fly. So this is essentially the background of the talk. I won't be talking more about new frameworks because we have more frameworks than stars in the sky probably. So what I would try to present would be the mental models that led to the building of Eclipse Almos and AGC in the experience that I had while heading the AI engineering for Dutch telecom group in Europe. So I'm the co-founder of uh Maicia Agentic Systems. I lead two projects open source. One is the Eclipse Elmos and then there is AGC which is agentic compute
Um well I'm also one of the contributors to the reliable AI uh manifesto as well. Yeah been building AI systems for since 2023 and prior to that my entire background has been distributed systems and large scale platforms. All right. So um enterprise agents are hard. I came up with this uh this this uh advanced graph. Building agents easy, reliable agents hard. Reliable agents at scale which means more agents, multi- aents and agents that change. This is even harder
And the hardest thing of course most likely other than AGI cracking might be reliable agents at scale in enterprises right enter. This is what we are going to talk about. So everything that I'm going to talk about here today was the experience that we had in building AI systems for Deutsche Telecom group. This is a picture except Germany is not colored here. All the countries Deutsche Telecom operates in Europe. That's almost 10 countries including Germany. So in 2023 I was tasked with building a team uh to bring genai into sales and service including customers facing um sales and service use cases across doa telecom in Europe. So um in 2023 uh there was a rudimentary version of something called as langchain and lang chain 4j was just emerging and I don't remember exactly lama index was there or not but the main point is it was all rack times the alchemy times uh no one knew how to actually build these kind of systems at scale especially if you want to do tool calling API calls etc
So we started building AI systems back then and we went live one of the first open source agentic platforms to go live in 2023 uh late 2023. Last year we had we moved it to Eclipse Foundation and the outcomes are more like industry-leading deflection rates uh launched in six countries. It was built as a pass platform for building agents for sales and service use cases which is used in multiple countries, multiple languages, multiple domains, uh and also multiple business processes. Um so what are those learnings? So we wanted to so like I said I wanted to focus more on the mental models that we used in building the system. So when we started of course as we call as a level two or level one agents we came up with the mental model that an a we need an agentic program back then at least in 2023 no one was talking a lot about agents. We said if you want to scale across 10 countries with almost 200 to 300 APIs and business processes which are completely different across different countries the only way you can do that is to build smaller computing units and we call them agents as a first class citizen into the platform that we were building. But what was the mental model that we used? What is an agent? Right? Back then we started to describe an agent as a fluid program which has certain fluidity in accepting the input and there is a certain fluidity in in doing the computation. A good example is uh the dialog trees were rigid before now with LLMs you have the flexibility to have a fluid kind of input and also on the computation how you weave together a couple of functions is now fluid
It can be fluid. So this is the definition of an agent at least that we used in our mental model when we started building this program. Now it brings in two challenges in enterprises especially enterprises don't like fluidity they need if something works it should work 100% of the time especially for customers especially for a company like telecom which is one of the top five brands in the world the brand cautious company you cannot mess up so any fluidity in input and computation has a natural implication on the fluidity on the output which is referred to as unrel reliable in in in enterprise terms. Also, no one knew how such a program's life cycle need to be managed. This is the critical reality of enterprises. Building is easy like we said, but let's say if you release a billing agent for your customers, how are you going to manage the release process on keeping on upgrading those agents or adding more use cases to the agents without breaking and with with the with the entire tracing lineage who made those change? This is all required in an enterprise. It's not a startup, right? There was no life cycle management. So these were the two primary challenges which usually is not there in in in startups which we started to observe uh in um in DT
Then there are also two more golden rules in enterprises. I don't know how many people here has ever worked in an enterprise. Don't introduce new teams. Conway's law. any new team is going to bring in with more sprint scrums scrum of scrums and don't do huge back and forth loops u so irrespective of AI in enterprise several programs fail you you would have heard of digital transformation programs moving from monoliths to microservices how many of such programs fail these are the realities of the enterprises so don't do huge back and forth loops this is absolutely critical in an enterprise Usually in an enterprise you have different roles. A data scientist and an engineer and u and a business process analyst. You cannot do the other person's job. You need to pass on the baton and the process in the making to the other person
So if you start to create an architecture or design which requires many people to move between it will never take off usually. Which brings me to from my experience for I've been um um reaching to several enterprises and even last year I was speaking at many conferences including CubeCon enterprise agent fails at least in my point of view are the results of these two realities enterprises don't like the fluid part of the agentic programs how to manage the life cycle and reliability and the rigidity of enterprise teams adds to the u to the to the challenges that it brings Now with that being said with these constraints how do you design an agentic platform which is different from if you start an agentic platform from scratch from green field. These constraints don't usually exist in uh outside of the enterprise. So we built something called as our agentic platform journey. Uh the project is called LMOS language models operating system. It was also based on the same construct that we need new computing paradigms uh because the computation itself is going to change and we started to see it as an operating system and um the new computing uh par primitives to be built on top. But the challenge was you need to you cannot choose your favorite language for example in an enterprise but raster is the greatest language I might say but you cannot use it because of the previous constraints that I said. What are your existing engineers using? Where are your 200 APIs and the client SDKs? What is a client SDK program which is available? Then what about all the tracing libraries? What about all the infrastructure investments that were made? What about all the hundreds of DevOps engineers sitting there with these constraints? You have to build it
This is exactly what we did. We built the platform which actually works with the existing groups. In 2023 it was hot because in a it was lchain times and rack times there was not spring AI was just emerging in the JVM world in do telecom almost all the enterprise stack was in Java. So we chose deliberately a stack to build which was based on cotlin which allows existing engineers to who knows these APIs who knows these domains to build AI agents. We absorbed the hard parts. The second part was we we started to see huge handovers between these groups in a in a in a typical program or um in a mobile app development or so the business owner might say I need a button green in color when it clicks on it book an book a book a ticket here you release an agent a billing agent to Croatia and you say the customer is saying why is my bill so freaking high and and the response was incorrect. potentially and now the business person started to write Jira tickets and this is never going to end. So we started to build something which optimizes this loop within Elmos
So this is what we did. We built a platform on the JVM stack in 2023 which absorbs the reliable agent execution constructs built-in tracing iterations integrations all all of that was baked in. There was no we even started building a new protocol called Elmos protocol uh which was scrapped later on but uh by the time MCPS came in but this is what Elmos was built on example like I said we don't need more agentic frameworks this is a good example of existing engineers what we build there is a framework that we built it's called arc it's built on cotlin and it has this construct if you look at versol's AI SDK these days this is exactly the same kind of construct which versal AI SDK uses the advantage is at least there it's just a simple script what we used the advantage of that was now the engineers let's say I'm a billing API domain expert in Dodge Telecom who who manages a portfolio of six APIs with 200 attributes very few people know what these attributes actually mean and only this engineer is is best poised to build an agent than the shiny new framework and a new engineer with a new team who keeps back and forth what is this API what is this attribute this is the key and then we started to collapse all the hard concerns in the platform in the in the framework such that the engineer doesn't have to know about how to do filtering how to do uh hallucination detection how to do retry loops all of that was absorbed in the framework itself. So the agent the engineer would write only a few lines of code. They wired the APIs and that was it. They also it allowed all the existing SDKs libraries to be deeply integrated into the platform. And one of the things that we came up with um I could have shown a demo maybe if there is time we will show you a demo but uh we came up with that problem statement like I described people were start uh folks started to write Jira tickets people on the front lines when we started to release let's say the billing agent or the product catalog agent um how do you keep on improving these agents by adding more behavior How do you verify this is correct or not? Not not I'm not referring to evaluation per se. I'm referring to defining the behavior which was the whole user story how as I as it used to be in traditional enterprises we figured out even if you have the best framework even if you have the best platform people are spending a lot of time in these so-called war rooms which was nothing a bunch of people sitting together and saying this is how it should be and this is how the bot should respond and business could not write the prompts because we could not allow promp prompts to be handwritten because of the um prompt injection plus all the reliable AI guardrails that we were started to add in
So we came up with a pseudo language which is again based on markdown which we refer to as ADL which is kind of like SOP standard operating procedure where the business the engineer wires the agent. Then within that construct you have the ADL where the user uh the business person would write in a semistructured format like this in markdown and this is not directly passed to the NLLM we are we actually built a compiler it's called the ADL compiler it's it's it's a it's a DSP version for business I would rather say all right and so that you provide the reliable guarantees there are super interesting constructs that we started to build. Uh if you are doing compiler design, there is something called as a tree shaking, right? In libraries. Um uh you you you tree shake the parts which are not used for a particular execution. The libraries, the same things we started to bring in into it to bring in the reliability aspect. So if there is a huge prompt broken down into what we call as ADL use cases for a particular use case we would we would know the intent and tree shake the rest of the things or at least it's constraint so that there is a guarantee that it works reliably. There is a two-pass rule for example uh how to prevent the loops right so for example if you provide uh use case to resolve a customer's complaint my bill is high you could leave it it out in an agentic way keep on repeating right or you say we only have a two pass rule it tries a solution then an alternate solution then hand over to the u u uh human right so these are intelligent uh god trail. So this is baked into the language itself so that you're only allowed to provide two alternative tracks and you're not actually collapsing into a single prompt
The best part was we distributed that chain and optimized the loop. The engineer just creates u the first version of the agent in cotlin wires the APIs or even there's a construct called as mock and after that the entire iteration is done by the business teams which created the entire acceleration we needed by end of 2024 we were in four countries when most of the enterprise AI was failing. So um it is not so clear but uh towards the end of the talk there is two open playgrounds if you like to check out what ADL is we have a hosted version I'll share the links you can check out test it you don't need to even provide the keys at least we have opened it up so this is ADL authoring environment as we call it within the same IDE kind of application that we built the engineer just creates a Java applications boots it up it comes comes up with the UI where the business person is able to manage version all these ADLs. This is a service booking appointment for folkswagen the standard operating procedure and this is another UI after the authoring the the business is able to test it with in an ID like environment. It shows which ADL use case was picked, what tools are called and what is going on the filtering and the business can also mock tools in a in a nonprogrammable in a nonprogrammer heavy way. So this actually was a gamecher in increasing the speed and velocity at which we were starting to release agents uh across these countries. Um so existing business existing engineers and there is something funny which we started to see when um the AI wave was taken off there's a lot of uh tooling which started to come in uh which allows you to just add as a library and say just add this decorator on top of some code and it actually does a remote call execution somewhere in the cloud and in the end you have a program with six decorators calling six different cloud vendors and each of these are remote code executions. This is madness and it's it's a nightmare of DevOps teams
So existing ops teams we started to start right from the CNCF stack that we already had and baked in all those constructs including the telemetry hotel. Yeah. Why do you need different u you know telemetry standards and then push the traces to five different places? No engineer need to know this. This is already baked in. So we started building custom operators where the engineer after building the agent is able to do just one push and then the pipeline takes care of it. It is deployed as a as a as a as an agent as a first class primitive on a Kubernetes cluster which manages the life cycle of the agent. So this is what I mentioned in the beginning. Enterprises don't need shiny frameworks
Enterprises need reliability. Enterprises need life cycle management and no new teams and reduce the loop. This this is a formula that with these constraints how do you build something? So that's Elmo and one more topic which I wanted to mention was the agent sprawl. So we started building uh even today most of the agent building is happening with frameworks right. So Elmo showed one way uh we started to realize after a certain point the number of agents what we refer to as agents as programmable execution units is increasing. So we again started to collapse it with ADLs within a single agent you can have two or three billing agent this and with ADLs. But then we started to also see a pattern. Why do you need um if if if the agent itself if you redefine the construct that I showed before can we look at agent as a universal new computing primitive
So this for example what is an agentic program? I showed a in the beginning a level two program. This is most likely going into the level four construct. If you have a loop construct that takes in if you attach the goal and the context and the constraints and the tools and if you do it over a loop where the context is continuously enriched with the constraints absorbed then you don't need to build agents all the time. You will have ephemeral agents. So all of these agents you could collapse into the agent loop primitive absorb all the hard parts in building agents which is telemetry which is tracing which is u observability all of that is if you bake all that into let's say a new computing unit and expose a single API then you get fml agents because you don't need those six agents you can you can just have one So this is what is AGC. There is a lot more to talk about AGC but this is agentic compute. The idea being just like how stripe did it right or S3. Let's take S3 right before S3 it was so hard to build file systems
They collapsed everything into a single API so that you don't have to worry about all those things. And um and u yeah of course it it was democratized. So this is what we did with AGC. Uh and it works not as a decorator pattern with your own custom u decorator. It's an open AI uh standard u uh completions API and there's also responses API which works with any framework that you have. So this is a AGC. So the idea was what if agents can be computed on demand that is agent compute underneath it that computing construct the same thing whichables and and uh cursors of the world is built you need only a small portion of it in enterprises if you can if you can ground it with the tools and constraints you will have your own allowables in enterprises including longunning executions and and all so that's pretty much it Um there is a lot of playgrounds. You can take a look at all these playgrounds and play around with AGC, play around with ADL those and all of this is open source and um yeah and AGC one more thing to talk about AGC is meant for platform teams
So if you're an engineering leader you want to set up an agentic platform for your whole company. This is what AGC is for not for building individual agents. It is one docker compos file, one helm chart and one single API which works with any framework. That's all you need. And you just start attaching your tools and your enterprise can connect to any any u observability stack or evaluation tool. They wire it. That's all you need. That's democratization
So that's pretty much it from the messy reality of enterprises. Um thank you questions. >> Thank you so much. Um, so if you have any questions, raise your hand and I'll bring over the mic to you. Okay. [Music] >> All right. That was a wonderful talk. Thank you
Uh, one question. So at I think it was kind of in the middle you were talking about basically you would have your business owners or I guess like the business stakeholders kind of own the definition of what the it was the ADL, right? >> Yeah. I guess kind of what was your experience of that because I think that's like a dream that I think we a lot of us have is like cool excellent the business priorities are kind of like over here and like we kind of more as like the tech technologist kind of only worry about the like engineering aspects to it. >> Yeah. >> But then I' I think I've kind of found in practice like that kind of split over because then all of a sudden you're running like >> live performance metrics. You're like how reliable is it? And it's kind of hard to >> kind of have people who are new to owning systems like that kind of do that. So yeah, basically what's your experience with that Ben? >> Yeah, good question. So essentially there are two two elements, two dimensions in there
>> Business teams never own IT systems. They they love to play around with it, but the moment someone says it's going to production and if something bombs, yeah, they won't take responsibility, which is okay. So what's the best way from giving them that freedom while saving engineer bandwidth? This is at least how we started to do it. When I was leading that program, I was looking into 10 different war rooms at one point early 2024. What are what were the all these war rooms doing? The war rooms were two business people, three engineers sitting around and doing prompt tuning. This is a waste of engineering resources. So we started to look at at least in the narrow scope what is the nature of these changes and said hey business you don't need to own it but what if we provide you something and you figure out what is the right SOP or the agent operating procedure once you lock it in give it back to the engineer we push it to production it's not like they are it's it's so essentially even the engineers were happy not to having to sit in wall rooms all day that's the whole point yeah but good question. [Music] So when uh you talked about AGC the last piece I think um >> I'm assuming it still acts as a platform and uh sends the actual LLM request to open AAI whomsoever wants it right I'm assuming it's just a pass layer kind of for agent >> yeah it's uh it's like a pass lay it's like it's indeed a pass layer but the difference yeah it is indeed a pass layer but there's a difference I would rather or frame it as see the the current way of building agents is you are writing still a lot of code right uh lang graph lang chain I'm not saying all of those agents will go away but from what we have seen right now this I had to change the laptop in between u that's the reason I don't have the API keys I'm using all laptop now so I wanted to show for example uh an agent which is a typical agent in a in a in a business use case uh which is uh AJ u let's say there is a this was an actual agent which was built u in a telecom company you needed to build an agent which creates sales proposals for the customers you need to work with the pricing data of the it's B2B pricing data proposal brand templates file search discount you need to do math uh in the end pricing all of that if you were if I were to ask you how would you do it today as an agent somebody will start to think okay I will have to pick X versus Y framework and build a program and this and that right what if you just provide the tools which is what happened here
These are all mock U tools including macros. We started to call macros for math math functions which were added. You attach the tools to a compute unit and let it figure out using that function loop construct. This is like the loable way of building that agent. So no one actually went in there and wrote any code to build this agent. You attach the tools. I'm not saying this is the UI. This is just a playground
And then you get a single call command which you can integrate in any of your framework. So if you want to build many such agents in this case this is not an agent which does only B2B sales you can use this this same construct to for example get the brand templates or what was my last call transcripts with Volkswagen. So you don't you are not building six or 10 different agents. It's only one construct. The compute becomes the agent. Not sure whether it >> Yeah. But this will be kind of similar to uh writing an agent with 10 tool calls or 10 tools given to it and a prompt. Right
If you see most of the agent code >> Yeah. >> Uh >> it's kind of looks similar. >> It looks similar. That's that's why it's uh it's a little it's a little tricky. Um >> and >> so for example, right? Um if I were to define a new computing unit, u if you have a if you have a constructor or a program which can naturally do this loop without having you write anything about what does it do? take a goal uh reason about it. This is sudo code right by the way. Select the right tools then update the context and if you're able to do this in a computing unit then you don't have to write as you described agent one agent two agent three etc. There is a certain class of agents you don't have to write
There is a certain class you'll have to write. >> So it's >> okay cool. So we we came down to this not by by by by actually building. So right now what we are building is actually operational intelligence platforms which is large scale decisioning systems. We started building 20 30 agents and then started to see why are it's the maintenance is going to spraw out and we started to see the best way would be to collapse it and reduce the moving parts and this is exactly how cursors and lovables work. So you collapse it to only API calls. What would you get? And this is that unit at least in our perspective. Yeah, good question
A great talker room. Thanks again for coming all over all the way from Cologne and being with us here. So you were head of AI at Deutsche Telecom which is an enormous company. I used to be director of open source science at IBM another enormous company and I'm amazed how anything gets done in an enormous company right and like it's very difficult to do anything a toy website a toy web page like go find out right so and you managed when being there to build this huge open source system. What are what is the secret? How do you get the elephant dance? How do you get the big company to resource this >> uh and put it out and use it in business? Like what does it take you know to build a major open source project inside of a big company? >> Thank you. So I would frame it as what worked in our advantage. First of all, it was an open field. It was chaos in 2023
No one knew what to do. But still even with the emerging two Python stacks I could not autopose it because it's an enterprise we are not a company building frameworks right so we started with lang chain and the madness of it there's no other way to put it and it was not working let's just put it this way and so there was no other option but at the same time to convince the management well I had this vision to build the next Heroku In 202 there were two options. The AI wave is taking off. Should I go back to the valley or should I find great distributed systems engineers? I was lucky to find some of the best distributed systems engineers in different corners in DT and we pulled them aside and then said let's build a pass platform uh over our weekends. So this was not officially started at DOA Telecom. We said let's do this over weekends and uh let's prove it at least for one use case. >> Skunk works. >> Yeah, exactly
So it's absolute skunk works which was done and one fine day we actually u put it in staging and convinced the management see it works even better than lang chain please allow us to build this and then it started to take off yeah >> all right with that let's thank our speaker one final time thank you so much >> thank you thanks everyone [Music]