Hands-On Workshop: Building Agents with Spring AI, MCP, Java, and Amazon Bedrock

Building Agents with Spring AI, MCP, Java, and Amazon Bedrock | Workshop| James Ward and Josh Long

Recording: Building Agents with Spring AI, MCP, Java, and Amazon Bedrock | Workshop| James Ward and Josh Long

Well, we are going to get into things today. Thank you for being here. I'm James Ward. >> I'm Josh Long. >> And [clears throat] we are going to get hands-on. But before we get into the hands-on part, uh, which will take, you know, most of the day, we did want to just get you familiar with the technologies that we're going to be diving into, um, and start with that. So, here's our >> what agenda for the day. >> Is that the only screen they've got the entire day? >> You got to sit close

You gota >> Yeah, I I >> I mean, luckily, we're not showing a whole lot on the screen, so >> Well, we've gotta, but yeah. Okay. So move if you need to if you can't see the screen because because yeah you're gonna we are going to show some stuff. >> We can move this maybe it's got wheels. >> Just wheel it around like like constantly roving >> just constantly. Yeah. Yeah. >> Have somebody moving the screen around

So did you catch this? Yeah. >> Uh okay. So here's our agenda for today. We are going to start with a little intro to Spring Boot. Um who here has used Spring? >> Oh okay. >> A few people have used Spring. Um, I'm sorry about your laptop. Uh, and so we'll start with that intro to Spring Boot just to make sure everyone kind of knows what Spring Boot is, dependency injection, all the parts of, uh, how that works

And then we'll give you a little intro to Spring AI. And that will go, I don't know, maybe an hour and a half or so. That'll be your like best opportunity to ask questions about the technology that we'll be diving into. >> Foundational stuff too, especially. >> Yeah. and then we'll dive into the hands-on part of the workshop. For that part, you will need a laptop. It will need to be connected to the internet

You'll need a browser and access to your email account. Um, and we'll we have AWS accounts that we'll be using for the workshop. We've got the VS Code all set up in those. And so, so everything you need is going to be available in that AWS account. Um, but you will need a laptop access to your email to be able to log into it. And I don't know what the power situation is, but um you know, share chargers, do whatever you need to to make sure your laptop stays powered on. >> Go Slack. >> Pro tip

>> Pro tip. >> Uh oh. So, we'll do the Spring AI part of the workshop and then we'll do uh the embable part of the workshop. >> Aentic. >> Yeah. >> Um I'm seriously, who can see who's having trouble seeing this? Can anybody see this >> in the back? Can you see this? So, if you're in the back and you can't see it, >> let us know because we we can >> want to at least temporarily move up during the workshop. You know, you won't need to to really see the screen, but >> yeah, I'm I'm I would have trouble seeing I'm having trouble seeing it from here. Uh so, gee

>> Okay. So, uh Josh is going to take over and give us our little intro to Spring. >> Let's make sure that you are all working on the screen here. [clears throat] It's all It's all Oh, yeah. Oh, here. You want to put it underneath that one? >> Oh, sure. >> You up a little bit. Do you want HDMI or >> Let me make sure I have nothing

>> Yeah. >> Hello. Hello. >> Is that correct? >> Okay. system display. Thank you, sir. >> Oh, that's a very light. Okay

Uh, who here has used Java? Okay, most people have some familiarity with Java. We are going to be use using Java today. Uh, we could use Cotlin, but um, we figured Java would be a little more familiar to most people. So, um, so those that's that's what we're going to be using for programming language side. Um, and oh, maybe I'll give our little spiel on on AI Java. Uh, you know, probably if you've been watching what's going on in the AI ecosystem, you see a lot of Python and Python, it's great, great language, but Josh and I work mostly in enterprise um, companies and we're enterprise companies and so their Java is pretty predominant. And so over the last six months with Spring AI and Coug and uh AA and and Babel and Lingchain forj, we've had a lot of the options in the JVM space really mature to the point where we don't necessarily have to use Python for AI anymore, which is good to have some options. Um so that's uh I think we're pretty excited about what's happening in the AI space around Java

I think it's a great place to be. A lot of the enterprises have already built a lot of stuff on Java and so being able to do the AI pieces also in Java I think makes a lot of sense for those those folks and we get into like in Babel the the paradigms that in Babel is actually bringing to AI are pretty um novel they're not just uh a lot of the the AI stuff in Python um maybe isn't bringing some of the enterprise best practices that we've been using in spring and other things for a while. So, so I think it's actually and Babel is actually a leader in the the space around how you build a gentic orchestrated um thing. So, we're going to dive into all that. >> No, that's about right. I think yeah, just yeah, it's the right we're in the right place at the right time. Um okay, so I'm going to do a quick just 101, you know, just to make sure we're all on the same page. How many of you have got previous uh spring experience? Some of you

Okay, good. So, we'll do a quick primer. I just want to make sure you understand the fundamentals of what's happening here. Because by the time we get to Embable, we're building on Spring AI, which is building on Spring Boot, which is building on Spring Framework, which is building on Java, which is, you know, building on decades and decades of suffering and pain, right? Uh, lower levels of abstraction. So, I want to make sure we're all kind of clear about the high-f flyier nature of what we're So, let's do a very quick intro. Spring and Spring Boot. So, we're going to build a new application here. Uh, as always, we build new software by going to my second favorite place on the internet, start.spring.io

Here you can see that. Oh. Oh, there we go. There it is. start.spring.io. Right. And so, uh, if you want to build a new application, start.spring.io. If you're if you're restless and can't sleep, start.spring.io

If your children have indigestion, start.spring.io. And we're going to build a new application. And we're going to we have some uh spec we have some choices we need to make when we build this application. What do we want to call this? I'm going to call it application because again, I'm I'm amazing with names, right? Um, and what version of Java do we want to use? Well, I'm going to be using Java 25. And look, uh, Java 25 is the version that is out right now. >> I think the workshop's using 24. >> 21. >> 20

>> Oh, don't say 20. Is it 20? >> I think it's 21. >> Oh, it's 21. Okay. Oh, >> we're going to get it updated, but >> Well, anyway, you should be using 25, right? 25 is the version today. And it has some of my favorite features, right? So, if I can finally do this, I can create this uh this uh script Java. And I can open this up. Let me make that font as big as humanly possible

So void main and I can say all right hello is it AI by the bay something up okay right easy there you go >> you're still getting used to those >> vings totally so what I've got there is now a uh if you look at that it's a it's a it's a program I wrote in Java so I'm going to run this right like so and what that is is it's a program uh written in Java but notice I didn't compile it I just interpreted it like a script. It actually gets compiled behind the scenes, but it's I'm using it like it's a script file, right? So, what this is is is it's a script written uh in Java. Okay, it's a it's a Java script. It's the good it's the very first good Java script. Now, this has been 30 years in the making, but we finally have it. Okay, it's here. So, so that's a script. And um look, pretty simple

The script has uh no class like me. Okay, no class at all. So, it's a very very simple language, very easy to understand getting started with uh behind the scenes that gets compiled and so on. But, nonetheless, so we're going to use Java 25. I like Java 25. We're going to build a new application. I just want to make sure we have some very very simple. We're not even doing not even not even doing the document stuff

This just like talk about the absolute basics here. So, we're going to build an application. Um, and that's it. Let's just open this up. Okay, here we are. >> [clears throat] >> That was an alias by the way. Uh it's alias to unzip and open. And if you look at that, that is a shell script

It's a script written in Java. It's a again it's a Java script uh that uh just finds it unzips the zip file and it finds the build file in that and then it runs idea space and then you know pom.xml or build.gradal or build griddle.ts. And I just want to be like 100% clear where we are. Spring is a framework that cares about how your objects are wired together. If you go the extra distance and tell it how things are wired together, it can do things for you on your behalf. Right? I know this sounds ridiculous. It just sounds so simple, but it really is just that. It's objects get wired together

There's a there's an opportunity between when you define the object and when somebody else consumes that object for the framework to do things on your behalf. And the way it does that is through polymorphism. It subclasses your types and and introduces new concerns. So, Spring has all these declarative concerns that you want to add to your applications. These concerns if you were doing pure object orientation would be weird to kind of add. You'd have to redundantly add them to every object you create and you'd end up cluttering those objects with those concerns. So, Spring can do that through we call it aspectoriented programming. But the idea is you create a type of object called fu

Somebody else expects a foo, but what they get is actually a dynamic subclass of fu that has all these extra all these extra capabilities woven in. Okay, it's a magic trick. It's a slight of hand. And if you understand that, then that's the foundation upon which everything else is built. So we have three pillars in Spring. There's a dependency injection, which is one object depends on the other. You have aspectoriented programming, which is one thing adds new capabilities to another in a generic single place. And then finally, you have um portable service abstractions in Spring, which are the concept of abstractions that you can reuse to greatly reduce the amount of code you write for a given context

Okay, that's it. Those are the three pillars of spring and they've been there since the day one. And if you hear people complaining about uh about they don't understand spring or there's like too much magic, whatever. Just refer back to those three bullet points. That's the underpinning of everything we're trying to do here. It's just wiring your objects together and doing it in such a way that you can take advantage of declarative concerns. So, what do I mean by that? Well, >> will you make your font as big as you can? >> Yes. How about that? Can you all see that in the back? >> Can you see that in San Francisco? Way >> threequarters of the way back

They can see. >> Yeah. Okay. Well, I this is there. How about that? I got to make it a little bit bigger, a little bit smarter. So, so there's several ways to tell Spring about how your objects are wired together. One is to be very explicit. So, you can create a a configuration class, right? I can call it my config

And um let's say I have a customer service, right? And the customer service type here uh depends on something else. It has a dependency on another thing. Let's create let's suppose I don't have on the class path here, do I? Uh the JDBC types. So, here we are. JDBC just add that there and I'll add H2. Okay, keeping it nice and simple. Okay, so now I've got a application that has an a dependency on a data source, right? I need a private SQL data source to do my work, right? Here's that. Now this data source is a a dependency

Now in good object-oriented programming, very simple object-oriented programming, you might have a class that wires these things up, right? So I you can imagine I have a a customer service object here and that provides the definition of the customer service. What is that? Here we go. And I might have another method that returns a data source which is a connection to a SQL database and I can use the new embedded database builder. Here we go. Like this. And then set type will be embedded database.h2 and then build. Right? So I'm building a data source and you know I've got these different wirings these different uh associations. Now I can stipulate that this customer service needs another thing

I could just say that that would actually work or in spring I could just I could say I have a collaborating object that is a dependency I'm going to declare that as an argument. So now in this case there is no spring but you can see how by taking the time to factor out the definitions of these different objects I'm already in a good place I can already start to take advantage of spring and the way you do that with spring is just say at configuration and then annotate these with at bean. Now that's that's you're telling Spring, hey, when you start up, look at these methods, invoke the methods, get the return value, keep them in a map in in effect by its bean name, which is the method here, and the type, which is the return value, and then uh resolve any dependencies that are required. So in this case, it'll say, hey, customer service requires a data source. This happens to produce a data source. I'll take the output of this and plug it into that. Right? Pretty simple so far. Same page, just basic dependency injection

You've probably seen this in Angular and Angular is not a, you know, not a new thing at all, right? Um, okay. So, that's just Java configuration. Now, of course, there's a couple problems with this. First of all, this customer service, this is the configuration I wrote for this type that I'm actually writing over here. So, you might want to keep the configuration next to the logical uh code for the type. So, in Spring, you can also discover things. We can actually discover uh your beans for you. We can use something called component scanning, right? This is redundant in this case

It's already on there. If I go back to my config class, I can put component [clears throat] scan there, right? And what that'll tell Spring to do is to discover other objects that have at component on them, right? And it'll use convention there. So here I can say at component. There you go. Add this. And that'll discover this class. This is a marker. It's a tag

Spring will look for anything in the current package and below. And it'll find anything that has at component on it. And it'll say, "Oh, okay. This thing requires a data source. I happen to have a definition for data source over here. I shall resolve this by creating an instance of this and then plugging in the the bean of the type data source. So in this case I don't need that. It's redundant

Right? So there's two different ways to describe to Spring how to create objects. Explicitly where you create a method and a return and you have a bean that gets returned or implicitly where you tell spring just discover it for me and kind of create these objects for me by convention. Okay. But ultimately they all end up in the same giant big bag of beans. Okay? They're all on the same map of bean IDs and types to instances of those objects. Fair enough. So far on the same page. Nothing special about this

This is not AI. This is not This is 20-year-old technology we're talking about here. But >> so real quick, one of the things that this gives us the ability to do is change our dependencies depending on the environment that we're running in. And so like why are we doing all this stuff to like create all these different kind of layers of beans and and wiring them together with Spring? The biggest reason is that then we can change those beans depending on our environment. So the most common environment that you're going to switch between is your like production code and then your testing code. And so at test time you could actually inject in different things. You could mock things. You could uh use a um a different data source in when you're running tests versus when you're running in production

Uh so is that fair to say like like testing is >> that indirection is super key. It's just object-oriented design. It's just polymorphism. I expect a thing of type foo, I get a foo. Who cares what subtype it is? As long as it meets that basic, you know, walks like, talks like, quacks like a duck, right? Contract. Um, okay. So, this is implicit. I'm using at component, but sometimes, you know, these annotations, there's a bunch of other annotations that you can use in lie of component

What you need to understand is that they all all are uh uh metaan annotated. So, at service does the same job as at component except it actually has some extra benefits. And if you look at at service itself, you look at the de definition of it, it still is just an at component, right? So it's we say that this annotation itself is annotated with at component. So any meta annotation will also get picked up, right? And these meta annotations are there for uh they have some technical implications like for example if I say at controller, right? [clears throat] If I say add controller, add controller tells spring that this is meant to serve HTTP requests or to serve GraphQL requests or to serve our soocket requests or to serve websocket requests. It depends on which layer you're working with, but it's a controller that adopts business logic to a protocol usually over the network, right? And so this at controller is still just at component, right? It's a meta at controller. So it has some it's at component at least and then some extra capabilities, right? So when we say that these we see these are uh uh annotations they're called stereotype annotations like UML they denote the technical concern that this thing addresses in the in the logical sense of the uh of the codebase. Okay so >> that essentially just becomes like metadata around this object that then other things will then use to look up and find things of a particular stereotype. >> Right? But uh but you can you can like if I wanted this to be a bean all three of these annotations will have at component and therefore they this will be a bean

And in fact at configuration if you look at it I know it's at component right every type in spring every annotation in spring that marks an object as being a bean for spring to manage is ultimately annotated with at component. So you've got two different ways to go. One is at beans the other is you know uh the stereotype annotations and you can mix and match. You can put the stereotype annotations on your own source code. And when you're defining instances of other people's objects that you don't have access to the source code for, do that here. Sorry. >> Oh, he's taking the phone call. >> Okay

>> Yeah. Well, good. Um, so yeah, this is just wiring. Okay. So far so good. Yeah. >> Like I just want to make sure we're on the same because it's all that's all we're doing, right? And the question then is that like >> at some point I might want I don't want to I I've got a database here. I've defined the bean of type data source

This is a a a thing that however is going to be common to all my applications. You're going to have a database. You're going to have a database. I'm going to have a database. Uh and so at some point we decided well those three things that we talked about the three pillars of Spring Framework, right? One is portable service abstractions, right? Two is dependency injection and three is uh a aspect oriented programming. So if you look at this like how what is portable service abstractions in this case? So I'm using a database, but this is the regular Java JDBC data source. Have you ever written code using ray straight JDBC? It's a waking nightmare. It's an actual terrible abstraction

It's good for implementations of other frameworks or libraries, but it's not something that application developers are meant to use directly. It's a terrible way to run a railroad. So what we have is portable service abstractions. Let's say for example, I wanted to simplify my use of JDBC. Instead of like let's just should I implement a simple here. I'm going to do something that I would never do in plight company just so you can see how we used to do it. Okay. So, I'm going to take that type and I'll implement a thing that returns all the customers in the database

Yeah. Get customers. Okay. Ready? Try var PS try con equals this dataource.get connection. Right. Var PS equals uh con.prepare statement select all from customers. Okay. and and then then throws exception because of course there's exceptions

Uh and then ps uh set int. Oh, just there's no >> no parameters, but you're going to have to map the >> where ID equals this. Now I've got ps set int is whatever, right? Get customer uh whatever I've got. You know, you can see how that would work. And then I want to actually get the query the result set. And now for each result, I'm going to go through that re that uh that uh result set and then add it to this customers list. And I'll say, okay, customers do add new customer rs.get int ID RS.get string name. Yeah

>> And this is the point where you're like, I wish I had an OM. >> Yeah. Yeah. I wish. Or or just something to make this tedium go away, right? So now you get rid of that. Get rid of this. And then what is that? Oh, it's here. Uh return customer

Now, of course, this ignores also what if I'm doing a a mutation and I need to do a transaction? What you know, how do I do that consistently? Sorry. Oh, yeah. Okay. What if I'm doing transactions? What if I'm doing a uh, you know, exception handling? What if something goes wrong and I need to roll back the transaction? All that gets a lot more muddy. So, instead of writing all this kind of code, we figured, okay, what's the actual business logic? We we care about this bit where you're mapping and we care about the query. Everything else is kind of just boilerplate. So we can use the spring framework JDBC client for example. Okay

So JDBC client. Here we go. And refactor that. There's this. And now we need a JDBC client. We'll define an instance of it over here. So being JDBC client JDBC client pass in the data source. Okay

Like that. And JDBC client.Crate. Okay. So there's this. Now I'm going to rewrite this code now to get to take advantage of that new technology from the distant distant past. Return this dot dhu. Okay. Select all from customers

>> This is just a higher level abstraction over the JDBC connection prepared statement. >> Right. Exactly. So now I can just say new row mapper customer. There we are. Return this. This is an interface implemented. And then the care the thing I care about is this right here

That one little bit of code. Put that in there. Okay. Now return a lambda. Good. Good. Good. So there's the >> better

>> Much better. Right. Like here you can actually kind of see it. It's just it's a What is that? >> There you are. Nope. What happened? It's unhappy. So sad. Wait, wait, wait

Okay. It's nice that your ID is just telling you when you're doing things wrong. >> Yeah. Uh oh, this. Okay. Okay. And then dot to list. There you go

So now I'm actually saying select all from customers. Here's the parameter. Here's the mapper of the results. Give it back to me as a list or as a single item or as a map or whatever, right? But I'm asking it for a list. So this is a lot cleaner than all this boiler plate code. And it's a lot less errorprone as well. So that's a portable service abstraction, right? I'm using JDBC client to go a level above the JDBC the the raw JDBC data source and I get that result. Uh we talked about dependency injection

We talked about portable service abstractions. What if I wanted declarative transaction management? Well, I can now say hey wrap each method I want each method to be transactional. So what I shall do is I I'll decorate it with this annotation saying hey when somebody calls this method start a transaction if there are no exceptions thrown then commit the transaction at the end of the method otherwise roll it back that's what the sanitation is that's AOP aspectoriented programming right so you understand these three things you can see how we solve problems by moving progressively up the abstraction stack but ultimately it's just objects being injected and used and referenced from others now at some point we said okay well this is boring this is something that everybody's going need if they're using JDBC. So in spring boot what what we've just looked at is spring framework but in spring boot we have this mechanism called auto configuration and when spring boot starts up it looks on the class path it looks in the class path for this text file metaimp spring and then this large text file uh autoconfigure autoconfiguration.imp imports whatever and it looks for this file data source. These are text files. These are this text file contains class names and it finds the classes and it tries to run them when the program starts up. So this is just data source auto configuration. It has the configuration annotation on it and it has other Java beans like we saw earlier, right? And all of this code is concerned with detecting whether you have an embedded database, whether you have a connection pool in the class path, whether you've got a well-known, you know, standalone database instead of an embedded one, whether you've specified properties

But basically, this will get run when the program starts up and it'll say, hey, it looks like you want a data source of some sort, and it looks like you've got an embedded database. So, it'll actually automatically create this code for you, right? So, the benefit is now it's doing things for you on your behalf. So that same program will still work even if I don't define any of this. Actually, I don't have to define this or this or this class at all. That's called auto configuration. That's the that's the thing that separates Spring Boot from Spring Framework. Okay, auto configuration. It's the next trunch

So we're in Spring Spring Boot here. And over here we're in >> So the first three were just Spring Framework, right? >> Where you're manually configuring things. And then when we get to Spring Boot, that's where it weaves in the auto configuration just to make life super easy. You don't have to do all the boilerplate code to do all the uh creation of all the things that your application needs to the services and those sorts of things that are >> right and these and these definitions are see this definition here this says hey this is this is spring saying hey I'm going to register this object of properties JDBC connection details for you but I'm only going to do it if nobody else has defined this type this annotation is a guard it's a conditional it says I'm going to do this thing based on the presence or lack of presence of this object So if you define a bean of this type in your own code, then Spring won't bother registering this for you. So it'll back off whenever you whenever you want to override any cog in the machine, you can do that just by defining the bean and taking full control. Right? So we provide the defaults, the convention, but you can provide the configuration and override these defaults at any stretch. It's not like Ruby and Rails where you get a lot a lot of code generated nonsense and you have to undo all of it if you want to make any changes to any of it, right? you can just replace one little part of it. So when you work with Spring AI, you're building on top of this

You're building on this auto configuration. You're building on top of dependency injection and portable service abstractions and uh aspectoriented programming. But again, it's just about knowing to look for the auto configuration and knowing which objects to swap out to get the behavior you want or which interfaces to implement to get the behavior you want. Any questions on this? This is not >> so one thing that often comes up is like okay so with auto configuration there's a lot of decisions being made for me just a lot of defaults which is helpful but how do how would I as a developer know what those defaults are like is the easiest way just turn on debug logs and and see you know when it starts up like how things are getting wired together >> so spring boot has this thing called the actuator this is the other thing is spring boot was a framework when we when when we debuted spring boot back in uh 2013 um Spring was already used in production, right? Like the largest companies in the world when they when they grow up and want to be successful, they use Spring. So, for example, Amazon, eBay, Tesla, uh Netflix, Alibaba, uh all of them have job listings for people using Spring. And you know, I'm infer from that what you will. But, uh the point is it's it's already in production. It's always been in production, right? I did a I I I spoke there's a small mom and pop uh company in uh in in China called Alibaba

Okay? Now in we just passed 11-11 a few days like a week ago, right? Singles day. In China, they have this uh sales holiday. It's kind of like our Black Friday or Cyber Monday or whatever. Uh they have the sales holiday in called 11-11. And in uh 2012, that one uh company made three times more than both Amazon and eBay combined had made the same year, right, in one day in China. And that's all powered by Spring. Every transaction, every service in that organization was Spring. So it's already very very widely used in production

But we wanted to make it easier. Right? This is 2012. Spring boot came out 2013, 2014. And uh one of the things we did was he added a lot. We have a lot of production worthy uh focus. Right? So we have the actuator. Okay. I have this on the class path

I'm going to export all the details here. I'm going to show all the details through this config file. This is by the way the other way you can configure things in Spring Boot is by modifying this text file called application.properties. It's right here. Right. And um so I've got I'm going to tell Spring Boot, hey, show everything. Don't do this in production, by the way. This is for the demo, not for production, okay? Because you're going to be it's going to be wide open

>> Okay? So, I'm going to now start the program. >> You can't put it behind security and export the metrics to your your observability platform of choice. >> Oops. I'm going to also add a web service so that the service doesn't just start up and then stop immediately. >> So, I'll add this web. That's the other thing. We have this concept of starters in Spring Boot. You choose a you care about a concern like web programming or database programming

Here, I've chosen web, right? When I add that web dependency, what I'm really getting is Spring Boot starter web. Then a web is just a dependency that brings in the right code to start up that particular bit of support that you want. So I've added that to the class path. I've done nothing else. Right? And now I've got an embedded web server started up on port uh port 8080. Right there it is. And it's started up. So now if I go here, localhost 8080 actuator, you have all these different endpoints

Actuator. Here we go. And it shows you information about the objects in your program, right? And you can see, for example, which loggers are available and what their log levels are. You can post to this to change the log levels of a given logger so that you can get debug output while the program is still running. For example, uh you can see what objects have been created, which spring objects are running in the program right now and how they're wired together. So here's an object of type handler mapping and it was defined in this class file and has this dependency and you know, etc. So I can see my customer service. Here's my customer service object and you can see that I created a customer service but I asked for transaction management

So what I got was a subclass, right? I got a sub I got a class that extends customer service dynamically, right? So the original >> the reason for that is that with the AOP we're actually wrapping your customer service with something that adds the transaction capabilities onto it >> because object-oriented object-oriented programming is nice but it's not good for like how would you add the transactions to something? You'd have to subclass it, right? single responsibility principle. I have my business logic here. But now I want to do transaction management. So I create a subclass. Ah, you know what about logging? Okay, maybe I have another subclass. What about uh uh you know um um you know auditing whatever I mean all sorts of stuff, right? It it gets very tedious. Object objected programming doesn't have a generic way to say apply this behavior to everything, right? There's no way to say generically here's transaction demarcation add it to all things type class. >> We someday we will have it in Java with type classes

>> Fingers crossed, right? Uh but we don't have it right now. So, Spring provides a way to do that through aspect oriented programming, right? And anyway, here's all the objects that are in Spring. Okay, go back here. Here's your health endpoint, right? So, you can see you get this actually returns 200 if your service is up and running. It returns 500 if there's an exception or if some component is failing. So, it says, hey, I've got a a database. The validation, you know, the connection query is is good. Here's a my file system

It's working so far. Here's my ping and my SSL certificates. They're all whatever. I have no SSL certificates configured, but if I did, they'd be shown here. So, there's that. You've got uh info. This is a Tavalarazza. This is for you to configure, right? It's empty

So, you go over here and you can put whatever identifying information you want in here. So, I can say go back. Go back. Info dot uh info dot uh what do I do? Management info defaults enable in enabled is true. And I can say info.mssage equals hi. Okay. Go here. And there's a message right >> now

Where that's really useful is if you go to your build here and I go up to the top spring boot started parent. Go here. Come on. Get commit. Okay. I'm going to copy and paste this build plugin. Go to my pom.xml. Go all the way to the bottom here

Add that in. Get rid of the version because it's implied. And I'm going to make this a git repository. Right? So now I say get init add get commit minus am yolo. Okay. Now maven skip test because I didn't write any tests uh uh uh package. Okay. >> So you're showing how we can propagate like get information into your info that's exposed through actuator

>> So what that did is I Yeah, exactly. So if I go over here to the target directory, you can see there's this new classes folder and then there's a property file called git.properties that has information about the current build, right? Well, now I go over here, refresh, and it shows the information there. So imagine you do some you make some changes to your code, go to CI, compile it, and then uh it gets deployed into production. Now you have this endpoint that says, hey, I'm running this version of the code, right? I can do get bisect. What about uh uh software bill of materials? You care about the providence of your dependencies, right? So, I can add a uh what is it? Cyclone, right? Add that. Here's another plugin. Add this. Go to my build

Okay. So, I'm adding that to the plugin here. And I go to my Do I have to do anything for that? I don't know. It's a bomb. I don't know. Let's just try it. We're going to go over here. Skip tests package

Okay. Reload. And now if I go back to here actuator uh bomb application cyclone bill uh do I have to enable it? Bill of materials cyclone bomb bomb >> sbomb sbomb. Yeah. Uh sbomb is unrestricted. So, do I have sbomb in here? >> No sbomb. >> Oh, there it is. >> Oh, there it was

>> Actuator sbomb. And it says there's there's an Sbomb called application. Click on that. Paste here. >> And here is the cyclone sbomb showing you the providence the Shaw of all the you know because your application is at the end of the day a lot of mystery meat, right? You want to know who built it, where it came from, what the Shaw of it all is, etc. And what the license is, do I have the right to use it? All that kind of stuff is shown here. or the Maven scopes, the the vendor, the publisher of the dependency, the location of it, which where it come from, uh you know, the version, all that kind of stuff, right? Like is shown here. So what what is all this to say? It's to say that this is a way to build production worthy services

So much of the discussion around AI ignores that a lot of this stuff doesn't ever make it to production for a couple reasons. First of all, uh you saw that we talked about that MIT thing a few weeks ago, >> MIT study that said 95% of AI projects have failed or something like that. And it's I think it's because you know getting to production is hard. >> Yeah. But it's also that they're trying to rebuild all this they've already got this institutional knowledge built up in these JVM and Springbased applications and now they're trying to rebuild new solutions on top of Python with to support AI when that means that they have to forego and abandon all this institutional knowledge that's built up over there. Right? Why why not just build it up on top of the JVM and uh on top of the JVM since it's already such a mature platform. We've got these ability to secure and stabilize and and observe and monitor and get providence about uh your dependencies. All that's built in

Why would you forego all that just for this AI engineering? So what we're going to do today is we're going to see how easy it is to pull all that in. By the way, speaking of deployment, at some point you're going to deploy this to production and you're going to care about a couple of things. So now let's go over here. CD downloads application, right? I want to build, for example, I want to take my application. Let me just build a simple controller so we we can see it worked. I'll go over here quickly. I'm going to build a simple HTTP controller at controller at responsebody class high controller. Okay

At get mapping. Okay. String. So I'm just creating a simple HTTP endpoint here. That's the that's not quite what I meant. >> Hello. Okay. Hi there

Good. Okay. We can actually just do that I guess. So let's just see if that works locally first. Local host 8080. >> There we go. Okay. So we've got a simple HTB endpoint

Now I want to I want to take this to production. There's a couple things you should know about. First of all, in the Java world, we have this thing called gravm. Uh skip tests minus p native native compile. This will take a second. So this is uh something that the JVM um has added support for being able to take a JVM application and compile it into native code. So instead of running in the JVM, you can just make it native >> like a C program or REST program or Go program whatever it's or yeah it's it's just a it is standalone self-contained statically linked no JDK required etc. And it's all transpiled into native code

Yes sir. No. Yeah, good question. So, so it gets rid of all of the just in time compilation pieces. It actually uh does ahead of time compilation of the JVM program. So, it takes a minute to actually create it because it's not just like packaging a JVM and running on the JVM. actually turn it into a static native binary and it does a lot of um inspection of the execution paths in order to be able to do that what would normally be just in time compilation of bite code into actual ahead of time compiled bite code. >> So this is 40 milliseconds

>> You still need an operating system. It's >> it has to run on an operating system. So I have some production applications that that uh run via docker essentially and just have a a I think it's called docker from scratch. So you don't even need like in the case of docker you don't even need to throw a whole operating system in the docker container and I just take the output from native image put it into the container and that's all that's in the container and then can start that up. >> Yeah. >> Oh sure. Yeah. Yeah

So that that question was about um do you what do you actually need to run these native images and you can target uh different operating systems. So if you're running in Docker then you're going to target a Linux based operating system but but Josh just built that on on his Mac and and that all ran fine. So that's a ARM 64 instead of a AMD 64. >> So it's 48 milliseconds to start the program now. So that's 20 times faster than it was on the JVM for the same program and much you know here's the same endpoint right go back there. But >> so it works just the same way that it did before. >> Now it's not gonna like there's a whole there's a whole like caveat. M I'm sorry

There's there's an asterisk there. There's some limitations that apply. Okay, we'll get into that if you have time later. But the point is for 99% of the use cases, this is a really great deployment choice. And here's the process identifier. Go over here. PS minus O RSS. Okay, that's the RAM

So measured in kilobytes divide by a thousand. That's 116 megs for web server with observability and our database. An embedded SQL database by the way. full SQL database running inside the JDK code plus my program. Right? So that's 116 megs of RAM. Uh versus Let's open up this pig dog. Here we go. [laughter] JavaScript

By the way, >> is the Wi-Fi slow? >> No. >> No. Okay. Oh, just just CNN. >> Just CNN just loading. Here we go. Here we go. What is that? What? 932 megs of RAM, right, >> for CNN.com

>> So my point is the JVM is an amazing place to build and deploy and scale services today. It's un it's unlike almost anything out there and certainly a lot easier than everything else out there. So that's the foundation upon which we're building. Okay, it's a really nice place to be. Oh, by the way, you might want a Docker image. Oh yeah, I forgot about that. So uh uh Spring Boot colon build image. If you do that, it'll eventually spit out a Docker image for you

I ran the tests. I think we're going to build Docker images in the workshop, but I think we're using JIB and I need to get that switched over. But whatever the point is, >> there's different ways to create. >> There's a built-in way. There's there's other ways that are that work as well. The point is there's a lot of ways to get a Docker image out of this without having to write a line of of of anything in this case. >> Docker files. >> So, okay

So, that's Spring Boot and Spring Framework, just foundational, nothing to do with AI yet. I just want you to understand the the the shoulders on which we stand when we build today. Any questions on that? Any uh comments? Any feedback? I just want to make sure we're all clear about this because it's going to help later. So clear. >> Okay. Ready to get into AI? >> I think I am. >> Let's do it. >> All right

>> Are you ready? >> Uh here's the Let me just run the Docker image. >> Oh, yeah. >> Come this far. >> Let's show them that it does actually work. >> Uh minus P880 is it? >> There you go. Okay. So, there's that. Oh, I didn't Oh, I forgot to compile a native image inside the Docker

I forgot to dop native whatever. Okay. So you can do either way native or JVM for the container but it works >> good. Is that Yeah, that's shut down. So the >> if I stop that >> still running something somewhere. >> Yeah. Question >> uh the like JMX management what that's all exposed through actuator. Yeah

>> Um do we I think we will get into some of that in the workshop. So yeah. Yeah. Yeah. So [clears throat] that the question for the the camera was um >> how do we how do we do all this with JMX? JMX is the built-in JVM kind of observability of how many threads you're using, how much memory you're using, all that kind of stuff. So um and then there are some pieces to JMX that also expose a kind of control plane into the JVM. Um, and I think Spring uses some of that JMX or you at least can use the the actual management pieces of JMX. >> Does anybody Yes

How does anybody remember how to publish a report? Is it before? >> Oh, yeah. Do it before. That was your problem because it was being passed as a parameter to you >> the program. Ah, there you go. Okay. >> So, it works. >> Painful. Yes, sir

>> And actually, I love that topic, but my memory Oh, gotcha. Okay. Sorry, I misunderstood that. Okay. So, so the question is how do you set the memory parameters on the container on the process within the container and so for that one >> fill the container >> and fill the container. So on that one uh uh we can answer that here because I don't think we go into that in the workshop but um for that the build packs that spring uses out of the box do uh a analysis on how much available memory is available to the container and then it has a default of picking like 80% of that uh as your XMX. Sorry I totally misunderstood your your question. So the X so it will automatically select that for you when you use the build packs from Spring

Uh you can override that and there's many different knobs you can turn. One of the knobs is to set uh to tell the build pack when it creates the kind of information that then determines how much memory to use. Uh you can say like I only want to use 50% of the available memory or something like that. And there's some different knobs that you can turn on the build pack kind of process that's doing that memory settings for you. The other option is you can just hardcode a Java ops environment variable and override any of those parameters as well. So, so some different knobs to turn, but the the the if you're using the Spring build packs, you're going to get good defaults out of the box. Uh this does use the available memory from that's that comes to the Docker container. And so uh the JVM supports reading that information from croups which most platforms that run docker containers will provide the available memory and available processors through crgroups

The JVM sees that and then you're you're able to to use that information um and that all just gets managed for you. But there have been cases where uh croups and in the past the information was not being propagated to the container correctly and I think old versions of the JDK weren't able to read that croup information. So all that resulted in you may uh you may the JVM may think that it has like a billion gigs of memory available when really that's like the host platform's available memory but in any if you're using all the modern stuff should just all be automatic. >> Thank you. Yeah, great question. Any other question? Like we're going to go into a quick 101 on Spring AI next, but I want to make sure we're all like copathetic here on the foundational Spring Spring boot 101 stuff. Any all that stuff is like uh >> we're going to be using like everything we just showed today. So >> at some point or another in this in the life cycle, you know

>> Yeah. >> Anybody any questions, comments, feedback? >> Okay. Okay. >> Let's dive into AI >> right now. Spring AI. We're going to talk about Spring AI. Uh Spring AI is just one module. So we we got Spring Boot upon which uh you know you can build almost anything

You've got batch processing. There's a project called Spring Match. Got messaging and integration called Spring integration. You got Spring for Kafka, Spring for Rabbit MQ, Spring uh uh Spring Security for Ooth and Webn and one-time tokens and Cararos and SAML and all that stuff, right? You've got I mean you've got the this huge ecosystem. Spring AI is a module. It's a very rich and that's why we're what we're here gathered here to talk about today is just about AI with Spring AI but you could do full day workshops on each of these projects right they're each in that domain in that particular context they're very very rich so one of the nice things about Spring AI is that it builds on spring boot which in turn brings in all that other stuff if you want it right so your spring applications aren't in a vacuum right you can use them to then pull in all these other interesting bits >> so you can use spring AI with any of those modules with security with whatever and and I think one of the great features with Spring, the Spring ecosystem, is that these things aren't created in isolation. Like at AWS, we have a a a good Python AI library called Strands. And Strands is great

It's a great library, but it it exists, you know, on its own. It's not part of the of a broader ecosystem. It's like, oh, how do I combine GraphQL with strands? Oh, yeah. Who knows, right? And so, like in the in the case of Spring, we can kind of pick and choose which modules you want to bring together into what you're building. >> That's a good point. We have Spring for Graph Go. We have spring for gRPC. We have spring like there's integrations for everything, right? Everything

So, okay. Um, we whenever we talk about uh an application, I just did the most foundational hello world kind of stuff. So, we just looked at objects and basics. That's a very contrived example, but it's more useful to have a domain that makes a lot of sense. So, and we struggled with this. We wanted to show you very quickly some of the AI stuff. And it's going to be repeat. You're going to see this stuff again in a few minutes as you dive into it

Hands and feet first. Our hands-on is a unicorn store, right? Customer service agent. So So we're we're that's what the hands-on will be, >> right? Well, then that that's a you know, in in in the Silicon Valley, these unicorns, they're real. >> Yeah, >> they are. We have real we have unicorn companies everywhere. Um >> but for most people outside the valley, uh there's you know, I wanted to have something a little more grounded, you know, something that can't fly. So we thought, you know, in the spring ecosystem, we have this spring pet clinic. This is a sample application from the days of Yore

The sample application uh was created back in like 2004 or whatever. It's don't get me wrong, it's been updated of late, right? Here's here it is five days ago. Um and it's such a such a popular demo application that it's in in turn inspired countless uh like uh Legion different implementations in terms of different technologies. So here's the spring pet clinic in terms of our favorite language cotlin and microservices and rest and AI and java Q and JDBC and angular and va and hila and react and and htmx and uh you know mustache and graphql and reactive and blah blah blah. Okay, so lots of different implications. I thought we could look at that, but that's a little too much. We we've only got seven hours. Uh not nearly enough time to look at all that

So we figured let's just find something in the same spirit as that, right? Something to do with like animals, dogs and and cats and stuff because I I mean who doesn't love animals, right? We love animals. We love dogs. We love cats. I love uh uh all [clears throat] I love all all I love all I love all almost all animals. Okay. Uh uh um except uh except except my dog. It's not even true. I I I did love him

I just Nobody We didn't like him. You know what I'm saying? He He was our He was our dog. We loved him. But he wasn't he wasn't a good boy. He was a whatever the diametric opposite of good is, right? He was that he was he was he was not good. But we kept him for three reasons. One is very simple reason is that he's he's an animal and he deserves a loving home. Simple

Okay. Uh that's part one. Part two is ego. In my home, both my partner and my daughter speak five languages. So if we didn't have that stupid dog, I would be the stupidest in the house. That that that will not stand. Okay? It will not stand. So we keep him for that reason alone

And then finally, just being 100% honest with y'all here, just being real with y'all here, he is uh very very cute, right? I mean, uh, I I know. So, so cute. So, so cute, >> but evil. >> Okay. Pure malevolence. Not a good boy. And And we think about him. I think about him a lot

He reminds me of this other dog about which we learned during the pandemic named Prancer. And this dog went viral. So, Prancer's owner was trying to find a new home for this dog. And she wrote this ad that went viral. She describes this dog thusly. She says, "Okay, I've tried. I've tried for the last several months to post this dog for adoption and make him sound palatable. The problem is he's just not

There's not a very big market for neurotic, man-hating, animal-hating, childrenhating dogs that look like gremlins. But I have to believe there's someone out there for Prancer because I'm tired and so is my family. Every day we live in the grips of the de demonic Chihuahua hellscape he has created in our home. And she continues on. She says, "He was quiet and just laid on the couch. Didn't bother anyone. I was excited to see him come out of his shell and become a real dog. I'm convinced at this point that he's not a real dog, but more like a vessel for a traumatized Victorian child that now haunts our home." Oh, this poor dog

Um, what else? Uh, we already addressed the men and children situation. If you have people over, he'd have to be put away like he's a vacuum. I know finding someone who wants a Chucky doll in a dog's body is hard, but I have to try. and she signs off warning, "Oh, he's only two years old and will probably live to be 21 through pure spite. So, take that into account if you're interested." And and by the way, who wouldn't be? Look at him. So cute. I know. Super cute

I would totally adopt that dog. Very, very cute. I might uh you know, I might lose a finger or whatever, but I would totally pet that dog, too. So, I think about this dog a lot. And by the way, this dog did eventually get adopted because, like I said, it went viral. So, here's People magazine. Prancer, the demonic Chihuahua's new owner, says he helped her heal from past trauma. Hi

Uh, here's um, USA Today demonic Chihuahua Hellscape. Meet Prancer, the puppy everyone wants after a brutally honest Facebook review. Right here is Buzzfeed. Buzzfeed. Uh, we caught up with Prancer, the viral nightmare Chihuahua. And then, of course, here is Prancer in the New York Times because a slow news day or something. I don't know. So, [clears throat] here's this open

There's that. Prancer, the demonic Chihuahua finds a home. Right. So, this dog went viral. eventually got adopted. But I like to think about this dog and I like to think that we should build a service to support finding and adopting terrible, terrible dogs like Peanut and Prancer. So that's what we're going to do here today. Okay

So we're going to go back to the console here >> using AI. >> The magic of AI. Yeah. >> The magic of AI >> because I mean that's most people don't find their dogs like that, right? Most people when they find a dog, they go to the shelter and they talk to the shelter and they interview the shelter about a potential dog, right? Um and uh and that that's how that works. So we're going to build the services to support that process. Hopefully, if you found your dog in the daily news, in the national news, that's probably an exception, not the rule, right? Most people aren't like Prancer. Uh, so we're going to build an application. As always, going back to where's my browser? Come on, browser

Do the thing here. Start that string down. We're going to build an application here. Uh, we're going to call this adoptions again because I'm great with names. Amazing with names. Like I said, when when I was a small boy, we had a small white dog and my father named him White Dog. again great with great with names. Okay

Uh we're going to add uh um >> so Bedrock. >> Yeah. Do you want to talk about this? >> Oh yeah, sure. So for today we're going to be using an AWS service called Bedrock for the AI side of things. Uh it's host models. There's like 200 plus models that we can use on there. We're going to be using two different models uh for this application and you will be as well. The first one is going to be for embeddings which allow us to take a string essentially turn it into a vectorzed form so that then we can do some similarity search on it

>> Oh, I'm just >> Oh, you're logging into log in. >> You don't want to show people your passwords. >> Maybe I should. >> So, we'll use an embedding model from Bedrock and then we're going to use a chat model from Bedrock as well uh to be able to give us the chat interface and be able to send text get text back from the LLM. So, two different models hosted on Bedrock. Uh, and we'll dive into some more details on that, but Josh has to get his two factor code to log into his AWS account. >> I'm a professional. >> So, yeah

So, while you're doing that, so Bedrock, it's going to be our LM provider. Uh, we have support in Spring, which we're going to see for being able to do both the embeddings and the chat uh client. And then also in Spring, which we're not going to go into, is things like image generation. Uh I think there's now audio generation in Spring exposed in Spring AI. Um there's all sorts of different other pieces, multimodal support pieces in Spring AI, but for today just embeddings and chat are going to be the things we do. >> Show them what >> Oh, okay. >> Oh, I did it. >> Okay

So, uh this this is in the AWS console. >> Can you see that in the back? We can make it bigger. Sorry, it's like >> So, here's our model catalog. you'll see 249 different models that are available. Uh there's Amazon has its own firstparty models. Uh we are going to be using I think both for both of those we're going to be using the Amazon models today. Um but you can pick whatever model you want to use. You can use the anthropic cloud models

Uh you can use the open um uh what's it called? GPT OSS models are in here. You can use DeepSeek models. Uh Jetrains even has models. >> I don't know what's that one about. I don't know. uh they're like for programming like uh coding models. So mistraw models so all sorts of of different options that you can pick from the Quen models. A lot of people really like the Quen models

Um so that's so we've already configured Josh's laptop to be able to be authenticated to AWS. Uh just released when was it last Wednesday, Thursday was a new version of Spring AI 1.1. We're gonna be using that today. uh the the very new version. Things may break because the workshop was created on Spring 10. We're going to be using Spring 1.1, right? And so we'll see how it goes. Um but I I've test done some testing. Hopefully things are good

But um so Spring AI 1.1 uh it adds a new feature. Oh yeah, and you can watch Josh and I's live stream we did on Friday. It was like the unboxing of Spring AI 1.1. We did this last Friday. Um so >> code unboxing. Yeah, this is we go through kind of all the features, but one of the features uh that is now in Springey 1.1 is go back to your AWS console real quick. You still have that open. Okay, there it is

Okay, so in um in the AWS console, it's now super easy with Spring 1.1 to be able to set up your local machine to talk to Bedrock. All you have to do is go into API keys and then you can generate a short-term or long-term API key. You just get a single token and you set that as an environment variable. It's like bedrock, I don't know, bearer token or something like that. Uh, it even tells you in the console what to set that to. So, that now works with Spring Eye. So, super easy. Sometimes it's kind of tricky to get your local machine off to AWS

This if you're just using Bedrock, this has become a super easy way to do that. So, we've already got Josh's m machine all authenticated to AWS and now we can build our our dog adoption service. >> Love it. Okay, good stuff. Good stuff. Okay. So now uh close that. Here we are over here

So we've got these. What is the difference between Bedrock and Converse by the way? Did we talk about that? >> Yeah. No, we didn't talk about that. So we're bringing in two different dependencies for the Spring application that we're going to build. And the first one is the Bedrock one. That's the one that we're going to use for embeddings. And then there's the Bedrock Converse one, which is used for chat. Uh it could also be used for multimodal

And so that's why we're bringing in two different ones here for Bedrock. the bedrock one and the Bedrock converse one and of course Spring AI supports direct access to cloud models you know to the cloud service to open AI you know what I don't know >> 20 20 different >> foundational many different different vendors you can use in spring AI so for today we're going to be using the bedrock stuff >> right and and by the way there's also spring AI community where you can find like there's a lot of models that are supported in the spring AI community uh that like there's a bunch of Chinese models right there are only you can only run them within the the geo move zone, right? So, it's very hard for us to test them because we don't have access to a data center there. But, >> so those are provided through the AI Spring community, >> right? So, you can find even more models there. You know, it's just it's just only so much we can manage at the same time. >> Uh, okay. So, we got both of these and we're going to add now some other things. We need a a uh observability, right? So, go back to the actuator. We want to have web support

So, we'll bring in the web support. We need some data access. So I'm going to bring in spring data JDBC which is going to be like an OM on top of the JDBC stuff going a step further than what I showed you even earlier. We need a database and obviously most of the time I reach for Postgress but you know what else we need? We need a vector store. A vector store is a data store optimized for semantic similarity search and there are many different vector stores out there. So you know you can take your pick. You can use Milvvis, you can use Neo forj, pine code. There's a bunch that are purpose-built for uh just being vector stores

And then there are many that are just add-ons or extensions or plugins or whatever for existing data store infrastructure. So for example, uh here's Pine Cone and here's Milvus and here's we cudrint and uh Chroma and whatever. These are all purpose-built vector stores. But then there's also, you know, you've got your elastic search, your MongoDB, your uh uh oh, these are new entries. That's nice. Open search. Cool. We should figure out what that is

Yeah, I have no idea. >> Great. >> Um, Azer PG Vector, that's what we're going to use as PG Vector. You have Elastic. Uh, you have Apache Cassandra, you've got uh Reddus, you've got Oracle, you've got uh my SQL, you've got uh Neo forj. So, you've got a bunch of data stores that are, you know, you've probably already got them deployed somewhere is my point, right? And so, we're going to use Postgress as a vector store. So, we're going to get two for one here. Okay, we're going to use PG vector

That'll give us both a SQL database driver and the capability to act as a vector store. Now, these are at the moment it's just a two for one, but logically we're getting two things. We're getting a vector store and a SQL data store. Okay, they're just happen to be the same thing at this time, but that's not always the case and it doesn't need to be the case. >> Okay, so far so good. Um, what version of J? >> We need MCP. [clears throat] >> Oh, yeah. You want to do that, too? Yeah, I like that

Okay, MCP client. Okay, hit enter. Okay, >> so we just created that new project. It's just the the stub with the dependencies and and now we'll start adding our functionality to this. >> First things first, we need to So what? Okay, let's try You want to try using the new Do I need a key or anything like that? >> Oh, let's not do that. >> Okay, we'll just do the old fashion. [laughter] >> Old fashion. >> Okay

Uh desktop reference uh source main resources application.properties. Is that it? So there's a little bit of configuration that we need to set up uh with this new application. It's really just around like which database are we going to connect to. Josh already has Postgress with PG vector running on his local machine and so we need to point our application to that one. There is another way to do this which we should um use at some point which is called test containers. Test containers is a nice feature that's supported in spring that allows you to like automatically spin up the containers for given services. So we could have a test container for PG vector um and do this more automatically than than we're doing it with a manual configuration. But I think the reason why we did the manual configuration is because we have our dogs for adoption in our database already kind of set and ready to go

So >> yep. >> Okay. So some configuration parameters here. First is which chat model do we want to use with Bedrock? We're going to use the Amazon Nova light model. Again, that could be claude, that could be the GPOSS, you know, DeepSeek, whatever. Um, but oh, did we oh, forgot one dependency, which was the uh we're going to do chat memory. So, you pulled in the chat memory that's also going to use the same database. Uh, okay

We then have the embedding model, which is going to be coher. So, that one's actually not using the Amazon one, um, which is called, uh, Titan or something. This one is the cohhere, which is a thirdparty model, uh, on bedrock for doing our embeddings. We need to set the PG vector dimensions to 1024. >> What? >> Yeah, we'll talk about how those get used in a little bit. >> Um, anything else to point out in here? Some database connection. >> We'll come back to these two in a bit. This is a Java feature and I want to talk about this

This is one of the So, this doesn't happen very often. So, just let's let's just enjoy it. This is there's very it's very rarely the case where we have something in Java that is not just as good but better than the equivalent in other languages. But this is one of those very few cases. Now, if you've ever used one of these other programming languages, and by those other programming languages, I mean things like JavaScript, not the the good kind, the bad kind, right? This is the good kind. This is the bad kind, right? So, it's very important to know the distinction. If you've ever used JavaScript, if you know TypeScript, if you know uh Python, uh even Cotlin, sorry, it's just got it's got the suspend functions. It's the same failing, you know

um you've got C sharp, you've got uh whatever a lot of these other languages, they have a thing called async await, right? And so the idea is let's say I'm writing a JavaScript program, the the bad kind, okay, not the not the good kind. So I want to get a customer here and uh how do I get the customer? I describe it as a promise and I say I'm going to return a customer here and then I return, you know, return uh window.fetch blah, right? I call this URL and I get the response back and it's a promise. And how do I consume that? Well, I can say con customer equals await get customer, right? And so what's happening behind the scenes is if if I could somehow print out, you know, the current thread or something like that. If I could do that, what you would see behind the scenes is that the thread here might be different than the thread over here because behind the scenes, you've made a a call to a network service that is going to block. It's going to take time waiting for those bytes to return, right? And uh and that that is a uh it's a you don't want to be stuck on this thread if all you're going to be doing is waiting for bytes to return, right? We call that blocking. And that's that frustrates your ability to scale, right? Because you can't you've only got limited threads in your system. And it's very important not to sit there and not do anything if you're just waiting. You got to give that thread back and let somebody else use it

>> In the case of Node, you have one thread, >> right? Yeah. Exactly. It's even worse in node. >> So do not block it. >> Do not block it ever. So this is a they have this this mechanism where you tell the runtime hey I plan on blocking I plan on doing something that would otherwise just sit here please feel free to take me off the main thread and let me resolve that in the background and then let the main thread continue. So the behind the scenes the com the runtime will rewrite this code so that the thread that you're executing on here might actually be different over here right uh because you're going to be doing this thing that might take a few seconds or milliseconds or whatever to resolve you're making a network call right and um oops come back well I lost it all I don't know how I lost it whatever the point is uh the point is >> those vi bindings >> yeah I gota like redo sequence oh god all right I'm taking a spiritual journey Uh, and it's it's uh I resolved this year I'm going to learn uh Vim motions, right? I've been an Emacs person for decades and this year I saw all the cool kids are using Vim and I'm like I'm going to I'm going to be cool. I'm going to learn it

But it's oh it's not going well. It's been a year and I'm not sure how well it's going. So anyway, >> you still haven't figured out how to get out of it. >> I still don't even know how to get out of Vim. It's been terrible. Just terrible. Um uh so anyway, we've got this like the point is this is lowlevel, right? I have to write everything in terms of these asynchronous types like promises and I have to decorate my functions. I have to say that this is async and I have to say I want to do a wait

Well, this is like low-level garbage management. Like if you're doing pointers in C++, this is the same thing for asynchronous composition. This is a waste of time, right? And >> there's a there's a great blog post about colored functions and why they're bad. This is what we call function coloring where we're having to delineate these are the functions that do async things and then these are the functions that don't. And the problem with that is that it's really hard to mix and match >> the you like often times can't mix those. And so um so there's another approach that we can do in the world of the JVM which is better. Right. >> Right

And and it's more transparent. So this is the you know if I wanted to make this call I would have to I'd have to do it inside of another async function. >> Right. >> It propagates that like anytime you want to call an async function that thing then becomes an async function. That's part of the challenge with >> it spreads like a virus, >> the function coloring, >> right? So, we don't like this. So, in Java, uh the the interesting what they did is they got rid of the there's no keyword. You just call get customers and as long as you are executing on a virtual thread, then it'll just automatically do the right thing. Right? So, I can demonstrate that here very quickly if you will permit me a brief digression here

Let's say I go here and I say at controller at response body class high controller, right? get mapping map of string. Uh actually, >> so while you're writing that, I'll I'll explain how this relates to AI [laughter] and you can keep coding. So in the world of AI, obviously we're going to be making calls to LLMs. And so that those calls are going to be network calls up to our model. We're going to be sitting there waiting for the model to start returning text. Maybe we're streaming the text back, whatever. And while we're kind of waiting for that LLM to respond, we don't want to actually have threads being used in our application. That's just as wasteful

Whenever whenever we we're just waiting for something else to happen, don't have threads actually allocated to that. And so there's been many techniques like the async await and JavaScript uh with effect libraries with futures and Scola, lots of different ways to get away from that that just blocking threads for no good reason. And so now built into the JVM is something called virtual threads. And Spring supports this so that all we have to do is turn on virtual threads. And automatically the JVM will know, okay, we're waiting on something to happen externally. Let's just take that virtual thread and and not have it allocated to the actual operating system thread that's actually going to be executing that thing. So it's really just an efficiency that in the past we accomplished through being reactive. And of course, if you want to use reactive stuff, uh, web flux in the spring world, you can still do that in Spring, but a lot of that stuff isn't really needed anymore with virtual threads

We just get the all the benefits that you want with not wasting resources just by setting that one parameter in the application. >> Uh, there we go. >> How's it going? You got your >> super duper good. I'm going to return I'm creating a simple endpoint that's going to call another one. This endpoint returns after 5 seconds. It's a HTTP. This is equivalent to httpbin.org, right? And so I go to localhost. This is a docker image that em emulates httbin

I go to localhost880 delay five and it'll artificially take 5 seconds to return. Sort of like a simulated call to an LLM or something where I'm waiting for bytes to return and it just takes a long time. Right? So I'm going to make a call now in this JVM program. I'm going to call that container. gonna make a note of the current thread before we make the call to the contain to the delayed service. I'll make a note of the thread afterwards and then I'll just print out the current thread. Right? So note and then reload. Okay

So let's see if that's happy. Okay. So I'm going to send 40 requests 20 concurrently to localhost 8080 slash. And did I call it anything else? No, just that. Okay, >> so it's just simulating making a bunch of requests to this service that's calling that other blocking service. And then what's happening with our actual threads underneath the covers? There you go. So you can see the logical thread is virtual thread number 98. Virtual thread number 98

They're the same, but the actual operating system thread under the hood has changed. It went from work fork join pool one worker four to fork join pool one worker 10. Right? You can see there's a there's the before and after has changed because I'm waiting for five seconds. It says, "Hey, you can't just be sitting here waiting. I'm going to park you in RAM and I'll wait for the bites to come back and I'll put you back on another thread as soon as you're done, as soon as you're ready to go." So, this happens automatically, right? It's a function of on what thread the framework is dispatching to this controller handler method, right? And uh we set that globally pretty simple like that, right? By setting Spring threads virtual enabled. So if I get rid of that then >> you just turned off the virtual threads. >> I did. Yeah

>> Okay. Making those requests again. >> Okay. >> Now >> everything is the same, right? It's a >> same thread before and after >> 1815 1850. Oh well 115 115 115 you know 165 165 etc. So this is the old way. This is blocking. This is not going to scale well right

So you just set that you get you just do this and you get the same thing as a async without having to worry about that. So, it's actually a really nice feature. Uh, you know, >> that's cool. >> Uh, what else do we want to show them here? Okay, >> we got the database. >> I guess that's it. >> Let's get our dogs adopted. >> Let's get them adopted. Okay, I'm going to connect to this database here

Hit test connection. Hit okay. Hit apply. Go over here. I've got this database tables. These tables. Let me just uh got maybe too much going on here. Let's just drop some

>> What is all that stuff? >> Oh, I'm working [laughter] on [clears throat] >> for tomorrow. >> Yeah. different day. Okay, so we've got our We don't need this either, do we? Project actions drop. Do we need the security stuff? No, we don't. Let's just get rid of that. >> Clean up. >> Yeah, I'll >> Okay, so we got our dogs

>> We got one in a database >> called dog. And in this table, we've got a bunch of dogs that are up for adoption. And here's our buddy Prancer. His ID is 45. His name is Prancer and is described as a demonic, neurotic, man-hating, animal-hating, children-hating dog that looks like a gremlin, right? And uh surprising surprising to everybody, no owner, right? So, we need to fix that, right? Uh so, let's go back and first make sure I've restored the virtual threats. I did. Good. Stop this

And we're going to do this by building a controller to support answering questions about the dogs in our fictitious dog adoption agency called Pooch Palace. So, class adoptions controller Is this we're going to build a chat bot. Is this the only thing you can do with AI? >> I we could do all sorts of things. We've mentioned some of the multimodal stuff. Um this one is going to be a chatbot. We're going to be able to chat with the pooch palace and and you know, figure out which dog we want to adopt, which will be Prancer. Um but you know, all sorts of things. Oh, and we'll show the structured data piece, which is Yeah

So, we'll we'll get into some of the other things that we can do, but >> um Okay. So, you're building right now just a basic controller. This is going to be allow us to make an HTTP request to the service and then interact with the LLM and start chatting. So the first thing we do is we add in our chat client. Our chat client uh comes in and this is all stuff you've seen before with beans the injection. So we're getting our chat client builder. This allows us to set some properties for when we actually set up our chat client. So right now we don't have any properties set up

We're just building that chat client. But um what else did we want to did we have to show on there? or you're you're just going ahead. Yeah. Okay, great. So, we got our our chat client. We got our HTTP request handler. Uh this is allowing us to make a request. We're primitizing it with a username, which we'll see why we're doing that for memory in a little bit

We're just trying to establish a dial tone here. So, hello Josh. It's nice to meet you. If there's anything you'd like to talk about or if you have any questions, feel free to let me ask. I like this. Right. Very, very friendly. We were connected

Clearly, we're friends already. I'm sure it's >> Wait, real quick before you do that, let me explain what just happened. We made an HTTP request to the Spring server. The Spring server then used the chat client to talk to Bedrock to make an LLM request. We got the response back from the LLM and then fed that back in the HTTP response. So, >> is that everything? >> Mhm. >> Okay. And we printed it out to the console and it's working, right? So, okay

And and this is this is the chat client, the chat client here. We're injecting the builder. We're building a brand new instance of it. uh you can create as many chat clients as you want, right? But when you use the preconfigured injectable builder, it has a pointer to something called a chat model. Okay? So there are chat models are stateful. They're heavy there. So we have a bedrock model, chat model implementation here. And uh you might have one or two models per application, but you might have dozens of chat clients

Chat clients are cheap. You can create them with different defaults, different security, different models, you know, different chat models, different uh different parameters. >> Yeah. So it's cheap and easy to create as many of these chat clients as you want, but they will point to the same model which is autoconfigured for you based on the properties and the environment and all that stuff. >> Okay, so far it's a good So we have a this application. We we've established that we're working. It's it's got a dial tone. Let's just make sure everything's fine, right? Let's just let's just see uh if it knows what's up

Oh, does not appear. It does. So I'm sorry, but I don't have access to personal information about individuals unless it's been shared with me during our conversation. So clearly it's got a memory problem, right? And that makes sense. These models are stateless. So what we want to do is give it uh access to a transcript of everything that's been said between us and the model. Uh and we want to correlate that conversation with a particular user for tenency, right? Multi-tenency. And one way to do that is to use something called an advisor

Uh advisers in Spring AI allow us to note and to intercept and to take note of and pre-process and post-process all requests and responses going to and from the model. Right? So I can take the request, I can use an adviser to change that request or to add to it or whatever. I can then take the response and then use that to add use an adviser to note that response and note the request as well. Advisers are like filters or interceptors or whatever. They have a chance to do anything with the request. So I'm going to use the prompt chat memory advisor, but we want to store it somewhere, right? Where do we store it? Well, probably a database, right? So uh I want to use a JDBC. JDBC of course is the Java database connectivity thing and the question then of course is how much do we store right so if I point it to this data store data source uh in theory we could store everything but I don't want to store everything so we'll create a message window chat memory that'll limit it to the last uh 20 I think it's the default is 20 windows so it's a rolling window um so this is just a out of the box uh advisor for spring AI so the advisor integrate with the chat client we'll show you where they hook in in a minute. But we can create our own advisors

There's what other advisors are available? >> Question answer, recursive, all sorts of great stuff. >> So now try this again. Oh, it's going to want to store something in the database. Like I said, we happened we already configured it to initialize a table to do that, right? So now we should actually have two tables there. Spring AI chat memory at the moment is empty, right? And we we created this uh advisor. Now we've injected it and configured it on the chat client. But we also need to tell the AI model uh sorry the AI chat client uh with which user or against which user to correlate this this this transcript. So I'll say uh that the chat memory dot conversation ID is this

Okay. Reload. So when we send the question from from you on this HTTP request up to the model, we're going to be able to essentially um hook into that advisor and correlate the user's ID uh so that you know if we have multiple users and are not going to get different memory >> conversation ID is Josh, right? So it's correlated to that. You can imagine using spring security and injecting the principle, the current user authenticated principle to get a different correlation ID, but I just hardcoded it in the path for the demo here. >> Yeah. Question. the user question and the assistant response. >> Yeah

So in the in the conversation, so the question for the camera was uh why in that table did we see two different rows created when we made that one request. So what happens with the L the conversation with the LLM is we send the message to the LLM. That's the user message. And then we get the response back from the LLM that becomes a second message that is labeled as assistant. >> Right? And here's the uh I asked it what's my name? and it remembered it because it was able to remember on each request we're sending everything that's been sent prior to that request so that it can refresh itself on the conversation, right? It has context. >> Okay, >> so this is just like the most simplest form of memory. There's other forms of memory. Maybe you want to do like automatic summarization

I think there's maybe even a built-in advisor for doing the like like summary uh keeping track of the summary and and that sort of thing. So um totally up to you like how you want to use memory with your LM. You likely for anything real need some kind of memory system because again the LLM has no memory between requests. We need to send all the context all the information on every request to the LM. >> What was that thing we saw in bedrock that just came out? The prompt >> prompt caching. Yeah. So Spring AI 1.1 just added support for prompt caching. Uh prompt caching is just giving you like a like a shortcut to a cache key on the actual LLM side of things

And so um we'll have to figure out how to integrate that into our pooch palace. But [laughter] but basically the model so we can do memory on the client but also the the models now are starting to debut this within the model themselves. So you can send a a question and it'll do a fuzzy match and find similar resp uh fuzzy responses and return like much faster. I think we saw it's like went from 10 seconds to three seconds for so many requests whatever like >> for like a whole book or something. >> Yeah. Like it was really really big gains in certain cases. So, that's actually pretty promising as well. And that also implies less token to spend as well

We'll talk about what that means in a second, but it's a good deal. Um, okay. So, this is it's worked. Here's the my my name is Josh. Okay, great. Got a response. What's my name? It says, "Hello, Josh." It's remembered me by looking at the transcript uh that we gave it before. Okay

So, I think we're doing good. But we've got a big problem here, right? Which is that we've now got a wideopen endpoint on the on the internet piped straight to my AWS account, right? And uh this isn't really helping. I mean, somebody could come along and use this to help get hope with their uh with their homework or something, you know, like this isn't really helping people find the dog of their dreams or in this case nightmare. So, what we want to do is to give this a focus, a mission, and we can do that by giving it a system prompt. So, we shall configure a system prompt. And I happen to have a pre-written system prompt, right? Because I am a professional. So, uh here's the system prompt. We're going to just cat that and put in the the clipboard buffer

And wait a minute. Wait a minute. That's wrong. [laughter] Much better. Okay. Copy and paste that var system. Paste that in. >> So, what is your what is the system prompt that you've just given it? >> It says you Oh, hello

Come back. It says, "You are an AI powered assistant to help people adopt a dog from the adoption agency named Pooch Palace with locations in Antworp, Soul, Tokyo, Singapore, Paris, uh, uh, Mumbai, New Delhi, Barcelona, San Francisco, and London." Hi, did I say answer? I'm in Oakland. Yes, it's not the same place. Uh, and London, information about the dogs available will be presented below. If there's no information, then return a polite response suggesting we don't have any dogs available. Okay, great. So, it's just this this is a system prompt. That's just a fancy word for the text that comes before our text

Literally, when you think about all of AI engineering, at the end of the day, 99% of it boils down to just the the almighty string concatenation operator, right? We're just concatenating strings in the right order, human language text. So, when somebody says, well, you need to do Python for AI, they're delirious. I mean, there's just no reason. It's just text over REST API, right? I mean, 99% of the time, this is not uh particularly endemic to a particular language. So, let's try this out now. Um, do you have any neurotic dogs? We don't have any dogs listed as neurotic. However, that was so fast, by the way. That was fast

>> That was so fast. Sometimes I do this with one of those other models and it's it's not like this. Um, uh, variety of dogs with different uh, temperaments and needs. If you're interested, I can provide more details about the dogs available at our different locations. Would you like to hear more about them? Sure thing. But the problem is it doesn't actually know how to resolve that request, right? It doesn't have access to the dogs. >> So now has the personality of of that we want knows what it's trying to do, what it's trying to help us with, but it doesn't have our dogs. >> Nope

Let's see if it mentions pooch pal. Sometimes it'll even mention like pooch palace. So it'll clearly I think it's caching it actually. [laughter] I think the caching is kicking in. Let's Let's see if we can delete all that. Oh, it's pulling it from memory. >> Uh, wow, that's so fast. I cannot tell you how fast that is

That's crazy. Okay, anyway, it didn't work. It It worked, but it doesn't mention pooch palace right now, but sometimes it does. The point is it's it it's acting like it should be able to help, but obviously it can't because it doesn't have access to the the data. And so, what we need to do is to give it access to that data um through a pattern called rag. >> So, rag is retrieval augmented generation. Is that get that right? Rag. Uh, so this is the way to get data into our prompt that's going to be sent to the LLM

So that now we can tell the LLM like here are the dogs that I have available. And uh, we've got a large database of dogs. We don't want to send all the dogs on every single request that we make to the LLM. So in the rag pattern, what we're going to do is we're going to be able to just pick a subset of the data to send into our prompt. So rag is just a fancy way pattern to be able to do that like similarity search. So we're going to take the prompt that comes in. We're going to similarity search that with our data and then we're going to inject automatically uh the data that matches into the prompt. Right? So what I'm going to do is I'm going to go through this one time when the program starts up

I'm going to go through and I'll create a new spring AI document. Okay. Now this is going to contain data about the dogs like this name, description, ID, etc. And it's just a string is there. There's no particular schema here. It's just a as long as we have the same text for all the records and it's consistent then that's all that matters. And we're going to write it out to the vector store. Now again I'm dealing with in terms of the implementation

This is another portable service abstraction. The implementation of the uh the vector store is for PG vector store but the abstract type is called vector store. We'll just work in terms of that lower level type. I'm going to create a spring AI document. Now I've got a a list of one here. It's a singleton list of one here that I'm passing to the vector store. Normally, if you have a large enough body of text, if you have like a Wikipedia page or an article or whatever, you'd tokenize it, right? You'd split it up into different chunks and then send that as individual documents uh as a list of individual documents. And there's a whole hierarchy of splitters in Spring AAI for exactly that kind of tokenization

But in this case, we just have a very simple little little bit of text. Um >> so in order to do the rag, just to hop back for a second, we need to get our our dogs that are in our adoption database vectorzed. And that vectorz is just creating a using a language model to create a matrix of numbers. And the reason why I'm going to get to that matrix of numbers is because later we're going to send a prompt. We're going to then do a match to our our our database of dogs from that user prompt and try to find which dog is is essentially matching the prompt that we've we've sent in. And so vectors and uh this uses cosine similarity search or something underneath the covers just becomes like a a simple math equation to go from how similar are these two vectors. >> We even have a simple vector store inmemory implementation of the vector store interface that's in spring AI. I wouldn't recommend you use it but it's kind of illustrative uh and interesting and you can see behind the scenes most of the interesting logic is actually this cosine similarity and just basic high school math you know matrix multiplication that kind of stuff

So okay, we've got this uh application. I've got a a document that has the data about the dogs and we shall call it a document. Okay. And now um finally I'm going to configure a question answer advisor. This is another adviser that will whenever we see a query first go consult the vector store to find any relevant data that might match the query. Right. So here we are. Come on

Question. Did I break something? return new question. I I just type it there. Fine. It's like okay. What >> outside the package? >> Oh, do you have a question answer somewhere else? >> Is it builder now? They have a builder. They built it. >> Ah, that's new

We've always I've just returned the constructor. Okay. 1.1. Like I said, there's some teething issues. 1.1 came out last week. Guess how much time we've had since last week. >> No, no time. Um, okay

So, here we go. Question enter advisor. We'll just inject that just as we did the other prompt chat memory advisor. Okay. And, uh, configure that down here on the default advisors. >> So, we just added rag to our chat client is just through that question answer advisor. So, it has the vector store. We're storing the vectors in our PG vector database and then that question answer advisor is giving us the hook into our chat client to be able to do the rag all for us

>> And by the way, notice that it happened when the program started up because I wrote the code to do so in the constructor and >> the actual like vector creating the vectors. So we did that just kind of manually. Usually you're going to have this as part of your uh like create and update process is when you're going to actually create your vectors. Um we didn't we didn't want to go that far. We're just manually creating the vectors uh here. >> Is that what you're pointing out? >> Yeah. Yeah. I'm >> And we only want to do that vectorization once

So now now we're false in that. >> So okay question >> 5.2 seconds here the first time. Let me restart it without the vector store capability without the vectorzation >> behind the scenes by the way. And that's 1.5. So it's a lot better. And by the way behind the scenes it's using the embedding model which points to cohhere. Right. Like >> the coher model on bedrock

Yeah. >> Right. Did you show them the table already for the >> uh here's the table and we'll get to your question in a second my friend here's the table here's the ID the content the original content the embedding that was created by the so we sent the text the plain text to the embedding model it sent us back this this matrix of numbers we then stored it in PG vector store which is acting as our vector store we stored it in the embedding column so vector stores are data store is optimized for storing and querying these matrices these vectors yes >> how does it know to ask it so it's advisor you it's like an interceptor where it's going to intercept the incoming and then also allow you to intercept and and manage the outgoing as well. also request response essentially and you a advisor the question answer advisor is going to intercept the the entry into that do the rag do the the cosign similarity search against our PG vector database and then inject into the the prompt that's going to go to the LLM it's going to inject our dog data >> so >> yeah another question >> what was the question by the way >> oh the question was uh how is how are we actually intercepting like what is the what is the actual way that that question answer advisor is hooking into the chat client. >> We got to get go to that repetition of the question before we go. >> Yes. >> Okay. Do you have any neurotic dogs? >> Oh, it found Prancer

>> Prancer. Prancer is described as a demonic neurotic dog that is man-hating, animal hating, and children hating and looks like a gremlin. If you're interested in learning more about Prancer, and I am uh or any other dogs we have available for adoption, please let me know. So, that's worked. Now uh this is text. Was there another question by the way? >> Oh yes another question. >> Yeah >> simple vector store. >> Yeah

it's this is not you would use in any way. So the simple vector store is just our our storage uh how we actually store those vectors and the simple vector store is just an in-memory storage of our vectors. >> You we're for this application we're not using the simple vector store. We will use it later in the embel workshel u but for this one we're using pg vector as that that data store. So, Spring has built into it the the PG vector-based version of the vector store, right? And so, that's storing the vectors and then also giving us the search on the those vectors as well, which in the case of PG vector is powered by post. >> Okay. >> Don't use No. One important thing to point out is that the simple vector store isn't responsible for creating the vectors

So it it doesn't do the actual tokenization of the string to the vector. We're using the bedrock coher model as a way to actually generate the vectors and then we're using the vector store uh in our case PG vector to store the vectors. >> It is yeah there is. So if you if you are storing your vectors in memory here with the simple vector store then you can use the simple vector stores cosine similarity search um it's probably incredibly slow and uh and depends on how many records you have uh it'll get you know significantly slower whereas you should be using you know generally external vector store that knows how to do cosine similar research on the data side not on the application >> using GPUs and things like that too usually So this is I just had deja vu by the way like what's this total I've been in this very moment. Have we have I spoke in this room before? I wonder. Um >> maybe years and years ago. >> Yeah. Anyway, uh here's the PG vector store

That's the one we're using. I just I only mention the simple vector store to mention that it's like vector the searching for vectors and finding things related is not all that difficult. It's 300 lines of Java code. Not very efficient code, but it's still there, you know. >> Um okay. So, so do you understand that? So basically we we have a a string. We took that string. We went to the embedding model

We said, "Hey, embedding model. Embed this text. Turn it into a float of numbers. Uh the embedding model is an interface. The interface has a it points to in this case the bedrock embedding models, right? Oh, >> the code here one. >> This guy. >> Oh, good work, buddy. >> Hey

Hey. uh and then once it tokenizes it, it then uh we it get then gets stored in the PG into the vector store implementation which in this case again is Postgres. And so if you go here, here's the embedded form of the text, right? So far so good. I know it's a a lot of moving parts, but all that to say, we tell Spring AI's chat client to consult that with this and we got back a string. Now, a string is no foundation upon which to build an abstraction. You you're going to want strongly typed data at some point, right? Or type people. Is that fair? Yeah. Yeah

I mean, we've been using strings in and strings out, but a lot of times when you're working with LLMs and you're building kind of hierarchies of of pieces of your agent, you don't want to just take text in and out. You may want to actually pass data in and get data out. And so, we're still going to pass text in in this case, but at this point, we can show, all right, let's say we wanted to not get text out. We wanted to get structured data. Let's now tell Spring AI to ask the LLM to get the data into a JSON form and then we're going to des serialize that JSON form into the object and all that is covered for us with just that dot entity instead of the content. So now we see oh so we actually get it into an object from the LLM. Uh and then we then serialize in this case our HTTP body as a JSON object. So it's kind of like des serializing JSON from the LLM into an object and then re serializing it into JSON coming out of the GP controller

>> But the point is we do have a strongly typeyped object. I can if I assign this to a variable it's a dog option suggestion var and then I can say entity name description and here's the integer ID. Right? And so the model was smart enough to bundle up the data for us in a form we can use. I can get access to that integer and I can use that to find the record. I don't have to parse this paragraph of text back. So that's called structured output mapping. Yes sir. >> Is that using structured output or is that using >> it's we have a generic one

It's getting better for certain models. Yeah. >> Yeah. So in we're using the bedrock uh the bedrock converse library for the for the bedrock models here. And bedrock converse has a structured output mode to it. And so this is just writing on top of that. Springi didn't have to do anything special there. It just, you know, used used that

If you're not using Bedrock, you're using other models that may not have structured output support. I don't know what Springey would do if like as a fallback if it's like like, okay, the the model that I'm using doesn't actually have a way to do this. I'm not sure what Spring >> I'm not sure. I think it'll >> we should experiment with that. >> I don't want to be in that situation. >> Yeah. >> So I don't Yeah. >> Um Okay

Uh, I'm going to go back to a string though because for now there's, you know, usually these things are multi-step, right? We just asked a question, we got a response back. That's called oneshot prompting, right? You sometimes though, more often than not, and we'll talk about this a lot more when we get to Babel, there's multiple steps and uh and we want to orchestrate that or at least move that step that workflow along. The next step here is that now that we found the dog of our dreams, right? Um, I go go back over here, reload. Do you have any neurotic dogs? Uh, yes, we found Prancer, but Would I like more information about Prancer? No, I don't want more information. I want to just adopt him. He sounds delightful, right? Sounds like the dog for me. So, what I want to do is I want to build a service to schedule a pickup, an appointment to come pick up this dog, right? And uh and take him home and adopt him, you know? So, we're going to in integrate here uh our dog adoption. Now, this is where I think the real opportunity is, right? We have we've integrated the AI with our data using rag and now we're going to integrate the business logic

Yes sir. >> So you said in the previous request before it went to the model and you added embedding. >> Yeah. >> But here's the second thing I'm ready to search on. >> Yeah. >> So how do you skip not? >> Yeah. So we are in this case, you're right, going to rembed the data and so that's unnecessary. Um, what we don't really know is like like when when we actually get this request, do we want to provide data? Do we want to provide tools? When would we decide like when to to provide one or the other of those? And so it actually gets tricky like like so the easiest approach which we're going to do here is just re-mbed the data even though it's not necessary but we don't know when we shouldn't embed the data

So I think it's one of the challenging pieces with AI is deciding like what do you put into context when and how do you make the decision on when to do that because it's all just dynamic free form text. When we get into agent orchestration with Babel you'll see where we get much more granular control about what actually goes into the context. >> But could you not have replaced just a bunch of tools and not even back. >> A great question. So yes, so what we're going to be doing is using tools to actually do the dog scheduling. You can also use tools to provide data into the prompt. The challenge with that and the reason why it's different from rag is that the LLM is going to decide when to call tools and if it decides that it needs to go fetch data from a database then it's going to you know result in a tool call which will then go do the fetch and so then we've got a few more kind of turns through the LM to get to the same point whereas rag is like pro proactive from my side no matter what I'm going to put that data into the prompt with rag and so there I think are different different times when you want to do rag versus you want to do tools like do you always want to have that data in and maybe like you're saying in this case maybe we don't always want to have that data in it should be a tool instead >> and you're optimizing do I want to reenccode something or do I want to make the LLM do work and I'm not sure which one creates more carbon for the planet but it'd be very hard to you know okay so I've got I've uh written this uh straw person dog adoption scheduleuler it's a method that takes an integer for the ID and the name and given that it's going to give us a date when we can come and pick up the dog I've just hardcoded the date for 3 days in the future and I've printed out scheduling the dog ID/dogname uh for pickup at and then I I printed out the time and uh we've annotated this with decorated these this business logic with the at tool annotation and we've been very clear about what the intent of this method of this tool is. This is a tool in the toolbox and this tool can do one thing and one thing only which is to help schedule an appointment to pick up a dog for adoption at I should have said at a pooch palace location right? Um, >> so this description is really important because the tool description gets sent to the LLM and that's how the LLM then decides when to call that tool

>> Yeah. >> So the uh the LM really needs this information so that it can make a good decision on when to call tools. So this is the tool calling pattern with LLMs. Most LMS now support the ability to provide into your context the list of tools that are available, the metadata about those tools like the description, the potential input parameters that are required, and then even potentially the output schema that's going to come out of that tool call. >> Yes, when you were young and your mom told you to use your words, this is what she was talking about. Okay, she's saying clearly document your AI inputs. Okay, this is very, very important. So, I've got a dog adoption

I'm going to make the model aware of that tool by injecting it into the uh constructor here just as I have so many other things and then uh pointing giving a pointer to it to the default tools method here. Okay, so I run that. Now we can let's go ahead and get rid of the uh cached data. Okay, so now we say do you have any neurotic dogs? Okay, good. Uh found our buddy Prancer. Fantastic. When can I >> So on that on that run, it didn't run your tool. It's the the model's like I don't need to I I know that I have this scheduling tool available, but I'm not going to call it this time

>> No output here. uh when uh from the Oakland location. Okay. And uh don't have does this uh got the scheduleuler default tools. >> Something changed with Springey 1.1. >> Oh boy. That's fun. >> Oh boy

>> Scheduler schedule. >> Check your check your memory real quick. See But didn't I? >> Let's try again. >> We can always roll back. Do you have any running dogs? Yes. >> Okay. Prancer. >> Prancer

So, >> when can we schedule it? >> When can I pick up uh what does the prompt say for the tool? Wait, wait. Schedule an appointment to pick up uh or adopt a dog. Oh, no. Schedule to pick up. >> We're live prompt engineering. [laughter] >> Uh, dog from a pooch palace location. That looks okay. Okay

>> Do you restart your Oh, this is all in the same process. >> Did it? So, we're gonna we're gonna do some uh Oh, no. I don't know. We don't know. Who knows? Was it one out of three? >> You're We're trying this. >> Yeah, I'm not gonna >> We could just skip to MCP. >> Well, I mean that. Who knows if that's gonna work? Oh my god

Okay, so uh when Okay, here we go. Take five. >> Wait, does it have the memory? >> Oh, uh let me see. Delete this. Goodbye to that. And now, do you have any dogs? Okay, good. It's PR. By the way, are we running on Spring AI 1? How do we know? Is that builder still working? How do I know which version I'm using? Uh even just make sure it's reloaded, you know

Okay. And now we go over here. >> It doesn't know about our tools. Wait, are you doing the tool setup right? >> Maybe not. I don't know. I've never used this technology before, clearly. Uh okay, hold on. We got the Default tools scheduler dog adoption scheduler schedule int dog dog ID string dog name

>> That looks right. Huh? >> Oh, wait. Okay. Color me confused. So, when can I pick up Prancer? Yeah, there is a way to to do debugging to see what the tools are. >> I am very confused about that actually. Wait, has anything I'm still not convinced >> that you're on >> the right version >> version >> 1.03 that looks right. Got the AI advisor web data JWBC writer vector store

simply type something differently. >> It's not MCP yet. >> Yeah, we haven't gone to MCP yet. So, that's part of what's >> making this more fun. >> Yeah. >> Dog adoption schedule. Schedule. That looks right

Schedule an appointment. ID of the dog. The name of the dog. Three days in the future. there is uh a way to turn on the the actual like prompts to see the raw prompt. So, we could do that. Hopefully, we don't have to do that live. But >> yeah, >> um let's let's should we skip to MCP? >> Yeah, let's try that

>> Let's try it. Let's try it because maybe >> Okay, whatever. Let's just skip to MC MCP. So, >> okay. So, this is like local to tool calling. So if something something is weird, we'll figure it out. But uh with with tool calling, your tool calls can be in the same process as your agent essentially. We've got our agent, our dog adoption agent, and we can do the tool annotations right in the same process and those will get called and Spring will manage all that for us

And we'll figure out why the why it's not working in a minute. Um, but we can also take those tools and make them remote tools. And MCP is the primary protocol to to be able to essentially have a microservices architecture so that all of our tools aren't running in our same process as our agent can be external. And we can have many MCP servers. We can use off-the-shelf MCP servers. We can build our MCP servers. But MCP becomes the standard protocol to externalize those tools and call them over HTTP. You may be familiar with MCP from uh code assistants

AI code assistants often uh use now MCP as a way to integrate with all sorts of different things. There's like thousands of MCP servers, but we can also use it for like this enterprise integration scenario where we're exposing tools to our LLM and doing that with a network protocol which is MCP. >> You're not going to like what I just discovered. I I rolled it back to I went went to 103 and I deleted the target directory. It wasn't seeing the new class. >> Oh, >> so it wasn't like act. It was just in a weird state. So I'm gonna It worked

>> So 1.1 works. >> No, 1.03 worked when I deleted the target directory. >> Okay, so 1.1 should work. >> Let's see if that works. I don't know. Who knows? Could have just been a some >> compiler got wonky somewhere. >> Yeah. Okay, fine

>> Just clean and and restart. That's, you know, things are not working as expected. >> Just clear the cache essentially. >> Yeah. Got Prancer. Good. >> Okay, found Prancer. When can we pick up for answer? Yay

November 17th. November 20th. Three days. >> And here's the scheduling. Bah. So, this is local tool calling. Haven't done the MCP stuff yet. Right

This is the stuff we were just trying to make work unsuccessfully for a few minutes there. It now works magically with no changes whatsoever except that I deleted my target directory. [sighs] This is why I'm not like worried about uh uh AI taking our jobs. doesn't even know to when to run. >> Well, and and the target directory is the cache for all the the built sources. And you know, you all know the joke that there are um what is it? There are two hard things in computer science. >> Naming caching and off by one. >> Exactly

>> So two hard things naming caching off by one. >> All right. Um >> so that's local. >> Okay. As you said, >> local tool calling. >> Let's take advantage of remote tool calling. So via MCP. If I went over here, created a new service, model context pro protocol and web

Uh, is this here and I haven't opened it yet. >> So, we're going to take that those tools that were running in the same process as our agent, move them out to an MCP server so that then we can have that uh remote architecture for our tools. So, you're going to run it on a different port so we don't get port conflicts. >> Yeah. And this is going to be new for me. Here we go. Paste that in there. I'm gonna try to do the old thing or is it new? Oh, we don't need to

Uh, >> how do we do it? Do we just annotate? >> Just annotate MCP tool. >> Yep. >> Okay. >> So, what Josh has done is taken that tool code out of our agent, moved it into an MCP server project and then just changing some annotations. I think we can actually use the same annotations with our local versus our MCP tools. Um, but there's some additional functionality in MCP that gets exposed when we use the MCP based annotations. So, um, so yeah, we've got our MCP tool. It's got a description

Same thing as before, our MCP arguments. Um, so this is all brand new as of last like Wednesday. So, >> basically, >> yeah. >> So, that should do it. I don't know if it's worked. We can go back over here uh and then call it from the client. Yeah. >> Um, >> yeah

So, now we need to tell our agent side to point to that MCP server. So we need to do some now configuration that says all right here's your MCP server that you're going to connect to. So Josh is going to create the MCP client. So usually our agent is an MCP client and so we have to configure it the transport that we want to use. Oh and a default is still service and event. So I think we'll be okay on that. So >> uh and we could also put this configuration in in a properties file instead of in code just like we started with. There's multiple ways to provide configuration

This is an easy one. Um if you want to do it programmatically we'll have to get the we'll have to do the the uh the properties based one because it's only one one line of property. >> Okay. >> Actually we will in the workshop. In the workshop we'll do uh an MCP client with >> that. So we've got our MCP sync client and then instead of providing default tools, we set the default tool callbacks and provide that there. >> Is that a builder now? >> Okay. List of Oh, this does not make me feel good

>> It's easier in properties. We'll have to >> Yeah, we'll do the properties in the workshop. >> Okay. bar. >> Okay, so you set up your your callbacks and then we add them into our chat client and now >> our agent should still know about our tools. It it goes off through MCP, fetches the tool descriptions from the MCP server, puts those into the prompt that gets sent over to the LLM, and now everything should work just like it did before, assuming we don't need to clear any target directories or caching, but fingers crossed. No. >> No

>> Did it denied? >> Client initialize. >> How do we know if this got exported or not? >> Uh, do you have MCP inspector? >> We can run it. Well, here. Let's just go back to the old thing and we'll do the new thing uh in a second here. At tool at tool param. Okay. Oh yeah, method tool callback provider. >> So there is a new way in Spring 1.1 that will just do all this for you

Um, but we know that this way works. So we're going to go back to the to the Spring 1.0 kind of more manual configuration way where we have to explicitly point it to the the things that are going to be our tools. >> Yep. But and the benefit of this approach is that you can take the code that we had for the local tool calling and just copy and paste it and it's >> there. Okay. Okay. Now take this refresh delete. Okay

So now do you have any dogs answer? >> Okay. >> Yay. It worked. >> Didn't. >> Yeah. No, you can pick it up from Oh, can't schedule an appointment. >> Why? >> Sadness. What's the issue with this uh tool callbacks scheduled MCP? Is that the right host and port? post import

>> Do you do you have to put uh does it automatically put the slash SSC on that? >> Yeah, it's always done. >> Oh, whatever. >> Something new in 1.1. >> We can figure this out in the meantime. >> We're gonna see all this working in your in the workshop and so we'll we'll get it working. But um actually in the break we're going to take a break and you know get some coffee hopefully or if there's coffee out there get some Java. Uh and then when we come back we are diving into handson. So, um, so yeah, that's but that's like your your like run through of everything

We're going to walk through for the first half of the workshop with Spring AI. We're going to do all that stuff and more. Uh, and then and then later today we'll get into agent orchestration with Embable. Um, and there'll be plenty of time for answering questions and diving in further. But anything else to add? >> Have a gonna get this working in the in the break. Let's do Let's do like 20 minutes. What time is it? >> Is that Is that Pacific time? I sure hope so. Yeah

118. Okay. So, let's let's resume in 20 minutes. So, 11:30 we'll start diving into hands-on code. >> Bye. >> Okay. >> You want to get this working? >> Yeah. Let's go sit down

>> Yeah. >> I hate code that doesn't work perfectly the first time. >> We did want to show you we actually got this point actually working. >> Uh the problem was like before caching something with the target directory. We had to just just clean it, restart it. And we're going to investigate why something is different because we've never needed to clean uh do those cleans before. But um but yeah, so this is uh Josh is showing you. Oh, do we want to show the properties one? Well, >> let me show you this first

Here's the manual way, right? You create the client manually and then you inject it uh thly >> the default tool call back. >> Callbacks, which is, you know, the whatever. So that that's wiring our agent side to the remote MCP server and doing the configuration in code. And we can also and this all work. Do you want to show show it working? >> Yeah, I got rid of this. I'm using Spring AI 1.1 over here. >> Three days old. >> Yeah

Creaking the old uh ancient software from the distant distant past. >> Okay. >> So >> Oh, also it's holding on to the socket. So, >> oh, graceful shutdown. >> Immediate. >> Great. Okay, so MCP server is running. Um, oh, and somebody while you're doing that, somebody asked like like what even is MCP? It just is a standard protocol developed by Anthropic that provides a way for a server to expose functionality to an LLM

And the the functionality that it exposes can be one of like six different things. Tools is the primary one. So it's the ability for the LM to know about and call tools. >> Let me just >> Oh, >> I just did. Fantastic. When can I pick up printer? November 20th at 7:34 a.m. Here's the adoption service. Here's the separate one with the IO

Right. So we got two modules running side by side and uh that actually has worked despite the uh >> Great. So we got tool calling over MCP. So MCP different pieces of functionality exposed to an LM. Tools being the primary one. That's the one that we're showing here. There's also something called resources which is like when you have an LLM that wants to, you know, read a file, that sort of thing. Um images, those are MCP resources

There's prompts, which are essentially shortcuts that the MCP server provides to the uh agent side, which are like a shortcut to a larger prompt. Um, and then there's a number of other other pieces of MCP which we won't go into today but or maybe we will later but um, yeah. So that's that's the the primary purpose for MCP. >> Have a server that provides things to an LM. >> And there's a very convenient little utility you can use to inspect MCP services. Just do this >> called the MCP inspector. >> Yeah. >> And that allows us to connect to an MCP server

In this case, we're connecting to our adoption MCP server. You can see the different pieces of functionality. Tools is the only thing that we did here. We get a schedule tool and then the MCP inspector allows us to test make uh calls to that just like the LLM would in an agent. >> Uh it was that we needed to clean our target directory, which is where the the compiled files get stored. And we've never had to do that before. So we're not sure why we had to do that now, but something was being cached incorrectly in that target directory. And we'll investigate why that was

But once we cleaned it, restarted it, everything worked fine. >> Yeah, >> it just like wasn't it wasn't seeing that the tool is an option that it could call. So when we called the LLM, it's like I don't know about any tools. Um, and it's uh we would have to enable some debugging to discover that that the tool list was not being correctly injected into the prompt. >> Right. So now once we did that, it rescanned the classes and it found the new definition for the tool annotations. It's now in the tool. It's in the prompt and everything worked as expected

>> Anyway, >> okay, it all works end to end. Let's dive into the workshop. Well, we have all day to do the workshop, but we are going to stop. We are going to stop for lunch um at noon, but I'm going to get you all started in the hands-on workshop and um yeah, let's do that. So, for the hands-on workshop, uh it's all going to be um you're going to go through the workshop. All the content is online. I'll I'll get you the the information where to go and how to get everything set up here um in a moment. >> Colonel Panic

>> Blue screen. >> No, no blue screens. Josh thinks it's hilarious that I use Windows. Um, >> it actually >> I mostly use Linux, but okay. Uh, here is So, what we're going to do, you all are you all are free to work at your own pace. We'll help you with any problems. All of the content for the workshop is in this this web page which I'll show you in a minute. But this is the most important thing that you need to keep track of is that URL because that is the URL to the workshop

So s12d.comspring-ai-by-st >> spring a by the bay with dashes. So that's your workshop content. Everything that you will do through the rest of the day is going to be in there. And so I want to show you how that looks, what that looks like, and what we're going to be doing. And don't forget that URL because >> we provision like two I'll do it again in a moment. But just to show you what's going to happen when you go to that URL. So I'm going to sign out here. Uh s12d.comspring by the bay

Okay. So when you first go there, you're going to get to this screen the which let me make that bigger. You can choose how you want to sign in. You do need to sign in to the workshop account. The easiest one is the email one-time password. That will email you a token to your email. You'll then be able to put in that token and then you'll be asked uh to agree to the the rules of using these workshop accounts on AWS. Essentially, it's saying like don't mine Bitcoin

Um I get in trouble if you mine Bitcoin on this account. Uh, it says don't store personal information in that account. A few other things. So, >> yep. And I'll show you that URL again here. I'll pull that up for you. Okay. So, once you once you get in, once you've done your your verification uh of that, >> which I now probably need to do again because I logged out

>> I take a photo. You can copy it. >> No problem. Okay. Let me get my two-factor code. It's really hard to see, isn't it? >> So small, right? Like I have a laptop that's almost big. Almost as big as that. >> Is it the Oh, here

Let me make it brighter. >> Sorry about that, man. >> Cheers. Circulate if they ask. Anybody need that URL? >> I'm coming. How are you? >> Good. >> Uh S12D >> Spring AI by the Bay. >> Oh, almost there

>> Okay. So, you'll see that after you get your two factor code, then it's going to have me review that terms and conditions. >> I agree with them. I will join the event. AI by the Bay. >> Once you have joined the event, >> you are going to be taken to a screen with an overview description, but there's at the bottom of this page, >> very important things which you can always get back to if you need to, but this is going to be the VS Code instance that you're going to use for the workshop. So, this is my values. You'll have your own, but I'm going to be able to launch that that URL

I do need to get my IDE password and put that in. >> And then I don't want to save it. This is my VS Code environment hooked to this workshop AWS account. This is where you'll be doing all of the inputting the code, running the terminal, all that kind of stuff is going to happen in there. So that's that's very important. That's on this homepage. You can always get back to it by clicking on this top >> magic password button here >> AWS ID >> but uh send yourself a link that's one >> if you do need to get back to your workshop uh login URL and login information it's it's here once you're logged in >> yeah question >> yes >> s12d.com >> back up for a momentai by the bay with dashes in between >> this one oh and let's make that super big. >> Yeah, it's that is a very small screen

I cannot believe how teeny tiny that screen is. I was expecting there would at least be two of them or maybe two of them back there, two in the front, but there's just one in the whole room and it's just slightly smaller than my 16-inch laptop. [laughter] All right, let's make this bigger. This is so cool. It's raining outside. You know what else would you want to be doing today? I want to write code and hang out with cool people. And we're doing it. >> We're doing it

>> Okay, there's your URL. >> Nice. We have >> big enough. It's as big as I can make it. [laughter] >> I think I walked around. I did. I think most people are connected. >> Okay

Getting into the workshop. >> So, all of the instructions for the whole rest of the day are in that workshop. >> Yeah. Uh and you're you're free >> for the first we have two workshops for the whole rest of this first workshop. >> Even the embable stuff is also in there. But uh if you're super fast and get through everything that you want to get through, pause at the embabel one. Don't go on to embable yet. At probably around I don't know 3:00 today is when we will go through the embabel one

And for that one, I'm going to walk through it with you so that I can explain all the agent orchestration, all the parts of Embable as we go through the Embable one. So, we'll kind of do that one all together. But for the Spring AI one, you've already seen your intro to Spring AI. You've seen all the things that you're going to be going through. And so, that one free to go uh at your own pace through the Spring AI stuff. Just stop at the MBA part. >> Does that make sense? >> That makes sense to me. >> Yeah

Cool. And then and then I'll be here if you run into problems, you have questions, we're here to help you. We will uh at noon go have lunch and then and then come back and continue. But we want to make sure that you have lots of time to get through the springi stuff, get all that working and then three o'clock we'll start the embable part. >> Say what do you three o'clock? That's pretty that's gonna So how long does this thing go? So, we've got essentially two hours for the Spring AI workshop and then um and then we'll we'll probably take a coffee break in the afternoon and then come back for the inbel. >> It's gonna fly just like that. It's already been two hours since we started and we haven't even >> we just got started. >> We're just getting in

>> Yeah, >> it's so uh people can usually get through the Spring AI workshop in about an hour, hour and a half. So, I think we have plenty of time to get through that. Um >> Yep. >> Yeah. Okay, how's everybody doing? You got logged in to your AWS account. You got access to the VS Code environment. You see the instructions. You're able to start walking through putting in some code

>> Once you see the instructions, we we can leave you alone. I just want to make sure everybody's there. And and if you run into things that don't work as expected, let me know because >> this workshop I did test it on Spring Eye 1.1, but there may be some nooks and crannies where things where I miss something. So >> imagine >> so this is all hot off the press. >> You and I you and I were both in different cities last week and now we're here. We are