Context is All You Need: The Art and Science of Information Design | Lena Hall, AI By the Bay25
Hi everyone. Thank you so much for coming to this talk. Um, raise your hand if you've ever had an LLM give you um, confident, beautifully formatted answer that was just wrong. [laughter] Okay. Um, that happens. Um, we have those powerful AI systems that can process huge amounts of information, search knowledge bases, and generate some really great responses. And still, we often get the results that don't meet our expectations. This happens in part not because the models aren't smart enough, but because we need to be much better at information design and the infrastructure around it
In many cases, we don't give models the full context, the right context, or we give it too much context. The models we have are powerful. The issue is that many developers or teams don't get most out of them because they don't realize that context is a design problem with learnable patterns. I'm Lena, um, senior director of developer relations at Alcomai. Uh, I help developers and companies with pragmatic AI solutions and um, I'll be happy to connect with you on LinkedIn as well. If you're curious what Akami does for developers outside of the traditional CDN, there is a quick summary for you. Uh, and today I'll share with you a practical mental model and framework that you can use to get better and more reliable results with AI. And we can do that by paying special attention to designing our context and having a thorough understanding of information architecture
So forewarning this this talk is not about writing better prompts. Uh it's about persistent version semantic infrastructure for correct context. So we will explore two big areas uh where context determines the success of or failure of applied LLMs. The first one is context for coding with AI agents where most of us have direct experience and the second one is context for integrating AI into products. So building domain specific AI agents where the stakes are super high and the failure patterns are similar. So these two areas context for coding agents and context for AI in the product they appear as different problems but they share fundamental patterns and understanding these patterns isn't intuitive at first but it's the skill that creates a massive difference in results. So we can start with the first one context with uh when coding with AI most commonly known as context engineering. There is a huge shift in software development running multiple instances of coding agents working in different branches, spending most of your day just writing specs, thinking about the new ways to optimize it, writing thousands of lines of code a day
And to me, the strangest thing about this new process is that when you try it at first and it works, it feels so unnatural and not intuitive, but you kind of like it and it works. So, you can't help but think like, am I crazy for doing all of that? So, it's it's a truly new skill. And I like the name context engineering for it. So, we can stick with that. So, what makes you good at this? Um, I worked with quite a few engineers with different levels of expertise and here's what I noticed. Beginners and more senior engineers use a completely different approach. Um, here's what the process looks like moving from a beginner to a more advanced AI coding engineer. At the beginning, people uh start with standard prompting in cursor, VS Code, um, their favorite editor
It's a back and forth conversation iteratively working on a problem with AI coding agent. It works for smallish tasks but it really breaks down as the complexity increases. We start seeing issues as we experiment. We discover that endless context isn't useful. Even million token windows uh they only have a fraction of usable capacity. Maybe it's 20%, maybe it's 40% before the quality goes down. And it works best when we stay below a certain threshold. Uh practically speaking, working at full speed, it takes five to 15 minutes of productive work before you need a restart
So you develop a pattern of starting over when things derail. We can call this a strategic restart. And when a session becomes a mess, we stop. We document learnings and we start fresh. This helps, but we need a more structured approach. uh we notice that when we restart there is no persistent plan. We are rebuilding context every single time. This means we need some mechanism for continuity of work and for structured memory
What we've tried, what worked, what didn't work. So we try a slightly improved approach, specdriven development. We write a detailed requirement specifications and rules in markdown files for to-do lists and progress tracking. this is better but you know you have a single source of truth persisting across sessions. Um there is a great GitHub repository to help bootstrap this process from Ryan Carson. Uh you can look look that up. Um show how to generate a product requirements document and also generate detailed uh tasks that emerge from that PRD. But specs uh become monolithic and hard to manage at scale especially when we discover incorrect functionality in the code
So we try to chase down the bugs in code. But we also discover we need a new workflow for correcting the output because our previous workflow doesn't scale. Um think of our traditional process. We write code, it generates compiled artifacts. Those artifacts then execute. When we um had a bug in the code, we wouldn't fix the compiled binaries. We would fix the code and rerun the app. In our new world, we write specifications in English
AI generates the code and that code becomes our application. So now we shouldn't always manually fix the AI code. We should go back to our specification, fix it and rerun it. This works. This works on complex code bases and for solving hard problems. But then we hit another wall. We discovered that reviewing huge AI generated PRs is very hard. Um and if our productivity is so high, we physically cannot review so much code all the time
And when reviewing becomes the bottleneck, it's time to rethink our process. If we think about it, the uh successful code review, it helps us discover any mental disconnects. So we come to a mental alignment and the technical adjustment in code follows. This gives us our next big insight. Plans are more reviewable than code. With the speed of AI, we should move the process of alignment to a higher level. So to the spec or to the plan when we catch errors at the planning level we prevent 10 times more errors that would we would otherwise run into at the implementation level. This leads to two more more advanced patterns
First one is phased development. We don't just write a requirements doc and send our agent on its mission. We start a brainstorming dock. We then have a structured plan. We split the plan into byite-sized tasks and finally we execute and at each stage we review the artifact before giving it to the next stage. So you see how a mistake here at the research stage propagates to a much greater impact to mistakes in the planning and in the implementation. We also use a sub agents to parallelize work. Um this keeps context window of each agent focused and helps the main agent coordinate without polluting its own context
But this too has a failure point. Keeping progress and coordination in markdown, it works under light usage. If we scale it up, markdown quickly becomes a mess. Um information gets stale, dependencies aren't tracked, the work gets duplicated. So how do we actually do this as humans? We coordinate. We delegate and track progress. We use issue trackers with dependency graphs. We version control everything
AI agents sometimes need kind of the same. We are starting to see new approaches for this problem. One is graph-based issue tracking. There is a new tool called beads for example. It allows agents to manage tasks on a graph while tracking dependencies. It is a graph-based um git backed issue tracker database to let coding agents track all your work like it's Jira. Another one is MCP agent mail for multi- aent coordination. Agents can send messages to each other and monitor the inbox for updates from other agents and can coordinate between each other on the progress between tasks, divide them and avoid conflicts like Gmail for agents
So, as you can see, it's a new discipline and without the proper technique, we can't blame AI for failing us. We just need to learn it. And um critical skill is shaping what goes into an agent to output quality code. Context isn't just our codebase. It's this accumulated set of architectural decisions, edge cases, requirement clarifications and also discovered constraints. This understanding gets lost when it lives within one session and uh not in an external persistent structure. Our domain expertise here is computer science and software engineering. And that expertise ma matters a ton, but how we apply it is changing as we're operating in a different level
If we step back and look at what we're actually doing with context engineering, the skills that you need are suspiciously similar to the skills you need to be a good engineering manager. And don't get me wrong, I don't want to manage AIs like people. I don't see AI agents um as employees. I see them as just smarter tools. Um I I want to engineer deterministic systems. But the mental model of an engineering manager specifically around coordination protocols and state management is super similar. So I'm really seeing principal engineers and engineering managers are naturally better at context engineering. Just comes to them more intuitively
So let me show you what I what I mean. A great engineering manager creates a safe space to say I don't know. They don't punish people for admitting gaps. Same with AI. AI needs to be comfortable to stop being a people pleaser and to say that it doesn't have enough information and hand it off to humans when it's supposed to. So we must build architectures that make uncertainty explicit. A great engineering manager makes it crystal clear which types of decisions can be made independently versus when to consult for advice and uh they teach their teams how to fail and learn from it. So if you do not control something structurally it will break
So we need to plan for that and when the failure happens we incorporate learnings into the persistent process. A great engineering manager has a good onboarding strategy. They don't throw new team members into a production on call on day one. They give structured context. They explain why decisions were made. They validate understanding. And that's exactly what we talked about like on boarding the LLM with structured context. Um when something goes wrong, a good engineering manager doesn't say just redo it or fix it
they understand where gaps in the process or documentation are and when they led to that outcome. So if AI fails, we don't just retry the same prompt. We fix the architecture, we update the plan and fix the structural gaps that caused the failure. A great engineering manager is excellent at delegation, splitting the work into manageable tasks. They break down complex problems into pieces that people can pro uh process. That's our phased development and also sub agents with focused context. Great engineering manager gives you just enough context to be productive without overwhelming you. They don't CC you on each email
They provide you what you need for a specific task. So structure over volume. And they also know when you need a reset, when a task was fundamentally misunderstood. So they don't let you continue down the wrong path. They recognize the moment to stop and restart with better understanding. Um, a great engineering manager isn't a single point of failure for communication. They provide us with tools, Slack, Docs, email, issue tracking so the team can coordinate. Like multiple agents need an infrastructure that doesn't bottleneck on you as a single or orchestrator
And when someone in their team is struggling, they don't just do the work for them. They correct the vision. They give you the new mental model. They teach you how to sol they teach you to see how you would potentially solve the problem. When your AI struggles, we don't manually fix every output. We fix the plan. So at the end of the day, engineering manager's main role is to steer the team, unblock them, and make sure they learn from mistakes. So that's context engineering
We as humans are the starting point that gives the LLM context or points to it and then while working we ideally become just the tool that LLM can use for guidance and at the end we validate the result. So you have to know exactly what you want to do. Communicate it well. Be deep enough in your field to know what correct looks like and give the necessary context and infrastructure to your agents to be successful. So we have explored context for AI in coding where we are the domain experts. Now let's look at context for AI in products where kind of the same patterns emerge but implementation looks different. In products or business, our domain isn't software. Usually, it's marketing, it's finance, it's medicine
And validating the work product here isn't as simple as reviewing the code. It really depends on understanding complex relationships between domain specific data. And only domain experts know what's correct. And right now we are integrating, a lot of us are integrating AI into our workflows with the same approaches that barely work for coding. Here's what most teams do when integrating LLMs into products. They use multi-agent architectures, orchestrators, sub aents, or role based agents. These agents call LLM APIs. They do some form of RAG, maybe convert text to SQL
We we influence model selection, temperature, tool access, and then we do some postfacto evals where we can adjust the prompts to improve our app. So let's be concrete about what can go wrong with this approach. The first one is the LLM call itself. It's a nondeterministic step. If we don't control these structural elements, they will break. So we need the infrastructure around it for structured outputs, retries, fallbacks. Excellent talk by Samuel this morning on this. This is I like to also call it like a classic to generals problem for distributed system where we can't be sure that the message was received and acted on correctly
Second, we have rag and the naive rag that we use in products is usually fundamentally flawed. It has no precision. If we ask it to calculate the average money spent on groceries this week or find the most recent transaction amount, semantic similarity fails. It has no temporal awareness. It's lossy and it's really prone to domain mismatch. Then we have tools. Tools are better for deterministic calculations. But in reality, with dozens of tools connected to an agent, there is no guarantee that the agent selects the right one or orchestrates the right path of tool calls
We also have text to SQL. It works for simple translations, but it can break down with nuanced phrasing. For example, if you say, "Find ice cream shops near my zip code with co with with names that remind me of superheroes." that logic doesn't directly translate to SQL. And then we have evals. We discover opportunities for improvement after doing error analysis on the data from real users. So it's more reactive. Um so if we analyze the biggest failure pattern across all production AI systems serving business users is the fact that success is determined by by the validation of our domain experts. Just like in our coding analogy, we have built systems that exclude them from the logic information loop
With coding agents, we are the domain experts butmemes for software. We don't let AI generate the code, deploy it to production, and then evaluate if it worked hopefully. Um, we review the architectural plan first. We validate the approach before execution. What happens now when we apply AI to our products? Our AI doesn't have deterministic code for all scenarios. There are exponentially more edge cases. So what is our alternative to test suites and user acceptance criteria for this wide array of nuances? Um for business t uh tasks having domain expertise during logic formation is critical for correctness not just evaluation afterwards. So how can we fix that? There are a few ways we can go from here
The first step the community took was advanced rag. It uses multiple representations. So instead of one lossy uh vector map, we build like an atlas. We create a keyword index, a semantic vector, a summary vector, and a graph of entities. And it can use an agendic router, uh an LLM that acts as a smart dispatcher. looking at the query and routing it to the right index. It uses smart retrievers models like Goldberg or late interaction type models, not just using a single vector. They keep a vector for every token that makes it more accurate and avoids the compression problem
It's um uh great at finding the precise like needle in the haststack. Um for high stakes business logic, there's still this critical flaw. We still need um to be in the loop, not out of the loop. The business logic is now hidden inside the the blackbox agendic router. The theme can't review it. Uh they can't approve it. They still can't fix it before it gets to the user. That's not an auditable fully reliable system
We also [clears throat] have some new approaches coming out that aim to improve thinking and reasoning like this one from Microsoft research. It's called enogentic organization. Most large language models still think like a single brain. And uh they're proposing a new reasoning protocol called async think that forms a mini civilization of minds delegating, merging, adopting in real time. And it learns this behavior through reinforcement learning. Learning to organize its own thoughts. This is great for reasoning and thinking, but it still doesn't give us reliability. So, how can we get there? We can verify LLM results with deterministic code
We can also use another LLM to verify the results of the first LLM. But we need to go further. We can go further in some cases. We can decouple planning from execution. So there is a new mental model for that. First, an LLM can create a comprehensive plan to accomplish the task. It knows everything about the underlying business entities. Second, and this is a critical step, domain experts can review and refine the plan
And third, only after the plan is approved, the system generates an executable deterministic artifact code that actually runs the logic. So this is a three tier architecture for context. In the tier one, we have the core data layer. Uh this is our semantic foundation. It's not just chunks and text embeddings. It's rich structural metadata. It includes entities and their definitions. The temporal status of data
Is it current? Is it historical or deprecated? Uh what are its relational graphs? What is its authority level? Is it is there some regulatory requirement or is it a brainstorming note? What are the dependencies of the data? So this is how we can close the gap between retrieval and true structural understanding. The tier two is the orchestration layer or the theme collaboration zone. This is where your domain experts operate where the LLM generates the plan and we can provide a way for anme to see it to see the reasoning and refine it. They can say no that's wrong. The definition of a patient record isn't capturing the new update from last month. Update the entity graph to know that patient record is now inclusive of the new recordkeeping system that we added. So you must check against like patient stats uh because we cross check for cancellations and so on. They iterate on the logic before any user sees it and then when they're happy they approve it that generates a deterministic reusable and versioned artifact that gets added to the product as an approved functionality
This is how we the system can learn capture institutional knowledge and scale the experts decisions. The layer three is just a user layer just how the results are surfaced to the user once the architecture underneath is solid. This part is pretty straightforward. Uh so this decoupled architecture changes the role of experts. We're not asking them to be the testers in a traditional sense. We're asking them to onboard the AI properly with the right context. Validate that it works and approve approve the work making it shippable. So you give it structure context, explain why decisions are made and validate the understanding before we ship to production
You can still choose the right architecture for the right job. So stick to the native rag for conceptual tasks like chat with your docs or internal uh knowledge based Q&A. This is uh low cost and high speed. Um you can use advanced agentic rag for exploratory complex ambiguous tasks for research discovery and um like like AI as a co-pilot when it's implicit reasoning is a feature but for a high stakes deterministic and auditable task finance legal medicine core um product features you should really try to decouple planning versus execution and getme validation as a non-negot negotiable before exposing the functionality to the users. So if we bring it all together, we've looked at two different worlds like context of AI for a coding agent and the context of AI in a business product. The path to reli reliability is similar in both scenarios. We started with a stateless implicit model. we would stuff the uh context into a prompt and get a non-deterministic blackbox result
This is low leverage. And in both scenarios, the only reliable solution was to evolve to a more stateful and explicit architecture. Um so giving us this more unified model. Um architecting context for AI has is a design problem with two core components. The first one is we must architect the state. This is our persistent context layer. In our coding example, this was the evolution from a simple chat history to structured specs, markdown files, and finally to graph-based issue trackers like beads. We externalized the context into persistent version structure
In our product example, this was our t tier [clears throat] one core data layer. So we stopped using just text chunks with embeddings and built a persistent graph of rich structural metadata and multiple representations with a definitive temporal relational descriptive enough to capture entities their relationships in addition to embeddings and in both cases you externalize what the AI knows into that versioned auditable system. The second we must architect our logic. This is our explicit orchestration layer in the coding example. This was our phased development. We stopped reviewing massive unrefable PRs and started reviewing the plan before the code was ever written. And in the product example, this was our theme orchestration layer where we decoupled planning from execution kind of forcing the AI to produce a reviewable plan that ourmemes validate before it becomes deterministic artifact. So in both cases we externalized what the AI does into this reviewable and more auditable process
So this ultimately changes our job to more being context engineers and context architects. Um so the new role is to be better at architecting the system that AI operates within. Um this actually not an easy role as it combines the element of data modeling, data governance, system design, distributed systems, AI and LLM engineering and structured communication. So we need to define clear boundaries. We need to onboard our AI with structured context. We need to be good at delegation, coordination, state management. And we must shift our focus from reviewing the output to reviewing the plan and have the right expertise to be able to judge the tradeoffs and know what the correct looks like as anme. So this should give you the mental model for thinking about context when it comes to LLM with coding agents and LLM integrated into your products
Um, thank you. Let's connect on LinkedIn and I'll be around after the talk today and tomorrow. Would love to chat with you. [applause]