From Experiment to Enterprise: How Block Operationalized MCP at Scale | Angie Jones, AI By the Bay25
So, I'm Angie. I lead uh AI tools and enablement at Block. And today, I'm going to share how engineering scaled AI across our entire 12,000 person company in only two months. It's ongoing, but we got really close in two months. So like most other enterprise companies, it's become abundantly clear um that we need to shift our mindset to become AI first, right? But we're a 16-year-old publicly traded like fintech company. Uh Block is the company that makes Square and Cash App uh that company. Um so we've got legacy systems, we've got longstanding habits, we have enough process to make innovation really hard or you know at least take a long time to accomplish. And so add to that the layers of regulation and compliance and risk management um that comes with handling other people's money and you can imagine how hard it is to move fast let alone try to rewire an entire company to think differently
So of course engineering was a great place to start. We'd been getting pretty good use out of uh our AI coding assistants like GitHub co-pilot. Uh and one of our principal machine learning engineers actually lives in San Francisco. His name is Bradley Axen. And Bradley set out to create this tool that would go beyond the IDE and could actually help automate some complex development tasks. So Brad started tinkering on a developer agent. I don't even know if we were calling them agents back then. Um, but something that could do things right, not just generate code snippets
And so this was before the LLMs actually supported function calling. So, as you can imagine, this was a really difficult task. But once OpenAI introduced these capabilities, everything changed. So Brad and a small team of volunteers around the company, volunteers, yes, um began experimenting with using tool calls to automate parts of the development workflow. And while this definitely made things more feasible, we quickly realized that without a standard, trying to create uh toolkits and integrations for the various APIs that we wanted to integrate with would be almost impossible to scale. So we started working on a protocol um and we shared it with our friends at Anthropic because we're no protocol people. Um and Anthropic let us know that they were working on something similar MCP model context protocol. So uh we read it
This is before it launched. We loved it. We aligned with them immediately and actually became a launch partner. So we uh developed our agent as an extensible MCP client that we call goose. Anyone heard of goose? Okay, great. So initially we designed goose for our internal engineers right and uh we added a few built-in MCP servers like git jet brains um memory computer controller and 75% of the engineers who use goose were saving about 8 to 10 hours a week just by offloading their busy work to goose within a month that adoption doubled without a formal rollout campaign people just saw other engineers using Goose and they wanted to try it as well. And as more of these use cases emerged, the weekly engagement grew by like 50%. So with all of these efficiency gains, we decided, huh, we should probably open source Goose
So the public release uh was really wellreceived. Uh this was back in January, so like a decade ago um in AI terms. Um, so in addition to coverage in like major tech publications, Google's also trended on Twitter. It was number one trending on GitHub. And with all of the buzz, our internal non-engineering team started asking, "Wait, hold on. What about us? Can we use Goose, too?" Now, seeing the engineering efficiencies, of course, we wanted all of our employees to move at this pace. So we realized that Goose couldn't just be a developer agent. It probably needed to be like a general purpose agent that anyone at the company could use regardless of their technical background
So we started assessing this and trying to figure out like what do we need to do in order to get this in their hands. But they couldn't wait. They picked it up anyway. They just started using it as is. And so all of a sudden now we had sales, legal, marketing, finance, design, uh customer service, our executive assistants, like everyone in the company started using Goose. And naturally, they ran into a few challenges. So the first challenge was with missing MCP servers. Like I said, we'd built developer centric MCPs, right? Um because that was our target audience
And now all of a sudden people wanted MCP servers for tools like Snowflake and Jira and Google Workspace to interact with their email and their calendar um and their Google Drive. And so they were out there in the wild wild west um trying to find MCP servers for their use cases. And once they would maybe find a server that met their needs, they run into the challenge of installing them. Right? So, what the heck is like a MPX command, right? Um, how do I run that? And even if they figured that part out, they'd then be prompted for API keys, which was a foreign concept for a lot of them. So, it was clear that we needed to pause, rethink the experience if we wanted everyone to benefit from this. So, how do you make an AI agent work for 12,000 12,000 employees across every function? Here's how we did it. So, before we even rolled Goose out, there were a couple of things that we just kind of got right by accident. Um, and these were choices that we made for developers that ended up being perfect for everyone else, too
So, the alpha version of Goose um before we rebuilt it as the MCP client was actually CLI only. So, uh Goose was like the first kind of like CLI agent type of thing. Um, but when we rebuilt it as the MCP client, we did so as both the CLI and then also a desktop app. So that was originally just for developer preference, but it ended up completely unlocking access for our non-engineers. Had we not done this, we would have excluded most of the company without even meaning to. Goose is also LLM agnostic, so it isn't tied to just one model. And that flexibility turned out to be critical because while models like claude are perfect for developers, right? Um other models like GPT, they're better for general purpose task. So internally we use data bricks as our model host and um this allows employees to choose from models from OpenAI, Anthropic, Google, Meta, um all of the the frontier models
And so giving people the power to choose this made the adoption even easier across the the various disciplines. And what we really love about this approach is that it balances that optionality with control, right? So we are still in control. We only expose the models that perform well on goose benchmarks. Um and when a new model hits that bar, we can just easily flip a switch and release it to the entire org. We also um pre-seelelect a recommended model. So that's the one by default and that's going to be the one that's hitting that 100 on the goose benchmark. Uh nowadays that's claude uh four or five. Um but let's say like there's an outage or something with one of the models which that happens sometimes
Uh this kind of shuts everybody down. So we could very easily just kind of switch that in data bricks to make the default model something else and and most times people don't even recognize that there there was an outage. So those are the things that we got right. Of course there are plenty of things that did not um scale so gracefully. For example, we thought about installing uh we thought like installing Goose itself was dead simple, right? It's just like a zip file. You download it, you unzip it. It's an application. We all use Max
Let's drag it to the applications folder, right? But for employees who are not techsavvy, like even that caused a bit of friction. Some people never unzipped the file. Others, they didn't move it to the right directory. Almost no one kept it updated, which meant that they were running into bugs that we had already fixed. Right? So, we changed course here. We added goose to our internal manage software center and now it's automatically installed on every employees computer and also automatically updated. So if you have a block laptop, you just automatically have the latest version of goops. And then the next hurdle uh was getting everyone the MCP servers that they needed to connect to their daily applications
So I often see discourse on Twitter about uh from developers about like I don't get MCP I don't know I don't know I don't know why I need it and I don't think that they're thinking broadly enough like this is supposed to be an implementation detail for the masses not for us right so um thinking a little bit broader you'll see that like other people this unlocks like use cases for them right um So with this, we didn't want I don't know if you all remember like the I guess it's still there now, but the very early days of the MCP service like you would go to someone's GitHub uh for that MCP and there was like the the JSON or like the you know the command and you're like trying to figure out how do I piece this together to put it in goose or whatever. Um and so they didn't understand that like running MPX commands and stuff like that. That's of course a security risk especially for folks outside of engineering who can't like understand what this code is doing. Um so it just wasn't safe to ask them to pull these MCP servers off the internet and install it locally. So we spun up this small task team to build the internal servers that people needed. Um and they moved really fast. They learned a lot and they even developed a playbook which I um have blogged about on the engineering uh the block engineering blog. But uh the timing of all of this was perfect because it aligned with our internal hack week
So this made space for other engineers who had been hearing about MCP. They wanted to try it out and so they jumped in during this week as well. By the end of that week, we had um over 60 6 MCP servers for our employees to use and now we're up to over 100. So, we basically have one for any application that people are using on the day-to-day. All of them are now bundled uh with internal goose by default. So, no one has to install anything. They toggle on what they need. So we don't really use external MCP servers much now
Maybe I I mean maybe five if that for like the really big complex ones because usually even if one exists we just recreate it internally because that gives us more control um over like off and versioning and stuff like that. So, one of my favorite moments from like that hack week time was when someone uh they came like in the Slack channel and they were like, "Hey, I vibe coded an MCP that I need. Where do I put this?" And we go, "Great. Here's a link to the GitHub where we're keeping them all." Um, and she replied like, "Oh, I don't have a GitHub. I'm not a I'm not an engineer." Right. And I just thought that was amazing how uh this basically just makes it accessible for for everyone to contribute. So now even though we have more than 100 MCP service for our employees, every day someone is kind of asking for something new, right? Um and sometimes they will resort to bad habits and go to the internet and look for one. And as you can imagine, this really freaks our infosc folks out
Um, so to keep things secure, our internal goose now includes an allow list. So if someone tries to install a server that isn't like pre-approved and on that list and goose will block that install. So that's an a a simple but effective safeguard that keeps our employees secure. The next hurdle that we desperately had to overcome was authentication. So for non-engineers being prompted for API keys and asked about tokens and scopes and all of this stuff, this was like way too overwhelming. So they were getting stuck on this OOTH approval flow which of course caused frustration, a lot of help requests. So today, all of our internal MCP servers are set up to use OOTH with short-lived credentials and they're integrated directly with our identity provider. So there's no need for anyone to manage API keys or copy tokens or anything out of a dashboard
When someone toggles on a server that requires authentication, they'll be prompted with our SSO flow in the browser, right? And so they just sign in with their normal block credentials. they approve that access and that's it. So, it's quick, it's secure, but most importantly, it's familiar to them, right? This setup gives all of the security and also the traceability that we need without overwhelming the employee. And honestly, this has been really nice for us engineers as well. So, solving the technical challenges was one thing, but helping people actually use Goose was another. So we set up two dedicated Slack channels to support Goose internally. One is a place where people can ask questions. Um they can report bugs, they can request features or just kind of get unstuck in real time
So everyone just kind of pitches out here um in this channel. Those who work on Goose as well as the power users in the company. We don't have like a dedicated customer support or anything to this. So it's just kind of like peer-to-peer help. Um, but I cannot overemphasize just how critical this has been to adoption, especially for the non-technical users because now they have a space where they can kind of quickly get unlocked. The other Slack channel is for inspiration also really big. So this is a space where employees they drop their success stories of like cool workflows or clever prompts or like just wow things that the agent was able to pull off. And this channel quickly became one of the best organic drivers of adoption
Um, people would scroll through this, they'd see what others were doing, they immediately want to try it out themselves. When you see the sales guy who has like zero technical background, say, "Hey, I met with uh a client who is not going to renew unless we have this feature." on the way home, I pulled out Goose, which is local. Use the local model and like vibe coded the the prototype for his customer, right? And so you see that kind of thing in a Slack channel. You're like, "Oh I gotta I got to get on my game, you know?" So that that played a a big role in building culture around Goose and just uh adoption of AI in general. Now while Slack gave people this place to connect, we knew we needed some education as well. So we set up an internal uh developer relations effort where every week we host brown bag sessions, workshops, office hours focused on AI enablement and these are open to everyone across the company and for those who are brand new to AI, we cover like the basics, right? So we can't leave people out. We have things like good prompting techniques and stuff like that. We also offer sessions on what you can do with these various MCP servers that are bundled with boost
And of course, we cover like new features as well. And then our Devro team also offers like tailored workshops to specific teams helping them figure out what can I automate, how can I use AI in my workflow. So it's not a standard workshop. It's customized to every team. they go to maybe an offsite or something and uh help that team understand how they might be able to use the app. Um and then finally we office we offer office hours um where people can come and they can get help uh troubleshoot issues. So basically yeah education and enablement this has been a huge unlock. Don't skip this step
Um, it's pretty exciting to see how the non- techies are using agents and I gotta say y'all, they are more creative than the engineers. Um, so here's a few impactful ways we've seen employees use MCP. So we use Snowflake as our cloud data provider. So by connecting Goose to the Snowflake MCP, employees can query that data using natural language instead of like SQL. So one employee used uh this MCP to detect fraud by just spotting suspicious patterns in the transaction history. So this would have been something maybe they might not have caught at all trying to do it uh manually or if they did it would have taken like quite a bit of time. Another employee they needed to analyze like a data set of over 81,000 sales leads. They went to this conference, they got all of these leads and they needed to figure out how to distribute them across different sales programs
So using goose with the datab bricks MCP, she accessed the lead data uh refined the logic just using like plain language, you know, and then landed on this clear recommendation that was backed by the data. So this would have taken her um days of manual analysis to do and she was able to do it in just one hour. So in her words, "This was so helpful I could cry." Um our pager duty MCP is also a fan favorite. So when you're pager usually like in firefighting mode, right? And you're focused on fixing that one issue at the at that one moment. Shout out to our friends at CloudFare yesterday. Um, but it's rare to have the time to or even the headsp space to kind of like zoom out and look at this like at from a a big picture view to detect like what patterns uh may be leading to these issues occurring in the first place. So employees use this MCP to analyze that incident data and they're able to um uncover like the top recurring issues that are triggering alerts and with that bird's eyee view they're able to take the actions that lead to eliminating uh the root cause and making our systems more reliable. So we're seeing like all sorts of powerful use cases across design and customer service, project management and engineering as well
Now how am I doing on time? I got 10 minutes. Okay. Um seeing what people are building with goose was inspiring but also it showed us like where the experience needed to evolve. So we started adding features to support the way people actually wanted to work. um employees were automating tasks that weren't just useful for them, but these are things like I I could share this workflow with everybody on my team. This would help them as well, right? Um but we didn't have that capability in Goose. So, we built a recipe feature that packages a custom uh system prompt as well as like a user prompt and then any of the required MCPs that were needed for that conversation. And you can turn this into like a mini agent that you can share with other people and they just invoke it on a on a click like just click the link and boom, now you have this agent that can do that thing
You can save that in goose. You can schedule it. It's really cool feature that I use all the time. Um non-engineers use goose to vibe code prototypes like that sales guy right um and they also make like internal productivity tools and things like that um but these were stuck on their laptops they don't know how to deploy this you know so they wanted to share these applications with other people we created an MCP that deploys static websites um that gives people that direct path from idea to the deployed app that they then could share with other people also game changer. And then being able to um work with everyday tools is great, but there were frequent questions about how to use Goose to like navigate internal processes like procurement and compliance onboarding, right? Things that typically require tribal knowledge or they're like stuck in like some Slack channel somewhere. So, we developed the MCP server for Glean. I don't know if you all have heard of Gleam, but it's absolutely amazing. Um, this one specializes in like search searching enterprise data
So, anytime I need to like find somebody in our company who does this, you know, or who would be a good person to talk to for that, I ask lean or how does this process work or how does how does this work? You know, it's super helpful in that way. In fact, um it helped me out like really recently where I needed to I wanted to sponsor um a hackathon, but the hackathon was h happening that weekend. So, I needed to like expedite the process. I don't know how to do all of that like with you know with the procurement and everything. So, I asked Goose like listen here's my situation. How do I like get this over the line? And it felt like Goose was whispering like come here. All right. You gonna tap this guy over here
Okay. And then you're going to open a jar ticket. These are the words you need to use so that they know to you want me to just do it for Yes, Goose do it. And so Goose open that on my behalf using a Jerry MCP. Um so we realized that while AI was spreading fast, engineering still had a ceiling. And I see this not just in block, I see this across the industry. Like I travel, I do conference talks. I'm like probably on the road about 80% of the time
And so I talk with a lot of engineers uh in the industry and I know that a lot of them are using AI for code generation but at block we wanted to go beyond just the IDE right how do we make this a part of our engineering fabric how do we bake this into how we build software so for this I started an AI champions program for engineering and I handpicked about 50 IC engineers across the company who could pioneer what AI native engineering looks like inside their repos. So this small but diverse group spans front end, backend, mobile, even hardware and they operate in uh repos of every size. small midsize those big nasty mono repos with the legacy code that we say oh AI doesn't work there all of that right um so while the champions are very different they share one trait they do not give up when AI doesn't behave on the first try right they're willing to do that hard work of shaping the context of crafting you know agent rules for their repos of maybe even building uh MCPs specific to their repos setting guard rails so it's not like catastrophic um and they do all of this so that uh everyone can benefit from it right so every champion works on their specific code base to identify where AI can remove friction whether that's automating tests uh improving documentation or integrating agents right into CI/CD right um and they all have three jobs make your repo AI native upskill your team by sharing what you know so it's not just you becoming a great engineer but help everybody else as well and then also collaborate and share that across the company so with other champions and they learn from each other this has been absolutely beautiful um so we measure this success by making sure that they're getting more done with less manual effort they are using AI to help maintain quality and that it's freeing us up to solve harder problems and like ship cooler things. So that's the bar, not activity impact. So to make this journey tangible, we gified it a bit with like a old school RPG. Um we call this repo quest. And so every repo can collect companions like a PR assistant, a CI agent, uh a quality guardian, things like that. And the idea is to help your repo reach this uh AI maturity level
So while we're like still in the early part of this journey, um we're already seeing a shift in our engineering culture. Let me give you an example of what like AI native engineering looks like in real life. So last week in a random Slack channel, a developer shared some odd behavior they were seeing with one of the products. They wanted another opinion. Um is this a bug? Is this intentional behavior? So, right there in Slack, developer 2 asked Goose like, "Hey, Goose, um, can you explore this behavior? Confirm if it's a bug or not." Goose identified that problem, open the issue, suggested three solutions to comp like complete with code snippets. So, right there in Slack, these two engineers look, third engineer comes in, they look over the the options, they go, "Oh, yeah, I like number one." And dev two says, "Yeah, yeah, me too." And so they decide, okay, yeah, let's go with option number one. Goose implemented and then goose opens a PR. So when we talk about developers and MCP, again, y'all are thinking inside your IDE
This took a Slack in a GitHub MCP server and then now we're able to like automate doing work right there in Slack. Um, we've also added agents to our sprints as team members. So we have connected goose to our project management systems using linear and um Jira MCPs. So if you assigned and issued a goose, it reads the ticket, it goes to the associated repo, makes a plan, writes the code, opens the PR, watches CI run, so if anything fails or there's like errors in the log, it iterates on that, makes more commits until the build is green, right? And this has been such a lift. One team um literally ran out of work mid-sprint, right? twice they had to pull in more tickets twice in a three-week sprint. So that's a normal workday at block when you combine these clear systems um AI ready repos. I don't want this to sound like magic. These champions did a lot of work to make their repos AI ready meaning context and rules and guard rails then you see like really amazing things
So in closing, um AI is helping thousands of employees across block work smarter, work faster, work more autonomously. And while a lot of the buzz that we're seeing in the industry is from developers, we've learned that we can scale AI to more than 15 different job functions. So we moved past the demos and the whatifs and Block is putting AI into practice and shaping what this can look like for enterprise companies. Thank you so much.