Name: The Balancing Act: Maximizing AI Governance, Minimizing Spends
Uploaded: 2026-08-20T06:16:21.724Z
Duration: 54 min 52 s
Description: The Balancing Act: Maximizing AI Governance, Minimizing Spends

Transcript for "The Balancing Act: Maximizing AI Governance, Minimizing Spends": Hi, everyone, and welcome to today's webinar. My name is Vinay, and I will be helping support today's live session. We are so excited to have you all join us. But before we begin, a little bit of housekeeping. On the top right corner of your screen, you will see the live chat option that you can use to interact with your fellow attendees. In the docs section that is located next to the live chat, we have attached a couple of links and documents for you in in case you wish to learn more about this session today. On the extreme right, you will also find the q and a box. We encourage you to leave all your questions in that box as we progress through this session. We'll make sure to address all the questions towards the end of this presentation. And a quick reminder, this webinar is being recorded, and we will share the recording with you within forty eight hours. Now please join me in welcoming today's speaker, Sam Bryce Johnson, principal solutions engineer at Cloudflare. Good day, everyone. Thank you, Vinay. Everyone, so thank you for for joining our session. Firstly, I I guess I just wanted to apologize. We we did have a a late reschedule at this session. Originally, it was meant to be on last week. I did a week and a half of of work travel and came back a little bit unwell. I do still have a little bit of a scratchy throat, so apologies if I I cough a little bit, but I do have water here with me. So, hopefully, we we should be okay. Now today's topic, is all around AI governance and and spend control. It's something that, to be honest, regardless of where we see customers are are at at their in their AI journey, it's something that everyone has some sort of interest in one way or another. I think we've we've all seen or many of us have seen the news. You see cases, Uber is a good example where they burn through something like, a year's worth of tokens, you know, the the imaginary costing mechanism that we have that is a token. They burn through a year's worth of tokens in four months. And it's a it's a real concern that, I guess, where where where finance is is, raising concerns with security who don't really know who's spending what, and attribution of it all becomes quite difficult. The CloudFlare solution to this is to be frank, it's it's very simple and elegant. And because of that, I could probably talk just about that in and be done in ten or fifteen minutes. So for that reason, we're actually going to to talk, a little bit a little bit broader than than just cost control. I wanted to touch on on, of course, the AI gateway, and and that's that's the mechanism that Cloudflare uses to to do cost attribution and cost control to requests. But we also had agents week. At CloudFlare, we do thematic, weeks where we do, a large number of product updates or product releases within the the one single week. I think it was maybe two weeks ago, two or three weeks ago where we had agents week. During agents week, it was five days of solid new releases and and updates. Because there were so many of them, I'm simply not gonna try to cover any all of them at all. I've just pulled out a couple that are a little bit more security focused, and address some of the concerns that we see, around from customers when it comes to broader AI adoption. And then lastly, I wanted to talk a little bit about Cloudflare OS. And Cloudflare OS is something that we have just open sourced. It's it's essentially what we're using internally, for for much of our nontechnical population lives in in Cloudflare OS, and that that's really their agentic playground of how they they perform their roles and and, use AI to make things easier for themselves. Now let's continue through. When it comes to cost control and cost visibility, what the pattern that we see constantly is a shared API key for the entire team. Now if you if you work somewhere where everybody like, maybe you're a small team, and everyone in the team has their own Claude accounts, like, maybe there's a a Claude team subscription and you will log in and and you do your work that way. I'm not necessarily talking, about you in and we apologize for that. Where this is more relevant is in environments where you were using an API key. And that API key will be a one shared key across everyone, and it gives zero visibility into who really, who who is doing what. And when you end up with that big invoice so, again, think about the the Uber example where I I think I read somewhere that they originally envisioned a budget amongst 5,000 engineers. They envisioned envisioned a budget of a few $100 per engineer per month. What they where they ended up was spending the whole year's budget within four months. They had to put a hard cap of $1,500 per engineer per month on it. I don't actually obviously, I don't know what they what they ended up spending within four months. I didn't find an article that said they spent x number of millions of dollars, but the math on the hard budget worked out to set works out $7,500,000 per month, which is obviously a huge amount of money. Now with when you're talking about companies that have, you know, for for want of a better term, they have all the resources in the world to to throw at these problems. They are still struggling with with, with this this issue. Where it gets hard with the shared API key and, you know, you can see the the examples up on the screen. And the more I thought about this slide, the the more I think that the the intern example is probably not fair on younger and early career workers because it's probably more likely to be someone my age who does something a little bit stupid and says, okay. Fable summarize this email for me. You you'll do it the best, so I'll just use you to do it. It's not exactly the most efficient use of, of resources. Now that then leads into basically, we end up with we have one budget with no no real visibility and the human the human nature, I guess, of of the way that we are is to just use the best tool for the job or the biggest tool for for want of a better term, for every job. So whether that's a a finance analyst who is summarizing a report, whether it's a HR manager, drafting a job description, whether it's someone like me who just needs a Python a basic Python script written, we just reach for Opus or Fable or or whatever it is, and use that to to perform the tool. I should say as well, I keep referring to the anthropic models, purely because that's simply what I I use, and I'm most familiar with their naming convention. No no reason other than that. I use use it in my personal life and and professionally as well. But what follow but with all that, now what follows on with people reaching for the the biggest, the baddest tool to do the the simplest of tasks? I I dug up a report that one of our blog post references, and that's the you can see on the screen there the Stanford HAI AI index. Now I I was I was looking through the report, and I I came to came to the models and and they're all stack ranked and and whatnot. And it it it's multiple it was multiple versions. Oh, so I thought, goodness. How how old is this report? It's four months old. So the the report references Opus four dot six, and I forgot that that's almost it feels like years ago. But, really, it's only four months ago. So just just to explain, mentioning Gemini three dot three Flash there as well. But I guess the theme and the point that I wanted to get across here for general tasks, not not for deep reasoning, for for general easy tasks. So summarize this email, write me a a a, a Python script, that sort of stuff. For general tasks, amongst the top 15 models that were ranked in the report, there was only four percentage points that separated number one and number 15 in terms of capability. A lightweight model, you can see on the screen there, Gemini, three Flash, that was ranked number five. So we don't need to be reaching for the biggest model all the time, probably almost never for for most of us, but due to human nature, it's simply what we do. Now, firstly, I'm I'm not trying to say that Gemini Flash is is as good as Opus. That's that's, of course, that's not the claim. It's for simple tasks that so many of us use AI for in our day to day lives. The the lightweight models are absolutely performing the functionality. Just to continue through. Now firstly, this this is not a screenshot. What I've done here is I've taken the anthropic again because it's it's just what I'm familiar with. I've taken the anthropic API and pulled out I've queried and looked at the fields and and pulled out what I can and said I and, just put a graphical representation as easy on the eyes. Put up what it is that a typical customer is able to query via the usage report from Anthropic. Okay? The first thing that jumps out here, just looking at that report because, obviously, you know, you're human. Your eyes have gone straight to the orange text, and, really, that that's what you've looked at. These are per minute buckets. These these are not calls. These are per minute buckets. There is nothing in this report that tells you what the calls were, who they were made by, what the cost of the individual calls were as well. In the example that I'm using, you can see in the none under case read, the none highlighted indicating that, okay. This this this is caching responses. Like, that's that's great. That's exactly what we wanted to do. Yeah. But it can also indicate problems as well, and, we we simply don't have that information. Think about the agent that's stuck in a loop. Right? If the agent's stuck in a loop and is asking the same question, that is going to end up with a lot of, a lot of case responses. That leads into rogue behavior is when because when we think about rogue behavior in cybersecurity terms, we think about things that people shouldn't people or agents shouldn't be doing. But rogue behavior, in this case, is not necessarily actions that agents or humans shouldn't be doing. It's just more of it. And this is where controlling the cost or even just having the visibility of the cost, becomes next to next to impossible. Now that brings us to two completely different problems. So we have governance on one side, and then we have, spend on the other. So they're they're not the same problem. They they are related in the sense that they can be solved somewhat, by the same mechanism. So they're two different problems, essentially, two sides of the one coin in in many ways, if if you will. So on one side, we have security. Security wants to know who made the request, whether they should have, what's the request trying to do. It's an access problem. Right? Finance wants to know what does this cost, who can I charge it to, like, what what department, what team can we attribute this cost? And that that, of course, is a budgeting problem. Then you you have the whole separate thing of, okay. Well, Jennifer has spent $500. Well, did we actually get value out of that? And that's that's a whole separate thing that I I'm not sure that, the world is answering correctly yet, but we continue to work on that. Anyway, the the two problems that we're looking at here, so who made the request and and what did the request cost and where can I charge it to, it's solved by the same foundation, and that is identity? So we can't enforce an access policy against an anonymous request, and we can't attribute attribute spend, to a budget if we don't know who was making that request in in the first place. Now from the Cloudflare view of the world, this is where AI gateway comes in effectively. Excuse me. Now AI gateway, very, very simple simple to implement at, at the highest level. It's really just one line of code. You take the the base URL of your model provider, and you swap it in you swap in the AI gateway endpoint. And from there, every call goes goes through the AI gateway. So that's how we get traffic to the AI gateway. At the the back end, we would put that API key that you're you're sharing amongst your however many users. That API key then goes into the AI gateway, through any one of the number of providers that we integrate with out of the box. And from there, you have your your frontier model, integration, essentially. I would suggest if you're going to do that, you then have a a backup provider as well. And for a backup provider, it could be, someone or a company who specializes in OSS models. So that's the the open source models. Cloudflare does that. Cloudflare is not the only OSS model provider that we integrate with out of the box. There are others like OpenRouter, Grok, and and so on and so on. That will give you, I guess, the full back control, essentially. But once once, the requests are going via AI gateway, it's that one control plane for observability, for security, for governance across all requests and all responses. Now before I get into the identity layer, the table stakes or what what we see as table stakes for an AI gateway. So analytics and login. Okay? So every request, every response, do we wanna do we want to layer in DLP? Do we want to layer in guardrails? And, again, on the DLP and guardrails on the response, on on the request and on the response. Costing, of course, token counting, like, these these magical things that are tokens that every provider has a different token count cost and every model has a different token consumption, but we're measuring token count as well. Caching. If if two users or two agents are asking similar requests rather than having those requests going all the way back to the LLM and incurring a cost, let's cache them and served up the case response as well. Rate limiting, I tend to think of the the best example with rate limiting, aside from a rogue AI agent is maybe your business has free users and paid users, and you wanted to rate limit the the free users, for example. And lastly, the last table stakes point here is, retries around model fallback. So if if one provider goes down, how do we switch over automatically to to our backup provider? Now the probably probably the most interesting, or an immediately useful thing for for teams, and and this is a this is a new feature, is around spend spend limits. Okay? So this is where we attribute and you can see from the the screenshot in the policy. This is where we attribute actual dollar costs, not not token questions, like, real real money, is is scoped via model, by provider, by person, or you can use custom attributes as well. So I should say, everything that I'm talking about is done very easily natively. There are things that we that we don't do natively today, but if you wanted to do them with a worker, essentially, the sky's the limit, basically. But I'll I'll keep it to things that we do natively. So, yeah, at the at the moment, identity with users is is where we at. Now if we do hit the budget, so let's say Sam can access anthropic models and will give Sam a $5 budget for every hour, for example. Sam sits there and basically outsources his job to Claude, so he gets that $5 limit within ten minutes every hour. Rather than cutting off Sam's access altogether, let's reroute Sam's requests to, to an OSS model, rather than the the very expensive, very expensive Frontier model. That means we're not cutting users off. We're still enabling users to to leverage AI, but they're just simply doing it in a more cost effective, manner. Now that then brings in the identity aware AI gateway and and how does this work? And now this is in this is in open beta, so it's still very new. But, essentially, what we do is we front the AI gateway endpoint. We front that with a Cloudflare access application. If you're not familiar with Cloudflare access, that's a it's essentially our z t n a solution. So we book Cloudflare access in the front. Every request authenticates to Cloudflare access. Cloudflare access integrates with what whatever I IDP it is using, whether it's, Entra, whether it's Okta, or any any SAML IDP. We authenticate there, and then we take out the the Cloudflare user ID field, and that passes through. That enables us to do the the cost attribution and the cost control, to an identity. Okay? Now let me keep going. Now that we're doing identity, we're also able to perform, user insights or or user analytics. Now with user analytics, we we look at things within the ninety fifth percentile. We we determine, well, what what is normal for these particular users? And when we step outside of that, how do we then flag that, flag that to the team? Now if you look closely at the screenshot and I apologize. It's a little bit blurry, but I I pulled it from a blog post. When we look at the screenshot, you can see, well, in this case, identity coverage, and I'm talking about the green box at the bottom right, identity coverage is sitting at 27%, purely because this is a this is a work work in progress. This is, this this customer who gave us this information, they've turned it on, but it's a it's a gradual gradual rollout. This customer is still using a shared key for many of their use cases. You can see in the bottom part of the screenshot, if you if if your eyesight's a little bit better than mine, maybe don't have to squint. But you can see in the bottom part, the unidentified, log line is the is at the top. So unidentified is, attributable to $1,600,000 worth of spend. So that's the majority of spend is unidentified, and that's the the shared API, shared API, issue. But then you can see below that we've got actual users, that have been picked up and and attributed here. Again, think about things like the the rogue agents or or or whatnot. This is what the user user insights is looking to looking to bubble up. Now I've alluded to cost optimization. So we're talking about cost control. Now let's talk about cost optimization. The easy easy way to think of it is, spending limits will stop the bleeding, but dynamic routing will, is where you can start optimizing, essentially. Now this is GA today. We've we've had this available for a while. This this screenshot excuse me. This screenshot is from Cloudflare dash. So you can build out dynamic routes via click ops or I mean, you can do it obviously via the API as well. But we can route requests by user plan, by, by the user ID. You can attach custom metadata as well. So if you wanted to look at, some of the keywords contained within the request and route the route to a given model based on that, that's available as well. One thing that's coming is, we are working on a an optimized LLM to assess the requests and then help have that, available to to help route request as well. So rather than just having keywords, so send it to an LLM. Says, this user looks like they wanna build an application, so let's send it to this, request. That's not available today, but it it's something that's been worked on. But just to give you an idea of of where we're taking, dynamic routing. Yay. Agents week. Now, as I touched on during the agenda at Cloudflare, we tend to have, thematic weeks with so security week, developers week, AI week. Our most recent one, it was agents week. I think, coming up next will be birthday week and sometime in the coming months. But agents week is what we had, a couple of a couple of weeks ago. Now agents week, many, many releases, the majority of them were more developer focused. Personally, my background is is on the cybersecurity side. So I tend to be more interested in the cyber side. But a a couple of the releases, really picked my interest purely because we had some events happen in in in re in recent weeks. And I'm I'm sure if you work in cyber yourself, you would be well aware of this. If you pay any attention to AI news, you you're probably aware of it as well. But, basically, we had the the two largest Frontier Lab vendors have come out and said that in their own internal testing, essentially running their own models in their own sandboxes, the the agents with guardrails removed managed to get out. And and they weren't breaking rules as such. They were just doing what they had to do in order to to, achieve their task. I think the OpenAI one is particularly interesting because OpenAI went, to ended up in Hugging Face, of course, as we all know, but went via another company called Modal. And then you've got Claude of Antropics. So the Claude models were writing malware for for goodness sake to to achieve their tasks. Again, just to just to reiterate, these these were tests by the Frontier Labs with the guard rails removed. But where the where this gets really interesting is, okay, the you and I don't have access to Claude or CHATGPT models. We we don't have access to those models without the guardrails, of course. Stands for reason. What we do, you know, potentially have access to are the very, very powerful, OSS models that we can download and run on our own infrastructure. Now I don't have access to $500,000 of compute sitting in my garage to run Gimme k three on, but $500,000 of compute is not a lot for a well funded, cybercrime organization, essentially. And then forget about a nation state, obviously. It's, yeah, it's it's, it's peanuts, essentially. Now why do we care about this, though? We we can't solve all the problems of the cyber criminal or we're not during this talk, we're not looking to solve the problems of the cyber criminals who are running these models on their own compute. But all of this sort of leads into organizations not rushing to adopt AI because security teams quite rightly, perhaps a little bit worried about the potential of not not so much the chatbot. Like, the the chatbot is just a chatbot, but worried about what if we give that agent access. What what is it that the agent can do, and how do we control that? How do we have visibility of that? And it's it's a difficult problem to answer. Some of our releases, over the last over agents week, a couple of weeks ago. So firstly, there were three that I wanted to call out. And, well, the first one is more of a framework. It's it's not a not a product release. But, essentially, the the agent access model being the framework of, well, how do we scope credentials to a task, expire with the task, and, and and make sure access is scoped not within the system prompt itself. Because if it's scoped within the prompt, this is where people can be really clever and talk around it, essentially. We've got two solutions, that are that are addressing this, and one one is an existing solution that's sort of been repurposed. But, firstly, Brightguard. Brightguard, as you can see on the screen, very focused around, providing risk analysis and risk controls around MCP tooling. Today, we have, we do have MCP portals available within our within our ZTNA solution or our CloudFlare one product. But you would use an AI gateway to authenticate and have the agent go via the MCP portal and say, well, these are the things that's that the m's these are the MCP tooling that the agent can access. Wireguard gives you visibility around what's appropriate, what might be risky. Okay? So this is this is very new, and it's in, private beta. Next is a gatekeeper. So a gatekeeper is essentially a worker that sits in front of a system of record. It it the credentials sit within the gatekeeper itself. We're not giving the credentials to the agent. That that is that that's the big thing there. It's holding the credentials. It's it's logging logging everything, and then and then providing that the data back as as needed. So what I wanted to to land on here is Cloudflare OS because Cloudflare OS for us internally at Cloudflare. And if you if you haven't seen Cloudflare OS, if you if this is the first time you're hearing on it, please please talk to your account team. Please have a look at blog posts. CloudFlare OS is something that we have open sourced. So it's essentially it's it's it's not a product that you buy. If you wanna use it, go ahead and use it. You can use Cloudflare hosted LLMs to do the inference or not. It's it's really up to you. It's it's incredibly configurable with what you what you do use and what you don't use and and how you use it, essentially. But it was all born, I I guess, from how how we started using AI internally at Cloudflare. And living through it was was quite quite interesting because I think it was it was around about May last year. I seem to remember. I I was on leave, and I came back from leave to an email saying, hey. Guess what? We've enabled, we've enabled Google Gemini because we're Cloudflow is a Google shop. We've enabled Google Gemini. We've got it there as a chatbot. And if you wanna use it to summarize an email I I don't think I've ever used it to summarize an email. If you wanted to to use it to summarize an email, it's it's there. And we all rushed out, and we we started using the I I personally use the chatbot functionality quite a bit. It helped in my role, when a customer came to me with a long technical requirement, it it helped sort of piecing that together. There were no and I'm I don't think Google still doesn't have MCP. So it was you you have to really check everything because it it would, you know, hallucinate a little bit. But then we moved from we moved from just having Gemini to then then we started to get the, probably, the more interesting tools and, having the ability for using an AI to effectively act as an IDE coding assistant. And this is where it really started taking off, I I would say, in the last few months of of last year because that brought with it MCP access. That brought with it if a customer, came to my team and with a a very complicated technical requirement, it became very easy to just just build the thing that the customer needs, and and there's your minimum viable product. And here's here's the flash demo, and we can give you the code, and it's just a place to start with. Right? And you would then change that as as you needed to to move the code into into production. Then the nontechnical team started using it. And that that's it it became it was interesting because you've got people that are really nontechnical at all, and they're asking questions around creating markdown files and and and skills, and they're asking, MCP server questions as well. And this was all sort of bubbling along, and people were creating their own apps. And you you had multiple teams doing the same thing because we're all we're all just operating in silos. Then the, I guess, the the, oh, oh, gosh moment came with, Sam Ray, and he he wrote a blog post where he talks about this. These are just quotes I've I've pulled from his blog. A someone in the sales organization approached Sam, our CIO, basically saying, well, we need I I I need I want access via API key to all these systems of record, which, obviously, my goodness. No. But maybe this is something that we should be thinking about. And the next step then became, well, let's let's work on Cloudflare approved MCP servers for us to use internally, and it was still very command line driven. And then let's build this into a nice Flash web portal, where our nontechnical users can log into and create agents for themselves, create applications, simple applications, if they need to to do their job. I've sat in on some of the the sales training on this. I I don't, must admit I don't attend enough of it. But I sat in on some of the sales training, and our our salespeople are being taught how to create, chief of staff agents to to just help them with with admin tasks. The message here is a obviously, we're Cloudflare. We use Cloudflare OS in, obviously, probably better than anybody else because it's our own product. We've managed to take this to a point where nontechnical users are creating agents that are operating in safe, safe ways, particularly when it comes to our customer data and things like that. Nontechnical users are creating these agents and getting real value out of them. Now how how this works, if we think back to excuse excuse me. When we when we think back to the slide that I had up before around, WireGuard and gatekeepers, So why, gatekeepers play a big big, big part here. So rather than when people asking for API keys, obviously, we don't give wanna give them unfettered access to a Salesforce API. Right? You you just don't wanna do that. Rather than just saying straight no is we put the the the gatekeeper functionality in front of every system of record. That way, the agent is always going through it. The agent will call the method, so something like, you know, list issues. We use Salesforce as our for for technical support cases. So list issues or get customer information. The gatekeeper receives that call, holds the credential, and then decides whether or not to execute it and returns only what the, what the agent is allowed to see. The credential never leaves, never leaves the gatekeeper, and the agent never actually touch it. So that's that's the left hand side of the diagram. Sorry. I've I've got this up on my monitor on the left hand side. That's the left hand side of of the flowchart there. The right hand side is really the action. So whether it can write, deploy, or delete, obviously, the gatekeeper, can can block or allow or queue it for human approval if needed. Okay? Now we also give, as I touched on, we also give our users the ability to build apps. And it it could be a simple app. Don't don't don't don't think too much about what these apps are. It could be basic things like, you know, shift schedulers or what whatever it is. But then they're also creating agents. Those agents can use those apps as well. Okay? And this is where that that application API, surface comes into it. So both the agents and the browser clients, using, in in the example in the blog is is captain, WebRTC. Captain WebRTC is a open source is is something that cloud for the open source, last year just around RPC tooling. Basically, what we're doing here is ensuring that the the browser, the user, when the user is doing the thing on the application they created, the agent has the same access as what the browser does. Now we then we have dynamic workers and and durable objects if we if we need to attach SQLite database, to to the to the user created app. That's all handled at that layer as well. Now what have we found with Cloudflare OS and and what's been created? So these are these are some big numbers up on the screen. Firstly, to be very clear, this isn't saying that we that we would have pushed 255 250,000 issues to production. Obviously, we're not pushing 250,000 issues to production in four months of using Cloudflare OS. What Cloudflare OS has done, though, is sped up the detection of those issues, within within the CICD pipelines, essentially essentially, and allowing the whole thing to to operate faster. In in my world, working as an engineer that sits in the sales organization, in my world, what I see with the salespeople that I work with is huge amounts of time saved through, from their their planning and their administrative tasks. So things like territory plans, things like customer proposals. A lot of this is used a lot of this is created with the help of, of AI, within Cloudflare OS. In terms of applications that are being created, so those those small apps that, that our users are are creating, you can see the number there. So 4,000, within a single month alone. Okay? So people are finding ways to, I guess, automate or or automate parts of functions that they perform, And these people aren't necessarily in technical functions of Cloudflare, which which is probably the most interesting part, to be honest. Now why are we talking about CloudFlare OS and dedicating time to this on a on a webinar that is essentially about AI gateway and cost control? Because AI gateway sits at the middle of this solution. It's not the it's not the brain. The the brain would be the the LLM, but maybe maybe the nervous system or or the heart. I don't know if the analogy stands or not. But what we're doing with AI gateway, it's it's the center point that holds all the the plumbing together, essentially. We can ensure that through through DLP controls in AI gateway, sensitive data is is not going to places that it shouldn't be going. It's it's just part of it. We can enforce cost controls for the users or teams, within AI gateway. You can do attribution of those costs. So my manager knows Sam's been an idiot and spending far too much money with Opus, or my my colleague has been much more efficient in the way that they work, which is probably not too far from the trip. And lastly, it's open source. So, again, I'll just hammer hammer that point home. If you Google Cloudflare OS, there's there's a landing page for it. Have a look at it. Talk to your talk to your account team, about it. That's that's not to say that, you know, the the more you use it, it will there will be a cost associated with it with it. That cloudflare OS itself is is actually open source. It's very cool. So five steps, I think, to get to a a better excuse me for for just a sec. Five steps to get to a a better place than having, cost overruns where you don't know where who's causing these problems or, whether it's an agent or whether it's a human. Five steps to get to a better place. Basically, front everything or route everything through AI gateway. It's it's a single line of code change to get the traffic to AI gateway. And then within AI gateway, we do our API connectivity to your preferred provider of choice. Always recommend having a a backup provider as well. That it doesn't have to be the two big, Frontier model providers. You you could also be using, a provider who specializes in OSS models as well. If we put Cloudflare access in front of the AI gateway, that forces every request to, that forces every request to have identity attached to it. And from there, we can do things like enforce spend limits, put fallbacks in place. We can look at baselines, to look for anomalous behavior and high cost overruns that, that that are out of the ordinary. And then lastly, if we wanted to extend this to agents, we can use gatekeepers to to do that, effectively and and safely. Now I can see there are a couple of questions, so I'll just I'll just go to the first question here. So if you route an existing session, say rate limiting from a Frontier model to an OSS, what becomes the session state and the workload behind? Okay. So how I've played with this, in my own testing with spend limit rules is each request is assessed on its own merit. So if you go request one, two, three, four, five can all go to the provider. Once you hit request number six, the dynamic route will kick in and send that to the OSS model, essentially. In terms of, on text, and I I guess, I I don't wanna don't wanna put, words into the person who's asked into the mouth of the person who's asked the question. If you if, I think about where, naturally, this question would go is, well, how do you remember context and things like that? This is where the the framework of CloudFlare OS, which has memory and and, you know, it's it's all sort of that those primitives are there, to be used, and and the the LLM could still reference the memory. The the OSS LS LLM could reference the memory that the Frontier Lab LLM was was also, remembering. You wouldn't necessarily want to keep the entire prompts and conversation history being sent to the LLM every time because your your token cost would just grow exponentially as you get to, request ten, twelve, 13, whatever. It'll become quite, cost prohibitive. Now the other question I have here. Okay. As new models, open sourced and paid, are released rapidly, are there any governance tools to help manage evolution of models, upgrades, validation, and testing of new models within the governance tooling? I don't have an answer for that. That is that is a really good question. I I wonder if that's there is popping up a category around I don't know if if pen testing is the right word, but where, and I know there are vendors out there that are doing things like let let us pen test your your LLM and and try to break through it. I reckon that's more more within the purview of of, not to be confused with Microsoft, but here, more within the purview of those vendors who are playing in that space. That's that is not something that we're we are addressing today. It's it's a good question because I don't have an answer for it. But it it could be something that we're looking at. I just I don't have that level of visibility. Another question that just come through. Is there a UI for managing external MCPs in Cloudflare OS? Yeah. Absolutely. So, firstly, with MCP servers, how I've played around with them in in Cloudflare is you you've basically got two one of two options. You can build the MCP server on Cloudflare. If you build an MCP server on Cloudflare, it is essentially it's a worker and a durable object, or you can use an external MCP server. And that external MCP server could be external to your network. It could be internal to your network. It it doesn't really matter. How we manage that is within our zero trust solution. So if you go into Cloudplay dash, got a zero trust, in the policy in the access policy section, there's there's a section to create a, a section to create a MCP portal. And from the MCP portal is where you you add the MCP servers themselves. And from there, you can control access using the product called Cloudflare Access, essentially. Another question that's come through, are there any scalability limitations when thousands of employees start using AI agents? This is what it's built for. We we are doing this internally today at Cloudflare. We we we are the test case, for this. Yeah. If if you if you need if you need demos of this and and more in-depth sessions of how Cloudflare how Cloudflare uses Cloudflare, in particular around AI agents and AI gateway and and Cloudflare OS. Talk to your account team. It's it's a little bit I I don't know if we have a solution yet for how to best demo Cloudflare OS because if I was to do a demo of Cloudflare OS, you see all the customer data that I've, I've I've got in there, and, obviously, we don't wanna show that. So I don't know if we have a a good solution for that today, but we can talk definitely around how we're using it. But talk talk to your account team. Yeah. The the the best outcome there is if if if you if you, if you're able to talk to the people who create it as well. Right? Now I'm not seeing any more questions come through. So with that, I think, I think we can end the session. Thank you so much, everyone, for for staying on and and listening and participating with the questions. Yeah. I enjoyed myself, and I hope you enjoyed it too. Cheers. Thank you.