Video: Beyond Downtime: Architecting for Proactive Network Resilience | Duration: 4876s | Summary: Beyond Downtime: Architecting for Proactive Network Resilience | Chapters: Introduction to Resilience (93.145004s), Digitalization at Edge (212.295s), Network Latency Challenges (384.67502s), Network Resiliency Importance (567.935s), Network Resilience Challenges (842.615s), Safe Change Management (1040.2849s), Change Management Importance (1494.7301s), IT Systems Management (1644.96s), Control vs Data Plane (1781.45s), Cloud Resilience Challenges (2001.365s), Network Diversity Benefits (2152.1702s), Fault-Tolerant Network Architecture (2568.875s), Anycast and Resilience (2672.73s), Cloudflare's Anycast Network (2892.63s), Final Thoughts and Conclusion (2983.09s)
Transcript for "Beyond Downtime: Architecting for Proactive Network Resilience": Hey, everybody. We're really excited to be having this webinar. Thank you all for joining. My name is David Tuber, director of product over here at Cloudflare, and I'm joined by Andre Katniss, who's a principal analyst over at Forrester. And we're gonna be talking to you about architecting your applications for proactive network resilience. So Andre thanks for joining. Oh, thank you David. And you know I'll start this off and kind of just you know people probably wondering why I'm here to discuss this with you. So my background is been working at Forrester for about the last fifteen years but you know worked in the aerospace industry designing pretty resilient systems. My background was aerospace engineering and so one of the things was you know working on pressure systems to feed liquid oxygen and kerosene into the rocket engines themselves. And if there's nothing that needs to have resiliency is systems like that. So that's a kind of a mindset that I have when I think about you know writing a lot of research and digging into things to share kind of best practices from one organization to another. So that's kind of like, you know, so why did Cloudflare invite me? You know, what's my background and why is it, network resilience such a passion of mine? And so, we start to think about that aspect and of my background. One of the things too is we need to really think about is in this kind of emerging world that's why the network and network resilience is so critical is the intertwining of the worlds together. So, you know, we think about twenty years ago, we didn't really have a network was good enough considered to be connectivity in itself. You know, it's just kind of think of as a commodity, not a lot of the organizations put a lot of effort into it from a business perspective. But what we've probably seen the last ten years and even more so accelerating the last couple of years with AI is intertwining between the network and the business itself. And then we can't think about and we think about security, you know, making this happen. Now, why? So, as we think about that aspect is why are these things intertwined together? It's really about what we start to see is kind of this digitalization of the edge. And I know it's a term that gets thrown around a lot. It's kind of like a marketing aspect itself, but I like to point to IoT and now we're going to talk about AI or elements of it. You know, we're You said the letters. You have to say the letters in every webinar. It's the rules. Internet of things and artificial intelligence, David. Yeah. All of the all of the acronyms, you know. All of the acronyms. I know. Oh, man. Then we're gonna get into software defined wide area networking, SD WAN with that. So, you know, we think about maybe like I mentioned about five to six years ago, started the idea that we needed to start to really think about this edge being different and attaching and it almost started with, you know, I had some client calls. One of them was from a rental company where as they started, you know, doing rentals and their dependency of, you know, at the rental site was, you know, set bring up reservations and changing things around became more the census weren't localized. It became more cloud based. They started having issues around just typing in someone's name and having some latency around that, type of letter and maybe a ten seconds go by before you type next letter itself or the fact is that, you know, I had other clients thinking in the manufacturing space, you know, their machines very dependent on, you know, connections to their data center and their inventory levels and making sure that things were in that area itself. And so the idea that, you know, things not being resilient, the network being down and or even having any type of latency at all started to rise in the concerns of the business because it was affecting the business itself. It's interesting because like I see the same thing. So it's like when I worked at Microsoft, we had, you know, obviously Outlook online mode, everybody's favorite thing. And, like, a lot of stock traders have to use online mode because they can't use dedicated PCs, so they have, like, a virtual PC that they log into when Outlook opens up online mode. And if you ever use Outlook online mode, which I did when I was debugging network latency for Outlook, don't use Outlook online mode. It's very, very slow because every single UI call is blocked on the network. And so if you're down streaming a whole bunch of data and your network is congested from all of the mail that you're downloading and you're trying to send mail, it's gonna take like five seconds to like type a letter or you're gonna click on a button and you're gonna wait five minutes for your email to send and your screen's gonna freeze. Yeah. Latency just late like that's a really good call out that like latency it is also a resiliency problem because at a certain point degradation of latency becomes a degradation of availability. Yeah. It's and even bandwidth itself became an issue whether it be VDI or had a client. So I had a client that was a downloading a 20 gig catalog. So the watch retailer and you would think the high end retailers would be in locations where they had good, you know, connectivity, you know, good amount of bandwidth but they didn't. I mean surprisingly throughout different parts of world even in, you know, tier one cities connectivity to the store itself, you know, high end watch retailer downloading 20 gigs and then try, you know, at the same time as other things are going on causing issues within the store of a mobile point of service. These things are starting to rise up with a lot of concerns of clients on the business side. This is affecting revenue, experience of clients in a store or what you think about acute care centers being dispersed everywhere and issues around, you know, having five nines reliability of medical information, not information itself, but let's just say we're monitoring your patients and, you know, baby heart monitor and things across the network. All those things started rising up when client calls about thinking about the network not just being good enough. So, David and bandwidth and just uptime, these all started rising, a lot of it because of that edge itself, there's so much happening. But interesting thing, most, the weakest point was actually links going in now to the site itself and not saying the link themselves, but the fact is they had to be attached if you're looking at the screen in front you to cloud resources or private data center resources and they're all intertwined together and all these things within that location was dependent on having good connectivity to these areas. And Those so bottlenecks those bottlenecks really get you. Like, it's like, you can have, like, 500 terabits of backbone. And if your customer is connected to a 10 meg uplink, that's, like, plugged into some DOCSIS box that's using like, you know, whatever, then that's going to be the limiting factor no matter what you do on the back end. Yeah, even, you know, routing to the back end, routing to the backbone can be a big difference. I've got even today I've got clients thinking about having performance issues of sending traffic to different security sites and there are different services, right? At different POP sites and then being overwhelmed and it does not have any ability to look at your throughputs or what was going on. The fact is the service is being overwhelmed that they could route it in another direction as well. So, I mean, these things were all these things are factors and clients are calling me about concerns about what's going on in this area itself. And so, kind of be looking at the screen in front of you, the idea that these remote locations, even the remote users, but acute care centers, manufacturing sites, retail, the more and more that they start to do, the more and more they're dependent on resources all over the place. And what I mean by all over the place is globally. And so what becomes the critical item in most organizations in the calls I'm getting today is the backbone and connectivity and kind of this WAN fabric is, you know, do I have the latency? Do I have the bandwidth? Do I have the uptime, the capabilities of having a resilient type of infrastructure? And when we think about it from, you know, having meeting regulatory, if you're in a healthcare area, you're thinking about, you know, customer experience from a hospitality or retail area and then even like downtime can make a big difference. I know there was, I've had airlines look at Delta or Southwest or any of them over the last six years, they've all had one to six hour outages mostly because of the network. We only think that maybe in those our ideas, but when you think about the business and this is where we're interrupting kind of business and network together, It's how long you bring not just the technology systems back up, your apps back up. The bigger picture is what does it take to get the business back to normal? And so, you know, think one airlines reported $150,000,000 in loss. Why? Because it's rerouting their planes, it's reticketing passengers and takes it off on average, you know, there's an issue to get the business back to normal after about seven days. So while you might have a minute, an hour of network issues, it could take a business seven days or even longer to get back to normal. And that's, these are the things that are starting to rise up and why? Because it's a passion of mine when I think about network resiliency and the importance behind it because it carries the entire business. It's like this business fabric and in itself, having security embedded in it, but this business fabric supporting the business itself and becoming a business differentiator. So, Dave, this is why you and I had some great conversations earlier around the capabilities of the importance of the network and how the sharing information and routing and how you're doing updates. I remember we talked about a few days ago on some of those elements. Yeah. I was like, go ahead, sorry. It's really interesting. I really liked that you brought up like network latency before bandwidth, you know, because, like, I as somebody who's very focused on network quality, every time you talk to an ISP, it's like, well, just download more bandwidth. And, like, I actually gave a talk at at one of the Nanogs with, with somebody who actually now is on who who now works at Cloudflare as well. And, basically, like, we were talking about network quality, and I asked a question. It was, during the pandemic, because this was, like, 2021. Like, during the pandemic, when you started working from home, how many of you called your ISP about Zoom problems, and how many of them told you to upgrade your network bandwidth connection? And, like, over half the rumors there. It like more money. It's this it's this weird incentivization structure, and I think that, like, maybe ten, fifteen years ago, bandwidth was king because everybody's like, wanna watch movies. I wanna do videos. I wanna do all of this stuff. But I think you pointed this out that, like, so many more things are moving onto the Internet, and so many more things that are weird to move or some of the things that feel very foreign to move are are moving. So, like, a really good example is, like, during one of like, there was a major cloud outage, and the the Ring services or the services for, like, like, smart doorbells and locks, some people were locked out of their homes. And it's like, those are things that you don't think matter until all of a sudden you wake up and you're like or all of sudden you're like trying to leave your house or all of sudden you're trying to get back into your house and you can't get in. You're like, what the heck? Like, I I'm I'm putting in the passcode. I've got my smart thing. I've got my phone. I'm putting it to my lock. Why isn't it working? It's like, oh, well, the cloud's down. There's a network outage. I can't get to so and so, and therefore, you don't get to get into your house. Like, it's this weird concern, but it's it's it's this weird use case that, like, you didn't think ten, fifteen years ago that that would be a thing, but it is not a thing. And Yeah. The network the Internet itself is an and I think you mentioned this at the beginning. The Internet itself is an inherently unreliable system. It is built for to tolerate failures. Like no network link is built to be a 100% up all of the time. No path is built to be a 100% up all the time. So like you so like folks on the internet have designed the internet to be tolerant of failures. And so we need, as internet providers, to build our networks to be tolerant of failures in ways that fail open and not closed. Basically, like, we can we can have problems, but that's okay because you, the end user, don't don't notice. And that's the thing that we really want to, you know, figure out and how we want to go solve that. I would even say that you probably would agree that, you know, it was built to be tolerant, but it wasn't built to be, you know, provide awesome experience. That was a marathon. This is where something, you know, as a internet provider, you're like, hey, we realize this is important. We're going to, you know, we're going to figure out how to make add in the resiliency and because resiliency is part of the experience and making sure that's a good experience. But back to your door lock thing, it could be a latency issue and it just the apps not working as quite as well, whatever reason it could be network could be the fact that you've got too many apps open on your phone and you've got memory over usage on it. So, there's this, you know, lot of different elements to making things better. The onion is like the onion is constantly being peeled. Right? Like, you're like, it's like, it's not just enough that, like, it's it's like the Internet, but it's also this other stuff, and it's also this other stuff, and it's like this very complex mesh of systems that all kinda work together. And like, I get it for a lot of folks who aren't like built on the Internet or who don't like necessarily work on the Internet or think about this stuff. It's very it's very difficult to parse. It's like, what do you mean some data what do mean some backhoe in Northern Virginia took out my ability to act get in my home in, like, Salt Lake City? Like, that doesn't make any sense to me. Explain how that happened. Like but the cascading effects are are very vast and, you know, the butterfly effect is very real in my job all the time. It's like, if I do this thing here, how will it actually impact stuff? You know? I use that all the time. That's why I created the idea that it's business wide fabric because you pull on one thread, it's going affect the rest of the network of your entire business. So, use that butterfly effect all the time. People are like, I'm like, everything's interconnected. You know, it's a tough thing to get my developers or analysts or developers, app developer analysts to think about things in that way. Even my counterparts on data center, they think about the data center, they don't think about the fact is it's dependent on all the other pieces. It's really interesting. So like, and even in within Cloudflare. So I'm gonna take a pause and do a little introduction of myself, because it's like, who is this who is this joker and why is he talking to me about network resilience? So I am a director of product at Cloudflare, I work on our infrastructure engineering team. And so, basically, my job is to build Cloudflare's network to be as performant and as reliant as possible and enable products to deploy on top of our network and get all of the goodness that comes under the hood. So, you know, there is a lot of thought that like me and my teams put into around this complex butterfly thing. And so one interesting like anecdote there is, so when we were deploying our deployment map, when we were rolling out our new deployment methodologies to kind of enforce safe change, one of the things that we thought about is, hey, every service needs an SLO, right? Like every service needs some amount of a service level objective. We need to be able to say this is the level of service that we can give, and we should define what that level of service is, or we should define what service is, and then we should set the threshold and alert. And every service gets one, and when we roll out, we look at the health of the service, and if it violates our SLO, if it goes under our threshold, we stop the change and roll back. But one of the things that is interesting there is, that's embedded in that, is if that service breaks, what does it actually impact? And what we started to find is that for some services, the mapping is like one to one on a product, right? Like got, our like, you've got, like, hey, like, if you're a magic transit and, know, there's an API that updates your config in magic transit and that breaks, well, chances are it's probably only going to break magic transit, so no big deal. But there are some services where if they break, the rollout is catastrophic. So for example, we've got a distributed key value store called Quicksilver, which we've blogged about. And if the ability for Quicksilver to distribute config updates to every single metal on our net edge metal on our network. If that breaks, what breaks? And, like, you start pulling at threads, and you're like, oh god. If this thing falls over, then Cloudflare is down. So, like, you have to think really hard about, like and and and oftentimes, you end up going to teams who are like, I'm just working on this service. There is no possible way that, like, this small change could have any impact, and you have to go and stop them. Feel like, here is this web of dependencies that your service has that like you can't just go willy nilly and make a change and then just deploy that everywhere. And that was a culture shift I think that like we really had to push then is definitely been well received, right? Like we need to be like our engineers and our teams definitely are aware that like the of the butterfly effect there. Like, they make a change in their service, and they're like, it's innocuous. No big deal. That change has impact It's interesting, Dave, because one of the things about it is that when I moved from the aerospace industry into manufacturing of switches, that's how I got into becoming analyst and got a networking world. When I became an analyst it was shocking how such a basic engineering concept like that which you're bringing up what we call fault tree analysis in the aerospace mechanical, the outside industry where it's exactly right. You're looking at each point and if it fails, what the cascading effect of it? And so you're designing for that resiliency around, you know, understanding and going through each, so as a fault tree, you're going through each level and each connection and seeing what's going on. It blew my mind the fact that that wasn't very that wasn't a common thing. It's still not common for most organizations out there and to see Cloudflare take that take that and do it is fantastic. It's definitely something it's definitely like so what we did, and the way that we did this, was that we took a look at kind of all of the times that we broke in, because, like, I think one of the things that I really love about Cloudflare is that we really value transparency. So this slide is very interesting, because it talks about, like, resilience and withstanding conditions, and a lot of vendors don't like to tell. We do love to tell. Like, we love to talk about this. And, you know, you can go on our blog and search outage and there's gonna be blogs and blogs. I've definitely written some of them. I've ghost written some of them. This basically like, here is how all of the time here are all of the times that we've broken and here's what happened. And some of them are like very silly. It's like, oh, we decided to route all of our like, made a change that accidentally routed all of our backbone traffic through one location. Oops. But some of them are very complex and parsing out the why is an interesting exercise, is interesting for you, but it's critical for us because those dependencies, like, for the for the easy ones, it's like, yeah, like, don't route your backbone through Atlanta. Like, don't route all of your backbone through Atlanta. Atlanta can't handle the world's traffic. But, like, some of them are very complex and difficult to parse. And so for for for customers and for end users, that's a very challenging thing to kind of wrap your head around. But like, that exercise is it's it's valuable doing it in in the light, right? It's valuable doing it for everybody so that you can see, but really what it is, is it's showing us that like, showing you this is how we need to think about this stuff. And like, most providers, I I imagine, do think about it some shape or degree, but I really like that we talk about it. Like, when we do this stuff, like, we also do it internally, and we look at all of the times when we break people, and we look about, like, here are all of the ways, like like, literally made a list of, like like for example, change related incidents, which are, which happened a fair bit, like one, two, three, four, here are the top four reasons, okay, we're gonna go lock these deployment conditions down, we're going to, you know, harden them, we're going to build that resilience, we're going to kind of do that fault tree analysis, we're gonna do the safe deployment methodologies, right? Like slow but slow and steady, leverage our self serve users while also, you know, tying that to health signals that are that are end user focused. Right? If use if this breaks, users can't do this. Right? And then that model, roll that out across the big across the big four or five. And then once that's done, then tackle, like, the long tail of, like, all of the other things that introduce change. Right? It's like it's the same reason, I think, for a lot of customers, why for a really long time, a lot of networks only had one ingress point. Right? If right. One ingress and ingress point into their networks because I had a big firewall box, and I need that big firewall box to be able to apply config and make sure that I can lock down whatever goes in and out of my network. The same thing happens with code. Right? Like if you have 50,000 ways to introduce change into your system, then something's gonna get through that you're not gonna see. And so we have over time been really locking those down and getting to a point where we can see and we're getting better at seeing, here are all of the ways we're introducing change and we're monitoring and logging and starting to crack down on, like, enforcing safe change. Like, for example, we talked about Quicksilver, like the key value store. Even that key value store is getting locked down, and we're enforcing safe change in versioning on it. So it's like every like, we wanna get every single one, and that's a process. And, like, I'm gonna be honest with you. Like, there's there's gonna be stuff we're gonna miss, and we're gonna go find it, and then we're gonna we're gonna go fix it because that that's how we think about it. Yep. Sorry, that was a little bit of like a just No, no. I think the elements you bring up are exactly, so myself and other counterparts are, you know, best practices and rolling out software back to the SL, you know, checking along the way, reducing your kind of blast radius. Those are exactly the, I think we've got probably eight kind of bullets we kind of recommend about the phase rollouts and sharing Q and A at each step and back to, I mean, that's unusual to hear that and that's something we have put in our recommendations for our for our clients as well. Yep. And definitely, like, you you look at some of the major outages that have happened, and, like, that that's a big reason why. And even for Cloudflare, like, a lot of that is can be tied back to, hey. Like, we need better change management. For a while, 50% of our outages were change related. And it's like the surface area is wide or the surface area is vast. I was going say what I just say thing was the rise of, well, SD WAN software defined WAN is for the acronyms or is was the amount of talk about configuration change management, just documenting like probably 98% of the enterprise working with had no idea of the traffic going in and out of site itself. So, a lot of times, you know, just shifting over to a much more resilient design is just having a documentation and let alone, you know, configuring it. So, you know, managing that world and documentation and it's so critical, you know, having that visibility and real time not, you know, give me something from five years ago it was ridiculous in this world. This is some of it goes back to, so yeah, bandwidth latency, but just keeping up on information and where your systems are at is another piece of the pie that we don't like, we forget to talk about. It's it's I I talked to a lot of my colleagues about, like, moving to, like, ERP, like, moving to a new ERP or moving to, like, a new, like, DSIM, a data center inventory manager. And that's like, it doesn't sound very like, it's not very cool. Right? It's like, oh man, like, I'm so excited to just implement a new ERP said nobody except our finance department. But like, that stuff is so critical because like really what it is, is it's a it's it's a source of truth. It's an accounting ledger of here's what's in our network and like it has to be there because we paid for it. Right? And this is true for for you for customers as well. Like, you need to have a very rigorous ledger of all of your stuff because you don't know where it is and you don't know what it's if you don't know where it is, you definitely know what don't know what it's doing. And if you don't know what it's doing, then that's a big problem. I like to call this the Jack from IT problem, and this we talk about this a lot when you get to, like, zero trust and, like, network services. It's like, hi. Like, I saw that we saw this a lot when you move when customers are moving to magic WAN. It's like, they wanna move to a cloud based WAN and they've got this firewall and they're looking at the firewall and they're looking at the policies and they're like, I don't know what this does. And they're like, well, nobody does. Jeff from IT wrote it ten years ago and Jeff's no longer here. But every time we try and touch this firewall policy, everything breaks. So Yeah. We have decided to just never touch it again and just don't worry about it because they don't know what it is, they don't know what it does, and they don't track it on a regular basis. So this stuff just kinda accumulates over time. And that's a real challenge with a lot of people. Yep. And and one of the value and, like, this is kind of, like, the double edged sword of, I think, a lot of, like, SaaS and data center SaaS and cloud providers is we have to do that. Right? Like, we are buying this hardware and so, like, that's that's that's capital that's capital expenditures for us. So we need to be able to track that, and I guess it's capital expenditures for for customers too, but, like, that's literally, like, what makes us money. So we have to track that stuff. And, like other provider and other customers might might not track it as much because it's like, it's there but it's not why they're doing the thing that they're doing. Like if you're talking to like a rental car company, like a rental car company doesn't exist so that they can run their, you know, Juniper firewall. They exist so they can, you know, sell me so they can rent me a Camry at ridiculous prices. And so, like, because we track that, that's very, very helpful and valuable. And because we know all of that stuff, it's very, very good. But like, there is a lack of control, right? You are giving up some control on that side, so it's a very double edged sword like that. But, like Yeah. The flip side is is that, like, we think about that stuff, and we and you pay us to think about that stuff. Not the camera, though. I I I don't really wanna think about the camera. But, like, I think one thing that's super interesting is, like, I think that and I think that, you know, I think a lot of customers and I'm curious about your thoughts here. Like, lot of customers maybe don't think about it like this, but, like, there's a there is a significant difference between configuration and actual end user traffic. Right? Like, you, like like and I'm speaking directly to the viewer here. You, a enterprise IT vendor or enterprise IT admin, are probably running some sort of network that's maybe SD WAN or maybe IoT or maybe insert, you know, acronym here, and you're configuring that. And you view that as critical traffic, but the traffic that's actually critical, and I think that a lot of ITMs get this, is this that the traffic that you are managing. Right? Like, you're managing, like, the ability for an end user to, you know, reserve a car or buy a thing. And so, like, that's data plane. You configuring is control plane, and they are both equally critical, but logically, we think about them as separate. At least at Cloudflare, think about them as separate. Yeah. And they fail and because we think about them as separate, logically, they fail in different ways. And sometimes that failure sometimes that separation is actually really good because, like, hey, like, is frustrating to not it's frustrating and sometimes absolutely critical to not be able to onboard, you know, websites to Cloudflare and not be able to configure, you know, SD WAN policies. But I think a lot of and I'm curious about your thoughts. My my personal supposition is that a lot of folks would much rather have that happen than have the ability to not, like, do critical business traffic through on the on the Internet. And that's data plane. And what are your what are your thoughts there? About where they would want their traffic to traverse? Or, like, how, like, how do people think about that? Like, is that, like, the the control plane versus data plane? Is that, like, a thing that people think about in your experience or or not so much? Not not as much in that in that respect. I mean, data center, data plane control plane management plane was a big hoop off for a while, but in the overall aspect of the apps across the entire business, things like that, as much. Interesting enough, in general, because there are so many different apps being used by a large enterprise, and I'm focusing on the large enterprises, that's become overwhelming for them and they thought and in many and you look at the last twenty or thirty years, a lot of that was shoved off to their providers and they don't really think about it, like they didn't really get into it because it required long conversations with the business on what the importance of whatever these apps. So mostly with our remote sites is what I ran into is just like, hey, this is get a bigger pipe, this is get a bigger pipe if we're having issues, but not really digging it down into the individual types of applications of traffic and where they're at and how to manage them and control them has only actually risen in the last three or four years, mostly because of now they have to start making decisions based on which link things have gone and more importantly, more dependency of the business on particular apps. Now they're starting to kind of dig into it but before it was more like, hey, let's get a bigger pipe so we can shove more things down and not get calls to our help desk to deal with issues. I'm starting to hear a little bit what you're bringing up and it's not as common right now. It's interesting and that phenomenon is definitely evidenced by, you know, Ashburn, Virginia being like 60% of global compute in the world or something like that or some, like, ridiculous number like that. But, like Yeah. Why why diversify your locations? Why think about that, like, physical topology when you could just buy more VMs in US East 1? Like IP addresses to one location. That's always works well. Build it into the app, you know, so that it's easier. Too hard to load balance. It's too hard to build that into the app. Load balancing for suckers when compute is fungible and you could just buy a bigger VM. Yeah. We could have we could have a whole another webinar on on just that alone. I mean, it's it's worth talking about. Right? Like, because I think that a lot of people and, like, I've had conversations with customers who are not necessarily impacted by cloud or outage, but were impacted by another cloud provider outage, and they're like, how can I be more resilient? And the first question I ask is, well, where are you provisioned? And then they say, well, I'm inter I'm at you in Ashburn. I'm like, anywhere else? And they're like Yeah. No. I'm like, well, I don't I don't know what to tell you. Yeah. Like, I'm not gonna go tell you to go spend a bunch of money and put, like, active active copies somewhere else in multiple places and then put those in front of a load balancer. Like, I'm not gonna not oh, I'm not gonna not tell you to do that, but I'm not, like, in control of your budget. Like, I know that that's an expensive thing to think about. Yep. But like But there's also developers in general and networking people, same thing. They don't, the network professionals, security professionals understand the development world and the same thing that developers don't understand networking. So you have that going on, but the cloud providers all have wackadoodle ways of within their VPCs and VNets and how they connect them together. And they do things very different than traditional within the data center. So, developers are kind of like doing whatever these cloud providers are doing in their unique environments. And so, there's money complexity, but they're also doing things weird so they can't have a conversation with networking because they're doing something different. So I think there's like multiple reasons why that's you're seeing that and all bad. All that. My app is in the a z. I can't possibly have an outage because they told me, like I mean, it's harder. It's harder to set up a national replicate them because they're already global. I I think that it's and especially with, like and, like, geopolitically, push for more sovereign clouds makes this conversation, like, 10 times more interesting. Right? Like, I don't wanna be in Ashburn, Virginia. I wanna be only in The United States. Okay. So where are you gonna be? Well, our options are Ashburn, Virginia or San Jose. Okay. Cool. So, like, for like, it's it's this weird thing where it's, like, the solution to better resilience and often better performance even is more diversity. Right? My favorite example of this is that I one of my first weeks at Cloudflare, I had a customer who was complaining about bad performance and they were they had users in Australia connecting to a VM in Dublin And I was like, so what do you think good looks like? And they're like, well, we want our users to connect locally in Australia. And I'm like, well, they are connecting locally in Australia. They're just going to Dublin. And they were like, well, it's expensive to get compute in Australia. I'm like, okay. I don't know how to tell you this any other way, but you have to traverse half the world, and I can make the Internet as well as resilient as possible. So, like, we've got plenty so, like, I said, we have plenty of backbone pads. We have plenty of transit providers and third parties, and, like, the way the Cloudflare operates is to like, as you talked about, like, referring a backbone, in the context of this example is a very good way to talk about how we think about this stuff is that, like, I don't necessarily think that there is a best path or I think the best path is the fastest one. Right? So like, if I have five paths and one of them is twenty milliseconds faster, then that's the one I'm gonna use. If that one's a third party transit, that's the one I'm gonna use. And the second that that stops being the fastest, then I'm gonna go find the next fastest one. And basically, the solution to performance is diversity. Right? I have I have like five different ways to get back to where you where at where you are, then I can just use the fastest one at any given time. And if they're all slow or if there's nothing working out, then I'll pick the one that's gonna be the most reliable and get you there. Right? Like, how many shortcuts do you try? How many failed shortcuts do you try before you just take the path that you know is correct? Is a very complicated question. I think in our case, the answer is three. But like, that diversity exists. And I told the customers like, I can get you from, you know, Sydney to Dublin. There are four different ways I can do it, and I can optimize for performance as much as I can. But you're also traversing over half the globe, and you're going across hemispheres and across two and across two oceans. So like, I don't know what to tell you other than maybe consider an endpoint or an origin or a VM that's in Sydney to improve performance for your Sydney users. And, you know, at the end of the day, that diversity is helpful from a resiliency and a performance perspective. Right? If Dublin went down, because Dublin's a major cloud compute in Europe, then okay, cool. Then they've got their Sydney instance, which is great. And then they can still run. But when you talk about sovereign and sovereign clouds and like kind of sovereign clouds, those sovereign clouds help with diversity, but also kind of impact resiliency. Because in Germany, there's maybe like one or two places in the world, there's more one or two places in the country that like have a lot of cloud compute and have a lot of data center space and that's Frankfurt and like Berlin or Munich. Cool. But that means that there's there's more opportunity like if Frankfurt fails, then all German customers who were already screwed but are gonna be extra screwed because their apps are just gonna be hard down as opposed to soft down or seeing performance issues. Because, like, you mentioned, like, the land size. Like, when you talk about, like, a lot of, like, compliance requirements, like, I had a I had a a customer call me. There was a I was talking about to a customer who's like, hey. I need resilient I need these data centers to be like 50 miles apart, like a 100 miles apart or something like that, and it was a pretty big distance. And the country they were asking about was Taiwan, and I'm like, okay. Let me point out so basically, we took our we took our we took, like, a a 100 a 100 mile radius thing, and we stuck the center of it in the Taiwan data center. And I was like, okay. So it can't be inside this circle. And it was, like, half the country. And, like, and, like, my favorite one is, like, Singapore, which is even smaller than that, and they're like, please, please build me resilient. Please build me fault tolerant data centers in Singapore, and they have to be 50 miles apart. And I'm like, what? Like, yes, I understand that, but like those are interesting problem and those are problems that like the more localized you get, these resiliency problems continue to compound. And it's like, again, butterfly, you flap your wings and say I want all of my traffic to be local or I want this or this has to be true and then the cascading effects are just very very difficult to consider. It's like, it's very it's very hard to comprehend, it's like a requirement like my data centers must be a 100 miles apart might impact on a might impact a statement like, I can't guarantee that your state will be online if that data center fails or if a data center fails. Like True. It's it's crazy. I know we don't have much time, so probably yeah. We got some more elements to go through. Right? I mean, like, we have slides to go through, but, like and we can talk about it. Like, I think a lot of I think I know. I can do I can do my marketing pitch right now. So one of the ways the Cloudflare actually solves this is by making our data centers physically redundant or making our identitators logically redundant and physically indistinct from each other, right? Our Anycast Network, we advertise all of our IPs from everywhere. And that's really helpful because that means that you're not necessarily tied to a specific data center. And even within our data center, you're not tied to specific location, specific sections of physical capacity, right? We logically segment our data centers into groups and we can basically make those fault domains independent of each other. So we can say, hey, if you know, you lose part of Frankfurt, that's okay, you got other parts of Frankfurt. And if for some reason we lose all of Frankfurt, it's like, okay, well you've got Amsterdam. Okay, you've got Munich, you've got, you know, Berlin, you've got all of these things. And the important one, and I think it's on the next slide, but the important one is honestly that every service can run everywhere. And that's really, really important because for a lot of people, the continuity is critical. And like we talked about Quicksilver, which talk which like distributes state, that distributes to every single machine everywhere. So basically every single machine has an understanding of has either has understanding or has access to understanding of what every other machine in the world has access to. So if you lose a machine in, you know, Singapore, that's okay. Another machine in Singapore has the same context and can still serve you properly. And so basically, from a logical perspective, our smallest fault domain is a single machine. We got a lot of those, which is really, really great. And like, we think about that in terms of just like, whatever is like the Futurama Zapp Brannigan reference. I'm gonna save send wave after wave of my own men to solve this problem. It's like, well, I'm gonna send machine after machine to just solve this problem. It's like, I don't care how many fail. Like, they will all be able to do the same thing and solve the problem and and and, you know, meet customer needs. And I think the thing that, like, we've really been focused on in the past five years since I've joined is number one, making sure that our physical network allows for us to segment even further, and basically the scope doesn't go from single machine to the rest of the world, but single machine to maybe a couple machines in a data center, to more machines in a data center, to a rack, to a couple of racks, to, you know, so on and so forth. Think the Anycast is important is an important thing. Just for people maybe who have joined and don't at a higher level on the executive team, explain, you know, the Anycast versus multicast versus just why is that Anycast different and what makes it different, your approach on using that protocol versus multicast scenario? Just just for for those who don't Yeah. I think it's very Well, unless you're unless you're like NBC or CBS, you're probably not using multicast anymore. But like but Anycast is basically the principle that we have IP addresses and we have ways to reach us and we tell everybody everywhere our addresses the same set of addresses and we basically provide the same basically, the war to the to the to everywhere in the world, you can connect to Cloudflare, and it will look the same. And this is really important because That's right. If you're connecting yeah. If you're connecting from, like, let's say, Telstra in Australia, you'll connect to Cloudflare dot com, and it'll give you an IP address. And you connect from Comcast in Seattle, and you'll get an IP address. And those IP addresses are gonna be the same. The paths to get there are going to be determined and that's because in Sydney, we advertise those IP addresses and in, you know, Seattle, we advertise those IP addresses. And those and your ISP, your last mile, just sends it to the closest place, and it just finds the best path. I'm putting best in quotes because the definition is the definition of best is complicated, it finds the best path to you and it will always find the best path to you. So Seattle could go down and you could type in cloudflare.com and then if you're in Seattle, Comcast will then route you to Portland. And then if Portland goes down, then Comcast will route you to San Jose and then Denver. And then, like, you might end up getting routed to Ashburn, but if that happens, then, like, 95% of The United States just went offline for some peculiar reason. Yeah. Like, that means that each location stands alone and can operate relatively independently, which is really, really helpful from a resiliency perspective because in a in a in a different world where each location has a set of addressable space, then that address is authoritative for that space. So in like another for other vendors, what they'll do is you'll go to DNS, you'll go type in cloudflare.com, and then your DNS provider, or what I like to call the Google Maps of the Internet, will basically return you an address that is specific to this location. And if that location goes down or is unavailable, then there is no other location that is advertising that address and you will fail. And you can go ask, you know, DNS for an you can go to DNS and type in, you know, site.com and it'll lead to a different location, but that takes time, right? And that takes you doing something. All of this happens under the hood and end users don't see anything which is And I think one of the one thing I like to bring up with customers is this is what I want to talk about why I brought up FiniCast. You're driving consistency. So, every time you do one off back to like fault tree analysis, it's end to end power of issues that have. Two, three ends to end power of mistakes could be made separate security holes but just compounds the complexity compounds as issues that could arise and bring and make you less resilient. So that's why I wanted to bring up is get rid of the one offs as much as possible, simplifying, create simplicity in it. So, the value of Anycast was what I want to try and get up to clients was that idea of driving that consistency and standardizing, getting rid of complexity within your operations or doing one offs everywhere separate of the whole DNS thing too. That's a whole other conversation. But you said something that was really, really important, and I think that that's that's really helpful. And one of the reasons why Cloudflare, especially for Zero Trust is very, very nice, because like, today, if you're an end user, you have different sites and all of those sites are identified by their IP addresses. Right? And this is actually kind of a problem because IP addresses can change. Right? Like, you know, and I don't know how many people are familiar with the HCP but like, so for example, like, for example, I have like my home server set up so that I can go log in and in order to do that I have to type in the IP address. When my computer restarts, that IP address changes because it connects to my home router and the ISP gives it a different IP address and all of sudden I'm broken because it's the leased model, right? So the leasing model makes it so that's like IP addresses can match. So why are we pinning all of our hopes and dreams on these blocks of 32 bit integers? Right? Like Cloudflare's Anycast Network is very helpful for that because it's like, you're gonna connect to the best path and you're gonna get to Cloudflare as fast as possible, and then the Internet is programmable from there. That, like, once you get here, we can figure out where the best path is to get you where you need to go as opposed to just putting in an IP address and having that be the the road map. And it it smooths over those those one offs that you're talking about. Yep. So we've got ninety seconds left. Any final thoughts? Any feelings? Anything you wanna you wanna share with the the group of people who is watching this? No, I think I kept on bringing it up and why, you know, I was excited to do this webinar with you. It's just something an area resiliency, especially with the network is such a part that I've been working with a lot of clients on the importance behind it. So, I'm glad we had a chance to do this and kind of dig into the various elements for clients to think about when it comes to our network and importance behind it. So, you know, more than just being up or down, you know, latency, bandwidth, how you do updates, documentation, you know, it's like you said, it's an onion, there's multiple things of why when you're looking for a provider, you're setting up your network, when you're architecting, you know, it's not one simple path to make that happen. But, you know, there's some right approaches behind it. Yeah. I love that. Well, it was amazing doing this with you. I had a great time. Yeah. Me too. I hope you had a great time. I hope I hope the viewers had a great time. It's great. They'll have an interesting time. I can't promise a good time, but I'll promise an interesting time. Well, thank you so much for watching. It's been great having you here, Andre. And it's been great having you on the other side of the screen watching this and see you next time. Yeah. Thank you, everyone. Cheers.