Video: Less Than a Day to Defend: Preparing for the Next Critical Vulnerability | Duration: 4860s | Summary: Less Than a Day to Defend: Preparing for the Next Critical Vulnerability | Chapters: Welcome & Introduction (75.139995s), Threat Landscape Overview (161.209995s), Attack Statistics Rising (250.289995s), Critical Vulnerabilities (418.06999499999995s), PHP Vulnerability Exploits (493.909995s), Instant Mitigation Strategies (646.659995s), WAF Protection Mechanisms (729.12s), Rule Creation Process (1015.72s), WAF Response Detection (1097.789995s), Faster Patching with AI (1255.409995s), Sustainable Vulnerability Management (1554.609995s)
Transcript for "Less Than a Day to Defend: Preparing for the Next Critical Vulnerability": Welcome everybody to this Cloudflare webinar, and thank you very much for joining us today wherever you're dialing in from. We... I'm delighted to welcome our chief security officer, Grant Bozicis, and, Daniele Montane, who is our director of security products based out of our London office. And they're gonna talk you through a whole range of topics that are very timely around, vulnerabilities, around the the the window to respond, shrinking, around what you should do to buy some time when a critical vulnerability hits, and then, you know, how to fix things without losing control. So we're looking forward very much to the content today. If you got any questions, put them in, in in chat. Please keep us, keep us entertained as we're as we're working through this incredibly important topic, and tell us what's top of mind and how you think about it. There will be, plenty of opportunity for our online team to respond, and I hope you enjoy, and get something out of the next, the next little while. So without a doubt, I'll come back in at the end and wrap it all up. But, Grant, do you wanna take it away? I will take it away. Thank you, Ben. Hello. Good morning. Good afternoon. Good evening. Maybe even... You know? Hello in the middle of the night. I'm Grant Borzicus, CloudFlare's chief security officer. So today, we're gonna talk about more abilities that's been said. And and I think the title of less than a day to defend is very relevant with what's going on, and we're gonna talk about that. So there are four things we're gonna talk about. Kind of the current threat landscape, the recent vulnerabilities, how WAF is really helping drive things, with what's going on, our web application firewall, and and how do we think about patching. That that is the topic, that everybody is going through today. So throughout landscape, what's changed? Why do we know so much? I think it's always the first question I think about. And so I always think that this is... It's relevant with what CloudFlare is because of the telemetry that we see. Right now, we see over 20% of the Internet traffic, every day, and it's pretty substantial. We see almost 10,000,000,000 DNS queries a day, HTTPS queries a day, and we're looking at 310,000,000,000 with a b attacks a day. So probably if I think that we've been talking for just, you know, probably about five minutes, we've already seen a billion attacks just in the time that that we've been on this webinar. And so when we think about this, I think this is a really interesting place to say, you know, hey, Grant. What is the formatory at Cloudflare telling us, and what kind of insights can you give us to really drive what's there? So like we said, 310,000,000,000 attacks is up 60% from a year ago, and it's three x the number from two years ago. Right? And and I think that that alone is a huge number. The other one, which I think is is very relevant, which is on the right hand side, which is the, 60% of all Internet traffic is bots. And and I think that is is very relevant. And when you think about the increase in attacks, the increase in bots, which is being driven from AI, we're now seeing that one in every four, ex... You know, vulnerabilities are being exploited, in the wild. And and I think this is this is such an interesting step that before there were vulnerabilities, but they were never really being exploited. But we're really seeing this really change the way that things are being done today. And so when I think about this, well, what has changed in this year from just last year? You know, we're almost at two times the amount of CVEs that that are... Or have been released in 2026 than we are in 2025. So by, you know, going into the the third quarter like we are, you know, kind of the middle of the third... Our fourth quarter today, we're gonna be, you know, more than two x that number. And and what we're seeing, right, is the advent of Mythos, the Daybreak models, the Frontier models, Researchers are are using testing harnesses like the the one we've talked a lot about in the four blogs that we've had with our Mythos experiment. But we also have a red team harness that we're using to to to look for vulnerabilities as well, you know, which would mean from a vendor standpoint, we're creating more vulnerabilities. Researchers are trying to find more vulnerabilities, and security teams are finding more vulnerabilities. Right? And that's all putting tremendous pressure, as I would say, you know, down the pipeline of the rest of the organizations when we find them more early. Now this statistic is even more interested... Interesting. I think we've always looked at, if we kinda go back a few years ago, that things are getting exploited faster. Right? And and, you know, it used to be it would take sixty three days. You know, back probably when I started, it was two or three hundred days that we would see the CVE from exploit. But what it's looking like today, on average, exploited, vulnerabilities are seven days before the patch even exists. Right? And so when we think about this, it's how do I defend my organization if there's no patch? Right? It... It's a different way of thinking about architecture. It's a different way of thinking about patching and mitigation versus remediation. And this is occurring because the attackers are using some of the same great tools that we have, but they're using it to their advantage to to find, holes in the organization to really drive, vulnerabilities and and drive more breaches. And I think that's that's the fundamental thing that is occurring in the organization. So let... Let's, you know... I think that's good from a broad landscape. Right? Like, we're seeing more abilities just significantly faster. So many, so fast. But, like, okay, Grant. How about what what... What's real in today? And so if we just look at the last five weeks, let's just go back to the week of October 1, which everybody loves the fall, East Of The United States because we all get the pumpkin things. But there have been 38 critical vulnerabilities, that have been exploited. Right? We'll talk about, PHP, but... And and some of the coding things. But when we look at this, right, 38 vulnerabilities in five weeks, actually, you have to fix or you'll get exploited. That... That's different. Right? I remember conversations when I was... Earlier in my career, hey. There's a vulnerability. Well, you can't actually exploit. It's too hard to exploit. That's just fake. Well, here we are, you know, maybe twenty years later, and we're seeing, you know, CVEs get exploited in real time, and and and that's the real challenge. And let's kinda zoom in on one because I think this is the one we... We've heard quite a bit about, this PHP vulnerability. There's two CVEs. You know, 70% of all websites are running PHP. The vulnerability was a remote code execution, and it's been there since 2012. So fourteen years, this vulnerability has existed. Right? And there's not a whole lot you can do at the time that was there, and we'll talk about what are the things there. And and and it's... It affected every organization. Right? When when this landed, it was... We were taking phone calls about what we were seeing and how to fix it and what was the vulnerability and how to patch it. Right? And and I think when we we get to that, it... That's very relevant. Now the reality of this was that before it was publicly released, we actually saw half million attempts to attack servers. A half million. Right? So that's not a small number. And and so when we think about this from, you know, kind of the eleven hours prior to, you know, why... What what happened? Was it the responsible disclosure program? Was it working? Did somebody tip it off? Were attackers finding new things? Because the way this vulnerability worked was at the time it was announced, you know, the CVE, you... The the, you know, a patch was released. And so that should have given us enough time to be able to patch our systems and be secure. But that's not what happened. Right? What actually happened was at the time that it was released and, you know, if we go out thirty hours from the time it was released, we saw over 21,000,000 attacks. And so that's fascinating. So just in a little over a day, people had taken exploits across the Internet in mass scale launching attacks. Right? So that means you you better be prepared at, you know, zero hours to be able to defend your organization. Well, that makes things significantly more difficult to do. Right? Because it kinda leaves us in two spots. Like, how do we have instant mitigation? Right? We wanna talk about that. And then how do we patch faster? And and I think as we we go through this, these are really the two things, and I always think, how do I put some sort of mitigation in place to buy me time to fix it? Right? Because if I tell any organization, if I'm in your organization and say, hey, CDE, drop everything and you got a patch. Well, is that gonna help, Grant? Well, maybe not because we may have been already beginning attack from this and we didn't even know. And so I think we're gonna explore both of these components about some things that we're doing internally with our web application firewall, and Daniela is gonna come on. And then I'm gonna kinda come back and talk about some of the things that we're doing from our standpoint to help mitigate this. So, Daniele, let's talk about how the WAF is helping drive massive change in your security posture. Great. Thank you very much, Grant. Hello, everyone. I'm Daniele. I lead our WAF and application security here at Cloudflare. And I've been working on the WAF since I joined Cloudflare more than six years ago. And I think this is the first time really where, there is so much pressure into creating protections and detections for new vulnerabilities, which, the WAF is actually, why it was created in the first place. So WAF is a perfect tool to, detect those vulnerabilities, those attacks and exploits early on, but, of course, it's not, it's not the solution. So the real solution, of course, is for you to patch, software and and your stack, but what WAF gives you is time. Right? It's time to get that patch in place and assess the impact on your on your infrastructure. So let's dig a little bit deeper into how WAF works and also how, we created the protection for the PHP vulnerability as an example. So this is a a, very short primer for for WAF. So WAF is essentially a filter. Right? The seats between the Internet, visitors like your normal users, but also attackers or agents and bots, and, your application on your server. And what WAF does is to inspect every HTTP request, look at the structure of the request, look at, metadata and and and signals, the the body, the header, the IP, where it comes from, whether there are also authentication headers or tokens, and then decide based on the request whether to allow or block the request or challenge the request. So in this example, here you see there are some some legitimate requests that's being sent to the origin that get through. But then when there is a, for example, a post request with an exploit, that's, where the WAF can identify the exploit and block that request from reaching the origin. And so helping, the the attacker or actually the the owner of the application to block that attack, until they can patch the application. So that exploit doesn't work anymore. So this is how our WAF really works from a high eye, level perspective. If we dig a little bit deeper, WAF has evolved over the years, by adding a number of different products and solutions to block malicious, or malformed requests. So it originally started as a collection of regexes that just look for specific signatures of attacks and expand it to add additional tools like machine learning or AI. We call it attack score, within within Cloudflare, which really helps detecting bypasses and variations of attacks by using machine learning model train on attack data. We also, some customers deploy, policy security filters, which rely on profiling your traffic, understanding the structure of your request, what type of fields you expect. Let's say you expect a field in the query, which is an integer. And then if you see a string, you block that because you don't expect, a string in that field and so on. So these additional filters help reducing the tax surface area so you can focus only on the, the part of the request where really the critical exploits can be can be sent. And, of course, at the... On top of that, you can also deploy other controls like rate limiting to protect your origin from from, from, volumetric attacks. So when a new vulnerability is like, PHP to shell, comes comes out, there are really a number of ways for us to react very quickly and before attackers can exploit it. And in this specific instance, different things came together. Right? So the first one is the... Our relationship with other partners, which includes software vendors like the PHP Foundation, but also researchers and other people in the security community. So for the PHP, we worked, with OpenAI, for example, and Wiz. And and together, we we, started coordinating weeks before the the official, release, the official advisory. That gives us time to really understand the exploit and create those protections. On top of that, we also add our own threat intelligence. So as Grant said, a cloud fair will block more than 300,000,000,000 threats every day, And that information really gives us an advantage to, map, basically, what's happening on the Internet, identifying trends, identifying new exploits. And this information can can be fed back into the WAF and creating protections against those rising, attacks. And then the third component is, of course, our own analysis. So using security researchers to analyze the impact of rules and detections on on live traffic and identify whether a rule works, whether it has low false positives, which means it doesn't affect weird traffic, and hide true positive... True positive, which means identify the path very specific... In a very specific way. And, of course... Sorry. Our security our security researcher... Excuse me. And, of course, our security researcher use AI and MLMs to create rules themselves. And these rules get, released on our network, across all our networks and globally, gradually for all our point of presence. So this actually is how we created, new protections and rules for PHP eleven hours ahead of the public disclosure. And this also would give us the time to block attacks and to monitor how the attacks evolved, while there was an active leak of that vulnerability on the Internet and where massive, exploitation was happening just hours before the actual, advisory was was public. So, one extra thing we have to build, though, for PHP. So traditional WAF looks at the request alone. So it looks whether there is a fingerprint or a pattern of an attack in, in some part of the body or the header. For PHP, the... Looking at the request alone was not enough. It was not enough. It was just giving us an indication whether there was an attack, but wasn't giving us the full picture whether the attack was successful. So we had to build a completely new feature in the WAF, which is the response, detection. So, after PHP, we could actually correlate request and response to detect whether an attempted exploit was actually leading to memory corruption and remote code execution on the server side. So using response allows us to be more precise, reduce, false positives, and really detect whether an attack is successful and block that before information are leaked to the attacker that then can be exploited further. Of course, attackers are evolving their techniques, and so they're using AI and NLM to get around our rules and protection. So this is the, this is the game. Right? It's a cat cat and mouse game. So we create new rules, new protection, and attackers try to evade them by, obfuscating their their attacks by changing their techniques. And so our response doesn't end here. So we have to, follow-up. We have to chase the new attack and the new variations. And for these reasons, we, of course, we're updating our rules, but also we launched, capture the flag challenge, which basically ask our researchers out there, our community out there to, identify gaps in our WAF and then submit those those, findings to to, and then we can update the rules accordingly. We have seen an an incredible response from the white hacker community, lots of good submissions, and, of course, this improved... Helped us improve the rule. So, really, this is an example of how, companies and partners, but also individuals out there that are aligned with the same goal of protecting the Internet, they come together and they help each other to get, to be one step ahead of attackers. Great. So, again, WAF is a great way to detect attacks, early and give more time to the, application owner to patch their their their software. And so now I'm going, handing back to Grant to talk about how we patch our Cloudflare, for PHP. Thank you. Thank you, Daniele. Hey, everyone. Grant back here. So let's talk about faster patching. And and let's actually talk about... Maybe I always like to talk about practitioner things of, like, how we actually do it and how we're thinking about it because our processes evolve, especially with our use of AI. If you watched my podcast, for webinar a couple... Probably about a month ago, we talked about some of the use cases around the AI, and this is actually one of them. We're gonna get into it of how we think about it and maybe there's some ways you can think about it as well. So, traditionally, right, the vulnerability process of today is relatively simple. We scan, assess, kinda, you know, propose, you know, what the patch is, we approve it, and ship it. Nothing's really changed since the beginning of of patching that's ever existed. But I think today, with what AI can do, which is really good at summarizing data, aggregating data, understanding data, driven from a human is a really good place to be. And so so what do we think about, or how do we think about it? So a couple really interesting things, because we have the same problems everyone has. Like, okay. And and I think we built our vulnerability harness with Mythos. We have our atomic red team harness that we're using Daybreak and various other models. We've even released a a platform and our call force one that we can scan for your own vulnerabilities and your source code and test and deploy a last rule. But, usually, the first question is, like, well, am I protected? First question. Right? Am I protected? And so we have a new, component called zero wire on CloudFlare radar that will tell you what, you know, the current status of the the large vulnerabilities. And I I think that's really cool. But the most important thing that I I I, you know, rebuilt was the question of, and we have all this telemetry. What vulnerabilities are actually being exploited? And this was something that my team asked for. We asked the Cloudforce one team, which was my other team, and we built something. And and I think this is really interesting, to to what actually occurred is is how did that that really drive the change with what's there. And I'll show you that here in a second. The the second piece is around assessing. Right? Like, where is it? Where is it in the ecosystem? And how critical is this vulnerability? How fast do we need to patch? Right? That becomes very relevant of what our current assessment is because that's what's really driving the change to this. This is the vulnerability intelligence slide that we talked about, and it will show you. Right? Once I understand the assessment, it's a critical vulnerability. It's exploitable on the Internet. You know? I don't have protection because it's a zero day. Is it being exploited? Right? This this this is, to me, something every organization should use. They should have an agent that checks every one of their vulnerabilities and pulls down, is this being exploited today? Right? And and I think from that standpoint, you you have all the tools in place to really be able to drive it. I understand kinda, you know, do I have protections? You know, is that vulnerability? Where is that vulnerability located? And is it being exploited? Or is it not being exploited? Do people not know about this vulnerability from a CVE standpoint that's moving forward? And to me, that really changes the game from a vulnerability management. Because in every organization, every board, every executive has asked me, you know, vulnerability and write in the newspaper, you know, is it being exploited? Where is it being exploited? What are the industries that it's being explored? And you can see right here. Right? This is actual screenshot of that information, and that really helps into drive, you know, the conversation forward. So the next thing we do, and and this is all automated, right, is we actually use AI to draft what the patch looks like, you know, make proposed changes. We we generate a laugh rule, and we test it. And so this is all occurring in real time. And so from the time we scan, assess, pull the the relevant CV information off of of... From a from a telemetry standpoint and actually get ready to make the change. This happens in minutes. Right? And so, you know, previous time, we had all our meetings and we had all these... This data aggregation, but the technology that we built is just aggregating. Right? And and this is something that we we did. I know from a WAF rule, and you can always try it, is is pull down a CVE and say, hey. You know, create this into a a a call for our WAF rule. It works. Right? And takes minutes. And so as you think about this, you know, rethink about how the work is there because I think that will really change your vulnerability management program. The the one thing that we do, and I talk a lot about this to every customer is we still have humans in the middle. The humans are driving our process. There's explicit approvals from humans. And so every decision, we're being really driven from a human. And and in this case, it's the same thing. Right? We we make the finding, we look at the fix, we look at the fallback plan, and then we approve it. Right? And I and I think in that case, that that really drives the request for the change to move things into kind of the order of of fixing. And then it shipped, tracked, and we confirm it and test. And and, again, that's done with an agent. That that takes away from a a lot of the mundane work that's there, but it's really because of the intelligence. Right? And so I think as we we think about this, you know, having that intelligence is a big piece. I think the the second piece is really around what is your process to look to patch and fix and making investments in that. That's something that we've done substantially. We need to be able to roll this out in real time based on what the the vulnerability looks like, based on the exploitability of the vulnerability, based on the current threat landscape of that vulnerability. And and that really changes the ecosystem where humans are using our brains to make intelligent decisions about what we need to do. And it's grounded in real intelligence data, and I think that's super powerful. And and I think what does what does that mean from kind of a metric standpoint? Right? We're still approving this. The person's still doing it. You know, we're we're at 98.8% of all of our SLAs, all the SLAs that all of our customers audit us against, we're hitting these. And for anything that is critical, high, we're running incidents to make sure that this is done immediately. And, you know, I think even from some of the bug bounty vulnerabilities that we've seen In in almost all cases, it's less than twenty four days to be... Or less than twenty four hours to be fixed. Right? We're actually seeing... I saw one the other day that it came in. We had our triage process through our bug bounty and validation, and the engineering team created the fix in four hours. Right? And then we began to roll it out. I I think that stuff that we think about, how how do we deploy our ecosystem, do it safely, do it securely, and make sure that that's really changing the the way that we operate with with today. Because, you know, I I I would say that this vulnerability management problem is not gonna go away. It's only gonna get worse. We already know it's gonna get worse. We've already been part of conversations with many of the frontier model companies that are finding these vulnerabilities. And so it's important to think about how do I build a sustainable process driven through automation, driven with automate... Driven through intelligence to get us where we need to be. And and I think, you know, the the things that I would take away from this, right, is one is... And it's what Daniel and I just talked about is you you have to have some sort of level of instant mitigation into the organization because you will not be able to patch in an hour or four hours or eight hours or twenty four hours. You have to have something that helps mitigate that risk today while you can get the organization moving to be able to fix. Now the days of, you know, patching in thirty days and sixty days and ninety days or quarterly or semiannually or annually or over, You think about how do we deploy images, how do we use serverless technology, how do we deploy, you know, you know, cattle versus pets, I think is a a very relevant thing because when we look at 310,000,000,000 attacks, up 60% from where we were last year, you know, the average time to exploit from a patch is less than... Is is seven days prior to a past being released, and we've seen almost 40 vulnerabilities in the last five weeks. We have to stay on top of this, and we have to make sure that we're ready. Right? And so when we think about just some of the amazing tools that we've created, right, with our web application firewall, here we do instant mitigation, You know, our new platform for threat events, I think, is is very relevant to what we're doing, and then our zero wire that's giving you insight to what actually we're blocking. So I think, you know, the the takeaway is look at your ecosystem, let us know what we can do to help, and and, you know, you know, really put some effort into making sure that this is in a really good place. That is all we have for today. I thank you for all of your, support. And if there's any questions, please put them in the chat window, and we'll get back with you. Thank you. Thank you very much, Grant, and thank you, Daniele. Hopefully, you all found that useful. Lots of opportunity there to get in touch with Cloudflare, to take advantage of our programs like Zero Wire. We are open to, helping you. Our mission at Cloudflow is to help build a better Internet, and that is one that is more secure. We will distribute slides that you saw. We will distribute a recording of this. And, I will say a quick plug for some work that Grant's team does around the world. We we we run AgenTic security workshops with senior security practitioners. We try and find people who are interested. And if you are interested, please get in touch, put a comment in the chat, and we will see what we can do about, about organizing something with you and, and your teams, wherever that might be. In the meantime, I hope you found this useful, and, we will see you on the next session. Thank you very much.