Name: Building Real-Time Voice AI for Customer Support
Uploaded: 2026-04-23T09:46:07.073Z
Duration: 1 h 13 min 4 s
Description: Building Real-Time Voice AI for Customer Support

Transcript for "Building Real-Time Voice AI for Customer Support": Hey, everyone. Nice to see you all. Hey, Musharraf. Hi. How are you? So yeah. Hello, everyone. So welcome to today's session. Really excited to have you all here. Also, this is my first one, so, wish me luck. And, yeah, lots of friends. Hey, Tim. How are you? Yeah. Let me know in your comments, like, where are you from? I'm from India, by the way. I work as a senior developer educator at Cloudflare. So let me quickly share my screen and show you what exactly we are working with today. Okay. If you go to the, doc section on your top right of the, stream, you should see a doc's tab. Click on it, and it see you should see a live demo of the agent option. So if you open that and if you just, like, append, slash hashtag slash workshop, you should be able to follow the slides along with me. And I just wanted to make sure that everyone's on the same page as me. And yeah. So today's topic is building voice agents with Cloudplay SDK. And quickly keep this. Yeah. And today, we are building, like, something fun together, a real time voice agent that you can actually talk to, and it'll talk back. And this is not a simple chatbot, that you have with your microphone on. So it's not just like adding a voice layer. Instead, it's a proper voice pipeline that does speech to text, and and LLM will make tool calls, into it. So it's like a proper agent to be agentic, voice agent. And it's all running on top of CloudFlare workers. So yeah. So what exactly are we building? So it's a customer support agent for a fictional ecommerce site called Acme Incorporated. If you have watched a lot of Looney Tunes, or the Roadrunner, you'll get the reference for it. So users can speak into the microphone. The agent will transcribe it in real time using a single WebSocket connection. And it'll call the LLM. So while this q and a and chat chat, that's happening. If the user invokes any intent, let's say he he likes to look up an order or let's say he wants to initiate a refund or he's just looking for a new product if they have available in their inventory. And, yeah, that's exactly what we'll be building today. I'll show you a demo in a quick minute. So what you'll learn, during this workshop. So we'll see how the new Cloudflare's voice SDK and agents SDK work, It's again built on top of durable objects. A lot of durable objects a lot of Cloudflare primitives and services are already built on top of durable objects. Again, this agents SDK too is also built on top of it. And we'll also see how the new Cloudflare slash voice, SDK works. And there's a lot of new features, involved in it. I'll show you I mean, I'll go we'll go through all of them one by one. And we'll also see how we can buy it up our tool calls within our AI agent, using ourselves, AI SDK. And we'll all use I mean, use all of these inside a worker plus a a simple React application. And I'll show you a quick demo of how this works. Yes. So I have the, agent open here, and I'll just talk to it. Hi there. Welcome to Acme support. I can help you track an order start, a return, or check if a product is in stock. What can I do for you? Hey. How are you? I'm doing great. Thanks for asking. How can I help you, TA? Do you have any notebooks available? Yes. We do. We have the classic line notebook by paper valve in stock for $26.79. Would you like anything else? Can you look up my order 9831? Can you look up my order? 9831. Sorry. I couldn't process that. Could you try again? Hi there. Welcome to Acme support. Let me restart it. Hey. Welcome to Acme can you look up order 9831? I found your order order 9831. It's been delivered and includes two pairs of suede Chelsea boots and three daily hydrating moisturizers. Is there anything else I can help you with? Can you initiate a refund for this? I can certainly help you with that. Could you please let me know the reason for the return? Yeah. The boots suck, and they don't fit me. All set. I've started that return for you. You should receive a prepaid return label in your email. Just make sure to ship the items back within fourteen days, and you'll see your refund in three to five business days. Thank you. That's all. You're very welcome. Thanks for reaching out to Acme Inc. Have a great day. Yeah. So you saw how the agent was able to do small talk. It was able to handle interruptions. It will also able to call external services like looking up inventory. It also looked up my order, and it also looked up, it also initiated a refund for me. So you can do all of this, with this new Cloudflare slash voice SDK. So these are the docs. You should be able to you I mean, see this, if you open. Yeah. On the top right corner, you should see the docs doc section, and you should be able to see, the documentation link for it. So here's a quick overview. So Cloudflare Voice is actually a set of abstractions that our team has built very carefully, with the right amount of abstraction level, but it it has also a lot of granular flexibility as well. So we have this method called with voice, which is a full, speech to text, pipeline. So with voice input, it's just transcription without response, so it will not respond back. There's also React hooks for if you're using it in React, there's abstractions for that too. So all you have to do is just, like, install it with Cloudflare slash once and agents. Okay. And before we begin, I mean, I want us to build this along. So there are a few pre prerequisites. First of all, you need a Cloudflare account. So make sure you sign up at -.cloudflare.com and everything is ready. You also need workers AI. This is included in the free plan itself, so there's no extra setup. And make sure you have Node. Js installed. So you can just do a check with node minus v or hyphen hyphen version and also PNPM or NPM, whichever you prefer. Make sure you're also logged into Wrangler so that you can deploy and use, I mean, connect to workers AI locally and also a microphone. If you are on a Mac mini or a CPU, make sure you have a microphone ready. So, yeah, before we begin, let's see how the application is working. So here's a quick little diagram. Actually, Claude came up with this. Once a user clicks the green call button, a WebSocket opens. I mean, the React app will initiate a WebSocket connection to the durable object. And you saw that it greeted us, as the first response. That's the durable I mean, that was set up in our durable object. So this is what happens when the loop starts, I mean, the agent loop. So then comes, what happens if the user speaks. So the CloudFlare I mean, voice SDK will automatically detect that the user is speaking, and it will not transcribe it in the client. Instead, it directly sends the binary, data over an over a WebSocket. So the user speaks, React SPA will send the audio stream as binary files binary chunks, and it does not wait for you to complete I mean, wait for the user to complete the entire sentence. Instead, it sends it in real time. So the durable object and workers' AI, they will work on transcribing it, and they'll also use the LLM to see if there are any tools invoked by the user. For example, we have intents for lookup order, return request or a refund request, and the check product. The check product one actually looks for the products in an inventory. I had actually created a small little fake mock CRMAPI. I mean, nothing complex. It just returns a bunch of JSON objects. So this way, we can simulate the CRM. So we this was just to showcase how you can make tool invocations from your agent in real time. So yeah. It just has, I'm just using two of them, slash products and slash orders. It just returns a bunch of order data. For example, yep, nothing special. So this was just for the sake of this demo. If you are interested in this, it's just fixers dot files dot workers dot dev. You can use this as well. And, yes. So this is like the I mean, we have five steps in total. Like, the tool call lookup order. I mean, it'll return the JSON result, and the tool result is then again sent back to workers.ai so that it will create a response as voice. And then the final response is sent back as an audio stream to the agent. And if, the user hangs up, the WebSocket will just close. So, the CloudFlare team has actually made very nice abstractions for all of this, so you don't need to handle any of this yourself. All of this will be handled by the durable object and worker CI. Okay. Let's learn a little bit about Cloudflare slash voice SDK. So it's still, in beta and experimental stage, and the voice pipeline is built on top of agents SDK. It has a few methods for you to use. For example, with voice, it's a method which will do speech to text plus LLM plus text to speech, including persistence, using a durable object, via SQLite storage. And, yeah, we also have with voice input. It just, like, takes in the audio and creates a respond, but it does not return back the voice. And there are React hooks as well. So use voice input and voice client. So, I mean, voice client will work everywhere, but use voice input just works for you. And there are also a few built in workers AI providers. For example, we have deep I mean, we defaulted to Deepgram slash flux. So if you use this method, workers AI flux speech to text method, it'll call this one. So We have workers AI catalog where this will be picked from. You also don't need API keys because since all of the application plus the SDK is built within the CloudFlare ecosystem, It uses the binding convention. No need of entering any API case. It'll be automatically picked up. We also have a default worker CI Noa model, again, from Deepgram. This will work. But you're not limited to just these models. You can actually use ElevenLabs or any other model from Deepgram as well. So, yeah, you are free to use any model. And all of this works within a single WebSocket connection. So There there are no multiple WebSocket connections for different things involved. Persistence is built in, using the durable object SQLite storage. Yeah. All of the streaming, happens, through throughout your session. And, yeah, I just want to, like, quickly start building the application. So, let's start by scaffolding a new project. So if you use p m k p n p m create CloudFlare at latest, so this is the c three CLI. Let's give it a name, and the framework is react since that work since that's what we are using. I'll just use I'll head over to my folder. Okay. I'll run this command. I'll just name this as voice works voice agent workshop slash live. This will actually scaffold everything for you, including the Wrangler file and some, default bindings. So we'll obviously select TypeScript, and it will also install all the dependencies as well as, it will also set up an agents dot MD file for you. It should prompt at any time. Do you do you want to use git? Yes, of course. Deploy? No. We'll deploy it afterwards. Okay. Let's open the project. I'll open it in my favorite ID. That's WinSCP. And there we go. It has scaffolded a bunch of files for us. I mean, the c three CLI. So we have wrangler dot json c file. Json c is just like JSON with comments. So we have I'll explain what all the files and these bindings are. Okay. The main is the entry point for the worker. So where exactly it need open from? Sorry sorry to disturb you. You're not seeing the right screen. Just wanted to let you know. Okay. Okay. Let me do that. I think I shared a tab. Let me just share the entire screen itself. Okay. I hope this is visible now. This is gonna zoom in a bit so that it's visible. Yeah. Awesome. Thank you. Yeah. Thank you. Thanks, Marshall. So yes. I actually ran the c three CLI command in my terminal. P n p m create CloudFlare at latest. I gave it a name, voice agent workshop live, and the framework is React. And it prompted me a few things. For example, if I wanted to use Git, I selected that. I also selected TypeScript. So if you're following along, make sure you select, TypeScript as well. And then we don't want to deploy, right now. We'll do this at the end. So, yeah, there we go. If you are setting it up, just run p n p m create CloudFlare at latest. Give it give your project a name and select the framework as react. And once you do that, let's open it up in our IDE. So it scaffolded a bunch of files for me. One obvious one is wrangler. Jsonc. This is the configuration for our CloudFlare Worker project. So it has a few things. For example, it has observability binding enabled by default, and it also has mean, which is the entry point for the worker, and it also has a name. So whatever name you insert here, so once you deploy, CloudFlare will automatically pick this up while deploying your agent. And the, we can just remove this, actually. So if I head back, to our slide now, let's install a few dependencies. The important ones over here are CloudFlare slash Voice, the new SDK CloudFlare has released the agents SDK. We also need the Versal AI SDK and the Workers AI provider, as well. Workers AI provider is the interface for working, CloudFlare Workers AI, with AI SDK from Ocel. And we also need ZOD since it's a peer dependency for AI SDK. Let me just copy this. We are on page number five. So if you're following along, copy the second command and paste. Okay. There we go. It installed all five dependencies for me. And for the UI bit, I am actually using, Tailwind CSS, with a CDN just to keep it simple. I'll just copy this and paste it in my index dot HTML file. There we go. Okay. Let's move on to the next one. We already went through the Wrangler configuration, but we still need a few bindings to be added. For example, one important one we need is the AI binding. So since workers works, with triangular configuration I mean, all CloudFlare services are accessible using bindings. You can still access it with rest APIs, but you'll need to set up your environment variables, etcetera. So j bindings. So okay. Let's add AI, the AI binding. Double quotes AI. And we can name I will give this a name. If you can add anything, you can also add voice AI. So yeah. Then we'll need two more things. We need a durable object so, our voice agent can process the conversations. Let's add that as well. Durable object. The I mean, auto complete. And yes, let's give this a name again. VoiceAI. And we also need a class name. Let's just, follow the slide. Agent. Yeah. This should be an object. And we'll also need a migration binding too, yeah, when serve auto completed. And, yeah, this is what all the bindings do. So the AI binding, will allow us to use workers' AI, And durable objects will create a new class of SQLite storage, which will also create the WebSocket connection as well as, the storage persistence. And then we'll obviously need migration, so the conversation I mean, the tables get created, etcetera. Okay. Let's head over to the next step. And, yeah, let's actually start building the app app itself. Okay. If we see look at this. So, the app that's, Wrangler scaffolded for us I mean, the create, CloudFlare command. So we have a worker, and index. So this is the serverless function that'll be invoked, by worker. And we also have a source where we have a main dot t x x, and an app dot t x x. So this is a typical React application. So within, index dot t s, let's actually open up our app. Okay. I have the references open over here. Okay. So let's begin, by importing a few things. We need the agent met class, from agents SDK and the route agent request method. So let's import these few things. Agent and route agent from agents. And then we'll also need to import a few, methods so the pipeline will work. We need with voice. And for the AI voice model, we are using, Deepgram Flux. And then we have the text to speech, worker AI method as well. So let's import these methods. And, yes, this is the interface I mean, the type interface of how our AI method will look like. Let's copy this too. And, yes, we'll need to, I mean, construct the agent class with this base voice agent. Let's just initialize this as well. Okay. Now for the voice agent class, what we can do is, it's we need to create a transcriber so that whenever someone speaks, it it'll automatically send that, binary files over the WebSocket connection. And then we'll also need text to speech as well. So whenever, the voice returns back, I mean, whenever the user speaks, we'll con I mean, transcribe text to speech. So and we'll we'll to connect it to our workers' AI, so this is how the binding will work, this dot en v dot a I. If you look at the Wrangler configuration file, yeah, this is the binding name that we have, and this is how we will access it. This .env.ai. So yes. Let's just create a new class. I'll just comment out the boilerplate, the scaffold created for us. Export Yes. Voice agent. Yes. And the voice agent class is extending the base voice agent, and we're also, passing in the type e n v, which we have right over here. So all of the bindings from workers I mean, Wrangler will be available throughout this class. Yes. We don't need a construction for now, constructor. Okay. Let's do let's create a transcriber. We don't need the types. New actually, let me just copy this so that it won't take a lot of time. And then we'll also need the TTS as well. Okay. Now, we need to also we let's create a few classes for when the call starts, and, we'll construct, the connection. And we'll also need to start the turn server by using a method called on turn. So if we just copy this, and then we'll just return, the agent through this method which we just imported from the agent SDK, the route agent request. So let's add that as well. And if no route is found, it will just return a four zero four. So this is the simplest base we have right now for our voice agent. Yeah. And, also, like, what's happening? That voice is a method that will create the entire pipeline, And voice AI, Flux STT is the speech to text transcriber. And workers' AI TTS will convert the audio sorry. Text to audio. On call start and on turn are the methods provided, by the new voice SDK. Route agent request, it just routes the durable object, and the default worker is an entry point. So this is where we are returning our durable object. Okay. Now that we have the base ready, let's actually start, including tools and, LLM for I mean, for the conversation. We already have their agent and route agent request. Next, we need to import a few things from the Versail's AI SDK. We need to generate text, and we'll also need to use the workers' AI provider since we are using workers AI, through ourselves AI SDK. Let's just import these. Okay. So this is where most of the I mean, work is happening inside the on turn method. I mean, that's where the text gets generated. And for tool invocation and conversation, I'm using Gemma four. You are happy to use you're open to use any model, from Workers AI platform. Okay. Let's create a Workers AI constructor. So inside on turn, I'll just come and do log. I'll create a new method called AI, and we'll use the standard generate text method. I'll explain what all of this does in a minute. Okay. We'll also change the system prompt as well, later on. Okay. We are I mean, we are passing in the model that we want to use. It's using the AI, I mean, workers AI model, cf slash gemma. So there are a lot of models in a workers AI catalog. Make sure you go and check it out and see whichever fits your use case and, your budget. So we have a very default, system prompt right now. So it just says your helpful voice assistant, which is fine. And then we are also passing all the messages the user and the assistant are talking through. So this is how we just append whatever's in. So user and assistant, these are the default this is the default payload, for creating and constructing your AI chat class. Yeah. And then, we just have a default return method. That's fine. Okay. So, yeah, basically, I mean, this slide also says the same thing. We just imported, generate text, and create workers AI, methods we which we had installed as dependencies. And then we also have I mean, we created a create workers AI class and constructed it with the Vercel AI SDK. And then we are, like, passing in all the messages through our sales AI SDK. Okay. Now, I have constructed the system prompt a bit different. I mean, I've, added a few very specific things inside the system prompt, and I'll tell you why I reached to these decisions. For example, the model kept, picking up numbers as strings. For example, if I kept, seeing it one two three four, it was literally writing one as o n e as string, two as string. So I had to add a very strict rule inside the system prompt itself to convert it as four digit numbers. I mean, we can change it to, like, however your order ID format looks like. So yeah. The system prompt says, you're a friendly customer support voice agent for this Acme Incorporated online store, and you will have tools to look up for orders, start return. So these are the tools that I have in my voice agent. So I have just returned the system prompt. We'll come to the, I mean, creating tools part a bit later. So, yeah, this is where, the order ID fixed that I have added. And sometimes it was also returning markdown and the voice, speech to text sorry. Text to speech transcribing kept actually outputting the markdown itself by using voice. For example, if it I mean, if something was bold, it kept saying star star star. So I just added this in the system prompt. Do not return markdown or code blocks in your responses because, during my initial test, Gemma four kept returning it. So yeah. This is how I have constructed the system prompt. So let me just copy this and paste it somewhere over here. So it's a very simple system prompt. You can obviously change this to however you like and modify it. Yes. And now we are, on this step of adding tools. So like I mentioned, I have this little site called fixtures.files.workers.dev. It has a few simple mock APIs which return a bunch of orders and products. That's it. So this is a very simple method. For example, fetch order by number. It's just making an API request to this, and it's passing in, parameter, the order number. That's it. So let's add this. That's it. Also, I hope everyone is able to follow along. If you have any questions, please, drop them in the chat that we have. There's also a q and a section, and, my friends and peers, Harshal and Arian and Vinay are all, in the backside. You cannot see them, but, yeah, they'll answer whatever questions that you have. So, yeah, please, ask any questions that you have. So, yes, back to the writing code. So we have, this base URL for this API that I have, And these are just the types. Okay. Let me just cover this. I'll not explain the types. These are just, like, basic, objects. It has a ID, it has a name, it has some quantity and some price, And the order and the product and also the APIs are paginated. So I I'm just passing that as well. I'll just copy the types. And then, we have the actual tools that the AI will invoke. For example, there's this I mean, I have one tool called fetch order by number. So let me just copy this. It's a simple API request. Nothing special over here. Now that we have, I mean, one tool written, we can, let's add and wire up all these tools inside the workers CI SDK. Yeah, Wartsale CI SDK. Sorry. So I'll tell you, like, what each of these tools do. So lookup order is actually it'll take an order ID from the agent, and it'll just create, I mean, make a fetch request, to my fake API. So I'm also normalizing it. So in case if something comes up with small I mean, lowercase or anything. So I'm just, like, normalizing it by, converting everything to upper case. And then we have fetch order by order I mean, order number, which we just wrote. It's the, fetch request. Then we have a request return tool as well. This, I mean, this is not making an API request, yet. For just for the sake of this demo, I'm just returning a random number over here. So and it says I mean, it'll ask for your, the reason for return, and it'll just respond back with a pre a prepaid return label has been emailed. So you can just ship the items back in fourteen days. So yeah. This is like a hard coded tool, just for the sake of this demo. And then we have check product. Again, this makes a request to the slash products API endpoint and will query it with whatever the user, had talked about. For example, I asked for, do you have any notebooks available? So it just, like, it'll pick up automatically that the user's intent is to look for notebooks. It'll pass notebooks as a parameter over here. So, yeah, that's the query is the product name. Again, very simple, fetch request. So let me copy these tools and add it over here. Yeah. Like I mentioned, like, all of the, I just explained all of the functions and the features. So let's head to the next step. Okay. Now let's actually wire up these tools. We have what the tool does, but we need to write it in a way that the LLM, understands it. So, yeah, let's do this. We'll need to import a few, few other things from the wholesale AI SDK. One is we need step count is. So how many steps, there are. I mean, the agent can look up and the tool method as well. And then we'll need Zod for creating this runtime validation of our tools. Okay. Let's import this as well. And then we'll also import Zard. By the way, this is I believe this is Zod three point o. Okay. Sorry. Yeah. This is for, version four. And, yes, this is the Zod schema for how the tools should be for Versal's AI SDK to pick up and execute. So we have a tool name called lookup order. So and this is what the tool I mean, this is the format, of the tool description, how it looks like. So if you have used Versal CI SDK, this should be very similar. It just takes a description. And, also, a tip while writing in I mean, writing tools, make sure the description is very solid, and it has some references. So it gets I mean, the right tool gets picked up, by your LLM. So I have written look, for the lookup order tool, I'm using, I'm saying lookup and order by its order ID, and I'm also giving it an example of how an order ID should look like. So if I say, can you please look up order, 1024? Or if I just say, can you look up Order 9831? It'll automatically pick up this. And since I'm already normalizing it and appending o r d hyphen, inside the tool just I just wrote, so this will automatically get picked up. And you can also pass multiple I mean, you can create the I mean, write the description as long as you want. So the in input schema, this is how it looks like. So it's I've just written a simple string. So the object and inside it is an order ID, which is a string. And then, again, the format for writing, Vercel AI SDKs, tools are, you add and execute. So if the description matches and the input schema is valid, it will invoke and execute the method. We need to pass in what method to be executed when this description and input schema is valid. We are passing in the order ID and we have an async method, which is the lookup order, which we just wrote. So if, a user says, can you please look up order 1234? It matched the description. 1234, also is a valid order ID. So input schema zed, is validated, then it'll execute the lookup orders method. So, yeah, this is how, tool calls work in general. And then we have return request, which is again a similar tool that I've written. So it's, again, a disc we have this description. I'll start a return for a delivered order. Only delivered orders can be returned. So my API, has statuses. For let me show you how the orders look like. So yes. It has a status field, and it'll only, return orders which have been delivered. Anything else, it'll just, say I mean, the agent will say that it cannot, return an order which is in the processing status because, I mean, that's just not possible. And then we have check product ability. Again, it's a simple, method similarly. Okay. Now that we have all the tools ready and, we have a lot of types. We wired up, the method. I mean, the lookup order, this is just for, like, normalizing. So, again, the return request, check product tool, and we have the sort schema as well. We have a system prompt. And, yes, this is where how, we are wiring up, the tools with our voice agent. So we have a class, base voice agent. It takes in the base voice agent, and then we have a transcriber and text to speech method, which will automatically connect to pipeline and execute, respectively. We have an on call start, which is earlier, we just wrote a simple welcome method. So let's here it is. Yeah. It just says, hi there. How can I help you today? So I just want to, like, make it more personal. So let's add on call. I mean, improve the greeting message. So we have, hello there. Welcome to Acme support. I can track an order. I believe this is what you heard when I was showing you the demo. And, yeah, there we go. We have the new greeting message. And in on turn, we actually have to pass in the new tools. We already had a simple system prompt. Let's remove this and use the one which we wrote above. And we'll also need to destructure steps. And, yes, we also need to pass in the tools that we just wrote. So let's use this as well. Right below the messages, we'll pass tools, econ tools. Let's see why this is showing in because we just routed. Oh, okay. We did not initialize it. Let's just add it right over here. This should be fixed, and we can just return the stop when is when the tool calls get changed, after five. So, yeah, let's pass this as in so that it does not keep on going into a loop, which I've already pasted. And we'll return the text. And, I had all added a few step logs. So let's do that as well. And this is just for, like, debugging. So if I open okay. Sorry. It was on the live version. Yeah. If I mean, this this was just for, like, logging. Whatever tools that I had, it's just logging. Like, for example, if a tool, for invoking return request, is called, it will just log that. So yeah. And, yeah, the we have the rest already done. This looks good enough. So we have a worker speech to text and text to speech pipeline almost ready. And this is just the worker side of things. I now want to move to building the UI side of things. So I have the index dot HTML. Nothing special over here. It just has the Tailwind script CDN. And then main.txs, it just has the app, I mean, DOM creation node. Okay. Let's open app.tx. Okay. This is the if you use React a lot, this is just the boiler plate we are all used to see. Okay. Let me open up a section. Okay. And I I had mentioned that there's a hook for using Cloudflare slash voice as I mean, SDK in React. So that's what we are going to use. Let's just import this. Let me also remove all these unused imports. Let me also just remove the CSS tool. I don't want to mess up whatever classes, Tailwind is providing. Okay. There are a few methods which use voice agent hook, provides. For example, it has, four statuses, when it's idle, when it's listening, thinking, and speaking so that you can build these really cool animations based on whatever status it is. So if it's idle, you can maybe show, like, a bot that's just, like, idle. If it's listening, you can show a different state of loading or, like, some speaker or waveform animation and thinking, when the LLM is actually, making I mean, generating and creating I mean, running its inference. And then we have speaking. So the these are the, four statuses. And then we have transcript. So this is just the message context that we have. And we also have is muted. When the mic is muted, it'll not try to invoke any messages or I mean, send over anything over the WebSocket connection. We also have mute, end call, start call, and we also have audio level. So in case if you're using some sort of waveform animations, you can just use the audio level variable. So at I mean, if the you talk, like, very loudly, it will show you how much I mean, you can show maybe a very high level volume or some sort of animation based on top of it. And then I have I just have an ease active flag. So if it's active, I'm, like, showing a bunch of things. For example, the red, call button for disconnecting or the green call button. If it's not active, I'll show just a green call icon button to do that. So, yeah, very simple stuff. And the transcript is just the messages that, the users, is talking during their conversation. And, yes, that's that's mostly it. Let me let's start adding the similar app. So there's a bunch of stuff that we need to import. Status, transcript. Let me just copy this, actually. Okay. I need to I already have it imported. I just need to restructure it. Okay. And there we go. And then, yeah, this is just a flag that we have before, I mean, when it's not idle. And then we have all the, markup, which is a button and the, the transcription that we have below. So if we head over here so, yeah, this is what we are be building basically. This was the button that we have I mean, which is green when it's idle and red when it's active so that you can disconnect it. Okay. Let's do that as well. I'll just write it now myself. Just a six, which seems fine. Let's call. And I have this UI where I'm showing, like, two different rows. So the other one is for just the button, the mute icon, and the, volume, the volume waveform. Yeah. So let's let's do that. Sorry. Windsor keeps adding random stuff. D I v. Okay. Yeah. Let me just copy this so that I don't keep writing a bunch of markup. So, yeah, just a button if it's active. So there's, like, I mean, when it's active, it'll call the end call method. And if it's in I mean, if it's I mean, the call has not yet started, we'll just call the start call method. So and based on the state, will this show red or green? That's it. And, yes, then we have the status. So below the button, let me show the status. Let me also add a little bit of gap as well. And then we'll show the, transcription messages. So it's I mean, we are already importing the transcript, which is, returned by use voice agent method from cloudless slash voice slash react, method. Yeah. Let's do that. Okay. This should show the role and the text. That seems good enough. Let's see. I hope this works. Also, if you have any questions, please drop them in the chat, or the q and a section, and we'll try to answer. I mean, I'll pick some of the questions at the end. In the meantime, my team will be able to answer you any questions that you have. Okay. Yeah. Super simple. Let's see if this work. Hey. How are you? Okay. I don't know if you are able to hear that, but the very bare bones version of a voice agent, is kind of working right now. So I mean, the small talk is working. I also want to see if it's able to invoke the tools that we have written. Let's try that. Hi. Can you look up order 9831? Okay. There we go. It's able to invoke the tools as well, which we just wrote. And the loop that we wrote over here you should see it here. Yeah. If I mean, it's just like logging a bunch of steps. You can see it was able to intercept all the logs and print them, whatever tool was being invoked. I'm not logging the entire conversation. This is just for me to help debug. And yes. There we go. We have this very bare bones version. If you want the better UI version where, it has, like, a much nicer UI, you can obviously go ahead and check out the app that we have. It's the open source, GitHub repo that's available in the doc section of, I mean, of the stream. You should I mean, on the top right, there's a doc tab. Just click on it, and you should see the GitHub repository, voice agents workshop. So yeah. It has, like, some, commands and a read me as well. And yeah. There we go. Like, we have a working version ready. Okay. Run and test. Yes. Of course. We we just did this. Okay. Now comes the CloudFlare deployment, the step. So now that we have our agent working locally, let's just go ahead and deploy this. Make sure you have, I mean, logged into Wrangler. So if you don't, just run n p x Wrangler login. I have already logged in, but, yeah, let me just do it again. Okay. Now that I'm logged in, I can just run build it and just deploy the application. Let me just see if deploy already has the build command appended. Yes. We do. Okay. If I run n p x wrangler deploy. It's saying the assets property in your configuration is missing. Let me add that. Okay. Change that. Let me open my original Wrangler file and see what's missing. Okay. It looks like everything is correct. Let me try again. I'm missing the direction. Okay. React Cloudflare, which is fine. Okay. Okay. I guess the demo guards are not in my favor right now. It should work. I mean, I have the assets not hand phone handling property. Let me see if I will be able to fix this. Okay. Or I can just do a PNPM done with the client. Oh, okay. I understood what failed. It's because I was directly running the Wrangler deploy through NPX. It should have been through through the backup dot JSON. Again, we'd, type scripting is throwing a few errors. Let me just fix them. Is muted. I did not use it for the bare bones app that we just built. I'll just remove all the unused transport, imports. Okay. There we have it. Let me try deploying it again. Okay. The app is built, which is a good sign. And now it's asking me to select the account. I mean, Wrangler is selecting I mean, asking me to select the account which, we we want the app to be deployed. And just do Cloudflare developer relations. And since these bindings I mean, since this is a worker, a really neat advantage is all of these resources will be automatically provisioned by just adding this one single line, in a Wrangler file. The worker CI binding, the durable objects, and the migrations, they'll all be automatically provisioned to the specific product project that we have. So there's no API keys involved. So that's one good security advantage. Okay. I think it's still being yeah, it was still, propagating. So we have the application built and deployed on Cloudflare Workers, which is awesome. I was really hoping nothing breaks, and it and it worked fast, which is good. But, yeah, let's also test it on the production live URL too. Hey. Can you look up order 9831? Okay. Perfect. Thank you. Thank you so much. And there we go. We have a voice agent, which is not just able to chat, but it's also able to execute tasks, via the agentic tools that we have written. And it's not just, stateless. You can actually ask its stuff. I mean, it has stuff because state because it's using a durable object. For example, if I am able to start the call, Hey. Can you look up order 9831? Can you tell me what my first message was about? Okay. Can you tell me what my second message was? Okay. Yeah. The exactly, I believe, I had talked I mean, I said that, but, I don't know. I think it picked up from my previous, sentence. But, yeah, it also has, state, and it is aware of the context of previous messages, as well, during the conversation. And yeah. There we go. That's what the workshop was about. And, yeah, and this is how the pipeline actually works. So when the browsers, when the I mean, a React application opened a WebSocket, we always send the greeting message. You can see over here. Yeah. This is, from a hook called on call start. So you'll whenever a new call starts, we will always greet the users with welcome to Acme support. You can make this anything, or make it even programmatically personalize it as well. So hi there. I mean, you can say the user's name based on how or what context your application is built in. And, yeah, the voice agent durable object, will send the audio, and we'll have we have a agent loop which will run all the tools, for example, like, invoking tools, like intents. It can also handle interruptions. I don't know if you saw this, but, okay. Let me show that again. Oh, hey. That's fine. Can you just tell me what do you do? Yeah. So yeah. There we go. We it can also handle interruptions very gracefully, and it will immediately pick up where the user, started the sentence. So that's why, I picked up Cloudflare's voice, SDK to build this demo. So and it has a lot of very nice abstractions built, by the team, but they are granular enough for you to change and customize it however you want. It has a lot of hooks, throughout its life cycle. So if you're interested, go through the docs. You'll find the link on the docs tab in the stream on the top right corner. And, yes, the entire conversation is also saved in a SQLite database, because it's using a durable object, which we are using it through the agents SDK. I mean, since, agents SDK is built on top of durable objects. And, yes, there we go. They we have our app built. We all we are also able to make it live using Wrangler as well. And, yeah, also, you're not just restricted to using workers' AI models. You can also use ElevenLabs. There's also an abstraction, built by the Cloudflare agents team, called ElevenLabs TTS. So just import Cloudflare slash voice hyphen 11 labs, and then you'll be able to use whatever model, ElevenLabs has. Just pass in your API key and the voice ID that you have created. There's also just voice input as well. For some reason, you don't want to have TTS, you can do that too using, with voice input instead of just with voice. And, yes, there you go. I hope, you are able to learn, something new from the workshop that we did today. If you want to access this slide, just scan this QR code, let me zoom in. I believe it will take you to this very specific page which has links to all the resources that we have here. The GitHub code base, live demo, the documentation as well. And there's also a very nice detailed blog written by Sunil Pai. If you don't know him, go follow him. He does a lot of amazing stuff. Go ahead and read this and how they built it, the decisions they took while building this, etc. And you can also follow me on GitHub. This is my GitHub handle. So, yeah, you'll find, my Twitter as well as LinkedIn. And, yeah, there we go. We were able to build a React application from scratch. We wrote a worker. Yeah. We wrote a worker which was able to wire up all the LLM tools. We're able to install AI SDK from Versal, and we also added, the Cloudflare slash voice SDK methods. And we wired up and created really nice route that will handle interruptions that will accept voice, and it will respond back with the Deepgram AI model. And we also wrote, the React application, using the voice SDK slash input, React hook that we had. And yes. If you have any questions, please, drop them in the chat or the q and a section. I'll try to answer them. Okay. Yeah. Lots of questions. Hey, Faye. So, we've got a lot of questions as you can already see. Has helped to answer most of them. Mhmm. But you will see a couple of questions we've highlighted for you which are in red. So maybe you can answer those first, and then Harshal is already working to reply to the rest. Most of them are already replied to, in the left, q and a section. You you can see, I think, there are four or five questions which are highlighted, Okay. which maybe you begin replying. Okay. Yeah. How to, Jayshree is asking how to terminate a durable object instance. Actually, if we leave it for some time, a durable object will sleep on its own. Or if you go to the -.cloudlet.com, I believe you can also remove a durable object. I mean, disconnect it from there. I hope that answers your question. Yeah. The demo didn't stream in the text and everything. Right? Is streaming configured in the demo? I'm right now struggling a bit with the hook protected Cloudflare WebSocket streaming problem in a worker. Actually, you can try it out by tuning the project that, I just, have published. So, I'll be able to like, maybe I can, I mean, I don't have full context on what error you're facing? So, yeah, go ahead and try and clone my GitHub repo. Hopefully, it will work and your problem gets solved. Alright. Where can I change the voice language? Actually, Deepgram can just understand English for now. So if you're using something like eleven Labs, yeah, they are, like, really good at handling a lot of different languages. You can definitely try that as well and handle other languages. Yeah. Again, a very similar question. Laurie is asking, how does the model handle other languages, and are we able to use 11 model? I believe this was asked a bit earlier. But, yes, the CloudFlare slash voice, SDK also has an Eleven Labs provider built right in. So you can just pass in your API key and the voice ID from Eleven Labs. It should just work fine. You don't need to change anything else. Okay. Tim is asking how to terminate a durable object instance. And is there an overview of the created instance? Yes. Yeah. I think if you head over to -.cloudcloud.dev actually, let me see if I can share you my dashboard. I'll just show you how to do that. Okay. If you head over to comp I mean, expand mute, select durable objects, and, yeah, you'll be able to see the voice agent durable object that we just created, and you'll see the durable object that we have, here. If you click on this three dot menu, you should see open in data studio. You'll all you'll actually be able to see all the messages, the conversation, created. So if you click here yeah. You can see that it has all these messages. I hope this answers your question. And okay. Okay. Looks like most of the answers have been answered by Hershel. Thanks, man. Really grateful. And, yeah, if you have any other questions, please, drop them in the comments. I'll I'll be able to okay. Vamsi is asking if we started a campaign call for 20 members. Can you tell me how actually worker works for the agent on the cloud? So workers is actually just the compute side of, your application. So durable objects will handle, the WebSocket connection. It adds state to the worker and also stay, save the conversations in a SQLite table. So workers is just, meant for your compute. Okay. Okay. I believe, yeah, I I believe we have answered most of the questions. And yeah. Thank you, everyone. If you have any questions, please drop in the chat. And, also, this, session is recorded and will be see shared with you all post, the session. And, yeah, happy like, I hope you learned something useful. And, yeah, you guys have been a great audience. Also, huge thank you to all my team members, Vinay, Hershel, Arian, and Prasad. Yeah. Thank you, guys. Just a quick shout out. I see there are a lot of people still online. If you guys have any more questions, please feel free to add them in the q and a box. Fayaz and the rest of the team will still be here hanging out in the backstage for the next couple more minutes. If you have anything else to ask us, please drop it in the q and a, and we'll be taking it up. If nothing else, we will be concluding today's session. Thank you. Okay. That's good.