6 ms·

Build an AI telephony agent for inbound and outbound calls

66 points by sagarkava 1y ago

sagarkava 1y ago
Hi I am Sagar, We just open-sourced a complete framework to build an AI-powered telephony agent that can handle both inbound and outbound calls—using Python, SIP, and cloud LLMs like OpenAI or Gemini. You can use it to create smart appointment bots, voice feedback collectors, or even enterprise IVR systems. It’s modular (plug in your SIP provider or AI model), production-ready, and extensible for real-time workflows. Features include: SIP & VoIP call handling (Twilio, Plivo, etc) LLM-integrated AI agent (customizable prompt & tools) FastAPI-based server for routing and control Plugins for STT, TTS, sentiment analysis Support for Agent2Agent and MCP protocols GitHub Repo:https://github.com/videosdk-live/agents https://github.com/videosdk-live/agents Full Blog: https://www.videosdk.live/blog/ai-telephony-agent-inbound-outbound-calls https://www.videosdk.live/blog/ai-telephony-agent-inbound-ou... Would love feedback from anyone working with telephony, LLMs, or real-time automation!
- gerdesj 1y ago
  I'm still getting to grips with all this AI stuff but I do know SIP n RTP rather well. Your project looks ideal for me to play with. My initial project will be to replace my aging parent's and my home FreePBX IVR with something a bit more useful. At the moment it requests you press 1 to make the phones ring and that drops pretty much all unwanted calls but that will soon be automatically defeated. I do also have some rather more commercial ideas but baby steps. Mind you a RPi based drop in box with a decent auto attendant for say £100 hardware, open source software and some fettling time seems a pretty good opportunity. You could go for somewhat cheaper hardware. The UK is being migrated away from copper A and B wires to what is called SOGEA and/or FTTP. Basically its going to be all VoIP for telephony from pretty much now on. The good old days of an analogue handset that is powered by the line will mostly go away within a few years. On the bright side our signalling was bloody weird and SIP is nearly eternal!
  - sagarkava 1y ago
    That is a fantastic idea. We’ve tested it on low-resource hardware. It’s SIP-agnostic and modular, making it ideal for home or SMB setups. We would love to highlight your RPi implementation if you publish it. GitHub: https://github.com/videosdk-live/agents https://github.com/videosdk-live/agents
methuselah_in 1y ago
No offense, but how many people will be jobless because of this?
- sagarkava 1y ago
  Totally valid concern. The goal isn’t to replace people, but to offload repetitive tasks so humans can focus on higher-value work. We’ve also built in Human-in-the-Loop support so a person can step in anytime the AI falls short. More on that here: https://docs.videosdk.live/ai_agents/human-in-the-loop https://docs.videosdk.live/ai_agents/human-in-the-loop It’s about shifting roles, not eliminating them and doing it responsibly.
  - methuselah_in 1y ago
    What you say is totally going to do opposite! In developing countries like India here people don't have too many skills are going to face the heat anyways
  - rwyinuse 1y ago
    I'm not convinced that most companies are going to choose shifting all those people to "higher value work", when the alternative is firing at least some of them them to improve short-term shareholder profits.
  - herval 1y ago
    it turns out many professions are essentially a long loop of repetitive tasks. Think telemarketing, or phone support, for instance. What kind of "higher-value work" would a phone support agent do?
  - 48terry 1y ago
    > The goal isn’t to replace people [...] It’s about shifting roles, not eliminating them and doing it responsibly. With what knowledge you have of the entire history of capitalism ever, do you, genuinely and earnestly, believe this is what's going to happen, and there isn't, perhaps, a different outcome that is more likely?
- 1y ago
mixdup 1y ago
I'll take an alternative view of someone else's comment. No offense, but how much shittier will my customer service experience with my power company or ISP be because of this?
- tguvot 1y ago
  as somebody who works in this area: depends on how much effort they will be willing to invest. it might be better than current service or it might be worse. big question will be availability of humans as fall back that actually have some skills
  - morkalork 1y ago
    I still can't wrap my head around how making functional online portals and well written documentation for customers is too hard, but laying off entire call centers and replacing them with a chat bot that relies on checks notes a functional set of APIs for tool use and well written documentation for a KB, is easier. It looks like all the same effort plus more!
    - cootsnuck 1y ago
      You're not wrong. Businesses that do that are going to be in for a rude surprise. Time will tell if they care about pissing off their customers more than they care about getting rid of workers.
    - tguvot 1y ago
      you overestimate customers. most of them not going to bother to read documentation . used to work closely with big telecoms. amount of support calls that are resolved by "make sure that connector is tight" is crazy high (they invented f-type connectors that work even when they are not tightened properly). "reboot it", solves most of remaining issues. imho, human populated callcenters may actually become a good business model in a few years, after everybody will fire their personal to replace it with AI, and will discover that in some cases you need humans
    - sagarkava 1y ago
nextworddev 1y ago
Just use pipecat
- sagarkava 1y ago
  Thanks for the mention. Curious—what challenges are you finding with Pipecat that you're hoping something else (like https://github.com/videosdk-live/agents https://github.com/videosdk-live/agents) might fix? Always looking to improve based on real gaps devs are facing.
Macha 1y ago
Personally, it's hard to see outbound calls generated by AI as anything other than robocall spam.
- TZubiri 1y ago
  I did some freelancing around LLMs for startups and I can't tell you the amount of times I've had to reject "great ideas" that were just about spamming people. Almost had a big red sign "no outbound!"
  - sagarkava 1y ago
    Completely agree. Our goal is to enable smart automation for legitimate, helpful use cases, not intrusive marketing. We also built in Human-in-the-Loop (HITL) support so humans can intervene any time. Docs on HITL: https://docs.videosdk.live/ai_agents/human-in-the-loop https://docs.videosdk.live/ai_agents/human-in-the-loop GitHub: https://github.com/videosdk-live/agents https://github.com/videosdk-live/agents
    - TZubiri 1y ago
      It's still illegal to place outbound automated calls, and it has been since like the 90s. There's even a simpsons episode about it
- Abishek_Muthian 1y ago
  I would like to use AI to talk to those spammers/scammers. When the adversary is using AI why shouldn't we to defend from it.
  - sagarkava 1y ago
    You absolutely can. Our framework can answer calls, run speech-to-text, analyze intent, and respond with LLMs, making it a great defensive tool against spam or scam calls. GitHub: https://github.com/videosdk-live/agents https://github.com/videosdk-live/agents
mmasu 1y ago
we tried to build something similar lately for outbound calls (for simple reminders to partners) and faced massive issues using gpt-4o-realtime-audio. Noise detection, turn detection, random telephony issues (we were using Twilio too), prompt not holding together, and more. We dropped the project because it would have resulted in a terrible experience for the person on the other side of the phone. Building these things is non trivial. The plan would have been to A/B test and see what the response would have been (watching NPS and business metrics uplift). Human handoff was always the plan in case things got too tricky for the LLM to handle. I see some hostility here towards this project and while I share many concerns, it is very naive to think that these services won’t be massively leveraged going forward. An AI agent can handle things as well as humans (not in our case but there are good services out there, i.e. Parloa) and the key elements are the same as all the other agentic based workflows: - narrow use cases - human in the loop ready to pick up/steer/correct we will see a lot more of this and as LLM capabilities improve, it will only get better - it is inevitable at this point and might (_might_) result in a better experience for customers in some cases. Nevertheless I also see the possibility that we will go full circle and we will always reach for a human, maybe showing up in person in a physical office to make sure cases or requests are handled well… or not :-)
- 48terry 1y ago
  I've read this comment twice and I genuinely can't understand it. Uh, so your own attempt at a similar project didn't work and was a terrible experience and the fundamentals of the system are specific and still require babysitting. But it's inevitable (???) that it'll get better... and this improvement only MIGHT make things better for people, only some of the time? I'm not alone in being unimpressed by this, right? Nothing about what was written here sounds... good? Even the most optimistic part is "well, maybe it might be good, sometimes". Like, this sucks. This is a bad system that doesn't work and makes things worse.
  - mmasu 1y ago
    what I mean is: building these systems is nontrivial, but if done well it can help. Imagine non being in an endless queue on a phone call when having to do a simple task through a customer center call, or having a phone reminder with more information and less noise than from a written notification. The fact that I failed at it (for lack of experience and resources) does not mean it should just be shrugged off as useless or impractical. Some companies offer this service and it works just fine for narrow use cases.

66 points by sagarkava 1y ago

Hi I am Sagar, We just open-sourced a complete framework to build an AI-powered telephony agent that can handle both inbound and outbound calls—using Python, SIP, and cloud LLMs like OpenAI or Gemini. You can use it to create smart appointment bots, voice feedback collectors, or even enterprise IVR systems. It’s modular (plug in your SIP provider or AI model), production-ready, and extensible for real-time workflows. Features include: SIP & VoIP call handling (Twilio, Plivo, etc) LLM-integrated AI agent (customizable prompt & tools) FastAPI-based server for routing and control Plugins for STT, TTS, sentiment analysis Support for Agent2Agent and MCP protocols GitHub Repo:https://github.com/videosdk-live/agents https://github.com/videosdk-live/agents Full Blog: https://www.videosdk.live/blog/ai-telephony-agent-inbound-outbound-calls https://www.videosdk.live/blog/ai-telephony-agent-inbound-ou... Would love feedback from anyone working with telephony, LLMs, or real-time automation!

gerdesj 1y ago
I'm still getting to grips with all this AI stuff but I do know SIP n RTP rather well. Your project looks ideal for me to play with. My initial project will be to replace my aging parent's and my home FreePBX IVR with something a bit more useful. At the moment it requests you press 1 to make the phones ring and that drops pretty much all unwanted calls but that will soon be automatically defeated. I do also have some rather more commercial ideas but baby steps. Mind you a RPi based drop in box with a decent auto attendant for say £100 hardware, open source software and some fettling time seems a pretty good opportunity. You could go for somewhat cheaper hardware. The UK is being migrated away from copper A and B wires to what is called SOGEA and/or FTTP. Basically its going to be all VoIP for telephony from pretty much now on. The good old days of an analogue handset that is powered by the line will mostly go away within a few years. On the bright side our signalling was bloody weird and SIP is nearly eternal!
- sagarkava 1y ago
  That is a fantastic idea. We’ve tested it on low-resource hardware. It’s SIP-agnostic and modular, making it ideal for home or SMB setups. We would love to highlight your RPi implementation if you publish it. GitHub: https://github.com/videosdk-live/agents https://github.com/videosdk-live/agents

methuselah_in 1y ago

No offense, but how many people will be jobless because of this?

sagarkava 1y ago
Totally valid concern. The goal isn’t to replace people, but to offload repetitive tasks so humans can focus on higher-value work. We’ve also built in Human-in-the-Loop support so a person can step in anytime the AI falls short. More on that here: https://docs.videosdk.live/ai_agents/human-in-the-loop https://docs.videosdk.live/ai_agents/human-in-the-loop It’s about shifting roles, not eliminating them and doing it responsibly.
- methuselah_in 1y ago
  What you say is totally going to do opposite! In developing countries like India here people don't have too many skills are going to face the heat anyways
- rwyinuse 1y ago
  I'm not convinced that most companies are going to choose shifting all those people to "higher value work", when the alternative is firing at least some of them them to improve short-term shareholder profits.
- herval 1y ago
  it turns out many professions are essentially a long loop of repetitive tasks. Think telemarketing, or phone support, for instance. What kind of "higher-value work" would a phone support agent do?
- 48terry 1y ago
  > The goal isn’t to replace people [...] It’s about shifting roles, not eliminating them and doing it responsibly. With what knowledge you have of the entire history of capitalism ever, do you, genuinely and earnestly, believe this is what's going to happen, and there isn't, perhaps, a different outcome that is more likely?
1y ago

mixdup 1y ago

I'll take an alternative view of someone else's comment. No offense, but how much shittier will my customer service experience with my power company or ISP be because of this?

tguvot 1y ago
as somebody who works in this area: depends on how much effort they will be willing to invest. it might be better than current service or it might be worse. big question will be availability of humans as fall back that actually have some skills
- morkalork 1y ago
  I still can't wrap my head around how making functional online portals and well written documentation for customers is too hard, but laying off entire call centers and replacing them with a chat bot that relies on checks notes a functional set of APIs for tool use and well written documentation for a KB, is easier. It looks like all the same effort plus more!
  - cootsnuck 1y ago
    You're not wrong. Businesses that do that are going to be in for a rude surprise. Time will tell if they care about pissing off their customers more than they care about getting rid of workers.
  - tguvot 1y ago
    you overestimate customers. most of them not going to bother to read documentation . used to work closely with big telecoms. amount of support calls that are resolved by "make sure that connector is tight" is crazy high (they invented f-type connectors that work even when they are not tightened properly). "reboot it", solves most of remaining issues. imho, human populated callcenters may actually become a good business model in a few years, after everybody will fire their personal to replace it with AI, and will discover that in some cases you need humans
  - sagarkava 1y ago

nextworddev 1y ago

Just use pipecat

sagarkava 1y ago
Thanks for the mention. Curious—what challenges are you finding with Pipecat that you're hoping something else (like https://github.com/videosdk-live/agents https://github.com/videosdk-live/agents) might fix? Always looking to improve based on real gaps devs are facing.

Macha 1y ago

Personally, it's hard to see outbound calls generated by AI as anything other than robocall spam.

TZubiri 1y ago
I did some freelancing around LLMs for startups and I can't tell you the amount of times I've had to reject "great ideas" that were just about spamming people. Almost had a big red sign "no outbound!"
- sagarkava 1y ago
  Completely agree. Our goal is to enable smart automation for legitimate, helpful use cases, not intrusive marketing. We also built in Human-in-the-Loop (HITL) support so humans can intervene any time. Docs on HITL: https://docs.videosdk.live/ai_agents/human-in-the-loop https://docs.videosdk.live/ai_agents/human-in-the-loop GitHub: https://github.com/videosdk-live/agents https://github.com/videosdk-live/agents
  - TZubiri 1y ago
    It's still illegal to place outbound automated calls, and it has been since like the 90s. There's even a simpsons episode about it
Abishek_Muthian 1y ago
I would like to use AI to talk to those spammers/scammers. When the adversary is using AI why shouldn't we to defend from it.
- sagarkava 1y ago
  You absolutely can. Our framework can answer calls, run speech-to-text, analyze intent, and respond with LLMs, making it a great defensive tool against spam or scam calls. GitHub: https://github.com/videosdk-live/agents https://github.com/videosdk-live/agents

mmasu 1y ago

we tried to build something similar lately for outbound calls (for simple reminders to partners) and faced massive issues using gpt-4o-realtime-audio. Noise detection, turn detection, random telephony issues (we were using Twilio too), prompt not holding together, and more. We dropped the project because it would have resulted in a terrible experience for the person on the other side of the phone. Building these things is non trivial. The plan would have been to A/B test and see what the response would have been (watching NPS and business metrics uplift). Human handoff was always the plan in case things got too tricky for the LLM to handle. I see some hostility here towards this project and while I share many concerns, it is very naive to think that these services won’t be massively leveraged going forward. An AI agent can handle things as well as humans (not in our case but there are good services out there, i.e. Parloa) and the key elements are the same as all the other agentic based workflows: - narrow use cases - human in the loop ready to pick up/steer/correct we will see a lot more of this and as LLM capabilities improve, it will only get better - it is inevitable at this point and might (_might_) result in a better experience for customers in some cases. Nevertheless I also see the possibility that we will go full circle and we will always reach for a human, maybe showing up in person in a physical office to make sure cases or requests are handled well… or not :-)

48terry 1y ago
I've read this comment twice and I genuinely can't understand it. Uh, so your own attempt at a similar project didn't work and was a terrible experience and the fundamentals of the system are specific and still require babysitting. But it's inevitable (???) that it'll get better... and this improvement only MIGHT make things better for people, only some of the time? I'm not alone in being unimpressed by this, right? Nothing about what was written here sounds... good? Even the most optimistic part is "well, maybe it might be good, sometimes". Like, this sucks. This is a bad system that doesn't work and makes things worse.
- mmasu 1y ago
  what I mean is: building these systems is nontrivial, but if done well it can help. Imagine non being in an endless queue on a phone call when having to do a simple task through a customer center call, or having a phone reminder with more information and less noise than from a written notification. The fact that I failed at it (for lack of experience and resources) does not mean it should just be shrugged off as useless or impractical. Some companies offer this service and it works just fine for narrow use cases.