deepgeek/gradio-webrtc
2
0
Fork 0
mirror of https://github.com/HumanAIGC-Engineering/gradio-webrtc.git synced 2026-02-05 01:49:23 +08:00
Code Issues Packages Projects Releases Wiki Activity

Rebrand to FastRTC (#60)

Browse Source 
* Add code

* add code

* add code

* Rename messages

* rename

* add code

* Add demo

* docs + demos + bug fixes

* add code

* styles

* user guide

* Styles

* Add code

* misc docs updates

* print nit

* whisper + pr

* url for images

* whsiper update

* Fix bugs

* remove demo files

* version number

* Fix pypi readme

* Fix

* demos

* Add llama code editor

* Update llama code editor and object detection cookbook

* Add more cookbook demos

* add code

* Fix links for PR deploys

* add code

* Fix the install

* add tts

* TTS docs

* Typo

* Pending bubbles for reply on pause

* Stream redesign (#63)

* better error handling

* Websocket error handling

* add code

---------

Co-authored-by: Freddy Boulton <freddyboulton@hf-freddy.local>

* remove docs from dist

* Some docs typos

* more typos

* upload changes + docs

* docs

* better phone

* update docs

* add code

* Make demos better

* fix docs + websocket start_up

* remove mention of FastAPI app

* fastphone tweaks

* add code

* ReplyOnStopWord fixes

* Fix cookbook

* Fix pypi readme

* add code

* bump versions

* sambanova cookbook

* Fix tags

* Llm voice chat

* kyutai tag

* Add error message to all index.html

* STT module uses Moonshine

* Not required from typing extensions

* fix llm voice chat

* Add vpn warning

* demo fixes

* demos

* Add more ui args and gemini audio-video

* update cookbook

* version 9

---------

Co-authored-by: Freddy Boulton <freddyboulton@hf-freddy.local>
This commit is contained in:
Freddy Boulton
2025-02-24 01:13:42 -05:00
committed by GitHub
parent 36190066ec
commit 853d6a06b5
131 changed files with 12349 additions and 4741 deletions
Download Patch File Download Diff File Expand all files Collapse all files

71
docs/advanced-configuration.md
View File

@@ -1,5 +1,9 @@
Any of the parameters for the `Stream` class can be passed to the [`WebRTC`](../userguide/gradio) component directly.
## Track Constraints
You can specify the `track_constraints` parameter to control how the data is streamed to the server. The full documentation on track constraints is [here](https://developer.mozilla.org/en-US/docs/Web/API/MediaTrackConstraints#constraints).
For example, you can control the size of the frames captured from the webcam like so:
@@ -10,21 +14,22 @@ track_constraints = {
"height": {"exact": 500},
"frameRate": {"ideal": 30},
}
webrtc = WebRTC(track_constraints=track_constraints,
modality="video",
mode="send-receive")
webrtc = Stream(
handler=...,
track_constraints=track_constraints,
modality="video",
mode="send-receive")
```
!!! warning
WebRTC may not enforce your constaints. For example, it may rescale your video
(while keeping the same resolution) in order to maintain the desired (or reach a better) frame rate. If you
really want to enforce height, width and resolution constraints, use the `rtp_params` parameter as set `"degradationPreference": "maintain-resolution"`.
WebRTC may not enforce your constraints. For example, it may rescale your video
(while keeping the same resolution) in order to maintain the desired frame rate (or reach a better one). If you really want to enforce height, width and resolution constraints, use the `rtp_params` parameter as set `"degradationPreference": "maintain-resolution"`.
```python
image = WebRTC(
label="Stream",
image = Stream(
modality="video",
mode="send",
track_constraints=track_constraints,
rtp_params={"degradationPreference": "maintain-resolution"}
@@ -36,7 +41,8 @@ webrtc = WebRTC(track_constraints=track_constraints,
You can configure how the connection is created on the client by passing an `rtc_configuration` parameter to the `WebRTC` component constructor.
See the list of available arguments [here](https://developer.mozilla.org/en-US/docs/Web/API/RTCPeerConnection/RTCPeerConnection#configuration).
When deploying on a remote server, an `rtc_configuration` parameter must be passed in. See [Deployment](/deployment).
!!! warning
When deploying on a remote server, the `rtc_configuration` parameter must be passed in. See [Deployment](../deployment).
## Reply on Pause Voice-Activity-Detection
@@ -50,19 +56,18 @@ The `ReplyOnPause` class runs a Voice Activity Detection (VAD) algorithm to dete
The following parameters control this argument:
```python
from gradio_webrtc import AlgoOptions, ReplyOnPause, WebRTC
from fastrtc import AlgoOptions, ReplyOnPause, Stream
options = AlgoOptions(audio_chunk_duration=0.6, # (1)
started_talking_threshold=0.2, # (2)
speech_threshold=0.1, # (3)
)
with gr.Blocks as demo:
audio = WebRTC(...)
audio.stream(ReplyOnPause(..., algo_options=algo_options)
)
demo.launch()
Stream(
handler=ReplyOnPause(..., algo_options=algo_options),
modality="audio",
mode="send-receive"
)
```
1. This is the length (in seconds) of audio chunks.
@@ -75,14 +80,13 @@ demo.launch()
You can configure the sampling rate of the audio passed to the `ReplyOnPause` or `StreamHandler` instance with the `input_sampling_rate` parameter. The current default is `48000`
```python
from gradio_webrtc import ReplyOnPause, WebRTC
from fastrtc import ReplyOnPause, Stream
with gr.Blocks as demo:
audio = WebRTC(...)
audio.stream(ReplyOnPause(..., input_sampling_rate=24000)
)
demo.launch()
stream = Stream(
handler=ReplyOnPause(..., input_sampling_rate=24000),
modality="audio",
mode="send-receive"
)
```
@@ -94,14 +98,13 @@ with the `output_sample_rate` and `output_frame_size` parameters.
The following code (which uses the default values of these parameters), states that each output chunk will be a frame of 960 samples at a frame rate of `24,000` hz. So it will correspond to `0.04` seconds.
```python
from gradio_webrtc import ReplyOnPause, WebRTC
from fastrtc import ReplyOnPause, Stream
with gr.Blocks as demo:
audio = WebRTC(...)
audio.stream(ReplyOnPause(..., output_sample_rate=24000, output_frame_size=960)
)
demo.launch()
stream = Stream(
handler=ReplyOnPause(..., output_sample_rate=24000, output_frame_size=960),
modality="audio",
mode="send-receive"
)
```
!!! tip
@@ -117,6 +120,10 @@ Pass any local path or url to an image (svg, png, jpeg) to the components `icon`
You can control the button color and pulse color with `icon_button_color` and `pulse_color` parameters. They can take any valid css color.
!!! warning
The `icon` parameter is only supported in the `WebRTC` component.
=== "Code"
``` python
audio = WebRTC(
@@ -148,6 +155,10 @@ You can control the button color and pulse color with `icon_button_color` and `p
You can supply a `button_labels` dictionary to change the text displayed in the `Start`, `Stop` and `Waiting` buttons that are displayed in the UI.
The keys must be `"start"`, `"stop"`, and `"waiting"`.
!!! warning
The `button_labels` parameter is only supported in the `WebRTC` component.
``` python
webrtc = WebRTC(
label="Video Chat",

1
docs/bolt.svg
View File

@@ -1 +0,0 @@
<svg xmlns="http://www.w3.org/2000/svg" height="24px" viewBox="0 -960 960 960" width="24px" fill="#e8eaed"><path d="m422-232 207-248H469l29-227-185 267h139l-30 208ZM320-80l40-280H160l360-520h80l-40 320h240L400-80h-80Zm151-390Z"/></svg>
Side by Side

Before

Width:  |  Height:  |  Size: 235 B

209
docs/cookbook.md
View File

@@ -1,6 +1,65 @@
<style>
.tag-button {
cursor: pointer;
opacity: 0.5;
transition: opacity 0.2s ease;
}
.tag-button > code {
color: var(--supernova);
}
.tag-button.active {
opacity: 1;
}
</style>
A collection of applications built with FastRTC. Click on the tags below to find the app you're looking for!
<div class="tag-buttons">
<button class="tag-button" data-tag="audio"><code>audio</code></button>
<button class="tag-button" data-tag="video"><code>video</code></button>
<button class="tag-button" data-tag="llm"><code>llm</code></button>
<button class="tag-button" data-tag="computer-vision"><code>computer-vision</code></button>
<button class="tag-button" data-tag="real-time-api"><code>real-time-api</code></button>
<button class="tag-button" data-tag="voice-chat"><code>voice chat</code></button>
<button class="tag-button" data-tag="code-generation"><code>code generation</code></button>
<button class="tag-button" data-tag="stopword"><code>stopword</code></button>
<button class="tag-button" data-tag="transcription"><code>transcription</code></button>
<button class="tag-button" data-tag="sambanova"><code>SambaNova</code></button>
<button class="tag-button" data-tag="groq"><code>Groq</code></button>
<button class="tag-button" data-tag="elevenlabs"><code>ElevenLabs</code></button>
<button class="tag-button" data-tag="elevenlabs"><code>Kyutai</code></button>
</div>
<script>
function filterCards() {
const activeButtons = document.querySelectorAll('.tag-button.active');
const selectedTags = Array.from(activeButtons).map(button => button.getAttribute('data-tag'));
const cards = document.querySelectorAll('.grid.cards > ul > li > p[data-tags]');
cards.forEach(card => {
const cardTags = card.getAttribute('data-tags').split(',');
const shouldShow = selectedTags.length === 0 || selectedTags.some(tag => cardTags.includes(tag));
card.parentElement.style.display = shouldShow ? 'block' : 'none';
});
}
document.querySelectorAll('.tag-button').forEach(button => {
button.addEventListener('click', () => {
button.classList.toggle('active');
filterCards();
});
});
</script>
<div class="grid cards" markdown>
- :speaking_head:{ .lg .middle }:eyes:{ .lg .middle } __Gemini Audio Video Chat__
{: data-tags="audio,video,real-time-api"}
---
@@ -8,35 +67,130 @@
<video width=98% src="https://github.com/user-attachments/assets/9636dc97-4fee-46bb-abb8-b92e69c08c71" controls style="text-align: center"></video>
[:octicons-arrow-right-24: Demo](https://huggingface.co/spaces/freddyaboulton/gemini-audio-video-chat)
[:octicons-arrow-right-24: Demo](https://huggingface.co/spaces/fastrtc/gemini-audio-video)
[:octicons-arrow-right-24: Gradio UI](https://huggingface.co/spaces/fastrtc/gemini-audio-video)
[:octicons-code-16: Code](https://huggingface.co/spaces/freddyaboulton/gemini-audio-video-chat/blob/main/app.py)
[:octicons-code-16: Code](https://huggingface.co/spaces/fastrtc/gemini-audio-video)
- :speaking_head:{ .lg .middle } __Google Gemini Real Time Voice API__
{: data-tags="audio,real-time-api,voice-chat"}
---
Talk to Gemini in real time using Google's voice API.
<video width=98% src="https://github.com/user-attachments/assets/da8c8a2a-5d99-4ac7-8927-0f7812e4146f" controls style="text-align: center"></video>
<video width=98% src="https://github.com/user-attachments/assets/ea6d18cb-8589-422b-9bba-56332d9f61de" controls style="text-align: center"></video>
[:octicons-arrow-right-24: Demo](https://huggingface.co/spaces/freddyaboulton/gemini-voice)
[:octicons-code-16: Code](https://huggingface.co/spaces/freddyaboulton/gemini-voice/blob/main/app.py)
[:octicons-arrow-right-24: Demo](https://huggingface.co/spaces/fastrtc/talk-to-gemini)
[:octicons-arrow-right-24: Gradio UI](https://huggingface.co/spaces/fastrtc/talk-to-gemini-gradio)
[:octicons-code-16: Code](https://huggingface.co/spaces/fastrtc/talk-to-gemini/blob/main/app.py)
- :speaking_head:{ .lg .middle } __OpenAI Real Time Voice API__
{: data-tags="audio,real-time-api,voice-chat"}
---
Talk to ChatGPT in real time using OpenAI's voice API.
<video width=98% src="https://github.com/user-attachments/assets/41a63376-43ec-496a-9b31-4f067d3903d6" controls style="text-align: center"></video>
<video width=98% src="https://github.com/user-attachments/assets/178bdadc-f17b-461a-8d26-e915c632ff80" controls style="text-align: center"></video>
[:octicons-arrow-right-24: Demo](https://huggingface.co/spaces/freddyaboulton/openai-realtime-voice)
[:octicons-arrow-right-24: Demo](https://huggingface.co/spaces/fastrtc/talk-to-openai)
[:octicons-arrow-right-24: Gradio UI](https://huggingface.co/spaces/fastrtc/talk-to-openai-gradio)
[:octicons-code-16: Code](https://huggingface.co/spaces/freddyaboulton/openai-realtime-voice/blob/main/app.py)
[:octicons-code-16: Code](https://huggingface.co/spaces/fastrtc/talk-to-openai/blob/main/app.py)
- :robot:{ .lg .middle } __Hello Computer__
{: data-tags="llm,stopword,sambanova"}
---
Say computer before asking your question!
<video width=98% src="https://github.com/user-attachments/assets/afb2a3ef-c1ab-4cfb-872d-578f895a10d5" controls style="text-align: center"></video>
[:octicons-arrow-right-24: Demo](https://huggingface.co/spaces/fastrtc/hello-computer)
[:octicons-arrow-right-24: Gradio UI](https://huggingface.co/spaces/fastrtc/hello-computer-gradio)
[:octicons-code-16: Code](https://huggingface.co/spaces/fastrtc/hello-computer/blob/main/app.py)
- :robot:{ .lg .middle } __Llama Code Editor__
{: data-tags="audio,llm,code-generation,groq,stopword"}
---
Create and edit HTML pages with just your voice! Powered by Groq!
<video width=98% src="https://github.com/user-attachments/assets/98523cf3-dac8-4127-9649-d91a997e3ef5" controls style="text-align: center"></video>
[:octicons-arrow-right-24: Demo](https://huggingface.co/spaces/fastrtc/llama-code-editor)
[:octicons-code-16: Code](https://huggingface.co/spaces/fastrtc/llama-code-editor/blob/main/app.py)
- :speaking_head:{ .lg .middle } __Talk to Claude__
{: data-tags="audio,llm,voice-chat"}
---
Use the Anthropic and Play.Ht APIs to have an audio conversation with Claude.
<video width=98% src="https://github.com/user-attachments/assets/fb6ef07f-3ccd-444a-997b-9bc9bdc035d3" controls style="text-align: center"></video>
[:octicons-arrow-right-24: Demo](https://huggingface.co/spaces/fastrtc/talk-to-claude)
[:octicons-arrow-right-24: Gradio UI](https://huggingface.co/spaces/fastrtc/talk-to-claude-gradio)
[:octicons-code-16: Code](https://huggingface.co/spaces/fastrtc/talk-to-claude/blob/main/app.py)
- :musical_note:{ .lg .middle } __LLM Voice Chat__
{: data-tags="audio,llm,voice-chat,groq,elevenlabs"}
---
Talk to an LLM with ElevenLabs!
<video width=98% src="https://github.com/user-attachments/assets/584e898b-91af-4816-bbb0-dd3216eb80b0" controls style="text-align: center"></video>
[:octicons-arrow-right-24: Demo](https://huggingface.co/spaces/fastrtc/llm-voice-chat)
[:octicons-arrow-right-24: Gradio UI](https://huggingface.co/spaces/fastrtc/llm-voice-chat-gradio)
[:octicons-code-16: Code](https://huggingface.co/spaces/fastrtc/llm-voice-chat/blob/main/app.py)
- :musical_note:{ .lg .middle } __Whisper Transcription__
{: data-tags="audio,transcription,groq"}
---
Have whisper transcribe your speech in real time!
<video width=98% src="https://github.com/user-attachments/assets/87603053-acdc-4c8a-810f-f618c49caafb" controls style="text-align: center"></video>
[:octicons-arrow-right-24: Demo](https://huggingface.co/spaces/fastrtc/whisper-realtime)
[:octicons-arrow-right-24: Gradio UI](https://huggingface.co/spaces/fastrtc/whisper-realtime-gradio)
[:octicons-code-16: Code](https://huggingface.co/spaces/fastrtc/whisper-realtime/blob/main/app.py)
- :robot:{ .lg .middle } __Talk to Sambanova__
{: data-tags="llm,stopword,sambanova"}
---
Talk to Llama 3.2 with the SambaNova API.
<video width=98% src="https://github.com/user-attachments/assets/92e4a45a-b5e9-45cd-b7f4-9339ceb343e1" controls style="text-align: center"></video>
[:octicons-arrow-right-24: Demo](https://huggingface.co/spaces/fastrtc/talk-to-sambanova)
[:octicons-arrow-right-24: Gradio UI](https://huggingface.co/spaces/fastrtc/talk-to-sambanova-gradio)
[:octicons-code-16: Code](https://huggingface.co/spaces/fastrtc/talk-to-sambanova/blob/main/app.py)
- :speaking_head:{ .lg .middle } __Hello Llama: Stop Word Detection__
{: data-tags="audio,llm,code-generation,stopword,sambanova"}
---
@@ -49,19 +203,9 @@
[:octicons-code-16: Code](https://huggingface.co/spaces/freddyaboulton/hey-llama-code-editor/blob/main/app.py)
- :robot:{ .lg .middle } __Llama Code Editor__
---
Create and edit HTML pages with just your voice! Powered by SambaNova systems.
<video width=98% src="https://github.com/user-attachments/assets/a09647f1-33e1-4154-a5a3-ffefda8a736a" controls style="text-align: center"></video>
[:octicons-arrow-right-24: Demo](https://huggingface.co/spaces/freddyaboulton/llama-code-editor)
[:octicons-code-16: Code](https://huggingface.co/spaces/freddyaboulton/llama-code-editor/blob/main/app.py)
- :speaking_head:{ .lg .middle } __Audio Input/Output with mini-omni2__
{: data-tags="audio,llm,voice-chat"}
---
@@ -73,19 +217,8 @@
[:octicons-code-16: Code](https://huggingface.co/spaces/freddyaboulton/mini-omni2-webrtc/blob/main/app.py)
- :speaking_head:{ .lg .middle } __Talk to Claude__
---
Use the Anthropic and Play.Ht APIs to have an audio conversation with Claude.
<video width=98% src="https://github.com/user-attachments/assets/650bc492-798e-4995-8cef-159e1cfc2185" controls style="text-align: center"></video>
[:octicons-arrow-right-24: Demo](https://huggingface.co/spaces/freddyaboulton/talk-to-claude)
[:octicons-code-16: Code](https://huggingface.co/spaces/freddyaboulton/talk-to-claude/blob/main/app.py)
- :speaking_head:{ .lg .middle } __Kyutai Moshi__
{: data-tags="audio,llm,voice-chat,kyutai"}
---
@@ -98,6 +231,7 @@
[:octicons-code-16: Code](https://huggingface.co/spaces/freddyaboulton/talk-to-moshi/blob/main/app.py)
- :speaking_head:{ .lg .middle } __Talk to Ultravox__
{: data-tags="audio,llm,voice-chat"}
---
@@ -111,6 +245,7 @@
- :speaking_head:{ .lg .middle } __Talk to Llama 3.2 3b__
{: data-tags="audio,llm,voice-chat"}
---
@@ -124,6 +259,7 @@
- :robot:{ .lg .middle } __Talk to Qwen2-Audio__
{: data-tags="audio,llm,voice-chat"}
---
@@ -137,18 +273,20 @@
- :camera:{ .lg .middle } __Yolov10 Object Detection__
{: data-tags="video,computer-vision"}
---
Run the Yolov10 model on a user webcam stream in real time!
<video width=98% src="https://github.com/user-attachments/assets/c90d8c9d-d2d5-462e-9e9b-af969f2ea73c" controls style="text-align: center"></video>
<video width=98% src="https://github.com/user-attachments/assets/f82feb74-a071-4e81-9110-a01989447ceb" controls style="text-align: center"></video>
[:octicons-arrow-right-24: Demo](https://huggingface.co/spaces/freddyaboulton/webrtc-yolov10n)
[:octicons-arrow-right-24: Demo](https://huggingface.co/spaces/fastrtc/object-detection)
[:octicons-code-16: Code](https://huggingface.co/spaces/freddyaboulton/webrtc-yolov10n/blob/main/app.py)
[:octicons-code-16: Code](https://huggingface.co/spaces/fastrtc/object-detection/blob/main/app.py)
- :camera:{ .lg .middle } __Video Object Detection with RT-DETR__
{: data-tags="video,computer-vision"}
---
@@ -159,6 +297,7 @@
[:octicons-code-16: Code](https://huggingface.co/spaces/freddyaboulton/rt-detr-object-detection-webrtc/blob/main/app.py)
- :speaker:{ .lg .middle } __Text-to-Speech with Parler__
{: data-tags="audio"}
---

45
docs/deployment.md
View File

@@ -1,9 +1,12 @@
When deploying in a cloud environment (like Hugging Face Spaces, EC2, etc), you need to set up a TURN server to relay the WebRTC traffic.
When deploying in a cloud environment (like Hugging Face Spaces, EC2, etc), you need to set up a TURN server to relay the WebRTC traffic. This guide will cover the different options for setting up a TURN server.
!!! tip
The `rtc_configuration` parameter of the `Stream` class also be passed to the [`WebRTC`](../userguide/gradio) component directly if you're building a standalone gradio app.
## Community Server
Hugging Face graciously provides a TURN server for the community.
In order to use it, you need to first create a Hugging Face account by going to the [huggingface.co](https://huggingface.co/).
In order to use it, you need to first create a Hugging Face account by going to [huggingface.co](https://huggingface.co/).
Then navigate to this [space](https://huggingface.co/spaces/freddyaboulton/turn-server-login) and follow the instructions on the page. You just have to click the "Log in" button and then the "Sign Up" button.
@@ -12,17 +15,18 @@ Then navigate to this [space](https://huggingface.co/spaces/freddyaboulton/turn-
Then you can use the `get_hf_turn_credentials` helper to get your credentials:
```python
from gradio_webrtc import get_hf_turn_credentials, WebRTC
from fastrtc import get_hf_turn_credentials, Stream
# Pass a valid access token for your Hugging Face account
# or set the HF_TOKEN environment variable
credentials = get_hf_turn_credentials(token=None)
with gr.Blcocks() as demo:
webrtc = WebRTC(rtc_configuration=credentials)
...
demo.launch()
Stream(
handler=...,
rtc_configuration=credentials,
modality="audio",
mode="send-receive"
)
```
!!! warning
@@ -38,6 +42,7 @@ The easiest way to do this is to use a service like Twilio.
Create a **free** [account](https://login.twilio.com/u/signup) and the install the `twilio` package with pip (`pip install twilio`). You can then connect from the WebRTC component like so:
```python
from fastrtc import Stream
from twilio.rest import Client
import os
@@ -53,13 +58,15 @@ rtc_configuration = {
"iceTransportPolicy": "relay",
}
with gr.Blocks() as demo:
...
rtc = WebRTC(rtc_configuration=rtc_configuration, ...)
...
Stream(
handler=...,
rtc_configuration=rtc_configuration,
modality="audio",
mode="send-receive"
)
```
!!! tip "Automatic Login"
!!! tip "Automatic login"
You can log in automatically with the `get_twilio_turn_credentials` helper
@@ -148,8 +155,7 @@ The `server-info.json` file will have the server's public IP and public DNS:
Finally, you can connect to your EC2 server from the gradio WebRTC component via the `rtc_configuration` argument:
```python
import gradio as gr
from gradio_webrtc import WebRTC
from fastrtc import Stream
rtc_configuration = {
"iceServers": [
{
@@ -159,7 +165,10 @@ rtc_configuration = {
},
]
}
with gr.Blocks() as demo:
webrtc = WebRTC(rtc_configuration=rtc_configuration)
Stream(
handler=...,
rtc_configuration=rtc_configuration,
modality="audio",
mode="send-receive"
)
```

37
docs/faq.md
View File

@@ -1,34 +1,37 @@
## Demo does not work when deploying to the cloud
Make sure you are using a TURN server. See [deployment](/deployment).
Make sure you are using a TURN server. See [deployment](../deployment).
## Recorded input audio sounds muffled during output audio playback
By default, the microphone is [configured](https://github.com/freddyaboulton/gradio-webrtc/blob/903f1f70bd586f638ad3b5a3940c7a8ec70ad1f5/backend/gradio_webrtc/webrtc.py#L575) to do echoCancellation.
By default, the microphone is [configured](https://github.com/freddyaboulton/gradio-webrtc/blob/903f1f70bd586f638ad3b5a3940c7a8ec70ad1f5/backend/gradio_webrtc/webrtc.py#L575) to do echo cancellation.
This is what's causing the recorded audio to sound muffled when the streamed audio starts playing.
You can disable this via the `track_constraints` (see [advanced configuration](./advanced-configuration])) with the following code:
You can disable this via the `track_constraints` (see [Advanced Configuration](../advanced-configuration)) with the following code:
```python
audio = WebRTC(
label="Stream",
track_constraints={
"echoCancellation": False,
"noiseSuppression": {"exact": True},
"autoGainControl": {"exact": True},
"sampleRate": {"ideal": 24000},
"sampleSize": {"ideal": 16},
"channelCount": {"exact": 1},
},
rtc_configuration=None,
mode="send-receive",
modality="audio",
)
stream = Stream(
track_constraints={
"echoCancellation": False,
"noiseSuppression": {"exact": True},
"autoGainControl": {"exact": True},
"sampleRate": {"ideal": 24000},
"sampleSize": {"ideal": 16},
"channelCount": {"exact": 1},
},
rtc_configuration=None,
mode="send-receive",
modality="audio",
)
```
## How to raise errors in the UI
You can raise `WebRTCError` in order for an error message to show up in the user's screen. This is similar to how `gr.Error` works.
!!! warning
The `WebRTCError` class is only supported in the `WebRTC` component.
Here is a simple example:
```python

BIN
docs/fastrtc_logo.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 918 KiB

BIN
docs/fastrtc_logo_small.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 2.9 KiB

203
docs/index.md
View File

@@ -1,30 +1,209 @@
<h1 style='text-align: center; margin-bottom: 1rem; color: white;'> Gradio WebRTC ⚡️ </h1>
<div style='text-align: center; margin-bottom: 1rem; display: flex; justify-content: center; align-items: center;'>
<h1 style='color: white; margin: 0;'>FastRTC</h1>
<img src="/fastrtc_logo.png"
onerror="this.onerror=null; this.src='https://huggingface.co/datasets/freddyaboulton/bucket/resolve/main/fastrtc_logo.png';"
alt="FastRTC Logo"
style="height: 40px; margin-right: 10px;">
</div>
<div style="display: flex; flex-direction: row; justify-content: center">
<img style="display: block; padding-right: 5px; height: 20px;" alt="Static Badge" src="https://img.shields.io/pypi/v/gradio_webrtc">
<a href="https://github.com/freddyaboulton/gradio-webrtc" target="_blank"><img alt="Static Badge" src="https://img.shields.io/badge/github-white?logo=github&logoColor=black"></a>
<img style="display: block; padding-right: 5px; height: 20px;" alt="Static Badge" src="https://img.shields.io/pypi/v/fastrtc">
<a href="https://github.com/freddyaboulton/fastrtc" target="_blank"><img alt="Static Badge" src="https://img.shields.io/badge/github-white?logo=github&logoColor=black"></a>
</div>
<h3 style='text-align: center'>
Stream video and audio in real time with Gradio using WebRTC.
The Real-Time Communication Library for Python.
</h3>
Turn any python function into a real-time audio and video stream over WebRTC or WebSockets.
## Installation
```bash
pip install gradio_webrtc
pip install fastrtc
```
to use built-in pause detection (see [ReplyOnPause](/user-guide/#reply-on-pause)), install the `vad` extra:
to use built-in pause detection (see [ReplyOnPause](userguide/audio/#reply-on-pause)), and text to speech (see [Text To Speech](userguide/audio/#text-to-speech)), install the `vad` and `tts` extras:
```bash
pip install gradio_webrtc[vad]
pip install fastrtc[vad, tts]
```
For stop word detection (see [ReplyOnStopWords](/user-guide/#reply-on-stopwords)), install the `stopword` extra:
```bash
pip install gradio_webrtc[stopword]
```
## Quickstart
Import the [Stream](userguide/streams) class and pass in a [handler](userguide/streams/#handlers).
The `Stream` has three main methods:
- `.ui.launch()`: Launch a built-in UI for easily testing and sharing your stream. Built with [Gradio](https://www.gradio.app/).
- `.fastphone()`: Get a free temporary phone number to call into your stream. Hugging Face token required.
- `.mount(app)`: Mount the stream on a [FastAPI](https://fastapi.tiangolo.com/) app. Perfect for integrating with your already existing production system.
=== "Echo Audio"
```python
from fastrtc import Stream, ReplyOnPause
import numpy as np
def echo(audio: tuple[int, np.ndarray]):
# The function will be passed the audio until the user pauses
# Implement any iterator that yields audio
# See "LLM Voice Chat" for a more complete example
yield audio
stream = Stream(
handler=ReplyOnPause(detection),
modality="audio",
mode="send-receive",
)
```
=== "LLM Voice Chat"
```py
from fastrtc import (
ReplyOnPause, AdditionalOutputs, Stream,
audio_to_bytes, aggregate_bytes_to_16bit
)
import gradio as gr
from groq import Groq
import anthropic
from elevenlabs import ElevenLabs
groq_client = Groq()
claude_client = anthropic.Anthropic()
tts_client = ElevenLabs()
# See "Talk to Claude" in Cookbook for an example of how to keep
# track of the chat history.
def response(
audio: tuple[int, np.ndarray],
):
prompt = groq_client.audio.transcriptions.create(
file=("audio-file.mp3", audio_to_bytes(audio)),
model="whisper-large-v3-turbo",
response_format="verbose_json",
).text
response = claude_client.messages.create(
model="claude-3-5-haiku-20241022",
max_tokens=512,
messages=[{"role": "user", "content": prompt}],
)
response_text = " ".join(
block.text
for block in response.content
if getattr(block, "type", None) == "text"
)
iterator = tts_client.text_to_speech.convert_as_stream(
text=response_text,
voice_id="JBFqnCBsd6RMkjVDRZzb",
model_id="eleven_multilingual_v2",
output_format="pcm_24000"
)
for chunk in aggregate_bytes_to_16bit(iterator):
audio_array = np.frombuffer(chunk, dtype=np.int16).reshape(1, -1)
yield (24000, audio_array)
stream = Stream(
modality="audio",
mode="send-receive",
handler=ReplyOnPause(response),
)
```
=== "Webcam Stream"
```python
from fastrtc import Stream
import numpy as np
def flip_vertically(image):
return np.flip(image, axis=0)
stream = Stream(
handler=flip_vertically,
modality="video",
mode="send-receive",
)
```
=== "Object Detection"
```python
from fastrtc import Stream
import gradio as gr
import cv2
from huggingface_hub import hf_hub_download
from .inference import YOLOv10
model_file = hf_hub_download(
repo_id="onnx-community/yolov10n", filename="onnx/model.onnx"
)
# git clone https://huggingface.co/spaces/fastrtc/object-detection
# for YOLOv10 implementation
model = YOLOv10(model_file)
def detection(image, conf_threshold=0.3):
image = cv2.resize(image, (model.input_width, model.input_height))
new_image = model.detect_objects(image, conf_threshold)
return cv2.resize(new_image, (500, 500))
stream = Stream(
handler=detection,
modality="video",
mode="send-receive",
additional_inputs=[
gr.Slider(minimum=0, maximum=1, step=0.01, value=0.3)
]
)
```
Run:
=== "UI"
```py
stream.ui.launch()
```
=== "Telephone"
```py
stream.fastphone()
```
=== "FastAPI"
```py
app = FastAPI()
stream.mount(app)
# Optional: Add routes
@app.get("/")
async def _():
return HTMLResponse(content=open("index.html").read())
# uvicorn app:app --host 0.0.0.0 --port 8000
```
Learn more about the [Stream](userguide/streams) in the user guide.
## Key Features
:speaking_head:{ .lg } Automatic Voice Detection and Turn Taking built-in, only worry about the logic for responding to the user.
:material-laptop:{ .lg } Automatic UI - Use the `.ui.launch()` method to launch the webRTC-enabled built-in Gradio UI.
:material-lightning-bolt:{ .lg } Automatic WebRTC Support - Use the `.mount(app)` method to mount the stream on a FastAPI app and get a webRTC endpoint for your own frontend!
:simple-webstorm:{ .lg } Websocket Support - Use the `.mount(app)` method to mount the stream on a FastAPI app and get a websocket endpoint for your own frontend!
:telephone:{ .lg } Automatic Telephone Support - Use the `fastphone()` method of the stream to launch the application and get a free temporary phone number!
:robot:{ .lg } Completely customizable backend - A `Stream` can easily be mounted on a FastAPI app so you can easily extend it to fit your production application. See the [Talk To Claude](https://huggingface.co/spaces/fastrtc/talk-to-claude) demo for an example on how to serve a custom JS frontend.
## Examples
See the [cookbook](/cookbook)
See the [cookbook](/cookbook).

63
docs/stylesheets/extra.css Normal file
View File

@@ -0,0 +1,63 @@
:root {
--white: #ffffff;
--galaxy: #393931;
--space: #2d2d2a;
--rock: #2d2d2a;
--cosmic: #ffdd00c5;
--radiate: #d6cec0;
--sun: #ffac2f;
--neutron: #F7F5F6;
--supernova: #ffdd00;
--asteroid: #d6cec0;
}
[data-md-color-scheme="fastrtc-dark"] {
--md-default-bg-color: var(--galaxy);
--md-default-fg-color: var(--white);
--md-default-fg-color--light: var(--white);
--md-default-fg-color--lighter: var(--white);
--md-primary-fg-color: var(--space);
--md-primary-bg-color: var(--white);
--md-accent-fg-color: var(--sun);
--md-typeset-color: var(--white);
--md-typeset-a-color: var(--supernova);
--md-typeset-mark-color: var(--sun);
--md-code-fg-color: var(--white);
--md-code-bg-color: var(--rock);
--md-code-hl-comment-color: var(--asteroid);
--md-code-hl-punctuation-color: var(--supernova);
--md-code-hl-generic-color: var(--supernova);
--md-code-hl-variable-color: var(--white);
--md-code-hl-string-color: var(--radiate);
--md-code-hl-keyword-color: var(--supernova);
--md-code-hl-operator-color: var(--supernova);
--md-code-hl-number-color: var(--radiate);
--md-code-hl-special-color: var(--supernova);
--md-code-hl-function-color: var(--neutron);
--md-code-hl-constant-color: var(--radiate);
--md-code-hl-name-color: var(--md-code-fg-color);
--md-typeset-del-color: hsla(6, 90%, 60%, 0.15);
--md-typeset-ins-color: hsla(150, 90%, 44%, 0.15);
--md-typeset-table-color: hsla(0, 0%, 100%, 0.12);
--md-typeset-table-color--light: hsla(0, 0%, 100%, 0.035);
}
[data-md-color-scheme="fastrtc-dark"] div.admonition {
color: var(--md-code-fg-color);
background-color: var(--galaxy);
}
[data-md-color-scheme="fastrtc-dark"] .grid.cards>ul>li {
border-color: var(--rock);
border-width: thick;
}
[data-md-color-scheme="fastrtc-dark"] .grid.cards>ul>li>hr {
border-color: var(--rock);
}

459
docs/userguide/api.md Normal file
View File

@@ -0,0 +1,459 @@
# Connecting via API
Before continuing, select the `modality`, `mode` of your `Stream` and whether you're using `WebRTC` or `WebSocket`s.
<div class="config-selector">
<div class="select-group">
<label for="connection">Connection</label>
<select id="connection" onchange="updateDocs()">
<option value="webrtc">WebRTC</option>
<option value="websocket">WebSocket</option>
</select>
</div>
<div class="select-group">
<label for="modality">Modality</label>
<select id="modality" onchange="updateDocs()">
<option value="audio">Audio</option>
<option value="video">Video</option>
<option value="audio-video">Audio-Video</option>
</select>
</div>
<div class="select-group">
<label for="mode">Mode</label>
<select id="mode" onchange="updateDocs()">
<option value="send-receive">Send-Receive</option>
<option value="receive">Receive</option>
<option value="send">Send</option>
</select>
</div>
</div>
### Sample Code
<div id="docs"></div>
### Message Format
Over both WebRTC and WebSocket, the server can send messages of the following format:
```json
{
"type": `send_input` | `fetch_output` | `stopword` | `error` | `warning` | `log`,
"data": string | object
}
```
- `send_input`: Send any input data for the handler to the server. See [`Additional Inputs`](#additional-inputs) for more details.
- `fetch_output`: An instance of [`AdditionalOutputs`](#additional-outputs) is sent to the server.
- `stopword`: The stopword has been detected. See [`ReplyOnStopWords`](../audio/#reply-on-stopwords) for more details.
- `error`: An error occurred. The `data` will be a string containing the error message.
- `warning`: A warning occurred. The `data` will be a string containing the warning message.
- `log`: A log message. The `data` will be a string containing the log message.
The `ReplyOnPause` handler can also send the following `log` messages.
```json
{
"type": "log",
"data": "pause_detected" | "response_starting"
}
```
!!! tip
When using WebRTC, the messages will be encoded as strings, so parse as JSON before using.
### Additional Inputs
When the `send_input` message is received, update the inputs of your handler however you like by using the `set_input` method of the `Stream` object.
A common pattern is to use a `POST` request to send the updated data. The first argument to the `set_input` method is the `webrtc_id` of the handler.
```python
from pydantic import BaseModel, Field
class InputData(BaseModel):
webrtc_id: str
conf_threshold: float = Field(ge=0, le=1)
@app.post("/input_hook")
async def _(data: InputData):
stream.set_input(data.webrtc_id, data.conf_threshold)
```
The updated data will be passed to the handler on the **next** call.
### Additional Outputs
The `fetch_output` message is sent to the client whenever an instance of [`AdditionalOutputs`](../streams/#additional-outputs) is available. You can access the latest output data by calling the `fetch_latest_output` method of the `Stream` object.
However, rather than fetching each output manually, a common pattern is to fetch the entire stream of output data by calling the `output_stream` method.
Here is an example:
```python
from fastapi.responses import StreamingResponse
@app.get("/updates")
async def stream_updates(webrtc_id: str):
async def output_stream():
async for output in stream.output_stream(webrtc_id):
# Output is the AdditionalOutputs instance
# Be sure to serialize it however you would like
yield f"data: {output.args[0]}\n\n"
return StreamingResponse(
output_stream(),
media_type="text/event-stream"
)
```
### Handling Errors
When connecting via `WebRTC`, the server will respond to the `/webrtc/offer` route with a JSON response. If there are too many connections, the server will respond with a 429 error.
```json
{
"status": "failed",
"meta": {
"error": "concurrency_limit_reached",
"limit": 10
}
```
Over `WebSocket`, the server will send the same message before closing the connection.
<style>
.config-selector {
margin: 1em 0;
display: flex;
gap: 2em;
}
.select-group {
display: flex;
flex-direction: column;
gap: 0.5em;
}
.select-group label {
font-size: 0.8em;
font-weight: 600;
color: var(--md-default-fg-color--light);
}
.select-group select {
padding: 0.5em;
border: 1px solid var(--md-default-fg-color--lighter);
border-radius: 4px;
background-color: var(--md-code-bg-color);
color: var(--md-code-fg-color);
width: 150px;
font-size: 0.9em;
}
/* Style code blocks to match site theme */
.rendered-content pre {
background-color: var(--md-code-bg-color) !important;
color: var(--md-code-fg-color) !important;
padding: 1em;
border-radius: 4px;
}
.rendered-content code {
font-family: var(--md-code-font-family);
background-color: var(--md-code-bg-color) !important;
color: var(--md-code-fg-color) !important;
}
</style>
<script>
// doT.js
// 2011-2014, Laura Doktorova, https://github.com/olado/doT
// Licensed under the MIT license.
var doT = {
name: "doT",
version: "1.1.1",
templateSettings: {
evaluate: /\{\{([\s\S]+?(\}?)+)\}\}/g,
interpolate: /\{\{=([\s\S]+?)\}\}/g,
encode: /\{\{!([\s\S]+?)\}\}/g,
use: /\{\{#([\s\S]+?)\}\}/g,
useParams: /(^|[^\w$])def(?:\.|\[[\'\"])([\w$\.]+)(?:[\'\"]\])?\s*\:\s*([\w$\.]+|\"[^\"]+\"|\'[^\']+\'|\{[^\}]+\})/g,
define: /\{\{##\s*([\w\.$]+)\s*(\:|=)([\s\S]+?)#\}\}/g,
defineParams: /^\s*([\w$]+):([\s\S]+)/,
conditional: /\{\{\?(\?)?\s*([\s\S]*?)\s*\}\}/g,
iterate: /\{\{~\s*(?:\}\}|([\s\S]+?)\s*\:\s*([\w$]+)\s*(?:\:\s*([\w$]+))?\s*\}\})/g,
varname: "it",
strip: false,
append: true,
selfcontained: false,
doNotSkipEncoded: false
},
template: undefined, //fn, compile template
compile: undefined, //fn, for express
log: true
}, _globals;
doT.encodeHTMLSource = function (doNotSkipEncoded) {
var encodeHTMLRules = { "&": "&#38;", "<": "&#60;", ">": "&#62;", '"': "&#34;", "'": "&#39;", "/": "&#47;" },
matchHTML = doNotSkipEncoded ? /[&<>"'\/]/g : /&(?!#?\w+;)|<|>|"|'|\//g;
return function (code) {
return code ? code.toString().replace(matchHTML, function (m) { return encodeHTMLRules[m] || m; }) : "";
};
};
_globals = (function () { return this || (0, eval)("this"); }());
/* istanbul ignore else */
if (typeof module !== "undefined" && module.exports) {
module.exports = doT;
} else if (typeof define === "function" && define.amd) {
define(function () { return doT; });
} else {
_globals.doT = doT;
}
var startend = {
append: { start: "'+(", end: ")+'", startencode: "'+encodeHTML(" },
split: { start: "';out+=(", end: ");out+='", startencode: "';out+=encodeHTML(" }
}, skip = /$^/;
function resolveDefs(c, block, def) {
return ((typeof block === "string") ? block : block.toString())
.replace(c.define || skip, function (m, code, assign, value) {
if (code.indexOf("def.") === 0) {
code = code.substring(4);
}
if (!(code in def)) {
if (assign === ":") {
if (c.defineParams) value.replace(c.defineParams, function (m, param, v) {
def[code] = { arg: param, text: v };
});
if (!(code in def)) def[code] = value;
} else {
new Function("def", "def['" + code + "']=" + value)(def);
}
}
return "";
})
.replace(c.use || skip, function (m, code) {
if (c.useParams) code = code.replace(c.useParams, function (m, s, d, param) {
if (def[d] && def[d].arg && param) {
var rw = (d + ":" + param).replace(/'|\\/g, "_");
def.__exp = def.__exp || {};
def.__exp[rw] = def[d].text.replace(new RegExp("(^|[^\\w$])" + def[d].arg + "([^\\w$])", "g"), "$1" + param + "$2");
return s + "def.__exp['" + rw + "']";
}
});
var v = new Function("def", "return " + code)(def);
return v ? resolveDefs(c, v, def) : v;
});
}
function unescape(code) {
return code.replace(/\\('|\\)/g, "$1").replace(/[\r\t\n]/g, " ");
}
doT.template = function (tmpl, c, def) {
c = c || doT.templateSettings;
var cse = c.append ? startend.append : startend.split, needhtmlencode, sid = 0, indv,
str = (c.use || c.define) ? resolveDefs(c, tmpl, def || {}) : tmpl;
str = ("var out='" + (c.strip ? str.replace(/(^|\r|\n)\t* +| +\t*(\r|\n|$)/g, " ")
.replace(/\r|\n|\t|\/\*[\s\S]*?\*\//g, "") : str)
.replace(/'|\\/g, "\\$&")
.replace(c.interpolate || skip, function (m, code) {
return cse.start + unescape(code) + cse.end;
})
.replace(c.encode || skip, function (m, code) {
needhtmlencode = true;
return cse.startencode + unescape(code) + cse.end;
})
.replace(c.conditional || skip, function (m, elsecase, code) {
return elsecase ?
(code ? "';}else if(" + unescape(code) + "){out+='" : "';}else{out+='") :
(code ? "';if(" + unescape(code) + "){out+='" : "';}out+='");
})
.replace(c.iterate || skip, function (m, iterate, vname, iname) {
if (!iterate) return "';} } out+='";
sid += 1; indv = iname || "i" + sid; iterate = unescape(iterate);
return "';var arr" + sid + "=" + iterate + ";if(arr" + sid + "){var " + vname + "," + indv + "=-1,l" + sid + "=arr" + sid + ".length-1;while(" + indv + "<l" + sid + "){"
+ vname + "=arr" + sid + "[" + indv + "+=1];out+='";
})
.replace(c.evaluate || skip, function (m, code) {
return "';" + unescape(code) + "out+='";
})
+ "';return out;")
.replace(/\n/g, "\\n").replace(/\t/g, '\\t').replace(/\r/g, "\\r")
.replace(/(\s|;|\}|^|\{)out\+='';/g, '$1').replace(/\+''/g, "");
//.replace(/(\s|;|\}|^|\{)out\+=''\+/g,'$1out+=');
if (needhtmlencode) {
if (!c.selfcontained && _globals && !_globals._encodeHTML) _globals._encodeHTML = doT.encodeHTMLSource(c.doNotSkipEncoded);
str = "var encodeHTML = typeof _encodeHTML !== 'undefined' ? _encodeHTML : ("
+ doT.encodeHTMLSource.toString() + "(" + (c.doNotSkipEncoded || '') + "));"
+ str;
}
try {
return new Function(c.varname, str);
} catch (e) {
/* istanbul ignore else */
if (typeof console !== "undefined") console.log("Could not create a template function: " + str);
throw e;
}
};
doT.compile = function (tmpl, def) {
return doT.template(tmpl, null, def);
};
// WebRTC template
const webrtcTemplate = doT.template(`
To connect to the server, you need to create a new RTCPeerConnection object and call the \`setupWebRTC\` function below.
{{? it.mode === "send-receive" || it.mode === "receive" }}
This code snippet assumes there is an html element with an id of \`{{=it.modality}}_output_component_id\` where the output will be displayed. It should be {{? it.modality === "audio"}}a \`<audio>\`{{??}}an \`<video>\`{{?}} element.
{{?}}
\`\`\`javascript
// pass any rtc_configuration params here
const pc = new RTCPeerConnection();
{{? it.mode === "send-receive" || it.mode === "receive" }}
const {{=it.modality}}_output_component = document.getElementById("{{=it.modality}}_output_component_id");
{{?}}
async function setupWebRTC(peerConnection) {
{{? it.mode === "send-receive" || it.mode === "send" }}
// Get {{=it.modality}} stream from webcam
const stream = await navigator.mediaDevices.getUserMedia({
{{=it.modality}}: true,
})
{{?}}
{{? it.mode === "send-receive" }}
// Send {{=it.modality}} stream to server
stream.getTracks().forEach(async (track) => {
const sender = pc.addTrack(track, stream);
})
{{?? it.mode === "send" }}
// Receive {{=it.modality}} stream from server
pc.addTransceiver({{=it.modality}}, { direction: "recvonly" })
{{?}}
{{? it.mode === "send-receive" || it.mode === "receive" }}
peerConnection.addEventListener("track", (evt) => {
if ({{=it.modality}}_output_component &&
{{=it.modality}}_output_component.srcObject !== evt.streams[0]) {
{{=it.modality}}_output_component.srcObject = evt.streams[0];
}
});
{{?}}
// Create data channel (needed!)
const dataChannel = peerConnection.createDataChannel("text");
// Create and send offer
const offer = await peerConnection.createOffer();
await peerConnection.setLocalDescription(offer);
// Send offer to server
const response = await fetch('/webrtc/offer', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
sdp: offer.sdp,
type: offer.type,
webrtc_id: Math.random().toString(36).substring(7)
})
});
// Handle server response
const serverResponse = await response.json();
await peerConnection.setRemoteDescription(serverResponse);
}
\`\`\`
`);
// WebSocket template
const wsTemplate = doT.template(`
{{? it.modality !== "audio" || it.mode !== "send-receive" }}
WebSocket connections are currently only supported for audio in send-receive mode.
{{??}}
To connect to the server via WebSocket, you'll need to establish a WebSocket connection and handle audio processing. The code below assumes there is an HTML audio element for output playback.
\`\`\`javascript
// Setup audio context and stream
const audioContext = new AudioContext();
const stream = await navigator.mediaDevices.getUserMedia({
audio: true
});
// Create WebSocket connection
const ws = new WebSocket(\`\${window.location.protocol === 'https:' ? 'wss:' : 'ws:'}//$\{window.location.host}/websocket/offer\`);
ws.onopen = () => {
// Send initial start message with unique ID
ws.send(JSON.stringify({
event: "start",
websocket_id: generateId() // Implement your own ID generator
}));
// Setup audio processing
const source = audioContext.createMediaStreamSource(stream);
const processor = audioContext.createScriptProcessor(2048, 1, 1);
source.connect(processor);
processor.connect(audioContext.destination);
processor.onaudioprocess = (e) => {
const inputData = e.inputBuffer.getChannelData(0);
const mulawData = convertToMulaw(inputData, audioContext.sampleRate);
const base64Audio = btoa(String.fromCharCode.apply(null, mulawData));
if (ws.readyState === WebSocket.OPEN) {
ws.send(JSON.stringify({
event: "media",
media: {
payload: base64Audio
}
}));
}
};
};
\`\`\`
{{?}}
`);
function updateDocs() {
// Get selected values
const modality = document.getElementById('modality').value;
const mode = document.getElementById('mode').value;
const connection = document.getElementById('connection').value;
// Context for templates
const context = {
modality: modality,
mode: mode,
additional_inputs: true,
additional_outputs: true
};
// Choose template based on connection type
const template = connection === 'webrtc' ? webrtcTemplate : wsTemplate;
// Render docs with syntax highlighting
const html = template(context);
const docsDiv = document.getElementById('docs');
docsDiv.innerHTML = marked.parse(html);
docsDiv.className = 'rendered-content';
// Initialize any code blocks that were just added
document.querySelectorAll('pre code').forEach((block) => {
hljs.highlightElement(block);
});
}
// Initial render
document.addEventListener('DOMContentLoaded', updateDocs);
</script>

27
docs/userguide/audio-video.md Normal file
View File

@@ -0,0 +1,27 @@
# Audio-Video Streaming
You can simultaneously stream audio and video using `AudioVideoStreamHandler` or `AsyncAudioVideoStreamHandler`.
They are identical to the audio `StreamHandlers` with the addition of `video_receive` and `video_emit` methods which take and return a `numpy` array, respectively.
Here is an example of the video handling functions for connecting with the Gemini multimodal API. In this case, we simply reflect the webcam feed back to the user but every second we'll send the latest webcam frame (and an additional image component) to the Gemini server.
Please see the "Gemini Audio Video Chat" example in the [cookbook](../../cookbook) for the complete code.
``` python title="Async Gemini Video Handling"
async def video_receive(self, frame: np.ndarray):
"""Send video frames to the server"""
if self.session:
# send image every 1 second
# otherwise we flood the API
if time.time() - self.last_frame_time > 1:
self.last_frame_time = time.time()
await self.session.send(encode_image(frame))
if self.latest_args[2] is not None:
await self.session.send(encode_image(self.latest_args[2]))
self.video_queue.put_nowait(frame)
async def video_emit(self) -> VideoEmitType:
"""Return video frames to the client"""
return await self.video_queue.get()
```

343
docs/userguide/audio.md Normal file
View File

@@ -0,0 +1,343 @@
# Audio Streaming
## Reply On Pause
Typically, you want to run a python function whenever a user has stopped speaking. This can be done by wrapping a python generator with the `ReplyOnPause` class and passing it to the `handler` argument of the `Stream` object.
=== "Code"
```python
from fastrtc import ReplyOnPause, Stream
def response(audio: tuple[int, np.ndarray]): # (1)
sample_rate, audio_array = audio
# Generate response
for audio_chunk in generate_response(sample_rate, audio_array):
yield (sample_rate, audio_chunk) # (2)
stream = Stream(
handler=ReplyOnPause(response),
modality="audio",
mode="send-receive"
)
```
1. The python generator will receive the **entire** audio up until the user stopped. It will be a tuple of the form (sampling_rate, numpy array of audio). The array will have a shape of (1, num_samples). You can also pass in additional input components.
2. The generator must yield audio chunks as a tuple of (sampling_rate, numpy audio array). Each numpy audio array must have a shape of (1, num_samples).
=== "Notes"
1. The python generator will receive the **entire** audio up until the user stopped. It will be a tuple of the form (sampling_rate, numpy array of audio). The array will have a shape of (1, num_samples). You can also pass in additional input components.
2. The generator must yield audio chunks as a tuple of (sampling_rate, numpy audio array). Each numpy audio array must have a shape of (1, num_samples).
The `ReplyOnPause` class will handle the voice detection and turn taking logic automatically!
!!! warning "Argument Order"
The first argument to the function must be the audio
!!! tip "Parameters"
You can customize the voice detection parameters by passing in `algo_options` and `model_options` to the `ReplyOnPause` class.
```python
from fastrtc import AlgoOptions, SileroVadOptions
stream = Stream(
handler=ReplyOnPause(
response,
algo_options=AlgoOptions(
audio_chunk_duration=0.6,
started_talking_threshold=0.2,
speech_threshold=0.1
),
model_options=SileroVadOptions(
threshold=0.5,
min_speech_duration_ms=250,
min_silence_duration_ms=100
)
)
)
```
## Reply On Stopwords
You can configure your AI model to run whenever a set of "stop words" are detected, like "Hey Siri" or "computer", with the `ReplyOnStopWords` class.
The API is similar to `ReplyOnPause` with the addition of a `stop_words` parameter.
=== "Code"
``` py
from fastrtc import Stream, ReplyOnStopWords
def response(audio: tuple[int, np.ndarray]):
"""This function must yield audio frames"""
...
for numpy_array in generated_audio:
yield (sampling_rate, numpy_array, "mono")
stream = Stream(
handler=ReplyOnStopWords(generate,
input_sample_rate=16000,
stop_words=["computer"]), # (1)
modality="audio",
mode="send-receive"
)
```
1. The `stop_words` can be single words or pairs of words. Be sure to include common misspellings of your word for more robust detection, e.g. "llama", "lamma". In my experience, it's best to use two very distinct words like "ok computer" or "hello iris".
=== "Notes"
1. The `stop_words` can be single words or pairs of words. Be sure to include common misspellings of your word for more robust detection, e.g. "llama", "lamma". In my experience, it's best to use two very distinct words like "ok computer" or "hello iris".
!!! tip "Extra Dependencies"
The `ReplyOnStopWords` class requires the the `stopword` extra. Run `pip install fastrtc[stopword]` to install it.
!!! warning "English Only"
The `ReplyOnStopWords` class is currently only supported for English.
## Stream Handler
`ReplyOnPause` and `ReplyOnStopWords` are implementations of a `StreamHandler`. The `StreamHandler` is a low-level abstraction that gives you arbitrary control over how the input audio stream and output audio stream are created. The following example echos back the user audio.
=== "Code"
``` py
import gradio as gr
from gradio_webrtc import WebRTC, StreamHandler
from queue import Queue
class EchoHandler(StreamHandler):
def __init__(self) -> None:
super().__init__()
self.queue = Queue()
def receive(self, frame: tuple[int, np.ndarray]) -> None: # (1)
self.queue.put(frame)
def emit(self) -> None: # (2)
return self.queue.get()
def copy(self) -> StreamHandler:
return EchoHandler()
def shutdown(self) -> None: # (3)
pass
def start_up(self) -> None: # (4)
pass
stream = Stream(
handler=EchoHandler(),
modality="audio",
mode="send-receive"
)
```
1. The `StreamHandler` class implements three methods: `receive`, `emit` and `copy`. The `receive` method is called when a new frame is received from the client, and the `emit` method returns the next frame to send to the client. The `copy` method is called at the beginning of the stream to ensure each user has a unique stream handler.
2. The `emit` method SHOULD NOT block. If a frame is not ready to be sent, the method should return `None`.
3. The `shutdown` method is called when the stream is closed. It should be used to clean up any resources.
4. The `start_up` method is called when the stream is first created. It should be used to initialize any resources. See [Talk To OpenAI](https://huggingface.co/spaces/fastrtc/talk-to-openai-gradio) or [Talk To Gemini](https://huggingface.co/spaces/fastrtc/talk-to-gemini-gradio) for an example of a `StreamHandler` that uses the `start_up` method to connect to an API.
=== "Notes"
1. The `StreamHandler` class implements three methods: `receive`, `emit` and `copy`. The `receive` method is called when a new frame is received from the client, and the `emit` method returns the next frame to send to the client. The `copy` method is called at the beginning of the stream to ensure each user has a unique stream handler.
2. The `emit` method SHOULD NOT block. If a frame is not ready to be sent, the method should return `None`.
3. The `shutdown` method is called when the stream is closed. It should be used to clean up any resources.
4. The `start_up` method is called when the stream is first created. It should be used to initialize any resources. See [Talk To OpenAI](https://huggingface.co/spaces/fastrtc/talk-to-openai-gradio) or [Talk To Gemini](https://huggingface.co/spaces/fastrtc/talk-to-gemini-gradio) for an example of a `StreamHandler` that uses the `start_up` method to connect to an API.
!!! tip
See this [Talk To Gemini](https://huggingface.co/spaces/fastrtc/talk-to-gemini-gradio) for a complete example of a more complex stream handler.
## Async Stream Handlers
It is also possible to create asynchronous stream handlers. This is very convenient for accessing async APIs from major LLM developers, like Google and OpenAI. The main difference is that `receive`, `emit`, and `start_up` are now defined with `async def`.
Here is a complete example of using `AsyncStreamHandler` for using the Google Gemini real time API:
=== "Code"
``` py
from fastrtc import AsyncStreamHandler
import asyncio
import base64
import os
import google.generativeai as genai
from google.generativeai.types import (
LiveConnectConfig, SpeechConfig,
VoiceConfig, PrebuiltVoiceConfig
)
class GeminiHandler(AsyncStreamHandler):
"""Handler for the Gemini API"""
def __init__(
self,
expected_layout: Literal["mono"] = "mono",
output_sample_rate: int = 24000,
output_frame_size: int = 480,
) -> None:
super().__init__(
expected_layout,
output_sample_rate,
output_frame_size,
input_sample_rate=16000,
)
self.input_queue: asyncio.Queue = asyncio.Queue()
self.output_queue: asyncio.Queue = asyncio.Queue()
self.quit: asyncio.Event = asyncio.Event()
def copy(self) -> "GeminiHandler":
return GeminiHandler(
expected_layout="mono",
output_sample_rate=self.output_sample_rate,
output_frame_size=self.output_frame_size,
)
async def start_up(self):
await self.wait_for_args()
api_key, voice_name = self.latest_args[1:]
client = genai.Client(
api_key=api_key or os.getenv("GEMINI_API_KEY"),
http_options={"api_version": "v1alpha"},
)
config = LiveConnectConfig(
response_modalities=["AUDIO"], # type: ignore
speech_config=SpeechConfig(
voice_config=VoiceConfig(
prebuilt_voice_config=PrebuiltVoiceConfig(
voice_name=voice_name,
)
)
),
)
async with client.aio.live.connect(
model="gemini-2.0-flash-exp", config=config
) as session:
async for audio in session.start_stream(
stream=self.stream(), mime_type="audio/pcm"
):
if audio.data:
array = np.frombuffer(audio.data, dtype=np.int16)
self.output_queue.put_nowait(array)
async def stream(self) -> AsyncGenerator[bytes, None]:
while not self.quit.is_set():
try:
audio = await asyncio.wait_for(self.input_queue.get(), 0.1)
yield audio
except (asyncio.TimeoutError, TimeoutError):
pass
async def receive(self, frame: tuple[int, np.ndarray]) -> None:
_, array = frame
array = array.squeeze()
audio_message = encode_audio(array)
self.input_queue.put_nowait(audio_message)
async def emit(self) -> tuple[int, np.ndarray]:
array = await self.output_queue.get()
return (self.output_sample_rate, array)
def shutdown(self) -> None:
self.quit.set()
self.args_set.clear()
```
## Text To Speech
You can use an on-device text to speech model if you have the `tts` extra installed.
Import the `get_tts_model` function and call it with the model name you want to use.
At the moment, the only model supported is `kokoro`.
The `get_tts_model` function returns an object with three methods:
- `tts`: Synchronous text to speech.
- `stream_tts_sync`: Synchronous text to speech streaming.
- `stream_tts`: Asynchronous text to speech streaming.
```python
from fastrtc import get_tts_model
model = get_tts_model(model="kokoro")
for audio in model.stream_tts_sync("Hello, world!"):
yield audio
async for audio in model.stream_tts("Hello, world!"):
yield audio
audio = model.tts("Hello, world!")
```
!!! tip
You can customize the audio by passing in an instace of `KokoroTTSOptions` to the method.
See [here](https://huggingface.co/hexgrad/Kokoro-82M/blob/main/VOICES.md) for a list of available voices.
```python
from fastrtc import KokoroTTSOptions, get_tts_model
model = get_tts_model(model="kokoro")
options = KokoroTTSOptions(
voice="af_heart",
speed=1.0,
lang="en-us"
)
audio = model.tts("Hello, world!", options=options)
```
## Speech To Text
You can use an on-device speech to text model if you have the `stt` or `stopword` extra installed.
Import the `get_stt_model` function and call it with the model name you want to use.
At the moment, the only models supported are `moonshine/base` and `moonshine/tiny`.
The `get_stt_model` function returns an object with the following method:
- `stt`: Synchronous speech to text.
```python
from fastrtc import get_stt_model
model = get_stt_model(model="moonshine/base")
audio = (16000, np.random.randint(-32768, 32768, size=(1, 16000)))
text = model.stt(audio)
```
!!! tip "Example"
See [LLM Voice Chat](https://huggingface.co/spaces/fastrtc/llm-voice-chat) for an example of using the `stt` method in a `ReplyOnPause` handler.
!!! warning "English Only"
The `stt` model is currently only supported for English.
## Requesting Inputs
In `ReplyOnPause` and `ReplyOnStopWords`, any additional input data is automatically passed to your generator. For `StreamHandler`s, you must manually request the input data from the client.
You can do this by calling `await self.wait_for_args()` (for `AsyncStreamHandler`s) in either the `emit` or `receive` methods. For a `StreamHandler`, you can call `self.wait_for_args_sync()`.
We can access the value of this component via the `latest_args` property of the `StreamHandler`. The `latest_args` is a list storing each of the values. The 0th index is the dummy string `__webrtc_value__`.
## Telephone Integration
In order for your handler to work over the phone, you must make sure that your handler is not expecting any additional input data besides the audio.
If you call `await self.wait_for_args()` your stream will wait forever for the additional input data.
The stream handlers have a `phone_mode` property that is set to `True` if the stream is running over the phone. You can use this property to determine if you should wait for additional input data.
```python
def emit(self):
if self.phone_mode:
self.latest_args = [None]
else:
await self.wait_for_args()
```
### `ReplyOnPause`
The generator you pass to `ReplyOnPause` must have default arguments for all arguments except audio.
If you yield `AdditionalOutputs`, they will be passed in as the input arguments to the generator the next time it is called.
!!! tip
See [Talk To Claude](https://huggingface.co/spaces/fastrtc/talk-to-claude) for an example of a `ReplyOnPause` handler that is compatible with telephone usage. Notice how the input chatbot history is yielded as an `AdditionalOutput` on each invocation.

217
docs/user-guide.md → docs/userguide/gradio.md
View File

@@ -1,19 +1,17 @@
# User Guide
# Gradio Component
To get started with WebRTC streams, all that's needed is to import the `WebRTC` component from this package and implement its `stream` event.
The automatic gradio UI is a great way to test your stream. However, you may want to customize the UI to your liking or simply build a standalone Gradio application.
This page will show how to do so with simple code examples.
For complete implementations of common tasks, see the [cookbook](/cookbook).
## The WebRTC Component
## Audio Streaming
To build a standalone Gradio application, you can use the `WebRTC` component and implement the `stream` event.
Similarly to the `Stream` object, you must set the `mode` and `modality` arguments and pass in a `handler`.
### Reply on Pause
Below are some common examples of how to use the `WebRTC` component.
Typically, you want to run an AI model that generates audio when the user has stopped speaking. This can be done by wrapping a python generator with the `ReplyOnPause` class
and passing it to the `stream` event of the `WebRTC` component.
=== "Code"
``` py title="ReplyonPause"
=== "Reply On Pause"
``` py
import gradio as gr
from gradio_webrtc import WebRTC, ReplyOnPause
@@ -54,122 +52,9 @@ and passing it to the `stream` event of the `WebRTC` component.
4. The `WebRTC` component must be the first input and output component.
5. Set a `time_limit` to control how long a conversation will last. If the `concurrency_count` is 1 (default), only one conversation will be handled at a time.
=== "Notes"
1. The python generator will receive the **entire** audio up until the user stopped. It will be a tuple of the form (sampling_rate, numpy array of audio). The array will have a shape of (1, num_samples). You can also pass in additional input components.
2. The generator must yield audio chunks as a tuple of (sampling_rate, numpy audio arrays). Each numpy audio array must have a shape of (1, num_samples).
3. The `mode` and `modality` arguments must be set to `"send-receive"` and `"audio"`.
4. The `WebRTC` component must be the first input and output component.
5. Set a `time_limit` to control how long a conversation will last. If the `concurrency_count` is 1 (default), only one conversation will be handled at a time.
### Reply On Stopwords
You can configure your AI model to run whenever a set of "stop words" are detected, like "Hey Siri" or "computer", with the `ReplyOnStopWords` class.
The API is similar to `ReplyOnPause` with the addition of a `stop_words` parameter.
=== "Code"
``` py title="ReplyonPause"
import gradio as gr
from gradio_webrtc import WebRTC, ReplyOnPause
def response(audio: tuple[int, np.ndarray]):
"""This function must yield audio frames"""
...
for numpy_array in generated_audio:
yield (sampling_rate, numpy_array, "mono")
with gr.Blocks() as demo:
gr.HTML(
"""
<h1 style='text-align: center'>
Chat (Powered by WebRTC ⚡️)
</h1>
"""
)
with gr.Column():
with gr.Group():
audio = WebRTC(
mode="send",
modality="audio",
)
webrtc.stream(ReplyOnStopWords(generate,
input_sample_rate=16000,
stop_words=["computer"]), # (1)
inputs=[webrtc, history, code],
outputs=[webrtc], time_limit=90,
concurrency_limit=10)
demo.launch()
```
1. The `stop_words` can be single words or pairs of words. Be sure to include common misspellings of your word for more robust detection, e.g. "llama", "lamma". In my experience, it's best to use two very distinct words like "ok computer" or "hello iris".
=== "Notes"
1. The `stop_words` can be single words or pairs of words. Be sure to include common misspellings of your word for more robust detection, e.g. "llama", "lamma". In my experience, it's best to use two very distinct words like "ok computer" or "hello iris".
### Stream Handler
`ReplyOnPause` is an implementation of a `StreamHandler`. The `StreamHandler` is a low-level
abstraction that gives you arbitrary control over how the input audio stream and output audio stream are created. The following example echos back the user audio.
=== "Code"
``` py title="Stream Handler"
import gradio as gr
from gradio_webrtc import WebRTC, StreamHandler
from queue import Queue
class EchoHandler(StreamHandler):
def __init__(self) -> None:
super().__init__()
self.queue = Queue()
def receive(self, frame: tuple[int, np.ndarray]) -> None: # (1)
self.queue.put(frame)
def emit(self) -> None: # (2)
return self.queue.get()
def copy(self) -> StreamHandler:
return EchoHandler()
with gr.Blocks() as demo:
with gr.Column():
with gr.Group():
audio = WebRTC(
mode="send-receive",
modality="audio",
)
audio.stream(fn=EchoHandler(),
inputs=[audio], outputs=[audio],
time_limit=15)
demo.launch()
```
1. The `StreamHandler` class implements three methods: `receive`, `emit` and `copy`. The `receive` method is called when a new frame is received from the client, and the `emit` method returns the next frame to send to the client. The `copy` method is called at the beginning of the stream to ensure each user has a unique stream handler.
2. The `emit` method SHOULD NOT block. If a frame is not ready to be sent, the method should return `None`.
=== "Notes"
1. The `StreamHandler` class implements three methods: `receive`, `emit` and `copy`. The `receive` method is called when a new frame is received from the client, and the `emit` method returns the next frame to send to the client. The `copy` method is called at the beginning of the stream to ensure each user has a unique stream handler.
2. The `emit` method SHOULD NOT block. If a frame is not ready to be sent, the method should return `None`.
### Async Stream Handlers
It is also possible to create asynchronous stream handlers. This is very convenient for accessing async APIs from major LLM developers, like Google and OpenAI. The main difference is that `receive` and `emit` are now defined with `async def`.
Here is a complete example of using `AsyncStreamHandler` for using the Google Gemini real time API:
=== "Code"
``` py title="AsyncStreamHandler"
=== "AsyncStreamHandler"
``` py
import asyncio
import base64
@@ -292,26 +177,11 @@ Here is a complete example of using `AsyncStreamHandler` for using the Google Ge
lambda: (gr.update(visible=False), gr.update(visible=True)),
None,
[api_key_row, row],
)
demo.launch()
)
```
=== "Server-To-Client Audio"
### Accessing Other Component Values from a StreamHandler
In the gemini demo above, you'll notice that we have the user input their google API key. This is stored in a `gr.Textbox` parameter.
We can access the value of this component via the `latest_args` prop of the `StreamHandler`. The `latest_args` is a list storing the values of each component in the WebRTC `stream` event `inputs` parameter. The value of the `WebRTC` component is the 0th index and it's always the dummy string `__webrtc_value__`.
In order to fetch the latest value from the user however, we `await self.wait_for_args()`. In a synchronous `StreamHandler`, we would call `self.wait_for_args_sync()`.
### Server-To-Client Only
To stream only from the server to the client, implement a python generator and pass it to the component's `stream` event. The stream event must also specify a `trigger` corresponding to a UI interaction that starts the stream. In this case, it's a button click.
=== "Code"
``` py title="Server-To-CLient"
``` py
import gradio as gr
from gradio_webrtc import WebRTC
from pydub import AudioSegment
@@ -334,21 +204,13 @@ To stream only from the server to the client, implement a python generator and p
trigger=button.click # (2)
)
```
1. Set `mode="receive"` to only receive audio from the server.
2. The `stream` event must take a `trigger` that corresponds to the gradio event that starts the stream. In this case, it's the button click.
=== "Notes"
1. Set `mode="receive"` to only receive audio from the server.
2. The `stream` event must take a `trigger` that corresponds to the gradio event that starts the stream. In this case, it's the button click.
## Video Streaming
### Input/Output Streaming
Set up a video Input/Output stream to continuosly receive webcam frames from the user and run an arbitrary python function to return a modified frame.
=== "Code"
=== "Video Streaming"
``` py title="Input/Output Streaming"
``` py
import gradio as gr
from gradio_webrtc import WebRTC
@@ -381,18 +243,9 @@ Set up a video Input/Output stream to continuosly receive webcam frames from the
2. The function must return a numpy array. It can take arbitrary values from other components.
3. Set the `modality="video"` and `mode="send-receive"`
4. The `inputs` parameter should be a list where the first element is the WebRTC component. The only output allowed is the WebRTC component.
=== "Notes"
1. The webcam frame will be represented as a numpy array of shape (height, width, RGB).
2. The function must return a numpy array. It can take arbitrary values from other components.
3. Set the `modality="video"` and `mode="send-receive"`
4. The `inputs` parameter should be a list where the first element is the WebRTC component. The only output allowed is the WebRTC component.
### Server-to-Client Only
Set up a server-to-client stream to stream video from an arbitrary user interaction.
=== "Code"
``` py title="Server-To-Client"
=== "Server-To-Client Video"
``` py
import gradio as gr
from gradio_webrtc import WebRTC
import cv2
@@ -419,39 +272,9 @@ Set up a server-to-client stream to stream video from an arbitrary user interact
1. The `stream` event's `fn` parameter is a generator function that yields the next frame from the video as a **numpy array**.
2. Set `mode="receive"` to only receive audio from the server.
3. The `trigger` parameter the gradio event that will trigger the stream. In this case, the button click event.
=== "Notes"
1. The `stream` event's `fn` parameter is a generator function that yields the next frame from the video as a **numpy array**.
2. Set `mode="receive"` to only receive audio from the server.
3. The `trigger` parameter the gradio event that will trigger the stream. In this case, the button click event.
## Audio-Video Streaming
You can simultaneously stream audio and video simultaneously to/from a server using `AudioVideoStreamHandler` or `AsyncAudioVideoStreamHandler`.
They are identical to the audio `StreamHandlers` with the addition of `video_receive` and `video_emit` methods which take and return a `numpy` array, respectively.
Here is an example of the video handling functions for connecting with the Gemini multimodal API. In this case, we simply reflect the webcam feed back to the user but every second we'll send the latest webcam frame (and an additional image component) to the Gemini server.
Please see the "Gemini Audio Video Chat" example in the [cookbook](/cookbook) for the complete code.
``` python title="Async Gemini Video Handling"
async def video_receive(self, frame: np.ndarray):
"""Send video frames to the server"""
if self.session:
# send image every 1 second
# otherwise we flood the API
if time.time() - self.last_frame_time > 1:
self.last_frame_time = time.time()
await self.session.send(encode_image(frame))
if self.latest_args[2] is not None:
await self.session.send(encode_image(self.latest_args[2]))
self.video_queue.put_nowait(frame)
async def video_emit(self) -> VideoEmitType:
"""Return video frames to the client"""
return await self.video_queue.get()
```
!!! tip
You can configure the `time_limit` and `concurrency_limit` parameters of the `stream` event similar to the `Stream` object.
## Additional Outputs

236
docs/userguide/streams.md Normal file
View File

@@ -0,0 +1,236 @@
# Core Concepts
The core of FastRTC is the `Stream` object. It can be used to stream audio, video, or both.
Here's a simple example of creating a video stream that flips the video vertically. We'll use it to explain the core concepts of the `Stream` object. Click on the plus icons to get a link to the relevant section.
```python
from fastrtc import Stream
import gradio as gr
import numpy as np
def detection(image):
return np.flip(image, axis=0)
stream = Stream(
handler=detection, # (1)
modality="video", # (2)
mode="send-receive", # (3)
additional_inputs=[
gr.Slider(minimum=0, maximum=1, step=0.01, value=0.3) # (4)
],
additional_outputs=None, # (5)
additional_outputs_handler=None # (6)
)
```
1. See [Handlers](#handlers) for more information.
2. See [Modalities](#modalities) for more information.
3. See [Stream Modes](#stream-modes) for more information.
4. See [Additional Inputs](#additional-inputs) for more information.
5. See [Additional Outputs](#additional-outputs) for more information.
6. See [Additional Outputs Handler](#additional-outputs) for more information.
7. Mount the `Stream` on a `FastAPI` app with `stream.mount(app)` and you can add custom routes to it. See [Custom Routes and Frontend Integration](#custom-routes-and-frontend-integration) for more information.
8. See [Built-in Routes](#built-in-routes) for more information.
Run:
=== "UI"
```py
stream.ui.launch()
```
=== "FastAPI"
```py
app = FastAPI()
stream.mount(app)
# uvicorn app:app --host 0.0.0.0 --port 8000
```
### Stream Modes
FastRTC supports three streaming modes:
- `send-receive`: Bidirectional streaming (default)
- `send`: Client-to-server only
- `receive`: Server-to-client only
### Modalities
FastRTC supports three modalities:
- `video`: Video streaming
- `audio`: Audio streaming
- `audio-video`: Combined audio and video streaming
### Handlers
The `handler` argument is the main argument of the `Stream` object. A handler should be a function or a class that inherits from `StreamHandler` or `AsyncStreamHandler` depending on the modality and mode.
| Modality | send-receive | send | receive |
|----------|--------------|------|----------|
| video | Function that takes a video frame and returns a new video frame | Function that takes a video frame and returns a new frame | Function that takes a video frame and returns a new frame |
| audio | `StreamHandler` or `AsyncStreamHandler` subclass | `StreamHandler` or `AsyncStreamHandler` subclass | Generator yielding audio frames |
| audio-video | `AudioVideoStreamHandler` or `AsyncAudioVideoStreamHandler` subclass | Not Supported Yet | Not Supported Yet |
## Methods
The `Stream` has three main methods:
- `.ui.launch()`: Launch a built-in UI for easily testing and sharing your stream. Built with [Gradio](https://www.gradio.app/). You can change the UI by setting the `ui` property of the `Stream` object. Also see the [Gradio guide](../gradio.md) for building Gradio apss with fastrtc.
- `.fastphone()`: Get a free temporary phone number to call into your stream. Hugging Face token required.
- `.mount(app)`: Mount the stream on a [FastAPI](https://fastapi.tiangolo.com/) app. Perfect for integrating with your already existing production system or for building a custom UI.
!!! warning
Websocket docs are only available for audio streams. Telephone docs are only available for audio streams in `send-receive` mode.
## Additional Inputs
You can add additional inputs to your stream using the `additional_inputs` argument. These inputs will be displayed in the generated Gradio UI and they will be passed to the handler as additional arguments.
!!! tip
For audio `StreamHandlers`, please read the special [note](../audio#requesting-inputs) on requesting inputs.
In the automatic gradio UI, these inputs will be the same python type corresponding to the Gradio component. In our case, we used a `gr.Slider` as the additional input, so it will be passed as a float. See the [Gradio documentation](https://www.gradio.app/docs/gradio) for a complete list of components and their corresponding types.
### Input Hooks
Outside of the gradio UI, you are free to update the inputs however you like by using the `set_input` method of the `Stream` object.
A common pattern is to use a `POST` request to send the updated data.
```python
from pydantic import BaseModel, Field
from fastapi import FastAPI
class InputData(BaseModel):
webrtc_id: str
conf_threshold: float = Field(ge=0, le=1)
app = FastAPI()
stream.mount(app)
@app.post("/input_hook")
async def _(data: InputData):
stream.set_input(data.webrtc_id, data.conf_threshold)
```
The updated data will be passed to the handler on the **next** call.
## Additional Outputs
You can return additional output from the handler by returning an instance of `AdditionalOutputs` from the handler.
Let's modify our previous example to also return the number of detections in the frame.
```python
from fastrtc import Stream, AdditionalOutputs
import gradio as gr
def detection(image, conf_threshold=0.3):
processed_frame, n_objects = process_frame(image, conf_threshold)
return processed_frame, AdditionalOutputs(n_objects)
stream = Stream(
handler=detection,
modality="video",
mode="send-receive",
additional_inputs=[
gr.Slider(minimum=0, maximum=1, step=0.01, value=0.3)
],
additional_outputs=[gr.Number()], # (5)
additional_outputs_handler=lambda component, n_objects: n_objects
)
```
We added a `gr.Number()` to the additional outputs and we provided an `additional_outputs_handler`.
The `additional_outputs_handler` is **only** needed for the gradio UI. It is a function that takes the current state of the `component` and the instance of `AdditionalOutputs` and returns the updated state of the `component`. In our case, we want to update the `gr.Number()` with the number of detections.
!!! tip
Since the webRTC is very low latency, you probably don't want to return an additional output on each frame.
### Output Hooks
Outside of the gradio UI, you are free to access the output data however you like by calling the `output_stream` method of the `Stream` object.
A common pattern is to use a `GET` request to get a stream of the output data.
```python
from fastapi.responses import StreamingResponse
@app.get("/updates")
async def stream_updates(webrtc_id: str):
async def output_stream():
async for output in stream.output_stream(webrtc_id):
# Output is the AdditionalOutputs instance
# Be sure to serialize it however you would like
yield f"data: {output.args[0]}\n\n"
return StreamingResponse(
output_stream(),
media_type="text/event-stream"
)
```
## Custom Routes and Frontend Integration
You can add custom routes for serving your own frontend or handling additional functionality once you have mounted the stream on a FastAPI app.
```python
from fastapi.responses import HTMLResponse
from fastapi import FastAPI
from fastrtc import Stream
stream = Stream(...)
app = FastAPI()
stream.mount(app)
# Serve a custom frontend
@app.get("/")
async def serve_frontend():
return HTMLResponse(content=open("index.html").read())
```
## Telephone Integration
FastRTC provides built-in telephone support through the `fastphone()` method:
```python
# Launch with a temporary phone number
stream.fastphone(
# Optional: If None, will use the default token in your machine or read from the HF_TOKEN environment variable
token="your_hf_token",
host="127.0.0.1",
port=8000
)
```
This will print out a phone number along with your temporary code you can use to connect to the stream. You are limited to **10 minutes** of calls per calendar month.
!!! warning
See this [section](../audio#telephone-integration) on making sure your stream handler is compatible for telephone usage.
!!! tip
If you don't have a HF token, you can get one [here](https://huggingface.co/settings/tokens).
## Concurrency
1. You can limit the number of concurrent connections by setting the `concurrency_limit` argument.
2. You can limit the amount of time (in seconds) a connection can stay open by setting the `time_limit` argument.
```python
stream = Stream(
handler=handler,
concurrency_limit=10,
time_limit=3600
)
```

57
docs/userguide/video.md Normal file
View File

@@ -0,0 +1,57 @@
# Video Streaming
## Input/Output Streaming
We already saw this example in the [Quickstart](../../#quickstart) and the [Core Concepts](../streams) section.
=== "Code"
``` py title="Input/Output Streaming"
from fastrtc import Stream
import gradio as gr
def detection(image, conf_threshold=0.3): # (1)
processed_frame = process_frame(image, conf_threshold)
return processed_frame # (2)
stream = Stream(
handler=detection,
modality="video",
mode="send-receive", # (3)
additional_inputs=[
gr.Slider(minimum=0, maximum=1, step=0.01, value=0.3)
],
)
```
1. The webcam frame will be represented as a numpy array of shape (height, width, RGB).
2. The function must return a numpy array. It can take arbitrary values from other components.
3. Set the `modality="video"` and `mode="send-receive"`
=== "Notes"
1. The webcam frame will be represented as a numpy array of shape (height, width, RGB).
2. The function must return a numpy array. It can take arbitrary values from other components.
3. Set the `modality="video"` and `mode="send-receive"`
## Server-to-Client Only
In this case, we stream from the server to the client so we will write a generator function that yields the next frame from the video (as a numpy array)
and set the `mode="receive"` in the `WebRTC` component.
=== "Code"
``` py title="Server-To-Client"
from fastrtc import Stream
def generation():
url = "https://download.tsi.telecom-paristech.fr/gpac/dataset/dash/uhd/mux_sources/hevcds_720p30_2M.mp4"
cap = cv2.VideoCapture(url)
iterating = True
while iterating:
iterating, frame = cap.read()
yield frame
stream = Stream(
handler=generation,
modality="video",
mode="receive"
)
```

160
docs/userguide/webrtc_docs.md Normal file
View File

@@ -0,0 +1,160 @@
# FastRTC Docs
## Connecting
To connect to the server, you need to create a new RTCPeerConnection object and call the `setupWebRTC` function below.
{% if mode in ["send-receive", "receive"] %}
This code snippet assumes there is an html element with an id of `{{ modality }}_output_component_id` where the output will be displayed. It should be {{ "a `<audio>`" if modality == "audio" else "an `<video>`"}} element.
{% endif %}
```js
// pass any rtc_configuration params here
const pc = new RTCPeerConnection();
{% if mode in ["send-receive", "receive"] %}
const {{modality}}_output_component = document.getElementById("{{modality}}_output_component_id");
{% endif %}
async function setupWebRTC(peerConnection) {
{%- if mode in ["send-receive", "send"] -%}
// Get {{modality}} stream from webcam
const stream = await navigator.mediaDevices.getUserMedia({
{{modality}}: true,
})
{%- endif -%}
{% if mode == "send-receive" %}
// Send {{ self.modality }} stream to server
stream.getTracks().forEach(async (track) => {
const sender = pc.addTrack(track, stream);
})
{% elif mode == "send" %}
// Receive {self.modality} stream from server
pc.addTransceiver({{modality}}, { direction: "recvonly" })
{%- endif -%}
{% if mode in ["send-receive", "receive"] %}
peerConnection.addEventListener("track", (evt) => {
if ({{modality}}_output_component &&
{{modality}}_output_component.srcObject !== evt.streams[0]) {
{{modality}}_output_component.srcObject = evt.streams[0];
}
});
{% endif %}
// Create data channel (needed!)
const dataChannel = peerConnection.createDataChannel("text");
// Create and send offer
const offer = await peerConnection.createOffer();
await peerConnection.setLocalDescription(offer);
// Send offer to server
const response = await fetch('/webrtc/offer', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
sdp: offer.sdp,
type: offer.type,
webrtc_id: Math.random().toString(36).substring(7)
})
});
// Handle server response
const serverResponse = await response.json();
await peerConnection.setRemoteDescription(serverResponse);
}
```
{%if additional_inputs %}
## Sending Input Data
Your python handler can request additional data from the frontend by calling the `fetch_args()` method (see [here](#add docs)).
This will send a `send_input` message over the WebRTC data channel.
Upon receiving this message, you should trigger the `set_input` hook of your stream.
A simple way to do this is with a `POST` request.
```python
@stream.post("/input_hook")
def _(data: PydanticBody):
stream.set_inputs(data.webrtc_id, data.inputs)
```
And then in your client code:
```js
const data_channel = peerConnection.createDataChannel("text");
data_channel.onmessage = (event) => {
event_json = JSON.parse(event.data);
if (event_json.type === "send_input") {
fetch('/input_hook', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
},
body: inputs
}
)
};
};
```
The `set_inputs` hook will set the `latest_args` property of your stream to whatever the second argument is.
NOTE: It is completely up to you how you want to call the `set_inputs` hook.
Here we use a `POST` request but you can use a websocket or any other protocol.
{% endif %}
{% if additional_outputs %}
## Fetching Output Data
Your python handler can send additional data to the front end by returning or yielding `AdditionalOutputs(...)`. See the [docs](https://freddyaboulton.github.io/gradio-webrtc/user-guide/#additional-outputs).
Your front end can fetch these outputs by calling the `get_outputs` hook of the `Stream`.
Here is an example using `server-sent-events`:
```python
@stream.get("/outputs")
def _(webrtc_id: str)
async def get_outputs():
while True:
for output in stream.get_output(webrtc_id):
# Serialize to a string prior to this step
yield f"data: {output}\n\n"
await
return StreamingResponse(get_outputs(), media_type="text/event-stream")
```
NOTE: It is completely up to you how you want to call the `get_output` hook.
Here we use a `server-sent-events` but you can use whatever protocol you want!
{% endif %}
## Stopping
You can stop the stream by calling the following function:
```js
function stop(pc) {
// close transceivers
if (pc.getTransceivers) {
pc.getTransceivers().forEach((transceiver) => {
if (transceiver.stop) {
transceiver.stop();
}
});
}
// close local audio / video
if (pc.getSenders()) {
pc.getSenders().forEach((sender) => {
if (sender.track && sender.track.stop) sender.track.stop();
});
}
// close peer connection
setTimeout(() => {
pc.close();
}, 500);
}
```

151
docs/userguide/websocket_docs.md Normal file
View File

@@ -0,0 +1,151 @@
# FastRTC WebSocket Docs
{% if modality != "audio" or mode != "send-receive" %}
WebSocket connections are currently only supported for audio in send-receive mode.
{% else %}
## Connecting
To connect to the server via WebSocket, you'll need to establish a WebSocket connection and handle audio processing. The code below assumes there is an HTML audio element for output playback.
```js
// Setup audio context and stream
const audioContext = new AudioContext();
const stream = await navigator.mediaDevices.getUserMedia({
audio: true
});
// Create WebSocket connection
const ws = new WebSocket(`${window.location.protocol === 'https:' ? 'wss:' : 'ws:'}//${window.location.host}/websocket/offer`);
ws.onopen = () => {
// Send initial start message with unique ID
ws.send(JSON.stringify({
event: "start",
websocket_id: generateId() // Implement your own ID generator
}));
// Setup audio processing
const source = audioContext.createMediaStreamSource(stream);
const processor = audioContext.createScriptProcessor(2048, 1, 1);
source.connect(processor);
processor.connect(audioContext.destination);
processor.onaudioprocess = (e) => {
const inputData = e.inputBuffer.getChannelData(0);
const mulawData = convertToMulaw(inputData, audioContext.sampleRate);
const base64Audio = btoa(String.fromCharCode.apply(null, mulawData));
if (ws.readyState === WebSocket.OPEN) {
ws.send(JSON.stringify({
event: "media",
media: {
payload: base64Audio
}
}));
}
};
};
// Handle incoming audio
const outputContext = new AudioContext({ sampleRate: 24000 });
let audioQueue = [];
let isPlaying = false;
ws.onmessage = (event) => {
const data = JSON.parse(event.data);
if (data.event === "media") {
// Process received audio
const audioData = atob(data.media.payload);
const mulawData = new Uint8Array(audioData.length);
for (let i = 0; i < audioData.length; i++) {
mulawData[i] = audioData.charCodeAt(i);
}
// Convert mu-law to linear PCM
const linearData = alawmulaw.mulaw.decode(mulawData);
const audioBuffer = outputContext.createBuffer(1, linearData.length, 24000);
const channelData = audioBuffer.getChannelData(0);
for (let i = 0; i < linearData.length; i++) {
channelData[i] = linearData[i] / 32768.0;
}
audioQueue.push(audioBuffer);
if (!isPlaying) {
playNextBuffer();
}
}
};
function playNextBuffer() {
if (audioQueue.length === 0) {
isPlaying = false;
return;
}
isPlaying = true;
const bufferSource = outputContext.createBufferSource();
bufferSource.buffer = audioQueue.shift();
bufferSource.connect(outputContext.destination);
bufferSource.onended = playNextBuffer;
bufferSource.start();
}
```
Note: This implementation requires the `alawmulaw` library for audio encoding/decoding:
```html
<script src="https://cdn.jsdelivr.net/npm/alawmulaw"></script>
```
## Handling Input Requests
When the server requests additional input data, it will send a `send_input` message over the WebSocket. You should handle this by sending the data to your input hook:
```js
ws.onmessage = (event) => {
const data = JSON.parse(event.data);
// Handle send_input messages
if (data?.type === "send_input") {
fetch('/input_hook', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
webrtc_id: websocket_id, // Use the same ID from connection
inputs: your_input_data
})
});
}
// ... existing audio handling code ...
};
```
## Receiving Additional Outputs
To receive additional outputs from the server, you can use Server-Sent Events (SSE):
```js
const eventSource = new EventSource('/outputs?webrtc_id=' + websocket_id);
eventSource.addEventListener("output", (event) => {
const eventJson = JSON.parse(event.data);
// Handle the output data here
console.log("Received output:", eventJson);
});
```
## Stopping
To stop the WebSocket connection:
```js
function stop(ws) {
if (ws) {
ws.send(JSON.stringify({
event: "stop"
}));
ws.close();
}
}
```
{% endif %}

48
docs/utils.md
View File

@@ -51,4 +51,50 @@ Example
>>> audio_tuple = (sample_rate, audio_data)
>>> file_path = audio_to_file(audio_tuple)
>>> print(f"Audio saved to: {file_path}")
```
```
## `aggregate_bytes_to_16bit`
Aggregate bytes to 16-bit audio samples.
This function takes an iterator of chunks and aggregates them into 16-bit audio samples.
It handles incomplete samples and combines them with the next chunk.
Parameters
```
chunks_iterator : Iterator[bytes]
An iterator of byte chunks to aggregate
```
Returns
```
Iterator[NDArray[np.int16]]
An iterator of 16-bit audio samples
```
Example
```python
>>> chunks_iterator = [b'\x00\x01', b'\x02\x03', b'\x04\x05']
>>> for chunk in aggregate_bytes_to_16bit(chunks_iterator):
>>> print(chunk)
```
## `async_aggregate_bytes_to_16bit`
Aggregate bytes to 16-bit audio samples asynchronously.
Parameters
```
chunks_iterator : Iterator[bytes]
An iterator of byte chunks to aggregate
```
Returns
```
Iterator[NDArray[np.int16]]
An iterator of 16-bit audio samples
```
Example
```python
>>> chunks_iterator = [b'\x00\x01', b'\x02\x03', b'\x04\x05']
>>> for chunk in async_aggregate_bytes_to_16bit(chunks_iterator):
>>> print(chunk)
```