diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index f58f8c4..3cb301a 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -3,8 +3,16 @@ on: push: branches: - main + pull_request: + branches: + - main + permissions: contents: write + pull-requests: write + deployments: write + pages: write + jobs: deploy: runs-on: ubuntu-latest @@ -24,5 +32,19 @@ jobs: path: .cache restore-keys: | mkdocs-material- - - run: pip install mkdocs-material - - run: mkdocs gh-deploy --force \ No newline at end of file + - run: pip install mkdocs-material + - name: Build docs + run: mkdocs build + + - name: Deploy to GH Pages (main) + if: github.event_name == 'push' + run: mkdocs gh-deploy --force + + - name: Deploy PR Preview + if: github.event_name == 'pull_request' + uses: rossjrw/pr-preview-action@v1 + with: + source-dir: ./site + preview-branch: gh-pages + umbrella-dir: pr-preview + action: auto \ No newline at end of file diff --git a/.gitignore b/.gitignore index 5a9f2b3..5ebb485 100644 --- a/.gitignore +++ b/.gitignore @@ -12,6 +12,9 @@ node_modules backend/**/templates/ demo/MobileNetSSD_deploy.caffemodel demo/MobileNetSSD_deploy.prototxt.txt +demo/scratch +.gradio +.vscode .DS_Store test/ .env \ No newline at end of file diff --git a/README.md b/README.md index 5ddb403..d7b85ca 100644 --- a/README.md +++ b/README.md @@ -1,57 +1,130 @@ -

Gradio WebRTC ⚡️

+
+

FastRTC

+ FastRTC Logo +
-Static Badge -Static Badge -Static Badge +Static Badge +Static Badge

-Stream video and audio in real time with Gradio using WebRTC. +The Real-Time Communication Library for Python.

+Turn any python function into a real-time audio and video stream over WebRTC or WebSockets. + ## Installation ```bash -pip install gradio_webrtc +pip install fastrtc ``` -to use built-in pause detection (see [ReplyOnPause](https://freddyaboulton.github.io/gradio-webrtc//user-guide/#reply-on-pause)), install the `vad` extra: +to use built-in pause detection (see [ReplyOnPause](https://fastrtc.org/)), and text to speech (see [Text To Speech](https://fastrtc.org/userguide/audio/#text-to-speech)), install the `vad` and `tts` extras: ```bash -pip install gradio_webrtc[vad] +pip install fastrtc[vad, tts] ``` -For stop word detection (see [ReplyOnStopWords](https://freddyaboulton.github.io/gradio-webrtc//user-guide/#reply-on-stopwords)), install the `stopword` extra: +## Key Features -```bash -pip install gradio_webrtc[stopword] -``` +- 🗣️ Automatic Voice Detection and Turn Taking built-in, only worry about the logic for responding to the user. +- 💻 Automatic UI - Use the `.ui.launch()` method to launch the webRTC-enabled built-in Gradio UI. +- 🔌 Automatic WebRTC Support - Use the `.mount(app)` method to mount the stream on a FastAPI app and get a webRTC endpoint for your own frontend! +- ⚡️ Websocket Support - Use the `.mount(app)` method to mount the stream on a FastAPI app and get a websocket endpoint for your own frontend! +- 📞 Automatic Telephone Support - Use the `fastphone()` method of the stream to launch the application and get a free temporary phone number! +- 🤖 Completely customizable backend - A `Stream` can easily be mounted on a FastAPI app so you can easily extend it to fit your production application. See the [Talk To Claude](https://huggingface.co/spaces/fastrtc/talk-to-claude) demo for an example on how to serve a custom JS frontend. ## Docs -https://freddyaboulton.github.io/gradio-webrtc/ +[https://fastrtc.org](https://fastrtc.org) ## Examples +See the [Cookbook](https://fastrtc.org/pr-preview/pr-60/cookbook/) for examples of how to use the library. + + + + + + + + + + + + + + + @@ -76,366 +149,169 @@ https://freddyaboulton.github.io/gradio-webrtc/

- - - - - - - - - - - - - - - - - - - -
-

🗣️ Audio Input/Output with mini-omni2

-

Build a GPT-4o like experience with mini-omni2, an audio-native LLM.

- +

🗣️👀 Gemini Audio Video Chat

+

Stream BOTH your webcam video and audio feeds to Google Gemini. You can also upload images to augment your conversation!

+

-Demo | -Code +Demo | +Code +

+		 +

🗣️ Google Gemini Real Time Voice API

+

Talk to Gemini in real time using Google's voice API.

+ +

+Demo | +Code +

+
+

🗣️ OpenAI Real Time Voice API

+

Talk to ChatGPT in real time using OpenAI's voice API.

+ +

+Demo | +Code +

+		 +

🤖 Hello Computer

+

Say computer before asking your question!

+ +

+Demo | +Code +

+
+

🤖 Llama Code Editor

+

Create and edit HTML pages with just your voice! Powered by SambaNova systems.

+ +

+Demo | +Code

🗣️ Talk to Claude

Use the Anthropic and Play.Ht APIs to have an audio conversation with Claude.

- +

-Demo | -Code +Demo | +Code +

+
+

🎵 Whisper Transcription

+

Have whisper transcribe your speech in real time!

+ +

+Demo | +Code +

+		 +

📷 Yolov10 Object Detection

+

Run the Yolov10 model on a user webcam stream in real time!

+ +

+Demo | +Code

-

🤖 Llama Code Editor

-

Create and edit HTML pages with just your voice! Powered by SambaNova systems.

- -

-Demo | -Code -

-		 -

🗣️ Talk to Ultravox

-

Talk to Fixie.AI's audio-native Ultravox LLM with the transformers library.

- -

-Demo | -Code -

-
-

🗣️ Talk to Llama 3.2 3b

-

Use the Lepton API to make Llama 3.2 talk back to you!

- -

-Demo | -Code -

-		 -

🤖 Talk to Qwen2-Audio

-

Qwen2-Audio is a SOTA audio-to-text LLM developed by Alibaba.

- -

-Demo | -Code -

-
-

📷 Yolov10 Object Detection

-

Run the Yolov10 model on a user webcam stream in real time!

- -

-Demo | -Code -

-		 -

📷 Video Object Detection with RT-DETR

-

Upload a video and stream out frames with detected objects (powered by RT-DETR) model.

-

-Demo | -Code -

-
-

🔊 Text-to-Speech with Parler

-

Stream out audio generated by Parler TTS!

-

-Demo | -Code -

-		 -
## Usage This is an shortened version of the official [usage guide](https://freddyaboulton.github.io/gradio-webrtc/user-guide/). -To get started with WebRTC streams, all that's needed is to import the `WebRTC` component from this package and implement its `stream` event. - -### Reply on Pause - -Typically, you want to run an AI model that generates audio when the user has stopped speaking. This can be done by wrapping a python generator with the `ReplyOnPause` class -and passing it to the `stream` event of the `WebRTC` component. - -```py -import gradio as gr -from gradio_webrtc import WebRTC, ReplyOnPause - -def response(audio: tuple[int, np.ndarray]): # (1) - """This function must yield audio frames""" - ... - for numpy_array in generated_audio: - yield (sampling_rate, numpy_array, "mono") # (2) +- `.ui.launch()`: Launch a built-in UI for easily testing and sharing your stream. Built with [Gradio](https://www.gradio.app/). +- `.fastphone()`: Get a free temporary phone number to call into your stream. Hugging Face token required. +- `.mount(app)`: Mount the stream on a [FastAPI](https://fastapi.tiangolo.com/) app. Perfect for integrating with your already existing production system. -with gr.Blocks() as demo: - gr.HTML( - """ -

- Chat (Powered by WebRTC ⚡️) -

- """ - ) - with gr.Column(): - with gr.Group(): - audio = WebRTC( - mode="send-receive", # (3) - modality="audio", - ) - audio.stream(fn=ReplyOnPause(response), - inputs=[audio], outputs=[audio], # (4) - time_limit=60) # (5) +## Quickstart -demo.launch() -``` - -1. The python generator will receive the **entire** audio up until the user stopped. It will be a tuple of the form (sampling_rate, numpy array of audio). The array will have a shape of (1, num_samples). You can also pass in additional input components. - -2. The generator must yield audio chunks as a tuple of (sampling_rate, numpy audio array). Each numpy audio array must have a shape of (1, num_samples). - -3. The `mode` and `modality` arguments must be set to `"send-receive"` and `"audio"`. - -4. The `WebRTC` component must be the first input and output component. - -5. Set a `time_limit` to control how long a conversation will last. If the `concurrency_count` is 1 (default), only one conversation will be handled at a time. - - -### Reply On Stopwords - -You can configure your AI model to run whenever a set of "stop words" are detected, like "Hey Siri" or "computer", with the `ReplyOnStopWords` class. - -The API is similar to `ReplyOnPause` with the addition of a `stop_words` parameter. - - -```py -import gradio as gr -from gradio_webrtc import WebRTC, ReplyOnPause - -def response(audio: tuple[int, np.ndarray]): - """This function must yield audio frames""" - ... - for numpy_array in generated_audio: - yield (sampling_rate, numpy_array, "mono") - - -with gr.Blocks() as demo: - gr.HTML( - """ -

- Chat (Powered by WebRTC ⚡️) -

- """ - ) - with gr.Column(): - with gr.Group(): - audio = WebRTC( - mode="send", - modality="audio", - ) - webrtc.stream(ReplyOnStopWords(generate, - input_sample_rate=16000, - stop_words=["computer"]), # (1) - inputs=[webrtc, history, code], - outputs=[webrtc], time_limit=90, - concurrency_limit=10) - -demo.launch() -``` - -1. The `stop_words` can be single words or pairs of words. Be sure to include common misspellings of your word for more robust detection, e.g. "llama", "lamma". In my experience, it's best to use two very distinct words like "ok computer" or "hello iris". - - -### Audio Server-To-Clien - -To stream only from the server to the client, implement a python generator and pass it to the component's `stream` event. The stream event must also specify a `trigger` corresponding to a UI interaction that starts the stream. In this case, it's a button click. - - - -```py -import gradio as gr -from gradio_webrtc import WebRTC -from pydub import AudioSegment - -def generation(num_steps): - for _ in range(num_steps): - segment = AudioSegment.from_file("audio_file.wav") - array = np.array(segment.get_array_of_samples()).reshape(1, -1) - yield (segment.frame_rate, array) - -with gr.Blocks() as demo: - audio = WebRTC(label="Stream", mode="receive", # (1) - modality="audio") - num_steps = gr.Slider(label="Number of Steps", minimum=1, - maximum=10, step=1, value=5) - button = gr.Button("Generate") - - audio.stream( - fn=generation, inputs=[num_steps], outputs=[audio], - trigger=button.click # (2) - ) -``` - -1. Set `mode="receive"` to only receive audio from the server. -2. The `stream` event must take a `trigger` that corresponds to the gradio event that starts the stream. In this case, it's the button click. - - -### Video Input/Output Streaming -Set up a video Input/Output stream to continuosly receive webcam frames from the user and run an arbitrary python function to return a modified frame. - -```py -import gradio as gr -from gradio_webrtc import WebRTC - - -def detection(image, conf_threshold=0.3): # (1) - ... your detection code here ... - return modified_frame # (2) - - -with gr.Blocks() as demo: - image = WebRTC(label="Stream", mode="send-receive", modality="video") # (3) - conf_threshold = gr.Slider( - label="Confidence Threshold", - minimum=0.0, - maximum=1.0, - step=0.05, - value=0.30, - ) - image.stream( - fn=detection, - inputs=[image, conf_threshold], # (4) - outputs=[image], time_limit=10 - ) - -if __name__ == "__main__": - demo.launch() -``` - -1. The webcam frame will be represented as a numpy array of shape (height, width, RGB). -2. The function must return a numpy array. It can take arbitrary values from other components. -3. Set the `modality="video"` and `mode="send-receive"` -4. The `inputs` parameter should be a list where the first element is the WebRTC component. The only output allowed is the WebRTC component. - -### Server-to-Client Only - -Set up a server-to-client stream to stream video from an arbitrary user interaction. - -```py -import gradio as gr -from gradio_webrtc import WebRTC -import cv2 - -def generation(): - url = "https://download.tsi.telecom-paristech.fr/gpac/dataset/dash/uhd/mux_sources/hevcds_720p30_2M.mp4" - cap = cv2.VideoCapture(url) - iterating = True - while iterating: - iterating, frame = cap.read() - yield frame # (1) - -with gr.Blocks() as demo: - output_video = WebRTC(label="Video Stream", mode="receive", # (2) - modality="video") - button = gr.Button("Start", variant="primary") - output_video.stream( - fn=generation, inputs=None, outputs=[output_video], - trigger=button.click # (3) - ) - demo.launch() -``` - -1. The `stream` event's `fn` parameter is a generator function that yields the next frame from the video as a **numpy array**. -2. Set `mode="receive"` to only receive audio from the server. -3. The `trigger` parameter the gradio event that will trigger the stream. In this case, the button click event. - - -### Additional Outputs - -In order to modify other components from within the WebRTC stream, you must yield an instance of `AdditionalOutputs` and add an `on_additional_outputs` event to the `WebRTC` component. - -This is common for displaying a multimodal text/audio conversation in a Chatbot UI. - - - -``` py title="Additional Outputs" -from gradio_webrtc import AdditionalOutputs, WebRTC - -def transcribe(audio: tuple[int, np.ndarray], - transformers_convo: list[dict], - gradio_convo: list[dict]): - response = model.generate(**inputs, max_length=256) - transformers_convo.append({"role": "assistant", "content": response}) - gradio_convo.append({"role": "assistant", "content": response}) - yield AdditionalOutputs(transformers_convo, gradio_convo) # (1) - - -with gr.Blocks() as demo: - gr.HTML( - """ -

- Talk to Qwen2Audio (Powered by WebRTC ⚡️) -

- """ - ) - transformers_convo = gr.State(value=[]) - with gr.Row(): - with gr.Column(): - audio = WebRTC( - label="Stream", - mode="send", # (2) - modality="audio", - ) - with gr.Column(): - transcript = gr.Chatbot(label="transcript", type="messages") - - audio.stream(ReplyOnPause(transcribe), - inputs=[audio, transformers_convo, transcript], - outputs=[audio], time_limit=90) - audio.on_additional_outputs(lambda s,a: (s,a), # (3) - outputs=[transformers_convo, transcript], - queue=False, show_progress="hidden") - demo.launch() -``` - - 1. Pass your data to `AdditionalOutputs` and yield it. - 2. In this case, no audio is being returned, so we set `mode="send"`. However, if we set `mode="send-receive"`, we could also yield generated audio and `AdditionalOutputs`. - 3. The `on_additional_outputs` event does not take `inputs`. It's common practice to not run this event on the queue since it is just a quick UI update. -=== "Notes" - 1. Pass your data to `AdditionalOutputs` and yield it. - 2. In this case, no audio is being returned, so we set `mode="send"`. However, if we set `mode="send-receive"`, we could also yield generated audio and `AdditionalOutputs`. - 3. The `on_additional_outputs` event does not take `inputs`. It's common practice to not run this event on the queue since it is just a quick UI update. - - -## Deployment - -When deploying in a cloud environment (like Hugging Face Spaces, EC2, etc), you need to set up a TURN server to relay the WebRTC traffic. -The easiest way to do this is to use a service like Twilio. +### Echo Audio ```python -from twilio.rest import Client -import os +from fastrtc import Stream, ReplyOnPause +import numpy as np -account_sid = os.environ.get("TWILIO_ACCOUNT_SID") -auth_token = os.environ.get("TWILIO_AUTH_TOKEN") +def echo(audio: tuple[int, np.ndarray]): + # The function will be passed the audio until the user pauses + # Implement any iterator that yields audio + # See "LLM Voice Chat" for a more complete example + yield audio -client = Client(account_sid, auth_token) +stream = Stream( + handler=ReplyOnPause(detection), + modality="audio", + mode="send-receive", +) +``` -token = client.tokens.create() +### LLM Voice Chat -rtc_configuration = { - "iceServers": token.ice_servers, - "iceTransportPolicy": "relay", -} +```py +from fastrtc import ( + ReplyOnPause, AdditionalOutputs, Stream, + audio_to_bytes, aggregate_bytes_to_16bit +) +import gradio as gr +from groq import Groq +import anthropic +from elevenlabs import ElevenLabs -with gr.Blocks() as demo: - ... - rtc = WebRTC(rtc_configuration=rtc_configuration, ...) - ... +groq_client = Groq() +claude_client = anthropic.Anthropic() +tts_client = ElevenLabs() + + +# See "Talk to Claude" in Cookbook for an example of how to keep +# track of the chat history. +def response( + audio: tuple[int, np.ndarray], +): + prompt = groq_client.audio.transcriptions.create( + file=("audio-file.mp3", audio_to_bytes(audio)), + model="whisper-large-v3-turbo", + response_format="verbose_json", + ).text + response = claude_client.messages.create( + model="claude-3-5-haiku-20241022", + max_tokens=512, + messages=[{"role": "user", "content": prompt}], + ) + response_text = " ".join( + block.text + for block in response.content + if getattr(block, "type", None) == "text" + ) + iterator = tts_client.text_to_speech.convert_as_stream( + text=response_text, + voice_id="JBFqnCBsd6RMkjVDRZzb", + model_id="eleven_multilingual_v2", + output_format="pcm_24000" + + ) + for chunk in aggregate_bytes_to_16bit(iterator): + audio_array = np.frombuffer(chunk, dtype=np.int16).reshape(1, -1) + yield (24000, audio_array) + +stream = Stream( + modality="audio", + mode="send-receive", + handler=ReplyOnPause(response), +) +``` + +### Webcam Stream + +```python +from fastrtc import Stream +import numpy as np + + +def flip_vertically(image): + return np.flip(image, axis=0) + + +stream = Stream( + handler=flip_vertically, + modality="video", + mode="send-receive", +) +``` + +### Object Detection + +```python +from fastrtc import Stream +import gradio as gr +import cv2 +from huggingface_hub import hf_hub_download +from .inference import YOLOv10 + +model_file = hf_hub_download( + repo_id="onnx-community/yolov10n", filename="onnx/model.onnx" +) + +# git clone https://huggingface.co/spaces/fastrtc/object-detection +# for YOLOv10 implementation +model = YOLOv10(model_file) + +def detection(image, conf_threshold=0.3): + image = cv2.resize(image, (model.input_width, model.input_height)) + new_image = model.detect_objects(image, conf_threshold) + return cv2.resize(new_image, (500, 500)) + +stream = Stream( + handler=detection, + modality="video", + mode="send-receive", + additional_inputs=[ + gr.Slider(minimum=0, maximum=1, step=0.01, value=0.3) + ] +) +``` + +## Running the Stream + +Run: + +### Gradio + +```py +stream.ui.launch() +``` + +### Telephone (Audio Only) + + ```py + stream.fastphone() + ``` + +### FastAPI + +```py +app = FastAPI() +stream.mount(app) + +# Optional: Add routes +@app.get("/") +async def _(): + return HTMLResponse(content=open("index.html").read()) + +# uvicorn app:app --host 0.0.0.0 --port 8000 ``` \ No newline at end of file diff --git a/backend/gradio_webrtc/__init__.py b/backend/fastrtc/__init__.py similarity index 81% rename from backend/gradio_webrtc/__init__.py rename to backend/fastrtc/__init__.py index 8f0db37..d966237 100644 --- a/backend/gradio_webrtc/__init__.py +++ b/backend/fastrtc/__init__.py @@ -5,7 +5,17 @@ from .credentials import ( ) from .reply_on_pause import AlgoOptions, ReplyOnPause, SileroVadOptions from .reply_on_stopwords import ReplyOnStopWords -from .speech_to_text import stt, stt_for_chunks +from .speech_to_text import MoonshineSTT, get_stt_model +from .stream import Stream +from .text_to_speech import KokoroTTSOptions, get_tts_model +from .tracks import ( + AsyncAudioVideoStreamHandler, + AsyncStreamHandler, + AudioEmitType, + AudioVideoStreamHandler, + StreamHandler, + VideoEmitType, +) from .utils import ( AdditionalOutputs, Warning, @@ -17,13 +27,7 @@ from .utils import ( audio_to_float32, ) from .webrtc import ( - AsyncAudioVideoStreamHandler, - AsyncStreamHandler, - AudioVideoStreamHandler, - StreamHandler, WebRTC, - VideoEmitType, - AudioEmitType, ) __all__ = [ @@ -44,11 +48,14 @@ __all__ = [ "ReplyOnPause", "ReplyOnStopWords", "SileroVadOptions", - "stt", - "stt_for_chunks", + "get_stt_model", + "MoonshineSTT", "StreamHandler", + "Stream", "VideoEmitType", "WebRTC", "WebRTCError", "Warning", + "get_tts_model", + "KokoroTTSOptions", ] diff --git a/backend/gradio_webrtc/credentials.py b/backend/fastrtc/credentials.py similarity index 100% rename from backend/gradio_webrtc/credentials.py rename to backend/fastrtc/credentials.py diff --git a/backend/gradio_webrtc/pause_detection/__init__.py b/backend/fastrtc/pause_detection/__init__.py similarity index 100% rename from backend/gradio_webrtc/pause_detection/__init__.py rename to backend/fastrtc/pause_detection/__init__.py diff --git a/backend/gradio_webrtc/pause_detection/vad.py b/backend/fastrtc/pause_detection/vad.py similarity index 100% rename from backend/gradio_webrtc/pause_detection/vad.py rename to backend/fastrtc/pause_detection/vad.py diff --git a/backend/gradio_webrtc/reply_on_pause.py b/backend/fastrtc/reply_on_pause.py similarity index 65% rename from backend/gradio_webrtc/reply_on_pause.py rename to backend/fastrtc/reply_on_pause.py index 5733fd4..60b8794 100644 --- a/backend/gradio_webrtc/reply_on_pause.py +++ b/backend/fastrtc/reply_on_pause.py @@ -4,12 +4,15 @@ from dataclasses import dataclass from functools import lru_cache from logging import getLogger from threading import Event -from typing import Any, Callable, Generator, Literal, Union, cast +from typing import Any, AsyncGenerator, Callable, Generator, Literal, cast +import click import numpy as np +from numpy.typing import NDArray -from gradio_webrtc.pause_detection import SileroVADModel, SileroVadOptions -from gradio_webrtc.webrtc import EmitType, StreamHandler +from .pause_detection import SileroVADModel, SileroVadOptions +from .tracks import EmitType, StreamHandler +from .utils import create_message, split_output logger = getLogger(__name__) @@ -18,8 +21,23 @@ counter = 0 @lru_cache def get_vad_model() -> SileroVADModel: - """Returns the VAD model instance.""" - return SileroVADModel() + """Returns the VAD model instance and warms it up with dummy data.""" + try: + import importlib.util + + mod = importlib.util.find_spec("onnxruntime") + if mod is None: + raise RuntimeError("Install fastrtc[vad] to use ReplyOnPause") + except (ValueError, ModuleNotFoundError): + raise RuntimeError("Install fastrtc[vad] to use ReplyOnPause") + model = SileroVADModel() + # Warm up the model with dummy data + print(click.style("INFO", fg="green") + ":\t Warming up VAD model.") + for _ in range(10): + dummy_audio = np.zeros(102400, dtype=np.float32) + model.vad((24000, dummy_audio), None) + print(click.style("INFO", fg="green") + ":\t VAD model warmed up.") + return model @dataclass @@ -40,19 +58,27 @@ class AppState: responding: bool = False stopped: bool = False buffer: np.ndarray | None = None + responded_audio: bool = False -ReplyFnGenerator = Union[ - # For two arguments +ReplyFnGenerator = ( Callable[ - [tuple[int, np.ndarray], list[dict[Any, Any]]], + [tuple[int, NDArray[np.int16]], list[dict[Any, Any]]], Generator[EmitType, None, None], - ], - Callable[ - [tuple[int, np.ndarray]], + ] + | Callable[ + [tuple[int, NDArray[np.int16]]], Generator[EmitType, None, None], - ], -] + ] + | Callable[ + [tuple[int, NDArray[np.int16]]], + AsyncGenerator[EmitType, None], + ] + | Callable[ + [tuple[int, NDArray[np.int16]], list[dict[Any, Any]]], + AsyncGenerator[EmitType, None], + ] +) async def iterate(generator: Generator) -> Any: @@ -152,6 +178,8 @@ class ReplyOnPause(StreamHandler): def reset(self): super().reset() + if self.phone_mode: + self.args_set.set() self.generator = None self.event.clear() self.state = AppState() @@ -164,25 +192,44 @@ class ReplyOnPause(StreamHandler): return None else: if not self.generator: + self.send_message_sync(create_message("log", "pause_detected")) if self._needs_additional_inputs and not self.args_set.is_set(): - asyncio.run_coroutine_threadsafe( - self.wait_for_args(), self.loop - ).result() + if not self.phone_mode: + self.wait_for_args_sync() + else: + self.latest_args = [None] + self.args_set.set() logger.debug("Creating generator") audio = cast(np.ndarray, self.state.stream).reshape(1, -1) if self._needs_additional_inputs: self.latest_args[0] = (self.state.sampling_rate, audio) - self.generator = self.fn(*self.latest_args) + self.generator = self.fn(*self.latest_args) # type: ignore else: self.generator = self.fn((self.state.sampling_rate, audio)) # type: ignore logger.debug("Latest args: %s", self.latest_args) self.state.responding = True try: if self.is_async: - return asyncio.run_coroutine_threadsafe( + output = asyncio.run_coroutine_threadsafe( self.async_iterate(self.generator), self.loop ).result() else: - return next(self.generator) + output = next(self.generator) # type: ignore + audio, additional_outputs = split_output(output) + if audio is not None: + self.send_message_sync(create_message("log", "response_starting")) + self.state.responded_audio = True + if self.phone_mode: + if additional_outputs: + self.latest_args = [None] + list(additional_outputs.args) + return output except (StopIteration, StopAsyncIteration): + if not self.state.responded_audio: + self.send_message_sync(create_message("log", "response_starting")) + self.reset() + except Exception as e: + import traceback + + traceback.print_exc() + logger.debug("Error in ReplyOnPause: %s", e) self.reset() diff --git a/backend/gradio_webrtc/reply_on_stopwords.py b/backend/fastrtc/reply_on_stopwords.py similarity index 90% rename from backend/gradio_webrtc/reply_on_stopwords.py rename to backend/fastrtc/reply_on_stopwords.py index 0f7f9ac..6b5b4a7 100644 --- a/backend/gradio_webrtc/reply_on_stopwords.py +++ b/backend/fastrtc/reply_on_stopwords.py @@ -12,8 +12,8 @@ from .reply_on_pause import ( ReplyOnPause, SileroVadOptions, ) -from .speech_to_text import get_stt_model, stt_for_chunks -from .utils import audio_to_float32 +from .speech_to_text import get_stt_model +from .utils import audio_to_float32, create_message logger = logging.getLogger(__name__) @@ -47,14 +47,16 @@ class ReplyOnStopWords(ReplyOnPause): ) self.stop_words = stop_words self.state = ReplyOnStopWordsState() - # Download Model - get_stt_model() + self.stt_model = get_stt_model("moonshine/base") def stop_word_detected(self, text: str) -> bool: for stop_word in self.stop_words: stop_word = stop_word.lower().strip().split(" ") if bool( - re.search(r"\b" + r"\s+".join(map(re.escape, stop_word)) + r"\b", text) + re.search( + r"\b" + r"\s+".join(map(re.escape, stop_word)) + r"[.,!?]*\b", + text.lower(), + ) ): logger.debug("Stop word detected: %s", stop_word) return True @@ -64,7 +66,7 @@ class ReplyOnStopWords(ReplyOnPause): self, ): if self.channel: - self.channel.send("stopword") + self.channel.send(create_message("stopword", "")) logger.debug("Sent stopword") def send_stopword(self): @@ -97,7 +99,9 @@ class ReplyOnStopWords(ReplyOnPause): self.model_options, return_chunks=True, ) - text = stt_for_chunks((16000, state.post_stop_word_buffer), chunks) + text = self.stt_model.stt_for_chunks( + (16000, state.post_stop_word_buffer), chunks + ) logger.debug(f"STT: {text}") state.stop_word_detected = self.stop_word_detected(text) if state.stop_word_detected: diff --git a/backend/fastrtc/speech_to_text/__init__.py b/backend/fastrtc/speech_to_text/__init__.py new file mode 100644 index 0000000..9dec76b --- /dev/null +++ b/backend/fastrtc/speech_to_text/__init__.py @@ -0,0 +1,3 @@ +from .stt_ import MoonshineSTT, get_stt_model + +__all__ = ["get_stt_model", "MoonshineSTT", "get_stt_model"] diff --git a/backend/fastrtc/speech_to_text/stt_.py b/backend/fastrtc/speech_to_text/stt_.py new file mode 100644 index 0000000..d528d0e --- /dev/null +++ b/backend/fastrtc/speech_to_text/stt_.py @@ -0,0 +1,81 @@ +from functools import lru_cache +from pathlib import Path +from typing import Literal, Protocol + +import click +import librosa +import numpy as np +from numpy.typing import NDArray + +from ..utils import AudioChunk, audio_to_float32 + +curr_dir = Path(__file__).parent + + +class STTModel(Protocol): + def stt(self, audio: tuple[int, NDArray[np.int16 | np.float32]]) -> str: ... + + def stt_for_chunks( + self, + audio: tuple[int, NDArray[np.int16 | np.float32]], + chunks: list[AudioChunk], + ) -> str: ... + + +class MoonshineSTT(STTModel): + def __init__( + self, model: Literal["moonshine/base", "moonshine/tiny"] = "moonshine/base" + ): + try: + from moonshine_onnx import MoonshineOnnxModel, load_tokenizer + except (ImportError, ModuleNotFoundError): + raise ImportError( + "Install fastrtc[stt] for speech-to-text and stopword detection support." + ) + + self.model = MoonshineOnnxModel(model_name=model) + self.tokenizer = load_tokenizer() + + def stt(self, audio: tuple[int, NDArray[np.int16 | np.float32]]) -> str: + sr, audio_np = audio # type: ignore + if audio_np.dtype == np.int16: + audio_np = audio_to_float32(audio) + if sr != 16000: + audio_np: NDArray[np.float32] = librosa.resample( + audio_np, orig_sr=sr, target_sr=16000 + ) + if audio_np.ndim == 1: + audio_np = audio_np.reshape(1, -1) + tokens = self.model.generate(audio_np) + return self.tokenizer.decode_batch(tokens)[0] + + def stt_for_chunks( + self, + audio: tuple[int, NDArray[np.int16 | np.float32]], + chunks: list[AudioChunk], + ) -> str: + sr, audio_np = audio + return " ".join( + [ + self.stt((sr, audio_np[chunk["start"] : chunk["end"]])) + for chunk in chunks + ] + ) + + +@lru_cache +def get_stt_model( + model: Literal["moonshine/base", "moonshine/tiny"] = "moonshine/base", +) -> STTModel: + import os + + os.environ["TOKENIZERS_PARALLELISM"] = "false" + m = MoonshineSTT(model) + from moonshine_onnx import load_audio + + audio = load_audio(str(curr_dir / "test_file.wav")) + print(click.style("INFO", fg="green") + ":\t Warming up STT model.") + + m.stt((16000, audio)) + print(click.style("INFO", fg="green") + ":\t STT model warmed up.") + return m diff --git a/backend/fastrtc/speech_to_text/test_file.wav b/backend/fastrtc/speech_to_text/test_file.wav new file mode 100644 index 0000000..a87858b Binary files /dev/null and b/backend/fastrtc/speech_to_text/test_file.wav differ diff --git a/backend/fastrtc/stream.py b/backend/fastrtc/stream.py new file mode 100644 index 0000000..1baddc8 --- /dev/null +++ b/backend/fastrtc/stream.py @@ -0,0 +1,634 @@ +import logging +from pathlib import Path +from typing import ( + Any, + AsyncContextManager, + Callable, + Literal, + TypedDict, + cast, +) + +import gradio as gr +from fastapi import FastAPI, Request, WebSocket +from fastapi.responses import HTMLResponse +from gradio import Blocks +from gradio.components.base import Component +from pydantic import BaseModel +from typing_extensions import NotRequired + +from .tracks import HandlerType, StreamHandlerImpl +from .webrtc import WebRTC +from .webrtc_connection_mixin import WebRTCConnectionMixin +from .websocket import WebSocketHandler + +logger = logging.getLogger(__name__) + +curr_dir = Path(__file__).parent + + +class Body(BaseModel): + sdp: str + type: str + webrtc_id: str + + +class UIArgs(TypedDict): + title: NotRequired[str] + """Title of the demo""" + icon: NotRequired[str] + """Icon to display on the button instead of the wave animation. The icon should be a path/url to a .svg/.png/.jpeg file.""" + icon_button_color: NotRequired[str] + """Color of the icon button. Default is var(--color-accent) of the demo theme.""" + pulse_color: NotRequired[str] + """Color of the pulse animation. Default is var(--color-accent) of the demo theme.""" + + +class Stream(WebRTCConnectionMixin): + def __init__( + self, + handler: HandlerType, + *, + additional_outputs_handler: Callable | None = None, + mode: Literal["send-receive", "receive", "send"] = "send-receive", + modality: Literal["video", "audio", "audio-video"] = "video", + concurrency_limit: int | None | Literal["default"] = "default", + time_limit: float | None = None, + rtp_params: dict[str, Any] | None = None, + rtc_configuration: dict[str, Any] | None = None, + additional_inputs: list[Component] | None = None, + additional_outputs: list[Component] | None = None, + ui_args: UIArgs | None = None, + ): + self.mode = mode + self.modality = modality + self.rtp_params = rtp_params + self.event_handler = handler + self.concurrency_limit = cast( + (int | float), + 1 if concurrency_limit in ["default", None] else concurrency_limit, + ) + self.time_limit = time_limit + self.additional_output_components = additional_outputs + self.additional_input_components = additional_inputs + self.additional_outputs_handler = additional_outputs_handler + self.rtc_configuration = rtc_configuration + self._ui = self._generate_default_ui(ui_args) + self._ui.launch = self._wrap_gradio_launch(self._ui.launch) + + def mount(self, app: FastAPI): + app.router.post("/webrtc/offer")(self.offer) + app.router.websocket("/telephone/handler")(self.telephone_handler) + app.router.post("/telephone/incoming")(self.handle_incoming_call) + app.router.websocket("/websocket/offer")(self.websocket_offer) + lifespan = self._inject_startup_message(app.router.lifespan_context) + app.router.lifespan_context = lifespan + + @staticmethod + def print_error(env: Literal["colab", "spaces"]): + import click + + print( + click.style("ERROR", fg="red") + + f":\t Running in {env} is not possible without providing a valid rtc_configuration. " + + "See " + + click.style("https://fastrtc.org/deployment/", fg="cyan") + + " for more information." + ) + raise RuntimeError( + f"Running in {env} is not possible without providing a valid rtc_configuration. " + + "See https://fastrtc.org/deployment/ for more information." + ) + + def _check_colab_or_spaces(self): + from gradio.utils import colab_check, get_space + + if colab_check() and not self.rtc_configuration: + self.print_error("colab") + if get_space() and not self.rtc_configuration: + self.print_error("spaces") + + def _wrap_gradio_launch(self, callable): + import contextlib + + def wrapper(*args, **kwargs): + lifespan = kwargs.get("app_kwargs", {}).get("lifespan", None) + + @contextlib.asynccontextmanager + async def new_lifespan(app: FastAPI): + if lifespan is None: + self._check_colab_or_spaces() + yield + else: + async with lifespan(app): + self._check_colab_or_spaces() + yield + + if "app_kwargs" not in kwargs: + kwargs["app_kwargs"] = {} + kwargs["app_kwargs"]["lifespan"] = new_lifespan + return callable(*args, **kwargs) + + return wrapper + + def _inject_startup_message( + self, lifespan: Callable[[FastAPI], AsyncContextManager] | None = None + ): + import contextlib + + import click + + def print_startup_message(): + self._check_colab_or_spaces() + print( + click.style("INFO", fg="green") + + ":\t Visit " + + click.style("https://fastrtc.org/userguide/api/", fg="cyan") + + " for WebRTC or Websocket API docs." + ) + + @contextlib.asynccontextmanager + async def new_lifespan(app: FastAPI): + if lifespan is None: + print_startup_message() + yield + else: + async with lifespan(app): + print_startup_message() + yield + + return new_lifespan + + def _generate_default_ui( + self, + ui_args: UIArgs | None = None, + ): + ui_args = ui_args or {} + same_components = [] + additional_input_components = self.additional_input_components or [] + additional_output_components = self.additional_output_components or [] + if additional_output_components and not self.additional_outputs_handler: + raise ValueError( + "additional_outputs_handler must be provided if there are additional output components." + ) + if additional_input_components and additional_output_components: + same_components = [ + component + for component in additional_input_components + if component in additional_output_components + ] + for component in additional_output_components: + if component not in same_components: + same_components.append(component) + if self.modality == "video" and self.mode == "receive": + with gr.Blocks() as demo: + gr.HTML( + f""" +

+ {ui_args.get("title", "Video Streaming (Powered by WebRTC ⚡️)")} +

+ """ + ) + with gr.Row(): + if additional_input_components: + with gr.Column(): + for component in additional_input_components: + component.render() + button = gr.Button("Start Stream", variant="primary") + with gr.Column(): + output_video = WebRTC( + label="Video Stream", + rtc_configuration=self.rtc_configuration, + mode="receive", + modality="video", + ) + for component in additional_output_components: + if component not in same_components: + component.render() + output_video.stream( + fn=self.event_handler, + inputs=self.additional_input_components, + outputs=[output_video], + trigger=button.click, + time_limit=self.time_limit, + concurrency_limit=self.concurrency_limit, # type: ignore + ) + if additional_output_components: + assert self.additional_outputs_handler + output_video.on_additional_outputs( + self.additional_outputs_handler, + outputs=additional_output_components, + ) + elif self.modality == "video" and self.mode == "send": + with gr.Blocks() as demo: + gr.HTML( + f""" +

+ {ui_args.get("title", "Video Streaming (Powered by WebRTC ⚡️)")} +

+ """ + ) + with gr.Row(): + if additional_input_components: + with gr.Column(): + for component in additional_input_components: + component.render() + with gr.Column(): + output_video = WebRTC( + label="Video Stream", + rtc_configuration=self.rtc_configuration, + mode="send", + modality="video", + ) + for component in additional_output_components: + if component not in same_components: + component.render() + output_video.stream( + fn=self.event_handler, + inputs=[output_video] + additional_input_components, + outputs=[output_video], + time_limit=self.time_limit, + concurrency_limit=self.concurrency_limit, # type: ignore + ) + if additional_output_components: + assert self.additional_outputs_handler + output_video.on_additional_outputs( + self.additional_outputs_handler, + outputs=additional_output_components, + ) + elif self.modality == "video" and self.mode == "send-receive": + css = """.my-group {max-width: 600px !important; max-height: 600 !important;} + .my-column {display: flex !important; justify-content: center !important; align-items: center !important};""" + + with gr.Blocks(css=css) as demo: + gr.HTML( + f""" +

+ {ui_args.get("title", "Video Streaming (Powered by WebRTC ⚡️)")} +

+ """ + ) + with gr.Column(elem_classes=["my-column"]): + with gr.Group(elem_classes=["my-group"]): + image = WebRTC( + label="Stream", + rtc_configuration=self.rtc_configuration, + mode="send-receive", + modality="video", + ) + for component in additional_input_components: + component.render() + if additional_output_components: + with gr.Column(): + for component in additional_output_components: + if component not in same_components: + component.render() + + image.stream( + fn=self.event_handler, + inputs=[image] + additional_input_components, + outputs=[image], + time_limit=self.time_limit, + concurrency_limit=self.concurrency_limit, # type: ignore + ) + if additional_output_components: + assert self.additional_outputs_handler + image.on_additional_outputs( + self.additional_outputs_handler, + inputs=additional_output_components, + outputs=additional_output_components, + ) + elif self.modality == "audio" and self.mode == "receive": + with gr.Blocks() as demo: + gr.HTML( + """ +

+ FastAPI (Powered by WebRTC ⚡️) +

+ """ + ) + with gr.Row(): + with gr.Column(): + for component in additional_input_components: + component.render() + button = gr.Button("Start Stream", variant="primary") + if additional_output_components: + with gr.Column(): + output_video = WebRTC( + label="Audio Stream", + rtc_configuration=self.rtc_configuration, + mode="receive", + modality="audio", + icon=ui_args.get("icon"), + icon_button_color=ui_args.get("icon_button_color"), + pulse_color=ui_args.get("pulse_color"), + ) + for component in additional_output_components: + if component not in same_components: + component.render() + output_video.stream( + fn=self.event_handler, + inputs=self.additional_input_components, + outputs=[output_video], + trigger=button.click, + time_limit=self.time_limit, + concurrency_limit=self.concurrency_limit, # type: ignore + ) + if additional_output_components: + assert self.additional_outputs_handler + output_video.on_additional_outputs( + self.additional_outputs_handler, + inputs=additional_output_components, + outputs=additional_output_components, + ) + elif self.modality == "audio" and self.mode == "send": + with gr.Blocks() as demo: + gr.HTML( + f""" +

+ {ui_args.get("title", "Audio Streaming (Powered by WebRTC ⚡️)")} +

+ """ + ) + with gr.Row(): + with gr.Column(): + with gr.Group(): + image = WebRTC( + label="Stream", + rtc_configuration=self.rtc_configuration, + mode="send-receive", + modality="audio", + ) + for component in additional_input_components: + if component not in same_components: + component.render() + if additional_output_components: + with gr.Column(): + for component in additional_output_components: + component.render() + image.stream( + fn=self.event_handler, + inputs=[image] + additional_input_components, + outputs=[image], + time_limit=self.time_limit, + concurrency_limit=self.concurrency_limit, # type: ignore + ) + if additional_output_components: + assert self.additional_outputs_handler + image.on_additional_outputs( + self.additional_outputs_handler, + inputs=additional_output_components, + outputs=additional_output_components, + ) + elif self.modality == "audio" and self.mode == "send-receive": + with gr.Blocks() as demo: + gr.HTML( + f""" +

+ {ui_args.get("title", "Audio Streaming (Powered by WebRTC ⚡️)")} +

+ """ + ) + with gr.Row(): + with gr.Column(): + with gr.Group(): + image = WebRTC( + label="Stream", + rtc_configuration=self.rtc_configuration, + mode="send-receive", + modality="audio", + icon=ui_args.get("icon"), + icon_button_color=ui_args.get("icon_button_color"), + pulse_color=ui_args.get("pulse_color"), + ) + for component in additional_input_components: + if component not in same_components: + component.render() + if additional_output_components: + with gr.Column(): + for component in additional_output_components: + component.render() + + image.stream( + fn=self.event_handler, + inputs=[image] + additional_input_components, + outputs=[image], + time_limit=self.time_limit, + concurrency_limit=self.concurrency_limit, # type: ignore + ) + if additional_output_components: + assert self.additional_outputs_handler + image.on_additional_outputs( + self.additional_outputs_handler, + inputs=additional_output_components, + outputs=additional_output_components, + ) + elif self.modality == "audio-video" and self.mode == "send-receive": + with gr.Blocks() as demo: + gr.HTML( + f""" +

+ {ui_args.get("title", "Audio Streaming (Powered by WebRTC ⚡️)")} +

+ """ + ) + with gr.Row(): + with gr.Column(): + with gr.Group(): + image = WebRTC( + label="Stream", + rtc_configuration=self.rtc_configuration, + mode="send-receive", + modality="audio-video", + icon=ui_args.get("icon"), + icon_button_color=ui_args.get("icon_button_color"), + pulse_color=ui_args.get("pulse_color"), + ) + for component in additional_input_components: + if component not in same_components: + component.render() + if additional_output_components: + with gr.Column(): + for component in additional_output_components: + component.render() + + image.stream( + fn=self.event_handler, + inputs=[image] + additional_input_components, + outputs=[image], + time_limit=self.time_limit, + concurrency_limit=self.concurrency_limit, # type: ignore + ) + if additional_output_components: + assert self.additional_outputs_handler + image.on_additional_outputs( + self.additional_outputs_handler, + inputs=additional_output_components, + outputs=additional_output_components, + ) + return demo + + @property + def ui(self) -> Blocks: + return self._ui + + @ui.setter + def ui(self, blocks: Blocks): + self._ui = blocks + + async def offer(self, body: Body): + return await self.handle_offer( + body.model_dump(), set_outputs=self.set_additional_outputs(body.webrtc_id) + ) + + async def handle_incoming_call(self, request: Request): + from twilio.twiml.voice_response import Connect, VoiceResponse + + response = VoiceResponse() + response.say("Connecting to the AI assistant.") + connect = Connect() + connect.stream(url=f"wss://{request.url.hostname}/telephone/handler") + response.append(connect) + response.say("The call has been disconnected.") + return HTMLResponse(content=str(response), media_type="application/xml") + + async def telephone_handler(self, websocket: WebSocket): + handler = cast(StreamHandlerImpl, self.event_handler.copy()) + handler.phone_mode = True + + async def set_handler(s: str, a: WebSocketHandler): + if len(self.connections) >= self.concurrency_limit: + await cast(WebSocket, a.websocket).send_json( + { + "status": "failed", + "meta": { + "error": "concurrency_limit_reached", + "limit": self.concurrency_limit, + }, + } + ) + await websocket.close() + return + + ws = WebSocketHandler( + handler, set_handler, lambda s: None, lambda s: lambda a: None + ) + await ws.handle_websocket(websocket) + + async def websocket_offer(self, websocket: WebSocket): + handler = cast(StreamHandlerImpl, self.event_handler.copy()) + handler.phone_mode = False + + async def set_handler(s: str, a: WebSocketHandler): + if len(self.connections) >= self.concurrency_limit: + await cast(WebSocket, a.websocket).send_json( + { + "status": "failed", + "meta": { + "error": "concurrency_limit_reached", + "limit": self.concurrency_limit, + }, + } + ) + await websocket.close() + return + + self.connections[s] = [a] # type: ignore + + def clean_up(s): + self.clean_up(s) + + ws = WebSocketHandler( + handler, set_handler, clean_up, lambda s: self.set_additional_outputs(s) + ) + await ws.handle_websocket(websocket) + + def fastphone( + self, + token: str | None = None, + host: str = "127.0.0.1", + port: int = 8000, + **kwargs, + ): + import secrets + import threading + import time + import urllib.parse + + import click + import httpx + import uvicorn + from gradio.networking import setup_tunnel + from gradio.tunneling import CURRENT_TUNNELS + from huggingface_hub import get_token + + app = FastAPI() + + self.mount(app) + + t = threading.Thread( + target=uvicorn.run, + args=(app,), + kwargs={"host": host, "port": port, **kwargs}, + ) + t.start() + + url = setup_tunnel( + host, port, share_token=secrets.token_urlsafe(32), share_server_address=None + ) + host = urllib.parse.urlparse(url).netloc + + URL = "https://api.fastrtc.org" + r = httpx.post( + URL + "/register", + json={"url": host}, + headers={"Authorization": token or get_token() or ""}, + ) + r.raise_for_status() + data = r.json() + code = f"{data['code']}" + phone_number = data["phone"] + reset_date = data["reset_date"] + print( + click.style("INFO", fg="green") + + ":\t Your FastPhone is now live! Call " + + click.style(phone_number, fg="cyan") + + " and use code " + + click.style(code, fg="cyan") + + " to connect to your stream." + ) + minutes = str(int(data["time_remaining"] // 60)).zfill(2) + seconds = str(int(data["time_remaining"] % 60)).zfill(2) + print( + click.style("INFO", fg="green") + + ":\t You have " + + click.style(f"{minutes}:{seconds}", fg="cyan") + + " minutes remaining in your quota (Resetting on " + + click.style(f"{reset_date}", fg="cyan") + + ")" + ) + print( + click.style("INFO", fg="green") + + ":\t Visit " + + click.style( + "https://fastrtc.org/userguide/audio/#telephone-integration", + fg="cyan", + ) + + " for information on making your handler compatible with phone usage." + ) + try: + while True: + time.sleep(0.1) + except (KeyboardInterrupt, OSError): + print( + click.style("INFO", fg="green") + + ":\t Keyboard interruption in main thread... closing server." + ) + r = httpx.post( + URL + "/unregister", + json={"url": host, "code": code}, + headers={"Authorization": token or get_token() or ""}, + ) + t.join(timeout=5) + for tunnel in CURRENT_TUNNELS: + tunnel.kill() diff --git a/backend/fastrtc/text_to_speech/__init__.py b/backend/fastrtc/text_to_speech/__init__.py new file mode 100644 index 0000000..2cc082a --- /dev/null +++ b/backend/fastrtc/text_to_speech/__init__.py @@ -0,0 +1,3 @@ +from .tts import KokoroTTSOptions, get_tts_model + +__all__ = ["get_tts_model", "KokoroTTSOptions"] diff --git a/backend/fastrtc/text_to_speech/tts.py b/backend/fastrtc/text_to_speech/tts.py new file mode 100644 index 0000000..318a7ae --- /dev/null +++ b/backend/fastrtc/text_to_speech/tts.py @@ -0,0 +1,90 @@ +import asyncio +import re +from dataclasses import dataclass +from functools import lru_cache +from typing import AsyncGenerator, Generator, Literal, Protocol + +import numpy as np +from huggingface_hub import hf_hub_download +from numpy.typing import NDArray + + +class TTSOptions: + pass + + +class TTSModel(Protocol): + def tts(self, text: str) -> tuple[int, NDArray[np.float32]]: ... + + async def stream_tts( + self, text: str, options: TTSOptions | None = None + ) -> AsyncGenerator[tuple[int, NDArray[np.float32]], None]: ... + + def stream_tts_sync( + self, text: str, options: TTSOptions | None = None + ) -> Generator[tuple[int, NDArray[np.float32]], None, None]: ... + + +@dataclass +class KokoroTTSOptions(TTSOptions): + voice: str = "af_heart" + speed: float = 1.0 + lang: str = "en-us" + + +@lru_cache +def get_tts_model(model: Literal["kokoro"] = "kokoro") -> TTSModel: + m = KokoroTTSModel() + m.tts("Hello, world!") + return m + + +class KokoroTTSModel(TTSModel): + def __init__(self): + from kokoro_onnx import Kokoro + + self.model = Kokoro( + model_path=hf_hub_download("fastrtc/kokoro-onnx", "kokoro-v1.0.onnx"), + voices_path=hf_hub_download("fastrtc/kokoro-onnx", "voices-v1.0.bin"), + ) + + def tts( + self, text: str, options: KokoroTTSOptions | None = None + ) -> tuple[int, NDArray[np.float32]]: + options = options or KokoroTTSOptions() + a, b = self.model.create( + text, voice=options.voice, speed=options.speed, lang=options.lang + ) + return b, a + + async def stream_tts( + self, text: str, options: KokoroTTSOptions | None = None + ) -> AsyncGenerator[tuple[int, NDArray[np.float32]], None]: + options = options or KokoroTTSOptions() + + sentences = re.split(r"(?<=[.!?])\s+", text.strip()) + + for s_idx, sentence in enumerate(sentences): + if not sentence.strip(): + continue + + chunk_idx = 0 + async for chunk in self.model.create_stream( + sentence, voice=options.voice, speed=options.speed, lang=options.lang + ): + if s_idx != 0 and chunk_idx == 0: + yield chunk[1], np.zeros(chunk[1] // 7, dtype=np.float32) + yield chunk[1], chunk[0] + + def stream_tts_sync( + self, text: str, options: KokoroTTSOptions | None = None + ) -> Generator[tuple[int, NDArray[np.float32]], None, None]: + loop = asyncio.new_event_loop() + + # Use the new loop to run the async generator + iterator = self.stream_tts(text, options).__aiter__() + while True: + try: + yield loop.run_until_complete(iterator.__anext__()) + except StopAsyncIteration: + break diff --git a/backend/fastrtc/tracks.py b/backend/fastrtc/tracks.py new file mode 100644 index 0000000..91104b8 --- /dev/null +++ b/backend/fastrtc/tracks.py @@ -0,0 +1,689 @@ +"""gr.WebRTC() component.""" + +from __future__ import annotations + +import asyncio +import functools +import inspect +import logging +import threading +import time +import traceback +from abc import ABC, abstractmethod +from collections.abc import Callable +from typing import ( + Any, + Generator, + Literal, + TypeAlias, + Union, + cast, +) + +import anyio.to_thread +import av +import numpy as np +from aiortc import ( + AudioStreamTrack, + MediaStreamTrack, + VideoStreamTrack, +) +from aiortc.contrib.media import AudioFrame, VideoFrame # type: ignore +from aiortc.mediastreams import MediaStreamError +from numpy import typing as npt + +from fastrtc.utils import ( + AdditionalOutputs, + DataChannel, + create_message, + current_channel, + player_worker_decode, + split_output, +) + +logger = logging.getLogger(__name__) + +VideoNDArray: TypeAlias = Union[ + np.ndarray[Any, np.dtype[np.uint8]], + np.ndarray[Any, np.dtype[np.uint16]], + np.ndarray[Any, np.dtype[np.float32]], +] + +VideoEmitType = ( + VideoNDArray | tuple[VideoNDArray, AdditionalOutputs] | AdditionalOutputs +) +VideoEventHandler = Callable[[npt.ArrayLike], VideoEmitType] + + +class VideoCallback(VideoStreamTrack): + """ + This works for streaming input and output + """ + + kind = "video" + + def __init__( + self, + track: MediaStreamTrack, + event_handler: VideoEventHandler, + channel: DataChannel | None = None, + set_additional_outputs: Callable | None = None, + mode: Literal["send-receive", "send"] = "send-receive", + ) -> None: + super().__init__() # don't forget this! + self.track = track + self.event_handler = event_handler + self.latest_args: str | list[Any] = "not_set" + self.channel = channel + self.set_additional_outputs = set_additional_outputs + self.thread_quit = asyncio.Event() + self.mode = mode + self.channel_set = asyncio.Event() + self.has_started = False + + def set_channel(self, channel: DataChannel): + self.channel = channel + current_channel.set(channel) + self.channel_set.set() + + def set_args(self, args: list[Any]): + self.latest_args = ["__webrtc_value__"] + list(args) + + def add_frame_to_payload( + self, args: list[Any], frame: np.ndarray | None + ) -> list[Any]: + new_args = [] + for val in args: + if isinstance(val, str) and val == "__webrtc_value__": + new_args.append(frame) + else: + new_args.append(val) + return new_args + + def array_to_frame(self, array: np.ndarray) -> VideoFrame: + return VideoFrame.from_ndarray(array, format="bgr24") + + async def process_frames(self): + while not self.thread_quit.is_set(): + try: + await self.recv() + except TimeoutError: + continue + + def start( + self, + ): + asyncio.create_task(self.process_frames()) + + def stop(self): + super().stop() + logger.debug("video callback stop") + self.thread_quit.set() + + async def wait_for_channel(self): + if not self.channel_set.is_set(): + await self.channel_set.wait() + if current_channel.get() != self.channel: + current_channel.set(self.channel) + + async def recv(self): # type: ignore + try: + try: + frame = cast(VideoFrame, await self.track.recv()) + except MediaStreamError: + self.stop() + return + + await self.wait_for_channel() + frame_array = frame.to_ndarray(format="bgr24") + if self.latest_args == "not_set": + return frame + + args = self.add_frame_to_payload(cast(list, self.latest_args), frame_array) + + array, outputs = split_output(self.event_handler(*args)) + if ( + isinstance(outputs, AdditionalOutputs) + and self.set_additional_outputs + and self.channel + ): + self.set_additional_outputs(outputs) + self.channel.send(create_message("fetch_output", [])) + if array is None and self.mode == "send": + return + + new_frame = self.array_to_frame(array) + if frame: + new_frame.pts = frame.pts + new_frame.time_base = frame.time_base + else: + pts, time_base = await self.next_timestamp() + new_frame.pts = pts + new_frame.time_base = time_base + + return new_frame + except Exception as e: + logger.debug("exception %s", e) + exec = traceback.format_exc() + logger.debug("traceback %s", exec) + + +class StreamHandlerBase(ABC): + def __init__( + self, + expected_layout: Literal["mono", "stereo"] = "mono", + output_sample_rate: int = 24000, + output_frame_size: int = 960, + input_sample_rate: int = 48000, + ) -> None: + self.expected_layout = expected_layout + self.output_sample_rate = output_sample_rate + self.output_frame_size = output_frame_size + self.input_sample_rate = input_sample_rate + self.latest_args: list[Any] = [] + self._resampler = None + self._channel: DataChannel | None = None + self._loop: asyncio.AbstractEventLoop + self.args_set = asyncio.Event() + self.channel_set = asyncio.Event() + self._phone_mode = False + + @property + def loop(self) -> asyncio.AbstractEventLoop: + return cast(asyncio.AbstractEventLoop, self._loop) + + @property + def channel(self) -> DataChannel | None: + return self._channel + + @property + def phone_mode(self) -> bool: + return self._phone_mode + + @phone_mode.setter + def phone_mode(self, value: bool): + self._phone_mode = value + + def set_channel(self, channel: DataChannel): + self._channel = channel + self.channel_set.set() + + async def fetch_args( + self, + ): + if self.channel: + self.channel.send(create_message("send_input", [])) + logger.debug("Sent send_input") + + async def wait_for_args(self): + if not self.phone_mode: + await self.fetch_args() + await self.args_set.wait() + else: + self.args_set.set() + + def wait_for_args_sync(self): + try: + asyncio.run_coroutine_threadsafe(self.wait_for_args(), self.loop).result() + except Exception: + import traceback + + traceback.print_exc() + + async def send_message(self, msg: str): + if self.channel: + self.channel.send(msg) + logger.debug("Sent msg %s", msg) + + def send_message_sync(self, msg: str): + asyncio.run_coroutine_threadsafe(self.send_message(msg), self.loop).result() + logger.debug("Sent msg %s", msg) + + def set_args(self, args: list[Any]): + logger.debug("setting args in audio callback %s", args) + self.latest_args = ["__webrtc_value__"] + list(args) + self.args_set.set() + + def reset(self): + self.args_set.clear() + + def shutdown(self): + pass + + def resample(self, frame: AudioFrame) -> Generator[AudioFrame, None, None]: + if self._resampler is None: + self._resampler = av.AudioResampler( # type: ignore + format="s16", + layout=self.expected_layout, + rate=self.input_sample_rate, + frame_size=frame.samples, + ) + yield from self._resampler.resample(frame) + + +EmitType: TypeAlias = ( + tuple[int, npt.NDArray[np.int16 | np.float32]] + | tuple[int, npt.NDArray[np.int16 | np.float32], Literal["mono", "stereo"]] + | AdditionalOutputs + | tuple[tuple[int, npt.NDArray[np.int16 | np.float32]], AdditionalOutputs] + | None +) +AudioEmitType = EmitType + + +class StreamHandler(StreamHandlerBase): + @abstractmethod + def receive(self, frame: tuple[int, npt.NDArray[np.int16]]) -> None: + pass + + @abstractmethod + def emit( + self, + ) -> EmitType: + pass + + @abstractmethod + def copy(self) -> StreamHandler: + pass + + def start_up(self): + pass + + +class AsyncStreamHandler(StreamHandlerBase): + @abstractmethod + async def receive(self, frame: tuple[int, npt.NDArray[np.int16]]) -> None: + pass + + @abstractmethod + async def emit( + self, + ) -> EmitType: + pass + + @abstractmethod + def copy(self) -> AsyncStreamHandler: + pass + + async def start_up(self): + pass + + +StreamHandlerImpl = StreamHandler | AsyncStreamHandler + + +class AudioVideoStreamHandler(StreamHandlerBase): + @abstractmethod + def video_receive(self, frame: VideoFrame) -> None: + pass + + @abstractmethod + def video_emit( + self, + ) -> VideoEmitType: + pass + + @abstractmethod + def copy(self) -> AudioVideoStreamHandler: + pass + + +class AsyncAudioVideoStreamHandler(StreamHandlerBase): + @abstractmethod + async def video_receive(self, frame: npt.NDArray[np.float32]) -> None: + pass + + @abstractmethod + async def video_emit( + self, + ) -> VideoEmitType: + pass + + @abstractmethod + def copy(self) -> AsyncAudioVideoStreamHandler: + pass + + +VideoStreamHandlerImpl = AudioVideoStreamHandler | AsyncAudioVideoStreamHandler +AudioVideoStreamHandlerImpl = AudioVideoStreamHandler | AsyncAudioVideoStreamHandler +AsyncHandler = AsyncStreamHandler | AsyncAudioVideoStreamHandler + +HandlerType = StreamHandlerImpl | VideoStreamHandlerImpl | VideoEventHandler | Callable + + +class VideoStreamHandler(VideoCallback): + async def process_frames(self): + while not self.thread_quit.is_set(): + try: + await self.channel_set.wait() + frame = cast(VideoFrame, await self.track.recv()) + frame_array = frame.to_ndarray(format="bgr24") + handler = cast(VideoStreamHandlerImpl, self.event_handler) + if inspect.iscoroutinefunction(handler.video_receive): + await handler.video_receive(frame_array) + else: + handler.video_receive(frame_array) # type: ignore + except MediaStreamError: + self.stop() + + def start(self): + if not self.has_started: + asyncio.create_task(self.process_frames()) + self.has_started = True + + async def recv(self): # type: ignore + self.start() + try: + handler = cast(VideoStreamHandlerImpl, self.event_handler) + if inspect.iscoroutinefunction(handler.video_emit): + outputs = await handler.video_emit() + else: + outputs = handler.video_emit() + + array, outputs = split_output(outputs) + if ( + isinstance(outputs, AdditionalOutputs) + and self.set_additional_outputs + and self.channel + ): + self.set_additional_outputs(outputs) + self.channel.send(create_message("fetch_output", [])) + if array is None and self.mode == "send": + return + + new_frame = self.array_to_frame(array) + + # Will probably have to give developer ability to set pts and time_base + pts, time_base = await self.next_timestamp() + new_frame.pts = pts + new_frame.time_base = time_base + + return new_frame + except Exception as e: + logger.debug("exception %s", e) + exec = traceback.format_exc() + logger.debug("traceback %s", exec) + + +class AudioCallback(AudioStreamTrack): + kind = "audio" + + def __init__( + self, + track: MediaStreamTrack, + event_handler: StreamHandlerBase, + channel: DataChannel | None = None, + set_additional_outputs: Callable | None = None, + ) -> None: + super().__init__() + self.track = track + self.event_handler = cast(StreamHandlerImpl, event_handler) + self.current_timestamp = 0 + self.latest_args: str | list[Any] = "not_set" + self.queue = asyncio.Queue() + self.thread_quit = asyncio.Event() + self._start: float | None = None + self.has_started = False + self.last_timestamp = 0 + self.channel = channel + self.set_additional_outputs = set_additional_outputs + + def set_channel(self, channel: DataChannel): + self.channel = channel + self.event_handler.set_channel(channel) + + def set_args(self, args: list[Any]): + self.event_handler.set_args(args) + + def event_handler_receive(self, frame: tuple[int, np.ndarray]) -> None: + current_channel.set(self.event_handler.channel) + return cast(Callable, self.event_handler.receive)(frame) + + def event_handler_emit(self) -> EmitType: + current_channel.set(self.event_handler.channel) + return cast(Callable, self.event_handler.emit)() + + async def process_input_frames(self) -> None: + while not self.thread_quit.is_set(): + try: + frame = cast(AudioFrame, await self.track.recv()) + for frame in self.event_handler.resample(frame): + numpy_array = frame.to_ndarray() + if isinstance(self.event_handler, AsyncHandler): + await self.event_handler.receive( + (frame.sample_rate, numpy_array) # type: ignore + ) + else: + await anyio.to_thread.run_sync( + self.event_handler_receive, (frame.sample_rate, numpy_array) + ) + except MediaStreamError: + logger.debug("MediaStreamError in process_input_frames") + break + + def start(self): + if not self.has_started: + loop = asyncio.get_running_loop() + if isinstance(self.event_handler, AsyncHandler): + callable = self.event_handler.emit + start_up = self.event_handler.start_up() + else: + callable = functools.partial( + loop.run_in_executor, None, self.event_handler_emit + ) + start_up = anyio.to_thread.run_sync(self.event_handler.start_up) + self.process_input_task = asyncio.create_task(self.process_input_frames()) + self.process_input_task.add_done_callback( + lambda _: logger.debug("process_input_done") + ) + self.start_up_task = asyncio.create_task(start_up) + self.start_up_task.add_done_callback( + lambda _: logger.debug("start_up_done") + ) + self.decode_task = asyncio.create_task( + player_worker_decode( + callable, + self.queue, + self.thread_quit, + lambda: self.channel, + self.set_additional_outputs, + False, + self.event_handler.output_sample_rate, + self.event_handler.output_frame_size, + ) + ) + self.decode_task.add_done_callback(lambda _: logger.debug("decode_done")) + self.has_started = True + + async def recv(self): # type: ignore + try: + if self.readyState != "live": + raise MediaStreamError + + if not self.event_handler.channel_set.is_set(): + await self.event_handler.channel_set.wait() + if current_channel.get() != self.event_handler.channel: + current_channel.set(self.event_handler.channel) + self.start() + + frame = await self.queue.get() + logger.debug("frame %s", frame) + + data_time = frame.time + + if time.time() - self.last_timestamp > 10 * ( + self.event_handler.output_frame_size + / self.event_handler.output_sample_rate + ): + self._start = None + + # control playback rate + if self._start is None: + self._start = time.time() - data_time # type: ignore + else: + wait = self._start + data_time - time.time() + await asyncio.sleep(wait) + self.last_timestamp = time.time() + return frame + except Exception as e: + logger.debug("exception %s", e) + exec = traceback.format_exc() + logger.debug("traceback %s", exec) + + def stop(self): + logger.debug("audio callback stop") + self.thread_quit.set() + super().stop() + + +class ServerToClientVideo(VideoStreamTrack): + """ + This works for streaming input and output + """ + + kind = "video" + + def __init__( + self, + event_handler: Callable, + channel: DataChannel | None = None, + set_additional_outputs: Callable | None = None, + ) -> None: + super().__init__() # don't forget this! + self.event_handler = event_handler + self.args_set = asyncio.Event() + self.latest_args: str | list[Any] = "not_set" + self.generator: Generator[Any, None, Any] | None = None + self.channel = channel + self.set_additional_outputs = set_additional_outputs + + def array_to_frame(self, array: np.ndarray) -> VideoFrame: + return VideoFrame.from_ndarray(array, format="bgr24") + + def set_channel(self, channel: DataChannel): + self.channel = channel + + def set_args(self, args: list[Any]): + self.latest_args = list(args) + self.args_set.set() + + async def recv(self): # type: ignore + try: + pts, time_base = await self.next_timestamp() + await self.args_set.wait() + if self.generator is None: + self.generator = cast( + Generator[Any, None, Any], self.event_handler(*self.latest_args) + ) + current_channel.set(self.channel) + try: + next_array, outputs = split_output(next(self.generator)) + if ( + isinstance(outputs, AdditionalOutputs) + and self.set_additional_outputs + and self.channel + ): + self.set_additional_outputs(outputs) + self.channel.send(create_message("fetch_output", [])) + except StopIteration: + self.stop() + return + + next_frame = self.array_to_frame(next_array) + next_frame.pts = pts + next_frame.time_base = time_base + return next_frame + except Exception as e: + logger.debug("exception %s", e) + exec = traceback.format_exc() + logger.debug("traceback %s ", exec) + + +class ServerToClientAudio(AudioStreamTrack): + kind = "audio" + + def __init__( + self, + event_handler: Callable, + channel: DataChannel | None = None, + set_additional_outputs: Callable | None = None, + ) -> None: + self.generator: Generator[Any, None, Any] | None = None + self.event_handler = event_handler + self.current_timestamp = 0 + self.latest_args: str | list[Any] = "not_set" + self.args_set = threading.Event() + self.queue = asyncio.Queue() + self.thread_quit = asyncio.Event() + self.channel = channel + self.set_additional_outputs = set_additional_outputs + self.has_started = False + self._start: float | None = None + super().__init__() + + def set_channel(self, channel: DataChannel): + self.channel = channel + + def set_args(self, args: list[Any]): + self.latest_args = list(args) + self.args_set.set() + + def next(self) -> tuple[int, np.ndarray] | None: + self.args_set.wait() + current_channel.set(self.channel) + if self.generator is None: + self.generator = self.event_handler(*self.latest_args) + if self.generator is not None: + try: + frame = next(self.generator) + return frame + except StopIteration: + self.thread_quit.set() + + def start(self): + if not self.has_started: + loop = asyncio.get_running_loop() + callable = functools.partial(loop.run_in_executor, None, self.next) + asyncio.create_task( + player_worker_decode( + callable, + self.queue, + self.thread_quit, + lambda: self.channel, + self.set_additional_outputs, + True, + ) + ) + self.has_started = True + + async def recv(self): # type: ignore + try: + if self.readyState != "live": + raise MediaStreamError + + self.start() + data = await self.queue.get() + if data is None: + self.stop() + return + + data_time = data.time + + # control playback rate + if data_time is not None: + if self._start is None: + self._start = time.time() - data_time # type: ignore + else: + wait = self._start + data_time - time.time() + await asyncio.sleep(wait) + + return data + except Exception as e: + logger.debug("exception %s", e) + exec = traceback.format_exc() + logger.debug("traceback %s", exec) + + def stop(self): + logger.debug("audio-to-client stop callback") + self.thread_quit.set() + super().stop() diff --git a/backend/gradio_webrtc/utils.py b/backend/fastrtc/utils.py similarity index 81% rename from backend/gradio_webrtc/utils.py rename to backend/fastrtc/utils.py index a2823ad..4c736e2 100644 --- a/backend/gradio_webrtc/utils.py +++ b/backend/fastrtc/utils.py @@ -5,10 +5,11 @@ import json import logging import tempfile from contextvars import ContextVar -from typing import Any, Callable, Protocol, TypedDict, cast +from typing import Any, Callable, Literal, Protocol, TypedDict, cast import av import numpy as np +from numpy.typing import NDArray from pydub import AudioSegment logger = logging.getLogger(__name__) @@ -31,6 +32,20 @@ class DataChannel(Protocol): def send(self, message: str) -> None: ... +def create_message( + type: Literal[ + "send_input", + "fetch_output", + "stopword", + "error", + "warning", + "log", + ], + data: list[Any] | str, +) -> str: + return json.dumps({"type": type, "data": data}) + + current_channel: ContextVar[DataChannel | None] = ContextVar( "current_channel", default=None ) @@ -48,7 +63,6 @@ def _send_log(message: str, type: str) -> None: ) if channel := current_channel.get(): - print("channel", channel) try: loop = asyncio.get_running_loop() asyncio.run_coroutine_threadsafe(_send(channel), loop) @@ -131,7 +145,7 @@ async def player_worker_decode( and channel() ): set_additional_outputs(outputs) - cast(DataChannel, channel()).send("change") + cast(DataChannel, channel()).send(create_message("fetch_output", [])) if frame is None: if quit_on_none: @@ -153,6 +167,9 @@ async def player_worker_decode( ) format = "s16" if audio_array.dtype == "int16" else "fltp" # type: ignore + if audio_array.ndim == 1: + audio_array = audio_array.reshape(1, -1) + # Convert to audio frame and resample # This runs in the same timeout context frame = av.AudioFrame.from_ndarray( # type: ignore @@ -167,7 +184,6 @@ async def player_worker_decode( processed_frame.time_base = audio_time_base audio_samples += processed_frame.samples await queue.put(processed_frame) - logger.debug("Queue size utils.py: %s", queue.qsize()) except (TimeoutError, asyncio.TimeoutError): logger.warning( @@ -178,12 +194,12 @@ async def player_worker_decode( import traceback exec = traceback.format_exc() - logger.debug("traceback %s", exec) - logger.error("Error processing frame: %s", str(e)) + print("traceback %s", exec) + print("Error processing frame: %s", str(e)) continue -def audio_to_bytes(audio: tuple[int, np.ndarray]) -> bytes: +def audio_to_bytes(audio: tuple[int, NDArray[np.int16 | np.float32]]) -> bytes: """ Convert an audio tuple containing sample rate and numpy array data into bytes. @@ -217,7 +233,7 @@ def audio_to_bytes(audio: tuple[int, np.ndarray]) -> bytes: return audio_buffer.getvalue() -def audio_to_file(audio: tuple[int, np.ndarray]) -> str: +def audio_to_file(audio: tuple[int, NDArray[np.int16 | np.float32]]) -> str: """ Save an audio tuple containing sample rate and numpy array data to a file. @@ -247,7 +263,9 @@ def audio_to_file(audio: tuple[int, np.ndarray]) -> str: return f.name -def audio_to_float32(audio: tuple[int, np.ndarray]) -> np.ndarray: +def audio_to_float32( + audio: tuple[int, NDArray[np.int16 | np.float32]], +) -> NDArray[np.float32]: """ Convert an audio tuple containing sample rate (int16) and numpy array data to float32. @@ -274,40 +292,64 @@ def audio_to_float32(audio: tuple[int, np.ndarray]) -> np.ndarray: def aggregate_bytes_to_16bit(chunks_iterator): - leftover = b"" # Store incomplete bytes between chunks + """ + Aggregate bytes to 16-bit audio samples. + This function takes an iterator of chunks and aggregates them into 16-bit audio samples. + It handles incomplete samples and combines them with the next chunk. + + Parameters + ---------- + chunks_iterator : Iterator[bytes] + An iterator of byte chunks to aggregate + + Returns + ------- + Iterator[NDArray[np.int16]] + """ + leftover = b"" for chunk in chunks_iterator: - # Combine with any leftover bytes from previous chunk current_bytes = leftover + chunk - # Calculate complete samples - n_complete_samples = len(current_bytes) // 2 # int16 = 2 bytes + n_complete_samples = len(current_bytes) // 2 bytes_to_process = n_complete_samples * 2 - # Split into complete samples and leftover to_process = current_bytes[:bytes_to_process] leftover = current_bytes[bytes_to_process:] - if to_process: # Only yield if we have complete samples + if to_process: audio_array = np.frombuffer(to_process, dtype=np.int16).reshape(1, -1) yield audio_array async def async_aggregate_bytes_to_16bit(chunks_iterator): - leftover = b"" # Store incomplete bytes between chunks + """ + Aggregate bytes to 16-bit audio samples. + + This function takes an iterator of chunks and aggregates them into 16-bit audio samples. + It handles incomplete samples and combines them with the next chunk. + + Parameters + ---------- + chunks_iterator : Iterator[bytes] + An iterator of byte chunks to aggregate + + Returns + ------- + Iterator[NDArray[np.int16]] + An iterator of 16-bit audio samples + """ + leftover = b"" async for chunk in chunks_iterator: - # Combine with any leftover bytes from previous chunk current_bytes = leftover + chunk - # Calculate complete samples - n_complete_samples = len(current_bytes) // 2 # int16 = 2 bytes + n_complete_samples = len(current_bytes) // 2 bytes_to_process = n_complete_samples * 2 - # Split into complete samples and leftover to_process = current_bytes[:bytes_to_process] leftover = current_bytes[bytes_to_process:] - if to_process: # Only yield if we have complete samples + if to_process: audio_array = np.frombuffer(to_process, dtype=np.int16).reshape(1, -1) yield audio_array diff --git a/backend/fastrtc/webrtc.py b/backend/fastrtc/webrtc.py new file mode 100644 index 0000000..a678e87 --- /dev/null +++ b/backend/fastrtc/webrtc.py @@ -0,0 +1,369 @@ +"""gr.WebRTC() component.""" + +from __future__ import annotations + +import logging +from collections.abc import Callable +from typing import ( + TYPE_CHECKING, + Any, + Concatenate, + Iterable, + Literal, + ParamSpec, + Sequence, + TypeVar, + cast, +) + +from gradio import wasm_utils +from gradio.components.base import Component, server +from gradio_client import handle_file + +from .tracks import ( + AudioVideoStreamHandlerImpl, + StreamHandler, + StreamHandlerBase, + StreamHandlerImpl, + VideoEventHandler, +) +from .webrtc_connection_mixin import WebRTCConnectionMixin + +if TYPE_CHECKING: + from gradio.blocks import Block + from gradio.components import Timer + +if wasm_utils.IS_WASM: + raise ValueError("Not supported in gradio-lite!") + + +logger = logging.getLogger(__name__) + + +# For the return type +R = TypeVar("R") +# For the parameter specification +P = ParamSpec("P") + + +class WebRTC(Component, WebRTCConnectionMixin): + """ + Creates a video component that can be used to upload/record videos (as an input) or display videos (as an output). + For the video to be playable in the browser it must have a compatible container and codec combination. Allowed + combinations are .mp4 with h264 codec, .ogg with theora codec, and .webm with vp9 codec. If the component detects + that the output video would not be playable in the browser it will attempt to convert it to a playable mp4 video. + If the conversion fails, the original video is returned. + + Demos: video_identity_2 + """ + + EVENTS = ["tick", "state_change"] + + def __init__( + self, + value: None = None, + height: int | str | None = None, + width: int | str | None = None, + label: str | None = None, + every: Timer | float | None = None, + inputs: Component | Sequence[Component] | set[Component] | None = None, + show_label: bool | None = None, + container: bool = True, + scale: int | None = None, + min_width: int = 160, + interactive: bool | None = None, + visible: bool = True, + elem_id: str | None = None, + elem_classes: list[str] | str | None = None, + render: bool = True, + key: int | str | None = None, + mirror_webcam: bool = True, + rtc_configuration: dict[str, Any] | None = None, + track_constraints: dict[str, Any] | None = None, + time_limit: float | None = None, + mode: Literal["send-receive", "receive", "send"] = "send-receive", + modality: Literal["video", "audio", "audio-video"] = "video", + rtp_params: dict[str, Any] | None = None, + icon: str | None = None, + icon_button_color: str | None = None, + pulse_color: str | None = None, + button_labels: dict | None = None, + ): + """ + Parameters: + value: path or URL for the default value that WebRTC component is going to take. Can also be a tuple consisting of (video filepath, subtitle filepath). If a subtitle file is provided, it should be of type .srt or .vtt. Or can be callable, in which case the function will be called whenever the app loads to set the initial value of the component. + format: the file extension with which to save video, such as 'avi' or 'mp4'. This parameter applies both when this component is used as an input to determine which file format to convert user-provided video to, and when this component is used as an output to determine the format of video returned to the user. If None, no file format conversion is done and the video is kept as is. Use 'mp4' to ensure browser playability. + height: The height of the component, specified in pixels if a number is passed, or in CSS units if a string is passed. This has no effect on the preprocessed video file, but will affect the displayed video. + width: The width of the component, specified in pixels if a number is passed, or in CSS units if a string is passed. This has no effect on the preprocessed video file, but will affect the displayed video. + label: the label for this component. Appears above the component and is also used as the header if there are a table of examples for this component. If None and used in a `gr.Interface`, the label will be the name of the parameter this component is assigned to. + every: continously calls `value` to recalculate it if `value` is a function (has no effect otherwise). Can provide a Timer whose tick resets `value`, or a float that provides the regular interval for the reset Timer. + inputs: components that are used as inputs to calculate `value` if `value` is a function (has no effect otherwise). `value` is recalculated any time the inputs change. + show_label: if True, will display label. + container: if True, will place the component in a container - providing some extra padding around the border. + scale: relative size compared to adjacent Components. For example if Components A and B are in a Row, and A has scale=2, and B has scale=1, A will be twice as wide as B. Should be an integer. scale applies in Rows, and to top-level Components in Blocks where fill_height=True. + min_width: minimum pixel width, will wrap if not sufficient screen space to satisfy this value. If a certain scale value results in this Component being narrower than min_width, the min_width parameter will be respected first. + interactive: if True, will allow users to upload a video; if False, can only be used to display videos. If not provided, this is inferred based on whether the component is used as an input or output. + visible: if False, component will be hidden. + elem_id: an optional string that is assigned as the id of this component in the HTML DOM. Can be used for targeting CSS styles. + elem_classes: an optional list of strings that are assigned as the classes of this component in the HTML DOM. Can be used for targeting CSS styles. + render: if False, component will not render be rendered in the Blocks context. Should be used if the intention is to assign event listeners now but render the component later. + key: if assigned, will be used to assume identity across a re-render. Components that have the same key across a re-render will have their value preserved. + mirror_webcam: if True webcam will be mirrored. Default is True. + rtc_configuration: WebRTC configuration options. See https://developer.mozilla.org/en-US/docs/Web/API/RTCPeerConnection/RTCPeerConnection . If running the demo on a remote server, you will need to specify a rtc_configuration. See https://freddyaboulton.github.io/gradio-webrtc/deployment/ + track_constraints: Media track constraints for WebRTC. For example, to set video height, width use {"width": {"exact": 800}, "height": {"exact": 600}, "aspectRatio": {"exact": 1.33333}} + time_limit: Maximum duration in seconds for recording. + mode: WebRTC mode - "send-receive", "receive", or "send". + modality: Type of media - "video" or "audio". + rtp_params: See https://developer.mozilla.org/en-US/docs/Web/API/RTCRtpSender/setParameters. If you are changing the video resolution, you can set this to {"degradationPreference": "maintain-framerate"} to keep the frame rate consistent. + icon: Icon to display on the button instead of the wave animation. The icon should be a path/url to a .svg/.png/.jpeg file. + icon_button_color: Color of the icon button. Default is var(--color-accent) of the demo theme. + pulse_color: Color of the pulse animation. Default is var(--color-accent) of the demo theme. + button_labels: Text to display on the audio or video start, stop, waiting buttons. Dict with keys "start", "stop", "waiting" mapping to the text to display on the buttons. + """ + self.time_limit = time_limit + self.height = height + self.width = width + self.mirror_webcam = mirror_webcam + self.concurrency_limit = 1 + self.rtc_configuration = rtc_configuration + self.mode = mode + self.modality = modality + self.icon_button_color = icon_button_color + self.pulse_color = pulse_color + self.rtp_params = rtp_params or {} + self.button_labels = { + "start": "", + "stop": "", + "waiting": "", + **(button_labels or {}), + } + if track_constraints is None and modality == "audio": + track_constraints = { + "echoCancellation": True, + "noiseSuppression": {"exact": True}, + "autoGainControl": {"exact": True}, + "sampleRate": {"ideal": 24000}, + "sampleSize": {"ideal": 16}, + "channelCount": {"exact": 1}, + } + if track_constraints is None and modality == "video": + track_constraints = { + "facingMode": "user", + "width": {"ideal": 500}, + "height": {"ideal": 500}, + "frameRate": {"ideal": 30}, + } + if track_constraints is None and modality == "audio-video": + track_constraints = { + "video": { + "facingMode": "user", + "width": {"ideal": 500}, + "height": {"ideal": 500}, + "frameRate": {"ideal": 30}, + }, + "audio": { + "echoCancellation": True, + "noiseSuppression": {"exact": True}, + "autoGainControl": {"exact": True}, + "sampleRate": {"ideal": 24000}, + "sampleSize": {"ideal": 16}, + "channelCount": {"exact": 1}, + }, + } + self.track_constraints = track_constraints + self.event_handler: Callable | StreamHandler | None = None + super().__init__( + label=label, + every=every, + inputs=inputs, + show_label=show_label, + container=container, + scale=scale, + min_width=min_width, + interactive=interactive, + visible=visible, + elem_id=elem_id, + elem_classes=elem_classes, + render=render, + key=key, + value=value, + ) + # need to do this here otherwise the proxy_url is not set + self.icon = ( + icon if not icon else cast(dict, self.serve_static_file(icon)).get("url") + ) + + def preprocess(self, payload: str) -> str: + """ + Parameters: + payload: An instance of VideoData containing the video and subtitle files. + Returns: + Passes the uploaded video as a `str` filepath or URL whose extension can be modified by `format`. + """ + return payload + + def postprocess(self, value: Any) -> str: + """ + Parameters: + value: Expects a {str} or {pathlib.Path} filepath to a video which is displayed, or a {Tuple[str | pathlib.Path, str | pathlib.Path | None]} where the first element is a filepath to a video and the second element is an optional filepath to a subtitle file. + Returns: + VideoData object containing the video and subtitle files. + """ + return value + + def on_additional_outputs( + self, + fn: Callable[Concatenate[P], R], + inputs: Block | Sequence[Block] | set[Block] | None = None, + outputs: Block | Sequence[Block] | set[Block] | None = None, + js: str | None = None, + concurrency_limit: int | None | Literal["default"] = "default", + concurrency_id: str | None = None, + show_progress: Literal["full", "minimal", "hidden"] = "full", + queue: bool = True, + ): + inputs = inputs or [] + if inputs and not isinstance(inputs, Iterable): + inputs = [inputs] + inputs = list(inputs) + + def handler(webrtc_id: str, *args): + if self.additional_outputs[webrtc_id].queue.qsize() > 0: + next_outputs = self.additional_outputs[webrtc_id].queue.get_nowait() + return fn(*args, *next_outputs.args) # type: ignore + return ( + tuple([None for _ in range(len(outputs))]) + if isinstance(outputs, Iterable) + else None + ) + + return self.state_change( # type: ignore + fn=handler, + inputs=[self] + cast(list, inputs), + outputs=outputs, + js=js, + concurrency_limit=concurrency_limit, + concurrency_id=concurrency_id, + show_progress=show_progress, + queue=queue, + trigger_mode="multiple", + ) + + def stream( + self, + fn: ( + Callable[..., Any] + | StreamHandlerImpl + | AudioVideoStreamHandlerImpl + | VideoEventHandler + | None + ) = None, + inputs: Block | Sequence[Block] | set[Block] | None = None, + outputs: Block | Sequence[Block] | set[Block] | None = None, + js: str | None = None, + concurrency_limit: int | None | Literal["default"] = "default", + concurrency_id: str | None = None, + time_limit: float | None = None, + trigger: Callable | None = None, + ): + from gradio.blocks import Block + + if inputs is None: + inputs = [] + if outputs is None: + outputs = [] + if isinstance(inputs, Block): + inputs = [inputs] + if isinstance(outputs, Block): + outputs = [outputs] + + self.concurrency_limit = cast( + int, (1 if concurrency_limit in ["default", None] else concurrency_limit) + ) + self.event_handler = fn # type: ignore + self.time_limit = time_limit + + if ( + self.mode == "send-receive" + and self.modality in ["audio", "audio-video"] + and not isinstance(self.event_handler, StreamHandlerBase) + ): + raise ValueError( + "In the send-receive mode for audio, the event handler must be an instance of StreamHandlerBase." + ) + + if self.mode == "send-receive" or self.mode == "send": + if cast(list[Block], inputs)[0] != self: + raise ValueError( + "In the webrtc stream event, the first input component must be the WebRTC component." + ) + + if ( + len(cast(list[Block], outputs)) != 1 + and cast(list[Block], outputs)[0] != self + ): + raise ValueError( + "In the webrtc stream event, the only output component must be the WebRTC component." + ) + for input_component in inputs[1:]: # type: ignore + if hasattr(input_component, "change"): + input_component.change( # type: ignore + self.set_input, + inputs=inputs, + outputs=None, + concurrency_id=concurrency_id, + concurrency_limit=None, + time_limit=None, + js=js, + ) + return self.tick( # type: ignore + self.set_input, + inputs=inputs, + outputs=None, + concurrency_id=concurrency_id, + concurrency_limit=None, + time_limit=None, + js=js, + ) + elif self.mode == "receive": + if isinstance(inputs, list) and self in cast(list[Block], inputs): + raise ValueError( + "In the receive mode stream event, the WebRTC component cannot be an input." + ) + if ( + len(cast(list[Block], outputs)) != 1 + and cast(list[Block], outputs)[0] != self + ): + raise ValueError( + "In the receive mode stream, the only output component must be the WebRTC component." + ) + if trigger is None: + raise ValueError( + "In the receive mode stream event, the trigger parameter must be provided" + ) + trigger(lambda: "start_webrtc_stream", inputs=None, outputs=self) + self.tick( # type: ignore + self.set_input, + inputs=[self] + list(inputs), + outputs=None, + concurrency_id=concurrency_id, + ) + + @server + async def offer(self, body): + return await self.handle_offer( + body, self.set_additional_outputs(body["webrtc_id"]) + ) + + def example_payload(self) -> Any: + return { + "video": handle_file( + "https://github.com/gradio-app/gradio/raw/main/demo/video_component/files/world.mp4" + ), + } + + def example_value(self) -> Any: + return "https://github.com/gradio-app/gradio/raw/main/demo/video_component/files/world.mp4" + + def api_info(self) -> Any: + return {"type": "number"} diff --git a/backend/fastrtc/webrtc_connection_mixin.py b/backend/fastrtc/webrtc_connection_mixin.py new file mode 100644 index 0000000..0f2e0e5 --- /dev/null +++ b/backend/fastrtc/webrtc_connection_mixin.py @@ -0,0 +1,273 @@ +"""Mixin for handling WebRTC connections.""" + +from __future__ import annotations + +import asyncio +import inspect +import logging +from collections import defaultdict +from collections.abc import Callable +from dataclasses import dataclass, field +from typing import ( + AsyncGenerator, + Literal, + ParamSpec, + TypeVar, + cast, +) + +from aiortc import ( + RTCPeerConnection, + RTCSessionDescription, +) +from aiortc.contrib.media import MediaRelay # type: ignore +from fastapi.responses import JSONResponse + +from fastrtc.tracks import ( + AudioCallback, + HandlerType, + ServerToClientAudio, + ServerToClientVideo, + StreamHandlerBase, + StreamHandlerImpl, + VideoCallback, + VideoStreamHandler, +) +from fastrtc.utils import ( + AdditionalOutputs, + DataChannel, + create_message, +) + +Track = ( + VideoCallback + | VideoStreamHandler + | AudioCallback + | ServerToClientAudio + | ServerToClientVideo +) + +logger = logging.getLogger(__name__) + + +# For the return type +R = TypeVar("R") +# For the parameter specification +P = ParamSpec("P") + + +@dataclass +class OutputQueue: + queue: asyncio.Queue[AdditionalOutputs] = field(default_factory=asyncio.Queue) + quit: asyncio.Event = field(default_factory=asyncio.Event) + + +class WebRTCConnectionMixin: + pcs: set[RTCPeerConnection] = set([]) + relay = MediaRelay() + connections: dict[str, list[Track]] = defaultdict(list) + data_channels: dict[str, DataChannel] = {} + additional_outputs: dict[str, OutputQueue] = defaultdict(OutputQueue) + handlers: dict[str, HandlerType | Callable] = {} + + concurrency_limit: int | float + event_handler: HandlerType + time_limit: float | int | None + modality: Literal["video", "audio", "audio-video"] + mode: Literal["send", "receive", "send-receive"] + + @staticmethod + async def wait_for_time_limit(pc: RTCPeerConnection, time_limit: float): + await asyncio.sleep(time_limit) + await pc.close() + + def clean_up(self, webrtc_id: str): + self.handlers.pop(webrtc_id, None) + connection = self.connections.pop(webrtc_id, []) + for conn in connection: + if isinstance(conn, AudioCallback): + if inspect.iscoroutinefunction(conn.event_handler.shutdown): + asyncio.create_task(conn.event_handler.shutdown()) + conn.event_handler.reset() + else: + conn.event_handler.shutdown() + conn.event_handler.reset() + output = self.additional_outputs.pop(webrtc_id, None) + if output: + logger.debug("setting quit for webrtc id %s", webrtc_id) + output.quit.set() + self.data_channels.pop(webrtc_id, None) + return connection + + def set_input(self, webrtc_id: str, *args): + if webrtc_id in self.connections: + for conn in self.connections[webrtc_id]: + conn.set_args(list(args)) + + async def output_stream( + self, webrtc_id: str + ) -> AsyncGenerator[AdditionalOutputs, None]: + outputs = self.additional_outputs[webrtc_id] + while not outputs.quit.is_set(): + try: + yield await asyncio.wait_for(outputs.queue.get(), 10) + except (asyncio.TimeoutError, TimeoutError): + logger.debug("Timeout waiting for output") + + async def fetch_latest_output(self, webrtc_id: str) -> AdditionalOutputs: + outputs = self.additional_outputs[webrtc_id] + return await asyncio.wait_for(outputs.queue.get(), 10) + + def set_additional_outputs( + self, webrtc_id: str + ) -> Callable[[AdditionalOutputs], None]: + def set_outputs(outputs: AdditionalOutputs): + self.additional_outputs[webrtc_id].queue.put_nowait(outputs) + + return set_outputs + + async def handle_offer(self, body, set_outputs): + logger.debug("Starting to handle offer") + logger.debug("Offer body %s", body) + if len(self.connections) >= cast(int, self.concurrency_limit): + return JSONResponse( + status_code=429, + content={ + "status": "failed", + "meta": { + "error": "concurrency_limit_reached", + "limit": self.concurrency_limit, + }, + }, + ) + + offer = RTCSessionDescription(sdp=body["sdp"], type=body["type"]) + + pc = RTCPeerConnection() + self.pcs.add(pc) + + if isinstance(self.event_handler, StreamHandlerBase): + handler = self.event_handler.copy() + else: + handler = cast(Callable, self.event_handler) + + self.handlers[body["webrtc_id"]] = handler + + @pc.on("iceconnectionstatechange") + async def on_iceconnectionstatechange(): + logger.debug("ICE connection state change %s", pc.iceConnectionState) + if pc.iceConnectionState == "failed": + await pc.close() + self.connections.pop(body["webrtc_id"], None) + self.pcs.discard(pc) + + @pc.on("connectionstatechange") + async def _(): + logger.debug("pc.connectionState %s", pc.connectionState) + if pc.connectionState in ["failed", "closed"]: + await pc.close() + connection = self.clean_up(body["webrtc_id"]) + if connection: + for conn in connection: + conn.stop() + self.pcs.discard(pc) + if pc.connectionState == "connected": + if self.time_limit is not None: + asyncio.create_task(self.wait_for_time_limit(pc, self.time_limit)) + + @pc.on("track") + def _(track): + relay = MediaRelay() + handler = self.handlers[body["webrtc_id"]] + + if self.modality == "video" and track.kind == "video": + cb = VideoCallback( + relay.subscribe(track), + event_handler=cast(Callable, handler), + set_additional_outputs=set_outputs, + mode=cast(Literal["send", "send-receive"], self.mode), + ) + elif self.modality == "audio-video" and track.kind == "video": + cb = VideoStreamHandler( + relay.subscribe(track), + event_handler=handler, # type: ignore + set_additional_outputs=set_outputs, + ) + elif self.modality in ["audio", "audio-video"] and track.kind == "audio": + eh = cast(StreamHandlerImpl, handler) + eh._loop = asyncio.get_running_loop() + cb = AudioCallback( + relay.subscribe(track), + event_handler=eh, + set_additional_outputs=set_outputs, + ) + else: + raise ValueError("Modality must be either video, audio, or audio-video") + if body["webrtc_id"] not in self.connections: + self.connections[body["webrtc_id"]] = [] + + self.connections[body["webrtc_id"]].append(cb) + if body["webrtc_id"] in self.data_channels: + for conn in self.connections[body["webrtc_id"]]: + conn.set_channel(self.data_channels[body["webrtc_id"]]) + if self.mode == "send-receive": + logger.debug("Adding track to peer connection %s", cb) + pc.addTrack(cb) + elif self.mode == "send": + cast(AudioCallback | VideoCallback, cb).start() + + if self.mode == "receive": + if self.modality == "video": + cb = ServerToClientVideo( + cast(Callable, self.event_handler), + set_additional_outputs=set_outputs, + ) + elif self.modality == "audio": + cb = ServerToClientAudio( + cast(Callable, self.event_handler), + set_additional_outputs=set_outputs, + ) + else: + raise ValueError("Modality must be either video or audio") + + logger.debug("Adding track to peer connection %s", cb) + pc.addTrack(cb) + self.connections[body["webrtc_id"]].append(cb) + cb.on("ended", lambda: self.clean_up(body["webrtc_id"])) + + @pc.on("datachannel") + def _(channel): + logger.debug(f"Data channel established: {channel.label}") + + self.data_channels[body["webrtc_id"]] = channel + + async def set_channel(webrtc_id: str): + while not self.connections.get(webrtc_id): + await asyncio.sleep(0.05) + logger.debug("setting channel for webrtc id %s", webrtc_id) + for conn in self.connections[webrtc_id]: + conn.set_channel(channel) + + asyncio.create_task(set_channel(body["webrtc_id"])) + + @channel.on("message") + def _(message): + logger.debug(f"Received message: {message}") + if channel.readyState == "open": + channel.send( + create_message("log", data=f"Server received: {message}") + ) + + # handle offer + await pc.setRemoteDescription(offer) + + # send answer + answer = await pc.createAnswer() + await pc.setLocalDescription(answer) # type: ignore + logger.debug("done handling offer about to return") + await asyncio.sleep(0.1) + + return { + "sdp": pc.localDescription.sdp, + "type": pc.localDescription.type, + } diff --git a/backend/fastrtc/websocket.py b/backend/fastrtc/websocket.py new file mode 100644 index 0000000..88ab003 --- /dev/null +++ b/backend/fastrtc/websocket.py @@ -0,0 +1,184 @@ +import asyncio +import audioop +import base64 +import logging +from typing import Any, Awaitable, Callable, Optional, cast + +import anyio +import librosa +import numpy as np +from fastapi import WebSocket + +from .tracks import AsyncStreamHandler, StreamHandlerImpl +from .utils import AdditionalOutputs, DataChannel, split_output + + +class WebSocketDataChannel(DataChannel): + def __init__(self, websocket: WebSocket, loop: asyncio.AbstractEventLoop): + self.websocket = websocket + self.loop = loop + + def send(self, message: str) -> None: + asyncio.run_coroutine_threadsafe(self.websocket.send_text(message), self.loop) + + +logger = logging.getLogger(__file__) + + +def convert_to_mulaw( + audio_data: np.ndarray, original_rate: int, target_rate: int +) -> bytes: + """Convert audio data to 8kHz mu-law format""" + + if audio_data.dtype != np.float32: + audio_data = audio_data.astype(np.float32) / 32768.0 + + if original_rate != target_rate: + audio_data = librosa.resample(audio_data, orig_sr=original_rate, target_sr=8000) + + audio_data = (audio_data * 32768).astype(np.int16) + + return audioop.lin2ulaw(audio_data, 2) # type: ignore + + +run_sync = anyio.to_thread.run_sync # type: ignore + + +class WebSocketHandler: + def __init__( + self, + stream_handler: StreamHandlerImpl, + set_handler: Callable[[str, "WebSocketHandler"], Awaitable[None]], + clean_up: Callable[[str], None], + additional_outputs_factory: Callable[ + [str], Callable[[AdditionalOutputs], None] + ], + ): + self.stream_handler = stream_handler + self.websocket: Optional[WebSocket] = None + self._emit_task: Optional[asyncio.Task] = None + self.stream_id: Optional[str] = None + self.set_additional_outputs_factory = additional_outputs_factory + self.set_additional_outputs: Callable[[AdditionalOutputs], None] + self.set_handler = set_handler + self.quit = asyncio.Event() + self.clean_up = clean_up + + def set_args(self, args: list[Any]): + self.stream_handler.set_args(args) + + async def handle_websocket(self, websocket: WebSocket): + await websocket.accept() + loop = asyncio.get_running_loop() + self.loop = loop + self.websocket = websocket + self.data_channel = WebSocketDataChannel(websocket, loop) + self.stream_handler._loop = loop + self.stream_handler.set_channel(self.data_channel) + self._emit_task = asyncio.create_task(self._emit_loop()) + if isinstance(self.stream_handler, AsyncStreamHandler): + start_up = self.stream_handler.start_up() + else: + start_up = anyio.to_thread.run_sync(self.stream_handler.start_up) # type: ignore + + self.start_up_task = asyncio.create_task(start_up) + try: + while not self.quit.is_set(): + message = await websocket.receive_json() + + if message["event"] == "media": + audio_payload = base64.b64decode(message["media"]["payload"]) + + audio_array = np.frombuffer( + audioop.ulaw2lin(audio_payload, 2), dtype=np.int16 + ) + + if self.stream_handler.input_sample_rate != 8000: + audio_array = audio_array.astype(np.float32) / 32768.0 + audio_array = librosa.resample( + audio_array, + orig_sr=8000, + target_sr=self.stream_handler.input_sample_rate, + ) + audio_array = (audio_array * 32768).astype(np.int16) + if isinstance(self.stream_handler, AsyncStreamHandler): + await self.stream_handler.receive( + (self.stream_handler.input_sample_rate, audio_array) + ) + else: + await run_sync( + self.stream_handler.receive, + (self.stream_handler.input_sample_rate, audio_array), + ) + + elif message["event"] == "start": + if self.stream_handler.phone_mode: + self.stream_id = cast(str, message["streamSid"]) + else: + self.stream_id = cast(str, message["websocket_id"]) + self.set_additional_outputs = self.set_additional_outputs_factory( + self.stream_id + ) + await self.set_handler(self.stream_id, self) + elif message["event"] == "stop": + self.quit.set() + self.clean_up(cast(str, self.stream_id)) + return + elif message["event"] == "ping": + await websocket.send_json({"event": "pong"}) + + except Exception as e: + print(e) + import traceback + + traceback.print_exc() + logger.debug("Error in websocket handler %s", e) + finally: + if self._emit_task: + self._emit_task.cancel() + if self.start_up_task: + self.start_up_task.cancel() + await websocket.close() + + async def _emit_loop(self): + try: + while not self.quit.is_set(): + if isinstance(self.stream_handler, AsyncStreamHandler): + output = await self.stream_handler.emit() + else: + output = await run_sync(self.stream_handler.emit) + + if output is not None: + frame, output = split_output(output) + if output is not None: + self.set_additional_outputs(output) + if not isinstance(frame, tuple): + continue + target_rate = ( + self.stream_handler.output_sample_rate + if not self.stream_handler.phone_mode + else 8000 + ) + mulaw_audio = convert_to_mulaw( + frame[1], frame[0], target_rate=target_rate + ) + audio_payload = base64.b64encode(mulaw_audio).decode("utf-8") + + if self.websocket and self.stream_id: + payload = { + "event": "media", + "media": {"payload": audio_payload}, + } + if self.stream_handler.phone_mode: + payload["streamSid"] = self.stream_id + await self.websocket.send_json(payload) + + await asyncio.sleep(0.02) + + except asyncio.CancelledError: + logger.debug("Emit loop cancelled") + except Exception as e: + import traceback + + traceback.print_exc() + logger.debug("Error in emit loop: %s", e) diff --git a/backend/gradio_webrtc/speech_to_text/__init__.py b/backend/gradio_webrtc/speech_to_text/__init__.py deleted file mode 100644 index 8569c11..0000000 --- a/backend/gradio_webrtc/speech_to_text/__init__.py +++ /dev/null @@ -1,3 +0,0 @@ -from .stt_ import get_stt_model, stt, stt_for_chunks - -__all__ = ["stt", "stt_for_chunks", "get_stt_model"] diff --git a/backend/gradio_webrtc/speech_to_text/stt_.py b/backend/gradio_webrtc/speech_to_text/stt_.py deleted file mode 100644 index 6b2f696..0000000 --- a/backend/gradio_webrtc/speech_to_text/stt_.py +++ /dev/null @@ -1,53 +0,0 @@ -from dataclasses import dataclass -from functools import lru_cache -from typing import Callable - -import numpy as np -from numpy.typing import NDArray - -from ..utils import AudioChunk - - -@dataclass -class STTModel: - encoder: Callable - decoder: Callable - - -@lru_cache -def get_stt_model() -> STTModel: - from silero import silero_stt - - model, decoder, _ = silero_stt(language="en", version="v6", jit_model="jit_xlarge") - return STTModel(model, decoder) - - -def stt(audio: tuple[int, NDArray[np.int16]]) -> str: - model = get_stt_model() - sr, audio_np = audio - if audio_np.dtype != np.float32: - print("converting") - audio_np = audio_np.astype(np.float32) / 32768.0 - try: - import torch - except ImportError: - raise ImportError( - "PyTorch is required to run speech-to-text for stopword detection. Run `pip install torch`." - ) - audio_torch = torch.tensor(audio_np, dtype=torch.float32) - if audio_torch.ndim == 1: - audio_torch = audio_torch.unsqueeze(0) - assert audio_torch.ndim == 2, "Audio must have a batch dimension" - print("before") - res = model.decoder(model.encoder(audio_torch)[0]) - print("after") - return res - - -def stt_for_chunks( - audio: tuple[int, NDArray[np.int16]], chunks: list[AudioChunk] -) -> str: - sr, audio_np = audio - return " ".join( - [stt((sr, audio_np[chunk["start"] : chunk["end"]])) for chunk in chunks] - ) diff --git a/backend/gradio_webrtc/webrtc.py b/backend/gradio_webrtc/webrtc.py deleted file mode 100644 index a722ca3..0000000 --- a/backend/gradio_webrtc/webrtc.py +++ /dev/null @@ -1,1151 +0,0 @@ -"""gr.WebRTC() component.""" - -from __future__ import annotations - -import asyncio -import functools -import inspect -import logging -import threading -import time -import traceback -from abc import ABC, abstractmethod -from collections import defaultdict -from collections.abc import Callable -from typing import ( - TYPE_CHECKING, - Any, - Concatenate, - Generator, - Iterable, - Literal, - ParamSpec, - Sequence, - TypeAlias, - TypeVar, - Union, - cast, -) - -import anyio.to_thread -import av -import numpy as np -from aiortc import ( - AudioStreamTrack, - MediaStreamTrack, - RTCPeerConnection, - RTCSessionDescription, - VideoStreamTrack, -) -from aiortc.contrib.media import AudioFrame, MediaRelay, VideoFrame # type: ignore -from aiortc.mediastreams import MediaStreamError -from gradio import wasm_utils -from gradio.components.base import Component, server -from gradio_client import handle_file -from numpy import typing as npt - -from gradio_webrtc.utils import ( - AdditionalOutputs, - DataChannel, - current_channel, - player_worker_decode, - split_output, -) - -if TYPE_CHECKING: - from gradio.blocks import Block - from gradio.components import Timer - from gradio.events import Dependency - - -if wasm_utils.IS_WASM: - raise ValueError("Not supported in gradio-lite!") - - -logger = logging.getLogger(__name__) - -VideoEmitType = Union[ - AdditionalOutputs, tuple[npt.ArrayLike, AdditionalOutputs], npt.ArrayLike, None -] -VideoEventHandler = Callable[[npt.ArrayLike], VideoEmitType] - - -class VideoCallback(VideoStreamTrack): - """ - This works for streaming input and output - """ - - kind = "video" - - def __init__( - self, - track: MediaStreamTrack, - event_handler: VideoEventHandler, - channel: DataChannel | None = None, - set_additional_outputs: Callable | None = None, - mode: Literal["send-receive", "send"] = "send-receive", - ) -> None: - super().__init__() # don't forget this! - self.track = track - self.event_handler = event_handler - self.latest_args: str | list[Any] = "not_set" - self.channel = channel - self.set_additional_outputs = set_additional_outputs - self.thread_quit = asyncio.Event() - self.mode = mode - self.channel_set = asyncio.Event() - self.has_started = False - - def set_channel(self, channel: DataChannel): - self.channel = channel - current_channel.set(channel) - self.channel_set.set() - - def set_args(self, args: list[Any]): - self.latest_args = ["__webrtc_value__"] + list(args) - - def add_frame_to_payload( - self, args: list[Any], frame: np.ndarray | None - ) -> list[Any]: - new_args = [] - for val in args: - if isinstance(val, str) and val == "__webrtc_value__": - new_args.append(frame) - else: - new_args.append(val) - return new_args - - def array_to_frame(self, array: np.ndarray) -> VideoFrame: - return VideoFrame.from_ndarray(array, format="bgr24") - - async def process_frames(self): - while not self.thread_quit.is_set(): - try: - await self.recv() - except TimeoutError: - continue - - def start( - self, - ): - asyncio.create_task(self.process_frames()) - - def stop(self): - super().stop() - logger.debug("video callback stop") - self.thread_quit.set() - - async def wait_for_channel(self): - if not self.channel_set.is_set(): - await self.channel_set.wait() - if current_channel.get() != self.channel: - current_channel.set(self.channel) - - async def recv(self): # type: ignore - try: - try: - frame = cast(VideoFrame, await self.track.recv()) - except MediaStreamError: - self.stop() - return - - await self.wait_for_channel() - frame_array = frame.to_ndarray(format="bgr24") - if self.latest_args == "not_set": - return frame - - args = self.add_frame_to_payload(cast(list, self.latest_args), frame_array) - - array, outputs = split_output(self.event_handler(*args)) - if ( - isinstance(outputs, AdditionalOutputs) - and self.set_additional_outputs - and self.channel - ): - self.set_additional_outputs(outputs) - self.channel.send("change") - if array is None and self.mode == "send": - return - - new_frame = self.array_to_frame(array) - if frame: - new_frame.pts = frame.pts - new_frame.time_base = frame.time_base - else: - pts, time_base = await self.next_timestamp() - new_frame.pts = pts - new_frame.time_base = time_base - - return new_frame - except Exception as e: - logger.debug("exception %s", e) - exec = traceback.format_exc() - logger.debug("traceback %s", exec) - - -class StreamHandlerBase(ABC): - def __init__( - self, - expected_layout: Literal["mono", "stereo"] = "mono", - output_sample_rate: int = 24000, - output_frame_size: int = 960, - input_sample_rate: int = 48000, - ) -> None: - self.expected_layout = expected_layout - self.output_sample_rate = output_sample_rate - self.output_frame_size = output_frame_size - self.input_sample_rate = input_sample_rate - self.latest_args: list[Any] = [] - self._resampler = None - self._channel: DataChannel | None = None - self._loop: asyncio.AbstractEventLoop - self.args_set = asyncio.Event() - self.channel_set = asyncio.Event() - - @property - def loop(self) -> asyncio.AbstractEventLoop: - return cast(asyncio.AbstractEventLoop, self._loop) - - @property - def channel(self) -> DataChannel | None: - return self._channel - - def set_channel(self, channel: DataChannel): - self._channel = channel - self.channel_set.set() - - async def fetch_args( - self, - ): - if self.channel: - self.channel.send("tick") - logger.debug("Sent tick") - - async def wait_for_args(self): - await self.fetch_args() - await self.args_set.wait() - - def wait_for_args_sync(self): - asyncio.run_coroutine_threadsafe(self.wait_for_args(), self.loop).result() - - def set_args(self, args: list[Any]): - logger.debug("setting args in audio callback %s", args) - self.latest_args = ["__webrtc_value__"] + list(args) - self.args_set.set() - - def reset(self): - self.args_set.clear() - - def shutdown(self): - pass - - @abstractmethod - def copy(self) -> "StreamHandlerBase": - pass - - def resample(self, frame: AudioFrame) -> Generator[AudioFrame, None, None]: - if self._resampler is None: - self._resampler = av.AudioResampler( # type: ignore - format="s16", - layout=self.expected_layout, - rate=self.input_sample_rate, - frame_size=frame.samples, - ) - yield from self._resampler.resample(frame) - - -EmitType: TypeAlias = Union[ - tuple[int, np.ndarray], - tuple[int, np.ndarray, Literal["mono", "stereo"]], - AdditionalOutputs, - tuple[tuple[int, np.ndarray], AdditionalOutputs], - None, -] -AudioEmitType = EmitType - - -class StreamHandler(StreamHandlerBase): - @abstractmethod - def receive(self, frame: tuple[int, np.ndarray]) -> None: - pass - - @abstractmethod - def emit( - self, - ) -> EmitType: - pass - - -class AsyncStreamHandler(StreamHandlerBase): - @abstractmethod - async def receive(self, frame: tuple[int, np.ndarray]) -> None: - pass - - @abstractmethod - async def emit( - self, - ) -> EmitType: - pass - - -StreamHandlerImpl = Union[StreamHandler, AsyncStreamHandler] - - -class AudioVideoStreamHandler(StreamHandlerBase): - @abstractmethod - def video_receive(self, frame: npt.NDArray) -> None: - pass - - @abstractmethod - def video_emit( - self, - ) -> VideoEmitType: - pass - - -class AsyncAudioVideoStreamHandler(StreamHandlerBase): - @abstractmethod - async def video_receive(self, frame: npt.NDArray) -> None: - pass - - @abstractmethod - async def video_emit( - self, - ) -> VideoEmitType: - pass - - -VideoStreamHandlerImpl = Union[AudioVideoStreamHandler, AsyncAudioVideoStreamHandler] -AudioVideoStreamHandlerImpl = Union[ - AudioVideoStreamHandler, AsyncAudioVideoStreamHandler -] -AsyncHandler = Union[AsyncStreamHandler, AsyncAudioVideoStreamHandler] - - -class VideoStreamHander(VideoCallback): - async def process_frames(self): - while not self.thread_quit.is_set(): - try: - await self.channel_set.wait() - frame = cast(VideoFrame, await self.track.recv()) - frame_array = frame.to_ndarray(format="bgr24") - handler = cast(VideoStreamHandlerImpl, self.event_handler) - if inspect.iscoroutinefunction(handler.video_receive): - await handler.video_receive(frame_array) - else: - handler.video_receive(frame_array) - except MediaStreamError: - self.stop() - - def start(self): - if not self.has_started: - asyncio.create_task(self.process_frames()) - self.has_started = True - - async def recv(self): # type: ignore - self.start() - try: - handler = cast(VideoStreamHandlerImpl, self.event_handler) - if inspect.iscoroutinefunction(handler.video_emit): - outputs = await handler.video_emit() - else: - outputs = handler.video_emit() - - array, outputs = split_output(outputs) - if ( - isinstance(outputs, AdditionalOutputs) - and self.set_additional_outputs - and self.channel - ): - self.set_additional_outputs(outputs) - self.channel.send("change") - if array is None and self.mode == "send": - return - - new_frame = self.array_to_frame(array) - - # Will probably have to give developer ability to set pts and time_base - pts, time_base = await self.next_timestamp() - new_frame.pts = pts - new_frame.time_base = time_base - - return new_frame - except Exception as e: - logger.debug("exception %s", e) - exec = traceback.format_exc() - logger.debug("traceback %s", exec) - - -class AudioCallback(AudioStreamTrack): - kind = "audio" - - def __init__( - self, - track: MediaStreamTrack, - event_handler: StreamHandlerBase, - channel: DataChannel | None = None, - set_additional_outputs: Callable | None = None, - ) -> None: - super().__init__() - self.track = track - self.event_handler = cast(StreamHandlerImpl, event_handler) - self.current_timestamp = 0 - self.latest_args: str | list[Any] = "not_set" - self.queue = asyncio.Queue() - self.thread_quit = asyncio.Event() - self._start: float | None = None - self.has_started = False - self.last_timestamp = 0 - self.channel = channel - self.set_additional_outputs = set_additional_outputs - - def set_channel(self, channel: DataChannel): - self.channel = channel - self.event_handler.set_channel(channel) - - def set_args(self, args: list[Any]): - self.event_handler.set_args(args) - - def event_handler_receive(self, frame: tuple[int, np.ndarray]) -> None: - current_channel.set(self.event_handler.channel) - return cast(Callable, self.event_handler.receive)(frame) - - async def process_input_frames(self) -> None: - while not self.thread_quit.is_set(): - try: - frame = cast(AudioFrame, await self.track.recv()) - for frame in self.event_handler.resample(frame): - numpy_array = frame.to_ndarray() - if isinstance(self.event_handler, AsyncHandler): - await self.event_handler.receive( - (frame.sample_rate, numpy_array) - ) - else: - await anyio.to_thread.run_sync( - self.event_handler_receive, (frame.sample_rate, numpy_array) - ) - except MediaStreamError: - logger.debug("MediaStreamError in process_input_frames") - break - - def start(self): - if not self.has_started: - loop = asyncio.get_running_loop() - if isinstance(self.event_handler, AsyncHandler): - callable = self.event_handler.emit - else: - callable = functools.partial( - loop.run_in_executor, None, self.event_handler.emit - ) - asyncio.create_task(self.process_input_frames()) - asyncio.create_task( - player_worker_decode( - callable, - self.queue, - self.thread_quit, - lambda: self.channel, - self.set_additional_outputs, - False, - self.event_handler.output_sample_rate, - self.event_handler.output_frame_size, - ) - ) - self.has_started = True - - async def recv(self): # type: ignore - try: - if self.readyState != "live": - raise MediaStreamError - - if not self.event_handler.channel_set.is_set(): - await self.event_handler.channel_set.wait() - if current_channel.get() != self.event_handler.channel: - current_channel.set(self.event_handler.channel) - - self.start() - - frame = await self.queue.get() - logger.debug("frame %s", frame) - - data_time = frame.time - - if time.time() - self.last_timestamp > 10 * ( - self.event_handler.output_frame_size - / self.event_handler.output_sample_rate - ): - self._start = None - - # control playback rate - if self._start is None: - self._start = time.time() - data_time # type: ignore - else: - wait = self._start + data_time - time.time() - await asyncio.sleep(wait) - self.last_timestamp = time.time() - return frame - except Exception as e: - logger.debug("exception %s", e) - exec = traceback.format_exc() - logger.debug("traceback %s", exec) - - def stop(self): - logger.debug("audio callback stop") - self.thread_quit.set() - super().stop() - - def shutdown(self): - self.event_handler.shutdown() - - -class ServerToClientVideo(VideoStreamTrack): - """ - This works for streaming input and output - """ - - kind = "video" - - def __init__( - self, - event_handler: Callable, - channel: DataChannel | None = None, - set_additional_outputs: Callable | None = None, - ) -> None: - super().__init__() # don't forget this! - self.event_handler = event_handler - self.args_set = asyncio.Event() - self.latest_args: str | list[Any] = "not_set" - self.generator: Generator[Any, None, Any] | None = None - self.channel = channel - self.set_additional_outputs = set_additional_outputs - - def array_to_frame(self, array: np.ndarray) -> VideoFrame: - return VideoFrame.from_ndarray(array, format="bgr24") - - def set_channel(self, channel: DataChannel): - self.channel = channel - - def set_args(self, args: list[Any]): - self.latest_args = list(args) - self.args_set.set() - - async def recv(self): # type: ignore - try: - pts, time_base = await self.next_timestamp() - await self.args_set.wait() - if self.generator is None: - self.generator = cast( - Generator[Any, None, Any], self.event_handler(*self.latest_args) - ) - current_channel.set(self.channel) - try: - next_array, outputs = split_output(next(self.generator)) - if ( - isinstance(outputs, AdditionalOutputs) - and self.set_additional_outputs - and self.channel - ): - self.set_additional_outputs(outputs) - self.channel.send("change") - except StopIteration: - self.stop() - return - - next_frame = self.array_to_frame(next_array) - next_frame.pts = pts - next_frame.time_base = time_base - return next_frame - except Exception as e: - logger.debug("exception %s", e) - exec = traceback.format_exc() - logger.debug("traceback %s ", exec) - - -class ServerToClientAudio(AudioStreamTrack): - kind = "audio" - - def __init__( - self, - event_handler: Callable, - channel: DataChannel | None = None, - set_additional_outputs: Callable | None = None, - ) -> None: - self.generator: Generator[Any, None, Any] | None = None - self.event_handler = event_handler - self.current_timestamp = 0 - self.latest_args: str | list[Any] = "not_set" - self.args_set = threading.Event() - self.queue = asyncio.Queue() - self.thread_quit = asyncio.Event() - self.channel = channel - self.set_additional_outputs = set_additional_outputs - self.has_started = False - self._start: float | None = None - super().__init__() - - def set_channel(self, channel: DataChannel): - self.channel = channel - - def set_args(self, args: list[Any]): - self.latest_args = list(args) - self.args_set.set() - - def next(self) -> tuple[int, np.ndarray] | None: - self.args_set.wait() - current_channel.set(self.channel) - if self.generator is None: - self.generator = self.event_handler(*self.latest_args) - if self.generator is not None: - try: - frame = next(self.generator) - return frame - except StopIteration: - self.thread_quit.set() - - def start(self): - if not self.has_started: - loop = asyncio.get_running_loop() - callable = functools.partial(loop.run_in_executor, None, self.next) - asyncio.create_task( - player_worker_decode( - callable, - self.queue, - self.thread_quit, - lambda: self.channel, - self.set_additional_outputs, - True, - ) - ) - self.has_started = True - - async def recv(self): # type: ignore - try: - if self.readyState != "live": - raise MediaStreamError - - self.start() - data = await self.queue.get() - if data is None: - self.stop() - return - - data_time = data.time - - # control playback rate - if data_time is not None: - if self._start is None: - self._start = time.time() - data_time # type: ignore - else: - wait = self._start + data_time - time.time() - await asyncio.sleep(wait) - - return data - except Exception as e: - logger.debug("exception %s", e) - exec = traceback.format_exc() - logger.debug("traceback %s", exec) - - def stop(self): - logger.debug("audio-to-client stop callback") - self.thread_quit.set() - super().stop() - - -# For the return type -R = TypeVar("R") -# For the parameter specification -P = ParamSpec("P") - - -class WebRTC(Component): - """ - Creates a video component that can be used to upload/record videos (as an input) or display videos (as an output). - For the video to be playable in the browser it must have a compatible container and codec combination. Allowed - combinations are .mp4 with h264 codec, .ogg with theora codec, and .webm with vp9 codec. If the component detects - that the output video would not be playable in the browser it will attempt to convert it to a playable mp4 video. - If the conversion fails, the original video is returned. - - Demos: video_identity_2 - """ - - pcs: set[RTCPeerConnection] = set([]) - relay = MediaRelay() - connections: dict[ - str, - list[VideoCallback | ServerToClientVideo | ServerToClientAudio | AudioCallback], - ] = defaultdict(list) - data_channels: dict[str, DataChannel] = {} - additional_outputs: dict[str, list[AdditionalOutputs]] = {} - handlers: dict[str, StreamHandlerBase | Callable] = {} - - EVENTS = ["tick", "state_change"] - - def __init__( - self, - value: None = None, - height: int | str | None = None, - width: int | str | None = None, - label: str | None = None, - every: Timer | float | None = None, - inputs: Component | Sequence[Component] | set[Component] | None = None, - show_label: bool | None = None, - container: bool = True, - scale: int | None = None, - min_width: int = 160, - interactive: bool | None = None, - visible: bool = True, - elem_id: str | None = None, - elem_classes: list[str] | str | None = None, - render: bool = True, - key: int | str | None = None, - mirror_webcam: bool = True, - rtc_configuration: dict[str, Any] | None = None, - track_constraints: dict[str, Any] | None = None, - time_limit: float | None = None, - mode: Literal["send-receive", "receive", "send"] = "send-receive", - modality: Literal["video", "audio", "audio-video"] = "video", - rtp_params: dict[str, Any] | None = None, - icon: str | None = None, - icon_button_color: str | None = None, - pulse_color: str | None = None, - button_labels: dict | None = None, - ): - """ - Parameters: - value: path or URL for the default value that WebRTC component is going to take. Can also be a tuple consisting of (video filepath, subtitle filepath). If a subtitle file is provided, it should be of type .srt or .vtt. Or can be callable, in which case the function will be called whenever the app loads to set the initial value of the component. - format: the file extension with which to save video, such as 'avi' or 'mp4'. This parameter applies both when this component is used as an input to determine which file format to convert user-provided video to, and when this component is used as an output to determine the format of video returned to the user. If None, no file format conversion is done and the video is kept as is. Use 'mp4' to ensure browser playability. - height: The height of the component, specified in pixels if a number is passed, or in CSS units if a string is passed. This has no effect on the preprocessed video file, but will affect the displayed video. - width: The width of the component, specified in pixels if a number is passed, or in CSS units if a string is passed. This has no effect on the preprocessed video file, but will affect the displayed video. - label: the label for this component. Appears above the component and is also used as the header if there are a table of examples for this component. If None and used in a `gr.Interface`, the label will be the name of the parameter this component is assigned to. - every: continously calls `value` to recalculate it if `value` is a function (has no effect otherwise). Can provide a Timer whose tick resets `value`, or a float that provides the regular interval for the reset Timer. - inputs: components that are used as inputs to calculate `value` if `value` is a function (has no effect otherwise). `value` is recalculated any time the inputs change. - show_label: if True, will display label. - container: if True, will place the component in a container - providing some extra padding around the border. - scale: relative size compared to adjacent Components. For example if Components A and B are in a Row, and A has scale=2, and B has scale=1, A will be twice as wide as B. Should be an integer. scale applies in Rows, and to top-level Components in Blocks where fill_height=True. - min_width: minimum pixel width, will wrap if not sufficient screen space to satisfy this value. If a certain scale value results in this Component being narrower than min_width, the min_width parameter will be respected first. - interactive: if True, will allow users to upload a video; if False, can only be used to display videos. If not provided, this is inferred based on whether the component is used as an input or output. - visible: if False, component will be hidden. - elem_id: an optional string that is assigned as the id of this component in the HTML DOM. Can be used for targeting CSS styles. - elem_classes: an optional list of strings that are assigned as the classes of this component in the HTML DOM. Can be used for targeting CSS styles. - render: if False, component will not render be rendered in the Blocks context. Should be used if the intention is to assign event listeners now but render the component later. - key: if assigned, will be used to assume identity across a re-render. Components that have the same key across a re-render will have their value preserved. - mirror_webcam: if True webcam will be mirrored. Default is True. - rtc_configuration: WebRTC configuration options. See https://developer.mozilla.org/en-US/docs/Web/API/RTCPeerConnection/RTCPeerConnection . If running the demo on a remote server, you will need to specify a rtc_configuration. See https://freddyaboulton.github.io/gradio-webrtc/deployment/ - track_constraints: Media track constraints for WebRTC. For example, to set video height, width use {"width": {"exact": 800}, "height": {"exact": 600}, "aspectRatio": {"exact": 1.33333}} - time_limit: Maximum duration in seconds for recording. - mode: WebRTC mode - "send-receive", "receive", or "send". - modality: Type of media - "video" or "audio". - rtp_params: See https://developer.mozilla.org/en-US/docs/Web/API/RTCRtpSender/setParameters. If you are changing the video resolution, you can set this to {"degradationPreference": "maintain-framerate"} to keep the frame rate consistent. - icon: Icon to display on the button instead of the wave animation. The icon should be a path/url to a .svg/.png/.jpeg file. - icon_button_color: Color of the icon button. Default is var(--color-accent) of the demo theme. - pulse_color: Color of the pulse animation. Default is var(--color-accent) of the demo theme. - button_labels: Text to display on the audio or video start, stop, waiting buttons. Dict with keys "start", "stop", "waiting" mapping to the text to display on the buttons. - """ - self.time_limit = time_limit - self.height = height - self.width = width - self.mirror_webcam = mirror_webcam - self.concurrency_limit = 1 - self.rtc_configuration = rtc_configuration - self.mode = mode - self.modality = modality - self.icon_button_color = icon_button_color - self.pulse_color = pulse_color - self.rtp_params = rtp_params or {} - self.button_labels = { - "start": "", - "stop": "", - "waiting": "", - **(button_labels or {}), - } - if track_constraints is None and modality == "audio": - track_constraints = { - "echoCancellation": True, - "noiseSuppression": {"exact": True}, - "autoGainControl": {"exact": True}, - "sampleRate": {"ideal": 24000}, - "sampleSize": {"ideal": 16}, - "channelCount": {"exact": 1}, - } - if track_constraints is None and modality == "video": - track_constraints = { - "facingMode": "user", - "width": {"ideal": 500}, - "height": {"ideal": 500}, - "frameRate": {"ideal": 30}, - } - if track_constraints is None and modality == "audio-video": - track_constraints = { - "video": { - "facingMode": "user", - "width": {"ideal": 500}, - "height": {"ideal": 500}, - "frameRate": {"ideal": 30}, - }, - "audio": { - "echoCancellation": True, - "noiseSuppression": {"exact": True}, - "autoGainControl": {"exact": True}, - "sampleRate": {"ideal": 24000}, - "sampleSize": {"ideal": 16}, - "channelCount": {"exact": 1}, - }, - } - self.track_constraints = track_constraints - self.event_handler: Callable | StreamHandler | None = None - super().__init__( - label=label, - every=every, - inputs=inputs, - show_label=show_label, - container=container, - scale=scale, - min_width=min_width, - interactive=interactive, - visible=visible, - elem_id=elem_id, - elem_classes=elem_classes, - render=render, - key=key, - value=value, - ) - # need to do this here otherwise the proxy_url is not set - self.icon = ( - icon if not icon else cast(dict, self.serve_static_file(icon)).get("url") - ) - - def set_additional_outputs( - self, webrtc_id: str - ) -> Callable[[AdditionalOutputs], None]: - def set_outputs(outputs: AdditionalOutputs): - if webrtc_id not in self.additional_outputs: - self.additional_outputs[webrtc_id] = [] - self.additional_outputs[webrtc_id].append(outputs) - - return set_outputs - - def preprocess(self, payload: str) -> str: - """ - Parameters: - payload: An instance of VideoData containing the video and subtitle files. - Returns: - Passes the uploaded video as a `str` filepath or URL whose extension can be modified by `format`. - """ - return payload - - def postprocess(self, value: Any) -> str: - """ - Parameters: - value: Expects a {str} or {pathlib.Path} filepath to a video which is displayed, or a {Tuple[str | pathlib.Path, str | pathlib.Path | None]} where the first element is a filepath to a video and the second element is an optional filepath to a subtitle file. - Returns: - VideoData object containing the video and subtitle files. - """ - return value - - def set_input(self, webrtc_id: str, *args): - if webrtc_id in self.connections: - for conn in self.connections[webrtc_id]: - conn.set_args(list(args)) - - def on_additional_outputs( - self, - fn: Callable[Concatenate[P], R], - inputs: Block | Sequence[Block] | set[Block] | None = None, - outputs: Block | Sequence[Block] | set[Block] | None = None, - js: str | None = None, - concurrency_limit: int | None | Literal["default"] = "default", - concurrency_id: str | None = None, - show_progress: Literal["full", "minimal", "hidden"] = "full", - queue: bool = True, - ): - inputs = inputs or [] - if inputs and not isinstance(inputs, Iterable): - inputs = [inputs] - inputs = list(inputs) - - def handler(webrtc_id: str, *args): - if ( - webrtc_id in self.additional_outputs - and len(self.additional_outputs[webrtc_id]) > 0 - ): - next_outputs = self.additional_outputs[webrtc_id].pop(0) - return fn(*args, *next_outputs.args) # type: ignore - return ( - tuple([None for _ in range(len(outputs))]) - if isinstance(outputs, Iterable) - else None - ) - - return self.state_change( # type: ignore - fn=handler, - inputs=[self] + cast(list, inputs), - outputs=outputs, - js=js, - concurrency_limit=concurrency_limit, - concurrency_id=concurrency_id, - show_progress=show_progress, - queue=queue, - trigger_mode="multiple", - ) - - def stream( - self, - fn: Callable[..., Any] - | StreamHandlerImpl - | AudioVideoStreamHandlerImpl - | None = None, - inputs: Block | Sequence[Block] | set[Block] | None = None, - outputs: Block | Sequence[Block] | set[Block] | None = None, - js: str | None = None, - concurrency_limit: int | None | Literal["default"] = "default", - concurrency_id: str | None = None, - time_limit: float | None = None, - trigger: Dependency | None = None, - ): - from gradio.blocks import Block - - if inputs is None: - inputs = [] - if outputs is None: - outputs = [] - if isinstance(inputs, Block): - inputs = [inputs] - if isinstance(outputs, Block): - outputs = [outputs] - - self.concurrency_limit = ( - 1 if concurrency_limit in ["default", None] else concurrency_limit - ) - self.event_handler = fn # type: ignore - self.time_limit = time_limit - - if ( - self.mode == "send-receive" - and self.modality in ["audio", "audio-video"] - and not isinstance(self.event_handler, StreamHandlerBase) - ): - raise ValueError( - "In the send-receive mode for audio, the event handler must be an instance of StreamHandlerBase." - ) - - if self.mode == "send-receive" or self.mode == "send": - if cast(list[Block], inputs)[0] != self: - raise ValueError( - "In the webrtc stream event, the first input component must be the WebRTC component." - ) - - if ( - len(cast(list[Block], outputs)) != 1 - and cast(list[Block], outputs)[0] != self - ): - raise ValueError( - "In the webrtc stream event, the only output component must be the WebRTC component." - ) - for input_component in inputs[1:]: # type: ignore - if hasattr(input_component, "change"): - input_component.change( # type: ignore - self.set_input, - inputs=inputs, - outputs=None, - concurrency_id=concurrency_id, - concurrency_limit=None, - time_limit=None, - js=js, - ) - return self.tick( # type: ignore - self.set_input, - inputs=inputs, - outputs=None, - concurrency_id=concurrency_id, - concurrency_limit=None, - time_limit=None, - js=js, - ) - elif self.mode == "receive": - if isinstance(inputs, list) and self in cast(list[Block], inputs): - raise ValueError( - "In the receive mode stream event, the WebRTC component cannot be an input." - ) - if ( - len(cast(list[Block], outputs)) != 1 - and cast(list[Block], outputs)[0] != self - ): - raise ValueError( - "In the receive mode stream, the only output component must be the WebRTC component." - ) - if trigger is None: - raise ValueError( - "In the receive mode stream event, the trigger parameter must be provided" - ) - trigger(lambda: "start_webrtc_stream", inputs=None, outputs=self) - self.tick( # type: ignore - self.set_input, - inputs=[self] + list(inputs), - outputs=None, - concurrency_id=concurrency_id, - ) - - @staticmethod - async def wait_for_time_limit(pc: RTCPeerConnection, time_limit: float): - await asyncio.sleep(time_limit) - await pc.close() - - def clean_up(self, webrtc_id: str): - self.handlers.pop(webrtc_id, None) - connection = self.connections.pop(webrtc_id, []) - for conn in connection: - if isinstance(conn, AudioCallback): - conn.event_handler.shutdown() - self.additional_outputs.pop(webrtc_id, None) - self.data_channels.pop(webrtc_id, None) - return connection - - @server - async def offer(self, body): - logger.debug("Starting to handle offer") - logger.debug("Offer body %s", body) - if len(self.connections) >= cast(int, self.concurrency_limit): - return {"status": "failed"} - - offer = RTCSessionDescription(sdp=body["sdp"], type=body["type"]) - - pc = RTCPeerConnection() - self.pcs.add(pc) - - if isinstance(self.event_handler, StreamHandlerBase): - handler = self.event_handler.copy() - else: - handler = cast(Callable, self.event_handler) - - self.handlers[body["webrtc_id"]] = handler - - set_outputs = self.set_additional_outputs(body["webrtc_id"]) - - @pc.on("iceconnectionstatechange") - async def on_iceconnectionstatechange(): - logger.debug("ICE connection state change %s", pc.iceConnectionState) - if pc.iceConnectionState == "failed": - await pc.close() - self.connections.pop(body["webrtc_id"], None) - self.pcs.discard(pc) - - @pc.on("connectionstatechange") - async def on_connectionstatechange(): - logger.debug("pc.connectionState %s", pc.connectionState) - if pc.connectionState in ["failed", "closed"]: - await pc.close() - connection = self.clean_up(body["webrtc_id"]) - if connection: - for conn in connection: - conn.stop() - self.pcs.discard(pc) - if pc.connectionState == "connected": - if self.time_limit is not None: - asyncio.create_task(self.wait_for_time_limit(pc, self.time_limit)) - - @pc.on("track") - def on_track(track): - relay = MediaRelay() - handler = self.handlers[body["webrtc_id"]] - - if self.modality == "video" and track.kind == "video": - cb = VideoCallback( - relay.subscribe(track), - event_handler=cast(VideoEventHandler, handler), - set_additional_outputs=set_outputs, - mode=cast(Literal["send", "send-receive"], self.mode), - ) - elif self.modality == "audio-video" and track.kind == "video": - cb = VideoStreamHander( - relay.subscribe(track), - event_handler=handler, # type: ignore - set_additional_outputs=set_outputs, - ) - elif self.modality in ["audio", "audio-video"] and track.kind == "audio": - eh = cast(StreamHandlerImpl, handler) - eh._loop = asyncio.get_running_loop() - cb = AudioCallback( - relay.subscribe(track), - event_handler=eh, - set_additional_outputs=set_outputs, - ) - else: - raise ValueError("Modality must be either video, audio, or audio-video") - if body["webrtc_id"] not in self.connections: - self.connections[body["webrtc_id"]] = [] - - self.connections[body["webrtc_id"]].append(cb) - if body["webrtc_id"] in self.data_channels: - for conn in self.connections[body["webrtc_id"]]: - conn.set_channel(self.data_channels[body["webrtc_id"]]) - if self.mode == "send-receive": - logger.debug("Adding track to peer connection %s", cb) - pc.addTrack(cb) - elif self.mode == "send": - cast(AudioCallback | VideoCallback, cb).start() - - if self.mode == "receive": - if self.modality == "video": - cb = ServerToClientVideo( - cast(Callable, self.event_handler), - set_additional_outputs=set_outputs, - ) - elif self.modality == "audio": - cb = ServerToClientAudio( - cast(Callable, self.event_handler), - set_additional_outputs=set_outputs, - ) - else: - raise ValueError("Modality must be either video or audio") - - logger.debug("Adding track to peer connection %s", cb) - pc.addTrack(cb) - self.connections[body["webrtc_id"]].append(cb) - cb.on("ended", lambda: self.clean_up(body["webrtc_id"])) - - @pc.on("datachannel") - def on_datachannel(channel): - logger.debug(f"Data channel established: {channel.label}") - - self.data_channels[body["webrtc_id"]] = channel - - async def set_channel(webrtc_id: str): - while not self.connections.get(webrtc_id): - await asyncio.sleep(0.05) - logger.debug("setting channel for webrtc id %s", webrtc_id) - for conn in self.connections[webrtc_id]: - conn.set_channel(channel) - - asyncio.create_task(set_channel(body["webrtc_id"])) - - @channel.on("message") - def on_message(message): - logger.debug(f"Received message: {message}") - if channel.readyState == "open": - channel.send(f"Server received: {message}") - - # handle offer - await pc.setRemoteDescription(offer) - - # send answer - answer = await pc.createAnswer() - await pc.setLocalDescription(answer) # type: ignore - logger.debug("done handling offer about to return") - await asyncio.sleep(0.1) - - return { - "sdp": pc.localDescription.sdp, - "type": pc.localDescription.type, - } - - def example_payload(self) -> Any: - return { - "video": handle_file( - "https://github.com/gradio-app/gradio/raw/main/demo/video_component/files/world.mp4" - ), - } - - def example_value(self) -> Any: - return "https://github.com/gradio-app/gradio/raw/main/demo/video_component/files/world.mp4" - - def api_info(self) -> Any: - return {"type": "number"} diff --git a/demo/README.md b/demo/README.md deleted file mode 100644 index 16e1307..0000000 --- a/demo/README.md +++ /dev/null @@ -1,44 +0,0 @@ ---- -license: mit -tags: -- object-detection -- computer-vision -- yolov10 -datasets: -- detection-datasets/coco -sdk: gradio -sdk_version: 5.0.0b1 ---- - -### Model Description -[YOLOv10: Real-Time End-to-End Object Detection](https://arxiv.org/abs/2405.14458v1) - -- arXiv: https://arxiv.org/abs/2405.14458v1 -- github: https://github.com/THU-MIG/yolov10 - -### Installation -``` -pip install supervision git+https://github.com/THU-MIG/yolov10.git -``` - -### Yolov10 Inference -```python -from ultralytics import YOLOv10 -import supervision as sv -import cv2 - -IMAGE_PATH = 'dog.jpeg' - -model = YOLOv10.from_pretrained('jameslahm/yolov10{n/s/m/b/l/x}') -model.predict(IMAGE_PATH, show=True) -``` - -### BibTeX Entry and Citation Info - ``` -@article{wang2024yolov10, - title={YOLOv10: Real-Time End-to-End Object Detection}, - author={Wang, Ao and Chen, Hui and Liu, Lihao and Chen, Kai and Lin, Zijia and Han, Jungong and Ding, Guiguang}, - journal={arXiv preprint arXiv:2405.14458}, - year={2024} -} -``` \ No newline at end of file diff --git a/demo/also_return_text.py b/demo/also_return_text.py deleted file mode 100644 index 85a682a..0000000 --- a/demo/also_return_text.py +++ /dev/null @@ -1,105 +0,0 @@ -import logging -import os - -import gradio as gr -import numpy as np -from gradio_webrtc import AdditionalOutputs, WebRTC -from pydub import AudioSegment -from twilio.rest import Client - -# Configure the root logger to WARNING to suppress debug messages from other libraries -logging.basicConfig(level=logging.WARNING) - -# Create a console handler -console_handler = logging.FileHandler("gradio_webrtc.log") -console_handler.setLevel(logging.DEBUG) - -# Create a formatter -formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s") -console_handler.setFormatter(formatter) - -# Configure the logger for your specific library -logger = logging.getLogger("gradio_webrtc") -logger.setLevel(logging.DEBUG) -logger.addHandler(console_handler) - - -account_sid = os.environ.get("TWILIO_ACCOUNT_SID") -auth_token = os.environ.get("TWILIO_AUTH_TOKEN") - -if account_sid and auth_token: - client = Client(account_sid, auth_token) - - token = client.tokens.create() - - rtc_configuration = { - "iceServers": token.ice_servers, - "iceTransportPolicy": "relay", - } -else: - rtc_configuration = None - - -def generation(num_steps): - for i in range(num_steps): - segment = AudioSegment.from_file( - "/Users/freddy/sources/gradio/demo/scratch/audio-streaming/librispeech.mp3" - ) - yield ( - ( - segment.frame_rate, - np.array(segment.get_array_of_samples()).reshape(1, -1), - ), - AdditionalOutputs( - f"Hello, from step {i}!", - "/Users/freddy/sources/gradio/demo/scratch/audio-streaming/librispeech.mp3", - ), - ) - - -css = """.my-group {max-width: 600px !important; max-height: 600 !important;} - .my-column {display: flex !important; justify-content: center !important; align-items: center !important};""" - - -with gr.Blocks() as demo: - gr.HTML( - """ -

- Audio Streaming (Powered by WebRTC ⚡️) -

- """ - ) - with gr.Column(elem_classes=["my-column"]): - with gr.Group(elem_classes=["my-group"]): - audio = WebRTC( - label="Stream", - rtc_configuration=rtc_configuration, - mode="receive", - modality="audio", - ) - num_steps = gr.Slider( - label="Number of Steps", - minimum=1, - maximum=10, - step=1, - value=5, - ) - button = gr.Button("Generate") - textbox = gr.Textbox(placeholder="Output will appear here.") - audio_file = gr.Audio() - - audio.stream( - fn=generation, inputs=[num_steps], outputs=[audio], trigger=button.click - ) - audio.on_additional_outputs( - fn=lambda t, a: (f"State changed to {t}.", a), - outputs=[textbox, audio_file], - ) - - -if __name__ == "__main__": - demo.launch( - allowed_paths=[ - "/Users/freddy/sources/gradio/demo/scratch/audio-streaming/librispeech.mp3" - ] - ) diff --git a/demo/app.py b/demo/app.py deleted file mode 100644 index 378a8a5..0000000 --- a/demo/app.py +++ /dev/null @@ -1,367 +0,0 @@ -import os - -import gradio as gr - -_docs = { - "WebRTC": { - "description": "Stream audio/video with WebRTC", - "members": { - "__init__": { - "rtc_configuration": { - "type": "dict[str, Any] | None", - "default": "None", - "description": "The configration dictionary to pass to the RTCPeerConnection constructor. If None, the default configuration is used.", - }, - "height": { - "type": "int | str | None", - "default": "None", - "description": "The height of the component, specified in pixels if a number is passed, or in CSS units if a string is passed. This has no effect on the preprocessed video file, but will affect the displayed video.", - }, - "width": { - "type": "int | str | None", - "default": "None", - "description": "The width of the component, specified in pixels if a number is passed, or in CSS units if a string is passed. This has no effect on the preprocessed video file, but will affect the displayed video.", - }, - "label": { - "type": "str | None", - "default": "None", - "description": "the label for this component. Appears above the component and is also used as the header if there are a table of examples for this component. If None and used in a `gr.Interface`, the label will be the name of the parameter this component is assigned to.", - }, - "show_label": { - "type": "bool | None", - "default": "None", - "description": "if True, will display label.", - }, - "container": { - "type": "bool", - "default": "True", - "description": "if True, will place the component in a container - providing some extra padding around the border.", - }, - "scale": { - "type": "int | None", - "default": "None", - "description": "relative size compared to adjacent Components. For example if Components A and B are in a Row, and A has scale=2, and B has scale=1, A will be twice as wide as B. Should be an integer. scale applies in Rows, and to top-level Components in Blocks where fill_height=True.", - }, - "min_width": { - "type": "int", - "default": "160", - "description": "minimum pixel width, will wrap if not sufficient screen space to satisfy this value. If a certain scale value results in this Component being narrower than min_width, the min_width parameter will be respected first.", - }, - "interactive": { - "type": "bool | None", - "default": "None", - "description": "if True, will allow users to upload a video; if False, can only be used to display videos. If not provided, this is inferred based on whether the component is used as an input or output.", - }, - "visible": { - "type": "bool", - "default": "True", - "description": "if False, component will be hidden.", - }, - "elem_id": { - "type": "str | None", - "default": "None", - "description": "an optional string that is assigned as the id of this component in the HTML DOM. Can be used for targeting CSS styles.", - }, - "elem_classes": { - "type": "list[str] | str | None", - "default": "None", - "description": "an optional list of strings that are assigned as the classes of this component in the HTML DOM. Can be used for targeting CSS styles.", - }, - "render": { - "type": "bool", - "default": "True", - "description": "if False, component will not render be rendered in the Blocks context. Should be used if the intention is to assign event listeners now but render the component later.", - }, - "key": { - "type": "int | str | None", - "default": "None", - "description": "if assigned, will be used to assume identity across a re-render. Components that have the same key across a re-render will have their value preserved.", - }, - "mirror_webcam": { - "type": "bool", - "default": "True", - "description": "if True webcam will be mirrored. Default is True.", - }, - }, - "events": {"tick": {"type": None, "default": None, "description": ""}}, - }, - "__meta__": {"additional_interfaces": {}, "user_fn_refs": {"WebRTC": []}}, - } -} - - -abs_path = os.path.join(os.path.dirname(__file__), "css.css") - -with gr.Blocks( - css_paths=abs_path, - theme=gr.themes.Default( - font_mono=[ - gr.themes.GoogleFont("Inconsolata"), - "monospace", - ], - ), -) as demo: - gr.Markdown( - """ -

Gradio WebRTC ⚡️

- -
-Static Badge -Static Badge -
-""", - elem_classes=["md-custom"], - header_links=True, - ) - gr.Markdown( - """ -## Installation - -```bash -pip install gradio_webrtc -``` - -## Examples: -1. [Object Detection from Webcam with YOLOv10](https://huggingface.co/spaces/freddyaboulton/webrtc-yolov10n) 📷 -2. [Streaming Object Detection from Video with RT-DETR](https://huggingface.co/spaces/freddyaboulton/rt-detr-object-detection-webrtc) 🎥 -3. [Text-to-Speech](https://huggingface.co/spaces/freddyaboulton/parler-tts-streaming-webrtc) 🗣️ -4. [Conversational AI](https://huggingface.co/spaces/freddyaboulton/omni-mini-webrtc) 🤖🗣️ - -## Usage - -The WebRTC component supports the following three use cases: -1. [Streaming video from the user webcam to the server and back](#h-streaming-video-from-the-user-webcam-to-the-server-and-back) -2. [Streaming Video from the server to the client](#h-streaming-video-from-the-server-to-the-client) -3. [Streaming Audio from the server to the client](#h-streaming-audio-from-the-server-to-the-client) -4. [Streaming Audio from the client to the server and back (conversational AI)](#h-conversational-ai) - - -## Streaming Video from the User Webcam to the Server and Back - -```python -import gradio as gr -from gradio_webrtc import WebRTC - - -def detection(image, conf_threshold=0.3): - ... your detection code here ... - - -with gr.Blocks() as demo: - image = WebRTC(label="Stream", mode="send-receive", modality="video") - conf_threshold = gr.Slider( - label="Confidence Threshold", - minimum=0.0, - maximum=1.0, - step=0.05, - value=0.30, - ) - image.stream( - fn=detection, - inputs=[image, conf_threshold], - outputs=[image], time_limit=10 - ) - -if __name__ == "__main__": - demo.launch() - -``` -* Set the `mode` parameter to `send-receive` and `modality` to "video". -* The `stream` event's `fn` parameter is a function that receives the next frame from the webcam -as a **numpy array** and returns the processed frame also as a **numpy array**. -* Numpy arrays are in (height, width, 3) format where the color channels are in RGB format. -* The `inputs` parameter should be a list where the first element is the WebRTC component. The only output allowed is the WebRTC component. -* The `time_limit` parameter is the maximum time in seconds the video stream will run. If the time limit is reached, the video stream will stop. - -## Streaming Video from the server to the client - -```python -import gradio as gr -from gradio_webrtc import WebRTC -import cv2 - -def generation(): - url = "https://download.tsi.telecom-paristech.fr/gpac/dataset/dash/uhd/mux_sources/hevcds_720p30_2M.mp4" - cap = cv2.VideoCapture(url) - iterating = True - while iterating: - iterating, frame = cap.read() - yield frame - -with gr.Blocks() as demo: - output_video = WebRTC(label="Video Stream", mode="receive", modality="video") - button = gr.Button("Start", variant="primary") - output_video.stream( - fn=generation, inputs=None, outputs=[output_video], - trigger=button.click - ) - -if __name__ == "__main__": - demo.launch() -``` - -* Set the "mode" parameter to "receive" and "modality" to "video". -* The `stream` event's `fn` parameter is a generator function that yields the next frame from the video as a **numpy array**. -* The only output allowed is the WebRTC component. -* The `trigger` parameter the gradio event that will trigger the webrtc connection. In this case, the button click event. - -## Streaming Audio from the Server to the Client - -```python -import gradio as gr -from pydub import AudioSegment - -def generation(num_steps): - for _ in range(num_steps): - segment = AudioSegment.from_file("/Users/freddy/sources/gradio/demo/audio_debugger/cantina.wav") - yield (segment.frame_rate, np.array(segment.get_array_of_samples()).reshape(1, -1)) - -with gr.Blocks() as demo: - audio = WebRTC(label="Stream", mode="receive", modality="audio") - num_steps = gr.Slider( - label="Number of Steps", - minimum=1, - maximum=10, - step=1, - value=5, - ) - button = gr.Button("Generate") - - audio.stream( - fn=generation, inputs=[num_steps], outputs=[audio], - trigger=button.click - ) -``` - -* Set the "mode" parameter to "receive" and "modality" to "audio". -* The `stream` event's `fn` parameter is a generator function that yields the next audio segment as a tuple of (frame_rate, audio_samples). -* The numpy array should be of shape (1, num_samples). -* The `outputs` parameter should be a list with the WebRTC component as the only element. - -## Conversational AI - -```python -import gradio as gr -import numpy as np -from gradio_webrtc import WebRTC, StreamHandler -from queue import Queue -import time - - -class EchoHandler(StreamHandler): - def __init__(self) -> None: - super().__init__() - self.queue = Queue() - - def receive(self, frame: tuple[int, np.ndarray] | np.ndarray) -> None: - self.queue.put(frame) - - def emit(self) -> None: - return self.queue.get() - - -with gr.Blocks() as demo: - with gr.Column(): - with gr.Group(): - audio = WebRTC( - label="Stream", - rtc_configuration=None, - mode="send-receive", - modality="audio", - ) - - audio.stream(fn=EchoHandler(), inputs=[audio], outputs=[audio], time_limit=15) - - -if __name__ == "__main__": - demo.launch() -``` - -* Instead of passing a function to the `stream` event's `fn` parameter, pass a `StreamHandler` implementation. The `StreamHandler` above simply echoes the audio back to the client. -* The `StreamHandler` class has two methods: `receive` and `emit`. The `receive` method is called when a new frame is received from the client, and the `emit` method returns the next frame to send to the client. -* An audio frame is represented as a tuple of (frame_rate, audio_samples) where `audio_samples` is a numpy array of shape (num_channels, num_samples). -* You can also specify the audio layout ("mono" or "stereo") in the emit method by retuning it as the third element of the tuple. If not specified, the default is "mono". -* The `time_limit` parameter is the maximum time in seconds the conversation will run. If the time limit is reached, the audio stream will stop. -* The `emit` method SHOULD NOT block. If a frame is not ready to be sent, the method should return None. - -## Deployment - -When deploying in a cloud environment (like Hugging Face Spaces, EC2, etc), you need to set up a TURN server to relay the WebRTC traffic. -The easiest way to do this is to use a service like Twilio. - -```python -from twilio.rest import Client -import os - -account_sid = os.environ.get("TWILIO_ACCOUNT_SID") -auth_token = os.environ.get("TWILIO_AUTH_TOKEN") - -client = Client(account_sid, auth_token) - -token = client.tokens.create() - -rtc_configuration = { - "iceServers": token.ice_servers, - "iceTransportPolicy": "relay", -} - -with gr.Blocks() as demo: - ... - rtc = WebRTC(rtc_configuration=rtc_configuration, ...) - ... -``` -""", - elem_classes=["md-custom"], - header_links=True, - ) - - gr.Markdown( - """ -## -""", - elem_classes=["md-custom"], - header_links=True, - ) - - gr.ParamViewer(value=_docs["WebRTC"]["members"]["__init__"], linkify=[]) - - demo.load( - None, - js=r"""function() { - const refs = {}; - const user_fn_refs = { - WebRTC: [], }; - requestAnimationFrame(() => { - - Object.entries(user_fn_refs).forEach(([key, refs]) => { - if (refs.length > 0) { - const el = document.querySelector(`.${key}-user-fn`); - if (!el) return; - refs.forEach(ref => { - el.innerHTML = el.innerHTML.replace( - new RegExp("\\b"+ref+"\\b", "g"), - `${ref}` - ); - }) - } - }) - - Object.entries(refs).forEach(([key, refs]) => { - if (refs.length > 0) { - const el = document.querySelector(`.${key}`); - if (!el) return; - refs.forEach(ref => { - el.innerHTML = el.innerHTML.replace( - new RegExp("\\b"+ref+"\\b", "g"), - `${ref}` - ); - }) - } - }) - }) -} - -""", - ) - -demo.launch() diff --git a/demo/app_.py b/demo/app_.py deleted file mode 100644 index 378a8a5..0000000 --- a/demo/app_.py +++ /dev/null @@ -1,367 +0,0 @@ -import os - -import gradio as gr - -_docs = { - "WebRTC": { - "description": "Stream audio/video with WebRTC", - "members": { - "__init__": { - "rtc_configuration": { - "type": "dict[str, Any] | None", - "default": "None", - "description": "The configration dictionary to pass to the RTCPeerConnection constructor. If None, the default configuration is used.", - }, - "height": { - "type": "int | str | None", - "default": "None", - "description": "The height of the component, specified in pixels if a number is passed, or in CSS units if a string is passed. This has no effect on the preprocessed video file, but will affect the displayed video.", - }, - "width": { - "type": "int | str | None", - "default": "None", - "description": "The width of the component, specified in pixels if a number is passed, or in CSS units if a string is passed. This has no effect on the preprocessed video file, but will affect the displayed video.", - }, - "label": { - "type": "str | None", - "default": "None", - "description": "the label for this component. Appears above the component and is also used as the header if there are a table of examples for this component. If None and used in a `gr.Interface`, the label will be the name of the parameter this component is assigned to.", - }, - "show_label": { - "type": "bool | None", - "default": "None", - "description": "if True, will display label.", - }, - "container": { - "type": "bool", - "default": "True", - "description": "if True, will place the component in a container - providing some extra padding around the border.", - }, - "scale": { - "type": "int | None", - "default": "None", - "description": "relative size compared to adjacent Components. For example if Components A and B are in a Row, and A has scale=2, and B has scale=1, A will be twice as wide as B. Should be an integer. scale applies in Rows, and to top-level Components in Blocks where fill_height=True.", - }, - "min_width": { - "type": "int", - "default": "160", - "description": "minimum pixel width, will wrap if not sufficient screen space to satisfy this value. If a certain scale value results in this Component being narrower than min_width, the min_width parameter will be respected first.", - }, - "interactive": { - "type": "bool | None", - "default": "None", - "description": "if True, will allow users to upload a video; if False, can only be used to display videos. If not provided, this is inferred based on whether the component is used as an input or output.", - }, - "visible": { - "type": "bool", - "default": "True", - "description": "if False, component will be hidden.", - }, - "elem_id": { - "type": "str | None", - "default": "None", - "description": "an optional string that is assigned as the id of this component in the HTML DOM. Can be used for targeting CSS styles.", - }, - "elem_classes": { - "type": "list[str] | str | None", - "default": "None", - "description": "an optional list of strings that are assigned as the classes of this component in the HTML DOM. Can be used for targeting CSS styles.", - }, - "render": { - "type": "bool", - "default": "True", - "description": "if False, component will not render be rendered in the Blocks context. Should be used if the intention is to assign event listeners now but render the component later.", - }, - "key": { - "type": "int | str | None", - "default": "None", - "description": "if assigned, will be used to assume identity across a re-render. Components that have the same key across a re-render will have their value preserved.", - }, - "mirror_webcam": { - "type": "bool", - "default": "True", - "description": "if True webcam will be mirrored. Default is True.", - }, - }, - "events": {"tick": {"type": None, "default": None, "description": ""}}, - }, - "__meta__": {"additional_interfaces": {}, "user_fn_refs": {"WebRTC": []}}, - } -} - - -abs_path = os.path.join(os.path.dirname(__file__), "css.css") - -with gr.Blocks( - css_paths=abs_path, - theme=gr.themes.Default( - font_mono=[ - gr.themes.GoogleFont("Inconsolata"), - "monospace", - ], - ), -) as demo: - gr.Markdown( - """ -

Gradio WebRTC ⚡️

- -
-Static Badge -Static Badge -
-""", - elem_classes=["md-custom"], - header_links=True, - ) - gr.Markdown( - """ -## Installation - -```bash -pip install gradio_webrtc -``` - -## Examples: -1. [Object Detection from Webcam with YOLOv10](https://huggingface.co/spaces/freddyaboulton/webrtc-yolov10n) 📷 -2. [Streaming Object Detection from Video with RT-DETR](https://huggingface.co/spaces/freddyaboulton/rt-detr-object-detection-webrtc) 🎥 -3. [Text-to-Speech](https://huggingface.co/spaces/freddyaboulton/parler-tts-streaming-webrtc) 🗣️ -4. [Conversational AI](https://huggingface.co/spaces/freddyaboulton/omni-mini-webrtc) 🤖🗣️ - -## Usage - -The WebRTC component supports the following three use cases: -1. [Streaming video from the user webcam to the server and back](#h-streaming-video-from-the-user-webcam-to-the-server-and-back) -2. [Streaming Video from the server to the client](#h-streaming-video-from-the-server-to-the-client) -3. [Streaming Audio from the server to the client](#h-streaming-audio-from-the-server-to-the-client) -4. [Streaming Audio from the client to the server and back (conversational AI)](#h-conversational-ai) - - -## Streaming Video from the User Webcam to the Server and Back - -```python -import gradio as gr -from gradio_webrtc import WebRTC - - -def detection(image, conf_threshold=0.3): - ... your detection code here ... - - -with gr.Blocks() as demo: - image = WebRTC(label="Stream", mode="send-receive", modality="video") - conf_threshold = gr.Slider( - label="Confidence Threshold", - minimum=0.0, - maximum=1.0, - step=0.05, - value=0.30, - ) - image.stream( - fn=detection, - inputs=[image, conf_threshold], - outputs=[image], time_limit=10 - ) - -if __name__ == "__main__": - demo.launch() - -``` -* Set the `mode` parameter to `send-receive` and `modality` to "video". -* The `stream` event's `fn` parameter is a function that receives the next frame from the webcam -as a **numpy array** and returns the processed frame also as a **numpy array**. -* Numpy arrays are in (height, width, 3) format where the color channels are in RGB format. -* The `inputs` parameter should be a list where the first element is the WebRTC component. The only output allowed is the WebRTC component. -* The `time_limit` parameter is the maximum time in seconds the video stream will run. If the time limit is reached, the video stream will stop. - -## Streaming Video from the server to the client - -```python -import gradio as gr -from gradio_webrtc import WebRTC -import cv2 - -def generation(): - url = "https://download.tsi.telecom-paristech.fr/gpac/dataset/dash/uhd/mux_sources/hevcds_720p30_2M.mp4" - cap = cv2.VideoCapture(url) - iterating = True - while iterating: - iterating, frame = cap.read() - yield frame - -with gr.Blocks() as demo: - output_video = WebRTC(label="Video Stream", mode="receive", modality="video") - button = gr.Button("Start", variant="primary") - output_video.stream( - fn=generation, inputs=None, outputs=[output_video], - trigger=button.click - ) - -if __name__ == "__main__": - demo.launch() -``` - -* Set the "mode" parameter to "receive" and "modality" to "video". -* The `stream` event's `fn` parameter is a generator function that yields the next frame from the video as a **numpy array**. -* The only output allowed is the WebRTC component. -* The `trigger` parameter the gradio event that will trigger the webrtc connection. In this case, the button click event. - -## Streaming Audio from the Server to the Client - -```python -import gradio as gr -from pydub import AudioSegment - -def generation(num_steps): - for _ in range(num_steps): - segment = AudioSegment.from_file("/Users/freddy/sources/gradio/demo/audio_debugger/cantina.wav") - yield (segment.frame_rate, np.array(segment.get_array_of_samples()).reshape(1, -1)) - -with gr.Blocks() as demo: - audio = WebRTC(label="Stream", mode="receive", modality="audio") - num_steps = gr.Slider( - label="Number of Steps", - minimum=1, - maximum=10, - step=1, - value=5, - ) - button = gr.Button("Generate") - - audio.stream( - fn=generation, inputs=[num_steps], outputs=[audio], - trigger=button.click - ) -``` - -* Set the "mode" parameter to "receive" and "modality" to "audio". -* The `stream` event's `fn` parameter is a generator function that yields the next audio segment as a tuple of (frame_rate, audio_samples). -* The numpy array should be of shape (1, num_samples). -* The `outputs` parameter should be a list with the WebRTC component as the only element. - -## Conversational AI - -```python -import gradio as gr -import numpy as np -from gradio_webrtc import WebRTC, StreamHandler -from queue import Queue -import time - - -class EchoHandler(StreamHandler): - def __init__(self) -> None: - super().__init__() - self.queue = Queue() - - def receive(self, frame: tuple[int, np.ndarray] | np.ndarray) -> None: - self.queue.put(frame) - - def emit(self) -> None: - return self.queue.get() - - -with gr.Blocks() as demo: - with gr.Column(): - with gr.Group(): - audio = WebRTC( - label="Stream", - rtc_configuration=None, - mode="send-receive", - modality="audio", - ) - - audio.stream(fn=EchoHandler(), inputs=[audio], outputs=[audio], time_limit=15) - - -if __name__ == "__main__": - demo.launch() -``` - -* Instead of passing a function to the `stream` event's `fn` parameter, pass a `StreamHandler` implementation. The `StreamHandler` above simply echoes the audio back to the client. -* The `StreamHandler` class has two methods: `receive` and `emit`. The `receive` method is called when a new frame is received from the client, and the `emit` method returns the next frame to send to the client. -* An audio frame is represented as a tuple of (frame_rate, audio_samples) where `audio_samples` is a numpy array of shape (num_channels, num_samples). -* You can also specify the audio layout ("mono" or "stereo") in the emit method by retuning it as the third element of the tuple. If not specified, the default is "mono". -* The `time_limit` parameter is the maximum time in seconds the conversation will run. If the time limit is reached, the audio stream will stop. -* The `emit` method SHOULD NOT block. If a frame is not ready to be sent, the method should return None. - -## Deployment - -When deploying in a cloud environment (like Hugging Face Spaces, EC2, etc), you need to set up a TURN server to relay the WebRTC traffic. -The easiest way to do this is to use a service like Twilio. - -```python -from twilio.rest import Client -import os - -account_sid = os.environ.get("TWILIO_ACCOUNT_SID") -auth_token = os.environ.get("TWILIO_AUTH_TOKEN") - -client = Client(account_sid, auth_token) - -token = client.tokens.create() - -rtc_configuration = { - "iceServers": token.ice_servers, - "iceTransportPolicy": "relay", -} - -with gr.Blocks() as demo: - ... - rtc = WebRTC(rtc_configuration=rtc_configuration, ...) - ... -``` -""", - elem_classes=["md-custom"], - header_links=True, - ) - - gr.Markdown( - """ -## -""", - elem_classes=["md-custom"], - header_links=True, - ) - - gr.ParamViewer(value=_docs["WebRTC"]["members"]["__init__"], linkify=[]) - - demo.load( - None, - js=r"""function() { - const refs = {}; - const user_fn_refs = { - WebRTC: [], }; - requestAnimationFrame(() => { - - Object.entries(user_fn_refs).forEach(([key, refs]) => { - if (refs.length > 0) { - const el = document.querySelector(`.${key}-user-fn`); - if (!el) return; - refs.forEach(ref => { - el.innerHTML = el.innerHTML.replace( - new RegExp("\\b"+ref+"\\b", "g"), - `${ref}` - ); - }) - } - }) - - Object.entries(refs).forEach(([key, refs]) => { - if (refs.length > 0) { - const el = document.querySelector(`.${key}`); - if (!el) return; - refs.forEach(ref => { - el.innerHTML = el.innerHTML.replace( - new RegExp("\\b"+ref+"\\b", "g"), - `${ref}` - ); - }) - } - }) - }) -} - -""", - ) - -demo.launch() diff --git a/demo/app_orig.py b/demo/app_orig.py deleted file mode 100644 index 31f3b3f..0000000 --- a/demo/app_orig.py +++ /dev/null @@ -1,73 +0,0 @@ -import os - -import cv2 -import gradio as gr -from gradio_webrtc import WebRTC -from huggingface_hub import hf_hub_download -from inference import YOLOv10 -from twilio.rest import Client - -model_file = hf_hub_download( - repo_id="onnx-community/yolov10n", filename="onnx/model.onnx" -) - -model = YOLOv10(model_file) - -account_sid = os.environ.get("TWILIO_ACCOUNT_SID") -auth_token = os.environ.get("TWILIO_AUTH_TOKEN") - -if account_sid and auth_token: - client = Client(account_sid, auth_token) - - token = client.tokens.create() - - rtc_configuration = { - "iceServers": token.ice_servers, - "iceTransportPolicy": "relay", - } -else: - rtc_configuration = None - - -def detection(image, conf_threshold=0.3): - image = cv2.resize(image, (model.input_width, model.input_height)) - new_image = model.detect_objects(image, conf_threshold) - return cv2.resize(new_image, (500, 500)) - - -css = """.my-group {max-width: 600px !important; max-height: 600 !important;} - .my-column {display: flex !important; justify-content: center !important; align-items: center !important};""" - - -with gr.Blocks(css=css) as demo: - gr.HTML( - """ -

- YOLOv10 Webcam Stream (Powered by WebRTC ⚡️) -

- """ - ) - gr.HTML( - """ -

- arXiv | github -

- """ - ) - with gr.Column(elem_classes=["my-column"]): - with gr.Group(elem_classes=["my-group"]): - image = WebRTC(label="Stream", rtc_configuration=rtc_configuration) - conf_threshold = gr.Slider( - label="Confidence Threshold", - minimum=0.0, - maximum=1.0, - step=0.05, - value=0.30, - ) - - image.stream( - fn=detection, inputs=[image, conf_threshold], outputs=[image], time_limit=10 - ) - -if __name__ == "__main__": - demo.launch() diff --git a/demo/audio_out.py b/demo/audio_out.py deleted file mode 100644 index 72dffc2..0000000 --- a/demo/audio_out.py +++ /dev/null @@ -1,71 +0,0 @@ -import os - -import gradio as gr -import numpy as np -from gradio_webrtc import WebRTC -from pydub import AudioSegment -from twilio.rest import Client - -account_sid = os.environ.get("TWILIO_ACCOUNT_SID") -auth_token = os.environ.get("TWILIO_AUTH_TOKEN") - -if account_sid and auth_token: - client = Client(account_sid, auth_token) - - token = client.tokens.create() - - rtc_configuration = { - "iceServers": token.ice_servers, - "iceTransportPolicy": "relay", - } -else: - rtc_configuration = None - - -def generation(num_steps): - for _ in range(num_steps): - segment = AudioSegment.from_file( - "/Users/freddy/sources/gradio/demo/audio_debugger/cantina.wav" - ) - yield ( - segment.frame_rate, - np.array(segment.get_array_of_samples()).reshape(1, -1), - ) - - -css = """.my-group {max-width: 600px !important; max-height: 600 !important;} - .my-column {display: flex !important; justify-content: center !important; align-items: center !important};""" - - -with gr.Blocks() as demo: - gr.HTML( - """ -

- Audio Streaming (Powered by WebRTC ⚡️) -

- """ - ) - with gr.Column(elem_classes=["my-column"]): - with gr.Group(elem_classes=["my-group"]): - audio = WebRTC( - label="Stream", - rtc_configuration=rtc_configuration, - mode="receive", - modality="audio", - ) - num_steps = gr.Slider( - label="Number of Steps", - minimum=1, - maximum=10, - step=1, - value=5, - ) - button = gr.Button("Generate") - - audio.stream( - fn=generation, inputs=[num_steps], outputs=[audio], trigger=button.click - ) - - -if __name__ == "__main__": - demo.launch() diff --git a/demo/audio_out_2.py b/demo/audio_out_2.py deleted file mode 100644 index eda279f..0000000 --- a/demo/audio_out_2.py +++ /dev/null @@ -1,64 +0,0 @@ -import os -import time - -import gradio as gr -import numpy as np -from gradio_webrtc import WebRTC -from pydub import AudioSegment -from twilio.rest import Client - -account_sid = os.environ.get("TWILIO_ACCOUNT_SID") -auth_token = os.environ.get("TWILIO_AUTH_TOKEN") - -if account_sid and auth_token: - client = Client(account_sid, auth_token) - - token = client.tokens.create() - - rtc_configuration = { - "iceServers": token.ice_servers, - "iceTransportPolicy": "relay", - } -else: - rtc_configuration = None - - -def generation(num_steps): - for _ in range(num_steps): - segment = AudioSegment.from_file( - "/Users/freddy/sources/gradio/demo/audio_debugger/cantina.wav" - ) - yield ( - segment.frame_rate, - np.array(segment.get_array_of_samples()).reshape(1, -1), - ) - time.sleep(3.5) - - -css = """.my-group {max-width: 600px !important; max-height: 600 !important;} - .my-column {display: flex !important; justify-content: center !important; align-items: center !important};""" - - -with gr.Blocks() as demo: - gr.HTML( - """ -

- Audio Streaming (Powered by WebRaTC ⚡️) -

- """ - ) - with gr.Row(): - with gr.Column(): - gr.Slider() - with gr.Column(): - # audio = gr.Audio(interactive=False) - audio = WebRTC( - label="Stream", - rtc_configuration=rtc_configuration, - mode="receive", - modality="audio", - ) - - -if __name__ == "__main__": - demo.launch() diff --git a/demo/css.css b/demo/css.css deleted file mode 100644 index 486edd8..0000000 --- a/demo/css.css +++ /dev/null @@ -1,161 +0,0 @@ -html { - font-family: Inter; - font-size: 16px; - font-weight: 400; - line-height: 1.5; - -webkit-text-size-adjust: 100%; - background: #fff; - color: #323232; - -webkit-font-smoothing: antialiased; - -moz-osx-font-smoothing: grayscale; - text-rendering: optimizeLegibility; -} - -:root { - --space: 1; - --vspace: calc(var(--space) * 1rem); - --vspace-0: calc(3 * var(--space) * 1rem); - --vspace-1: calc(2 * var(--space) * 1rem); - --vspace-2: calc(1.5 * var(--space) * 1rem); - --vspace-3: calc(0.5 * var(--space) * 1rem); -} - -.app { - max-width: 748px !important; -} - -.prose p { - margin: var(--vspace) 0; - line-height: var(--vspace * 2); - font-size: 1rem; -} - -code { - font-family: "Inconsolata", sans-serif; - font-size: 16px; -} - -h1, -h1 code { - font-weight: 400; - line-height: calc(2.5 / var(--space) * var(--vspace)); -} - -h1 code { - background: none; - border: none; - letter-spacing: 0.05em; - padding-bottom: 5px; - position: relative; - padding: 0; -} - -h2 { - margin: var(--vspace-1) 0 var(--vspace-2) 0; - line-height: 1em; -} - -h3, -h3 code { - margin: var(--vspace-1) 0 var(--vspace-2) 0; - line-height: 1em; -} - -h4, -h5, -h6 { - margin: var(--vspace-3) 0 var(--vspace-3) 0; - line-height: var(--vspace); -} - -.bigtitle, -h1, -h1 code { - font-size: calc(8px * 4.5); - word-break: break-word; -} - -.title, -h2, -h2 code { - font-size: calc(8px * 3.375); - font-weight: lighter; - word-break: break-word; - border: none; - background: none; -} - -.subheading1, -h3, -h3 code { - font-size: calc(8px * 1.8); - font-weight: 600; - border: none; - background: none; - letter-spacing: 0.1em; - text-transform: uppercase; -} - -h2 code { - padding: 0; - position: relative; - letter-spacing: 0.05em; -} - -blockquote { - font-size: calc(8px * 1.1667); - font-style: italic; - line-height: calc(1.1667 * var(--vspace)); - margin: var(--vspace-2) var(--vspace-2); -} - -.subheading2, -h4 { - font-size: calc(8px * 1.4292); - text-transform: uppercase; - font-weight: 600; -} - -.subheading3, -h5 { - font-size: calc(8px * 1.2917); - line-height: calc(1.2917 * var(--vspace)); - - font-weight: lighter; - text-transform: uppercase; - letter-spacing: 0.15em; -} - -h6 { - font-size: calc(8px * 1.1667); - font-size: 1.1667em; - font-weight: normal; - font-style: italic; - font-family: "le-monde-livre-classic-byol", serif !important; - letter-spacing: 0px !important; -} - -#start .md > *:first-child { - margin-top: 0; -} - -h2 + h3 { - margin-top: 0; -} - -.md hr { - border: none; - border-top: 1px solid var(--block-border-color); - margin: var(--vspace-2) 0 var(--vspace-2) 0; -} -.prose ul { - margin: var(--vspace-2) 0 var(--vspace-1) 0; -} - -.gap { - gap: 0; -} - -.md-custom { - overflow: hidden; -} diff --git a/demo/docs.py b/demo/docs.py deleted file mode 100644 index dde0b2e..0000000 --- a/demo/docs.py +++ /dev/null @@ -1,99 +0,0 @@ -_docs = { - "WebRTC": { - "description": "Stream audio/video with WebRTC", - "members": { - "__init__": { - "rtc_configuration": { - "type": "dict[str, Any] | None", - "default": "None", - "description": "The configration dictionary to pass to the RTCPeerConnection constructor. If None, the default configuration is used.", - }, - "height": { - "type": "int | str | None", - "default": "None", - "description": "The height of the component, specified in pixels if a number is passed, or in CSS units if a string is passed. This has no effect on the preprocessed video file, but will affect the displayed video.", - }, - "width": { - "type": "int | str | None", - "default": "None", - "description": "The width of the component, specified in pixels if a number is passed, or in CSS units if a string is passed. This has no effect on the preprocessed video file, but will affect the displayed video.", - }, - "label": { - "type": "str | None", - "default": "None", - "description": "the label for this component. Appears above the component and is also used as the header if there are a table of examples for this component. If None and used in a `gr.Interface`, the label will be the name of the parameter this component is assigned to.", - }, - "show_label": { - "type": "bool | None", - "default": "None", - "description": "if True, will display label.", - }, - "container": { - "type": "bool", - "default": "True", - "description": "if True, will place the component in a container - providing some extra padding around the border.", - }, - "scale": { - "type": "int | None", - "default": "None", - "description": "relative size compared to adjacent Components. For example if Components A and B are in a Row, and A has scale=2, and B has scale=1, A will be twice as wide as B. Should be an integer. scale applies in Rows, and to top-level Components in Blocks where fill_height=True.", - }, - "min_width": { - "type": "int", - "default": "160", - "description": "minimum pixel width, will wrap if not sufficient screen space to satisfy this value. If a certain scale value results in this Component being narrower than min_width, the min_width parameter will be respected first.", - }, - "interactive": { - "type": "bool | None", - "default": "None", - "description": "if True, will allow users to upload a video; if False, can only be used to display videos. If not provided, this is inferred based on whether the component is used as an input or output.", - }, - "visible": { - "type": "bool", - "default": "True", - "description": "if False, component will be hidden.", - }, - "elem_id": { - "type": "str | None", - "default": "None", - "description": "an optional string that is assigned as the id of this component in the HTML DOM. Can be used for targeting CSS styles.", - }, - "elem_classes": { - "type": "list[str] | str | None", - "default": "None", - "description": "an optional list of strings that are assigned as the classes of this component in the HTML DOM. Can be used for targeting CSS styles.", - }, - "render": { - "type": "bool", - "default": "True", - "description": "if False, component will not render be rendered in the Blocks context. Should be used if the intention is to assign event listeners now but render the component later.", - }, - "key": { - "type": "int | str | None", - "default": "None", - "description": "if assigned, will be used to assume identity across a re-render. Components that have the same key across a re-render will have their value preserved.", - }, - "mirror_webcam": { - "type": "bool", - "default": "True", - "description": "if True webcam will be mirrored. Default is True.", - }, - "postprocess": { - "value": { - "type": "typing.Any", - "description": "Expects a {str} or {pathlib.Path} filepath to a video which is displayed, or a {Tuple[str | pathlib.Path, str | pathlib.Path | None]} where the first element is a filepath to a video and the second element is an optional filepath to a subtitle file.", - } - }, - "preprocess": { - "return": { - "type": "str", - "description": "Passes the uploaded video as a `str` filepath or URL whose extension can be modified by `format`.", - }, - "value": None, - }, - }, - "events": {"tick": {"type": None, "default": None, "description": ""}}, - }, - "__meta__": {"additional_interfaces": {}, "user_fn_refs": {"WebRTC": []}}, - } -} diff --git a/demo/echo_audio/README.md b/demo/echo_audio/README.md new file mode 100644 index 0000000..c02c1c3 --- /dev/null +++ b/demo/echo_audio/README.md @@ -0,0 +1,15 @@ +--- +title: Echo Audio +emoji: 🪩 +colorFrom: purple +colorTo: red +sdk: gradio +sdk_version: 5.16.0 +app_file: app.py +pinned: false +license: mit +short_description: Simple echo stream - simplest FastRTC demo +tags: [webrtc, websocket, gradio, secret|TWILIO_ACCOUNT_SID, secret|TWILIO_AUTH_TOKEN] +--- + +Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference \ No newline at end of file diff --git a/demo/echo_audio/app.py b/demo/echo_audio/app.py new file mode 100644 index 0000000..18f1e71 --- /dev/null +++ b/demo/echo_audio/app.py @@ -0,0 +1,45 @@ +import numpy as np +from fastapi import FastAPI +from fastapi.responses import RedirectResponse +from fastrtc import ReplyOnPause, Stream, get_twilio_turn_credentials +from gradio.utils import get_space + + +def detection(audio: tuple[int, np.ndarray]): + # Implement any iterator that yields audio + # See "LLM Voice Chat" for a more complete example + yield audio + + +stream = Stream( + handler=ReplyOnPause(detection), + modality="audio", + mode="send-receive", + rtc_configuration=get_twilio_turn_credentials() if get_space() else None, + concurrency_limit=5 if get_space() else None, + time_limit=90 if get_space() else None, +) + +app = FastAPI() + +stream.mount(app) + + +@app.get("/") +async def index(): + return RedirectResponse( + url="/ui" if not get_space() else "https://fastrtc-echo-audio.hf.space/ui/" + ) + + +if __name__ == "__main__": + import os + + if (mode := os.getenv("MODE")) == "UI": + stream.ui.launch(server_port=7860) + elif mode == "PHONE": + stream.fastphone(port=7860) + else: + import uvicorn + + uvicorn.run(app, host="0.0.0.0", port=7860) diff --git a/demo/echo_audio/requirements.txt b/demo/echo_audio/requirements.txt new file mode 100644 index 0000000..eef4c22 --- /dev/null +++ b/demo/echo_audio/requirements.txt @@ -0,0 +1,3 @@ +fastrtc[vad] +twilio +python-dotenv diff --git a/demo/echo_conversation.py b/demo/echo_conversation.py deleted file mode 100644 index be58de7..0000000 --- a/demo/echo_conversation.py +++ /dev/null @@ -1,61 +0,0 @@ -import logging -from queue import Queue - -import gradio as gr -import numpy as np -from gradio_webrtc import StreamHandler, WebRTC - -# Configure the root logger to WARNING to suppress debug messages from other libraries -logging.basicConfig(level=logging.WARNING) - -# Create a console handler -console_handler = logging.StreamHandler() -console_handler.setLevel(logging.DEBUG) - -# Create a formatter -formatter = logging.Formatter("%(name)s - %(levelname)s - %(message)s") -console_handler.setFormatter(formatter) - -# Configure the logger for your specific library -logger = logging.getLogger("gradio_webrtc") -logger.setLevel(logging.DEBUG) -logger.addHandler(console_handler) - - -class EchoHandler(StreamHandler): - def __init__(self) -> None: - super().__init__() - self.queue = Queue() - - def receive(self, frame: tuple[int, np.ndarray] | np.ndarray) -> None: - self.queue.put(frame) - - def emit(self) -> None: - return self.queue.get() - - def copy(self) -> StreamHandler: - return EchoHandler() - - -with gr.Blocks() as demo: - gr.HTML( - """ -

- Conversational AI (Powered by WebRTC ⚡️) -

- """ - ) - with gr.Column(): - with gr.Group(): - audio = WebRTC( - label="Stream", - rtc_configuration=None, - mode="send-receive", - modality="audio", - ) - - audio.stream(fn=EchoHandler(), inputs=[audio], outputs=[audio], time_limit=15) - - -if __name__ == "__main__": - demo.launch() diff --git a/demo/gemini_audio_video/README.md b/demo/gemini_audio_video/README.md new file mode 100644 index 0000000..9a622c5 --- /dev/null +++ b/demo/gemini_audio_video/README.md @@ -0,0 +1,15 @@ +--- +title: Gemini Audio Video +emoji: ♊️ +colorFrom: purple +colorTo: red +sdk: gradio +sdk_version: 5.16.0 +app_file: app.py +pinned: false +license: mit +short_description: Gemini understands audio and video! +tags: [webrtc, websocket, gradio, secret|TWILIO_ACCOUNT_SID, secret|TWILIO_AUTH_TOKEN, secret|GEMINI_API_KEY] +--- + +Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference \ No newline at end of file diff --git a/demo/gemini_audio_video/app.py b/demo/gemini_audio_video/app.py new file mode 100644 index 0000000..7fbe44d --- /dev/null +++ b/demo/gemini_audio_video/app.py @@ -0,0 +1,207 @@ +import asyncio +import base64 +import os +import time +from io import BytesIO + +import gradio as gr +from gradio.utils import get_space +import numpy as np +from google import genai +from dotenv import load_dotenv +from fastrtc import ( + AsyncAudioVideoStreamHandler, + Stream, + get_twilio_turn_credentials, + WebRTC, +) +from PIL import Image + +load_dotenv() + + +def encode_audio(data: np.ndarray) -> dict: + """Encode Audio data to send to the server""" + return { + "mime_type": "audio/pcm", + "data": base64.b64encode(data.tobytes()).decode("UTF-8"), + } + + +def encode_image(data: np.ndarray) -> dict: + with BytesIO() as output_bytes: + pil_image = Image.fromarray(data) + pil_image.save(output_bytes, "JPEG") + bytes_data = output_bytes.getvalue() + base64_str = str(base64.b64encode(bytes_data), "utf-8") + return {"mime_type": "image/jpeg", "data": base64_str} + + +class GeminiHandler(AsyncAudioVideoStreamHandler): + def __init__( + self, + ) -> None: + super().__init__( + "mono", + output_sample_rate=24000, + output_frame_size=480, + input_sample_rate=16000, + ) + self.audio_queue = asyncio.Queue() + self.video_queue = asyncio.Queue() + self.quit = asyncio.Event() + self.session = None + self.last_frame_time = 0 + self.quit = asyncio.Event() + + def copy(self) -> "GeminiHandler": + return GeminiHandler() + + async def start_up(self): + client = genai.Client( + api_key=os.getenv("GEMINI_API_KEY"), http_options={"api_version": "v1alpha"} + ) + config = {"response_modalities": ["AUDIO"]} + try: + async with client.aio.live.connect( + model="gemini-2.0-flash-exp", config=config + ) as session: + self.session = session + print("set session") + while not self.quit.is_set(): + turn = self.session.receive() + async for response in turn: + if data := response.data: + audio = np.frombuffer(data, dtype=np.int16).reshape(1, -1) + self.audio_queue.put_nowait(audio) + except Exception as e: + import traceback + + traceback.print_exc() + + async def video_receive(self, frame: np.ndarray): + try: + print("out") + if self.session: + print("here") + # send image every 1 second + print(time.time() - self.last_frame_time) + if time.time() - self.last_frame_time > 1: + self.last_frame_time = time.time() + print("sending image") + await self.session.send(input=encode_image(frame)) + print("sent image") + if self.latest_args[1] is not None: + print("sending image2") + await self.session.send(input=encode_image(self.latest_args[1])) + print("sent image2") + except Exception as e: + print(e) + import traceback + + traceback.print_exc() + self.video_queue.put_nowait(frame) + + async def video_emit(self): + return await self.video_queue.get() + + async def receive(self, frame: tuple[int, np.ndarray]) -> None: + _, array = frame + array = array.squeeze() + audio_message = encode_audio(array) + if self.session: + try: + await self.session.send(input=audio_message) + except Exception as e: + print(e) + import traceback + + traceback.print_exc() + + async def emit(self): + array = await self.audio_queue.get() + return (self.output_sample_rate, array) + + async def shutdown(self) -> None: + if self.session: + self.quit.set() + await self.session._websocket.close() + self.quit.clear() + + +stream = Stream( + handler=GeminiHandler(), + modality="audio-video", + mode="send-receive", + rtc_configuration=get_twilio_turn_credentials() + if get_space() == "spaces" + else None, + time_limit=90 if get_space() else None, + additional_inputs=[ + gr.Image(label="Image", type="numpy", sources=["upload", "clipboard"]) + ], + ui_args={ + "icon": "https://www.gstatic.com/lamda/images/gemini_favicon_f069958c85030456e93de685481c559f160ea06b.png", + "pulse_color": "rgb(35, 157, 225)", + "icon_button_color": "rgb(35, 157, 225)", + "title": "Gemini Audio Video Chat", + }, +) + +css = """ +#video-source {max-width: 600px !important; max-height: 600 !important;} +""" + +with gr.Blocks(css=css) as demo: + gr.HTML( + """ +
+
+ +
+
+

Gen AI SDK Voice Chat

+

Speak with Gemini using real-time audio + video streaming

+

Powered by Gradio and WebRTC⚡️

+

Get an API Key here

+
+
+ """ + ) + with gr.Row() as row: + with gr.Column(): + webrtc = WebRTC( + label="Video Chat", + modality="audio-video", + mode="send-receive", + elem_id="video-source", + rtc_configuration=get_twilio_turn_credentials() + if get_space() == "spaces" + else None, + icon="https://www.gstatic.com/lamda/images/gemini_favicon_f069958c85030456e93de685481c559f160ea06b.png", + pulse_color="rgb(35, 157, 225)", + icon_button_color="rgb(35, 157, 225)", + ) + with gr.Column(): + image_input = gr.Image( + label="Image", type="numpy", sources=["upload", "clipboard"] + ) + + webrtc.stream( + GeminiHandler(), + inputs=[webrtc, image_input], + outputs=[webrtc], + time_limit=60 if get_space() else None, + concurrency_limit=2 if get_space() else None, + ) + +stream.ui = demo + + +if __name__ == "__main__": + if (mode := os.getenv("MODE")) == "UI": + stream.ui.launch(server_port=7860) + elif mode == "PHONE": + raise ValueError("Phone mode not supported for this demo") + else: + stream.ui.launch(server_port=7860) diff --git a/demo/gemini_audio_video/requirements.txt b/demo/gemini_audio_video/requirements.txt new file mode 100644 index 0000000..e8cbeb2 --- /dev/null +++ b/demo/gemini_audio_video/requirements.txt @@ -0,0 +1,4 @@ +fastrtc +python-dotenv +google-genai +twilio diff --git a/demo/hello_computer/README.md b/demo/hello_computer/README.md new file mode 100644 index 0000000..e85e2de --- /dev/null +++ b/demo/hello_computer/README.md @@ -0,0 +1,15 @@ +--- +title: Hello Computer +emoji: 💻 +colorFrom: purple +colorTo: red +sdk: gradio +sdk_version: 5.16.0 +app_file: app.py +pinned: false +license: mit +short_description: Say computer before asking your question +tags: [webrtc, websocket, gradio, secret|TWILIO_ACCOUNT_SID, secret|TWILIO_AUTH_TOKEN, secret|SAMBANOVA_API_KEY] +--- + +Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference \ No newline at end of file diff --git a/demo/hello_computer/README_gradio.md b/demo/hello_computer/README_gradio.md new file mode 100644 index 0000000..843605f --- /dev/null +++ b/demo/hello_computer/README_gradio.md @@ -0,0 +1,15 @@ +--- +title: Hello Computer (Gradio) +emoji: 💻 +colorFrom: purple +colorTo: red +sdk: gradio +sdk_version: 5.16.0 +app_file: app.py +pinned: false +license: mit +short_description: Say computer (Gradio) +tags: [webrtc, websocket, gradio, secret|TWILIO_ACCOUNT_SID, secret|TWILIO_AUTH_TOKEN, secret|SAMBANOVA_API_KEY] +--- + +Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference \ No newline at end of file diff --git a/demo/hello_computer/app.py b/demo/hello_computer/app.py new file mode 100644 index 0000000..0f5a1dc --- /dev/null +++ b/demo/hello_computer/app.py @@ -0,0 +1,153 @@ +import base64 +import json +import os +from pathlib import Path + +import gradio as gr +import numpy as np +import openai +from dotenv import load_dotenv +from fastapi import FastAPI +from fastapi.responses import HTMLResponse, StreamingResponse +from fastrtc import ( + AdditionalOutputs, + ReplyOnStopWords, + Stream, + WebRTCError, + get_stt_model, + get_twilio_turn_credentials, +) +from gradio.utils import get_space +from pydantic import BaseModel + +load_dotenv() + +curr_dir = Path(__file__).parent + + +client = openai.OpenAI( + api_key=os.environ.get("SAMBANOVA_API_KEY"), + base_url="https://api.sambanova.ai/v1", +) +model = get_stt_model() + + +def response( + audio: tuple[int, np.ndarray], + gradio_chatbot: list[dict] | None = None, + conversation_state: list[dict] | None = None, +): + gradio_chatbot = gradio_chatbot or [] + conversation_state = conversation_state or [] + try: + text = model.stt(audio) + print("STT in handler", text) + sample_rate, array = audio + gradio_chatbot.append( + {"role": "user", "content": gr.Audio((sample_rate, array.squeeze()))} + ) + yield AdditionalOutputs(gradio_chatbot, conversation_state) + + conversation_state.append({"role": "user", "content": text}) + + request = client.chat.completions.create( + model="Meta-Llama-3.2-3B-Instruct", + messages=conversation_state, # type: ignore + temperature=0.1, + top_p=0.1, + ) + response = {"role": "assistant", "content": request.choices[0].message.content} + + except Exception as e: + import traceback + + traceback.print_exc() + raise WebRTCError(str(e) + "\n" + traceback.format_exc()) + + conversation_state.append(response) + gradio_chatbot.append(response) + + yield AdditionalOutputs(gradio_chatbot, conversation_state) + + +chatbot = gr.Chatbot(type="messages", value=[]) +state = gr.State(value=[]) +stream = Stream( + ReplyOnStopWords( + response, # type: ignore + stop_words=["computer"], + input_sample_rate=16000, + ), + mode="send", + modality="audio", + additional_inputs=[chatbot, state], + additional_outputs=[chatbot, state], + additional_outputs_handler=lambda *a: (a[2], a[3]), + concurrency_limit=5 if get_space() else None, + time_limit=90 if get_space() else None, + rtc_configuration=get_twilio_turn_credentials() if get_space() else None, +) +app = FastAPI() +stream.mount(app) + + +class Message(BaseModel): + role: str + content: str + + +class InputData(BaseModel): + webrtc_id: str + chatbot: list[Message] + state: list[Message] + + +@app.get("/") +async def _(): + rtc_config = get_twilio_turn_credentials() if get_space() else None + html_content = (curr_dir / "index.html").read_text() + html_content = html_content.replace("__RTC_CONFIGURATION__", json.dumps(rtc_config)) + return HTMLResponse(content=html_content) + + +@app.post("/input_hook") +async def _(data: InputData): + body = data.model_dump() + stream.set_input(data.webrtc_id, body["chatbot"], body["state"]) + + +def audio_to_base64(file_path): + audio_format = "wav" + with open(file_path, "rb") as audio_file: + encoded_audio = base64.b64encode(audio_file.read()).decode("utf-8") + return f"data:audio/{audio_format};base64,{encoded_audio}" + + +@app.get("/outputs") +async def _(webrtc_id: str): + async def output_stream(): + async for output in stream.output_stream(webrtc_id): + chatbot = output.args[0] + state = output.args[1] + data = { + "message": state[-1], + "audio": audio_to_base64(chatbot[-1]["content"].value["path"]) + if chatbot[-1]["role"] == "user" + else None, + } + yield f"event: output\ndata: {json.dumps(data)}\n\n" + + return StreamingResponse(output_stream(), media_type="text/event-stream") + + +if __name__ == "__main__": + import os + + if (mode := os.getenv("MODE")) == "UI": + stream.ui.launch(server_port=7860) + elif mode == "PHONE": + raise ValueError("Phone mode not supported") + else: + import uvicorn + + uvicorn.run(app, host="0.0.0.0", port=7860) diff --git a/demo/hello_computer/index.html b/demo/hello_computer/index.html new file mode 100644 index 0000000..3e453b4 --- /dev/null +++ b/demo/hello_computer/index.html @@ -0,0 +1,486 @@ + + + + + + + Hello Computer 💻 + + + + + +
+
+ +
+
+
+
+
+
+
+
+
+
+
+ +
+
+ + + + + + \ No newline at end of file diff --git a/demo/hello_computer/requirements.txt b/demo/hello_computer/requirements.txt new file mode 100644 index 0000000..c0920dd --- /dev/null +++ b/demo/hello_computer/requirements.txt @@ -0,0 +1,4 @@ +fastrtc[stopword] +python-dotenv +openai +twilio \ No newline at end of file diff --git a/demo/llama_code_editor/README.md b/demo/llama_code_editor/README.md new file mode 100644 index 0000000..8608630 --- /dev/null +++ b/demo/llama_code_editor/README.md @@ -0,0 +1,16 @@ +--- +title: Llama Code Editor +emoji: 🦙 +colorFrom: indigo +colorTo: pink +sdk: gradio +sdk_version: 5.16.0 +app_file: app.py +pinned: false +license: mit +short_description: Create interactive HTML web pages with your voice +tags: [webrtc, websocket, gradio, secret|TWILIO_ACCOUNT_SID, secret|TWILIO_AUTH_TOKEN, +secret|SAMBANOVA_API_KEY, secret|GROQ_API_KEY] +--- + +Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference diff --git a/demo/llama_code_editor/app.py b/demo/llama_code_editor/app.py new file mode 100644 index 0000000..5e8aa1c --- /dev/null +++ b/demo/llama_code_editor/app.py @@ -0,0 +1,45 @@ +from fastapi import FastAPI +from fastapi.responses import RedirectResponse +from fastrtc import Stream +from gradio.utils import get_space + +try: + from demo.llama_code_editor.handler import ( + CodeHandler, + ) + from demo.llama_code_editor.ui import demo as ui +except (ImportError, ModuleNotFoundError): + from handler import CodeHandler + from ui import demo as ui + + +stream = Stream( + handler=CodeHandler, + modality="audio", + mode="send-receive", + concurrency_limit=10 if get_space() else None, + time_limit=90 if get_space() else None, +) + +stream.ui = ui + +app = FastAPI() + + +@app.get("/") +async def _(): + url = "/ui" if not get_space() else "https://fastrtc-llama-code-editor.hf.space/ui/" + return RedirectResponse(url) + + +if __name__ == "__main__": + import os + + if (mode := os.getenv("MODE")) == "UI": + stream.ui.launch(server_port=7860, server_name="0.0.0.0") + elif mode == "PHONE": + stream.fastphone(host="0.0.0.0", port=7860) + else: + import uvicorn + + uvicorn.run(app, host="0.0.0.0", port=7860) diff --git a/demo/llama_code_editor/assets/sandbox.html b/demo/llama_code_editor/assets/sandbox.html new file mode 100644 index 0000000..39326ac --- /dev/null +++ b/demo/llama_code_editor/assets/sandbox.html @@ -0,0 +1,37 @@ +
+
+
📦
+
+

No Application Created

+
\ No newline at end of file diff --git a/demo/llama_code_editor/assets/spinner.html b/demo/llama_code_editor/assets/spinner.html new file mode 100644 index 0000000..0621d44 --- /dev/null +++ b/demo/llama_code_editor/assets/spinner.html @@ -0,0 +1,60 @@ +
+ +
+ +
+ +
+
+ + +

Generating your application...

+ +

This may take a few moments

+ + +
\ No newline at end of file diff --git a/demo/llama_code_editor/handler.py b/demo/llama_code_editor/handler.py new file mode 100644 index 0000000..fee8dfc --- /dev/null +++ b/demo/llama_code_editor/handler.py @@ -0,0 +1,73 @@ +import base64 +import os +import re +from pathlib import Path + +import numpy as np +import openai +from dotenv import load_dotenv +from fastrtc import ( + AdditionalOutputs, + ReplyOnPause, + audio_to_bytes, +) +from groq import Groq + +load_dotenv() + +groq_client = Groq(api_key=os.environ.get("GROQ_API_KEY")) + +client = openai.OpenAI( + api_key=os.environ.get("SAMBANOVA_API_KEY"), + base_url="https://api.sambanova.ai/v1", +) + +path = Path(__file__).parent / "assets" + +spinner_html = open(path / "spinner.html").read() + + +system_prompt = "You are an AI coding assistant. Your task is to write single-file HTML applications based on a user's request. Only return the necessary code. Include all necessary imports and styles. You may also be asked to edit your original response." +user_prompt = "Please write a single-file HTML application to fulfill the following request.\nThe message:{user_message}\nCurrent code you have written:{code}" + + +def extract_html_content(text): + """ + Extract content including HTML tags. + """ + match = re.search(r".*?", text, re.DOTALL) + return match.group(0) if match else None + + +def display_in_sandbox(code): + encoded_html = base64.b64encode(code.encode("utf-8")).decode("utf-8") + data_uri = f"data:text/html;charset=utf-8;base64,{encoded_html}" + return f'' + + +def generate(user_message: tuple[int, np.ndarray], history: list[dict], code: str): + yield AdditionalOutputs(history, spinner_html) + + text = groq_client.audio.transcriptions.create( + file=("audio-file.mp3", audio_to_bytes(user_message)), + model="whisper-large-v3-turbo", + response_format="verbose_json", + ).text + + user_msg_formatted = user_prompt.format(user_message=text, code=code) + history.append({"role": "user", "content": user_msg_formatted}) + + response = client.chat.completions.create( + model="Meta-Llama-3.1-70B-Instruct", + messages=history, # type: ignore + temperature=0.1, + top_p=0.1, + ) + + output = response.choices[0].message.content + html_code = extract_html_content(output) + history.append({"role": "assistant", "content": output}) + yield AdditionalOutputs(history, html_code) + + +CodeHandler = ReplyOnPause(generate) # type: ignore diff --git a/demo/llama_code_editor/requirements.in b/demo/llama_code_editor/requirements.in new file mode 100644 index 0000000..8a5354d --- /dev/null +++ b/demo/llama_code_editor/requirements.in @@ -0,0 +1,5 @@ +fastrtc[vad] +groq +openai +python-dotenv +twilio \ No newline at end of file diff --git a/demo/llama_code_editor/requirements.txt b/demo/llama_code_editor/requirements.txt new file mode 100644 index 0000000..008acf3 --- /dev/null +++ b/demo/llama_code_editor/requirements.txt @@ -0,0 +1,295 @@ +# This file was autogenerated by uv via the following command: +# uv pip compile demo/llama_code_editor/requirements.in -o demo/llama_code_editor/requirements.txt +aiofiles==23.2.1 + # via gradio +aiohappyeyeballs==2.4.6 + # via aiohttp +aiohttp==3.11.12 + # via + # aiohttp-retry + # twilio +aiohttp-retry==2.9.1 + # via twilio +aioice==0.9.0 + # via aiortc +aiortc==1.10.1 + # via fastrtc +aiosignal==1.3.2 + # via aiohttp +annotated-types==0.7.0 + # via pydantic +anyio==4.6.2.post1 + # via + # gradio + # groq + # httpx + # openai + # starlette +attrs==25.1.0 + # via aiohttp +audioread==3.0.1 + # via librosa +av==12.3.0 + # via aiortc +certifi==2024.8.30 + # via + # httpcore + # httpx + # requests +cffi==1.17.1 + # via + # aiortc + # cryptography + # pylibsrtp + # soundfile +charset-normalizer==3.4.0 + # via requests +click==8.1.7 + # via + # typer + # uvicorn +coloredlogs==15.0.1 + # via onnxruntime +cryptography==43.0.3 + # via + # aiortc + # pyopenssl +decorator==5.1.1 + # via librosa +distro==1.9.0 + # via + # groq + # openai +dnspython==2.7.0 + # via aioice +fastapi==0.115.5 + # via gradio +fastrtc==0.0.2.post4 + # via -r demo/llama_code_editor/requirements.in +ffmpy==0.4.0 + # via gradio +filelock==3.16.1 + # via huggingface-hub +flatbuffers==24.3.25 + # via onnxruntime +frozenlist==1.5.0 + # via + # aiohttp + # aiosignal +fsspec==2024.10.0 + # via + # gradio-client + # huggingface-hub +google-crc32c==1.6.0 + # via aiortc +gradio==5.16.0 + # via fastrtc +gradio-client==1.7.0 + # via gradio +groq==0.18.0 + # via -r demo/llama_code_editor/requirements.in +h11==0.14.0 + # via + # httpcore + # uvicorn +httpcore==1.0.7 + # via httpx +httpx==0.27.2 + # via + # gradio + # gradio-client + # groq + # openai + # safehttpx +huggingface-hub==0.28.1 + # via + # gradio + # gradio-client +humanfriendly==10.0 + # via coloredlogs +idna==3.10 + # via + # anyio + # httpx + # requests + # yarl +ifaddr==0.2.0 + # via aioice +jinja2==3.1.4 + # via gradio +jiter==0.7.1 + # via openai +joblib==1.4.2 + # via + # librosa + # scikit-learn +lazy-loader==0.4 + # via librosa +librosa==0.10.2.post1 + # via fastrtc +llvmlite==0.43.0 + # via numba +markdown-it-py==3.0.0 + # via rich +markupsafe==2.1.5 + # via + # gradio + # jinja2 +mdurl==0.1.2 + # via markdown-it-py +mpmath==1.3.0 + # via sympy +msgpack==1.1.0 + # via librosa +multidict==6.1.0 + # via + # aiohttp + # yarl +numba==0.60.0 + # via librosa +numpy==2.0.2 + # via + # gradio + # librosa + # numba + # onnxruntime + # pandas + # scikit-learn + # scipy + # soxr +onnxruntime==1.20.1 + # via fastrtc +openai==1.54.4 + # via -r demo/llama_code_editor/requirements.in +orjson==3.10.11 + # via gradio +packaging==24.2 + # via + # gradio + # gradio-client + # huggingface-hub + # lazy-loader + # onnxruntime + # pooch +pandas==2.2.3 + # via gradio +pillow==11.0.0 + # via gradio +platformdirs==4.3.6 + # via pooch +pooch==1.8.2 + # via librosa +propcache==0.2.1 + # via + # aiohttp + # yarl +protobuf==5.28.3 + # via onnxruntime +pycparser==2.22 + # via cffi +pydantic==2.9.2 + # via + # fastapi + # gradio + # groq + # openai +pydantic-core==2.23.4 + # via pydantic +pydub==0.25.1 + # via gradio +pyee==12.1.1 + # via aiortc +pygments==2.18.0 + # via rich +pyjwt==2.10.1 + # via twilio +pylibsrtp==0.10.0 + # via aiortc +pyopenssl==24.2.1 + # via aiortc +python-dateutil==2.9.0.post0 + # via pandas +python-dotenv==1.0.1 + # via -r demo/llama_code_editor/requirements.in +python-multipart==0.0.20 + # via gradio +pytz==2024.2 + # via pandas +pyyaml==6.0.2 + # via + # gradio + # huggingface-hub +requests==2.32.3 + # via + # huggingface-hub + # pooch + # twilio +rich==13.9.4 + # via typer +ruff==0.9.6 + # via gradio +safehttpx==0.1.6 + # via gradio +scikit-learn==1.5.2 + # via librosa +scipy==1.14.1 + # via + # librosa + # scikit-learn +semantic-version==2.10.0 + # via gradio +shellingham==1.5.4 + # via typer +six==1.16.0 + # via python-dateutil +sniffio==1.3.1 + # via + # anyio + # groq + # httpx + # openai +soundfile==0.12.1 + # via librosa +soxr==0.5.0.post1 + # via librosa +starlette==0.41.3 + # via + # fastapi + # gradio +sympy==1.13.3 + # via onnxruntime +threadpoolctl==3.5.0 + # via scikit-learn +tomlkit==0.12.0 + # via gradio +tqdm==4.67.0 + # via + # huggingface-hub + # openai +twilio==9.4.5 + # via -r demo/llama_code_editor/requirements.in +typer==0.13.1 + # via gradio +typing-extensions==4.12.2 + # via + # fastapi + # gradio + # gradio-client + # groq + # huggingface-hub + # librosa + # openai + # pydantic + # pydantic-core + # pyee + # typer +tzdata==2024.2 + # via pandas +urllib3==2.2.3 + # via requests +uvicorn==0.32.0 + # via gradio +websockets==12.0 + # via gradio-client +yarl==1.18.3 + # via aiohttp diff --git a/demo/llama_code_editor/ui.py b/demo/llama_code_editor/ui.py new file mode 100644 index 0000000..cfe08d9 --- /dev/null +++ b/demo/llama_code_editor/ui.py @@ -0,0 +1,75 @@ +from pathlib import Path + +import gradio as gr +from dotenv import load_dotenv +from fastrtc import WebRTC, get_twilio_turn_credentials +from gradio.utils import get_space + +try: + from demo.llama_code_editor.handler import ( + CodeHandler, + display_in_sandbox, + system_prompt, + ) +except (ImportError, ModuleNotFoundError): + from handler import CodeHandler, display_in_sandbox, system_prompt + +load_dotenv() + +path = Path(__file__).parent / "assets" + +with gr.Blocks(css=".code-component {max-height: 500px !important}") as demo: + history = gr.State([{"role": "system", "content": system_prompt}]) + with gr.Row(): + with gr.Column(scale=1): + gr.HTML( + """ +

+ Llama Code Editor +

+

+ Powered by SambaNova and Gradio-WebRTC ⚡️ +

+

+ Create and edit single-file HTML applications with just your voice! +

+

+ Each conversation is limited to 90 seconds. Once the time limit is up you can rejoin the conversation. +

+ """ + ) + webrtc = WebRTC( + rtc_configuration=get_twilio_turn_credentials() + if get_space() + else None, + mode="send", + modality="audio", + ) + with gr.Column(scale=10): + with gr.Tabs(): + with gr.Tab("Sandbox"): + sandbox = gr.HTML(value=open(path / "sandbox.html").read()) + with gr.Tab("Code"): + code = gr.Code( + language="html", + max_lines=50, + interactive=False, + elem_classes="code-component", + ) + with gr.Tab("Chat"): + cb = gr.Chatbot(type="messages") + + webrtc.stream( + CodeHandler, + inputs=[webrtc, history, code], + outputs=[webrtc], + time_limit=90 if get_space() else None, + concurrency_limit=10 if get_space() else None, + ) + webrtc.on_additional_outputs( + lambda history, code: (history, code, history), outputs=[history, code, cb] + ) + code.change(display_in_sandbox, code, sandbox, queue=False) + +if __name__ == "__main__": + demo.launch() diff --git a/demo/llm_voice_chat/README.md b/demo/llm_voice_chat/README.md new file mode 100644 index 0000000..355a28b --- /dev/null +++ b/demo/llm_voice_chat/README.md @@ -0,0 +1,15 @@ +--- +title: LLM Voice Chat +emoji: 💻 +colorFrom: purple +colorTo: red +sdk: gradio +sdk_version: 5.16.0 +app_file: app.py +pinned: false +license: mit +short_description: Talk to an LLM with ElevenLabs +tags: [webrtc, websocket, gradio, secret|TWILIO_ACCOUNT_SID, secret|TWILIO_AUTH_TOKEN, secret|GROQ_API_KEY, secret|ELEVENLABS_API_KEY] +--- + +Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference \ No newline at end of file diff --git a/demo/llm_voice_chat/README_gradio.md b/demo/llm_voice_chat/README_gradio.md new file mode 100644 index 0000000..f36f353 --- /dev/null +++ b/demo/llm_voice_chat/README_gradio.md @@ -0,0 +1,15 @@ +--- +title: LLM Voice Chat (Gradio) +emoji: 💻 +colorFrom: purple +colorTo: red +sdk: gradio +sdk_version: 5.16.0 +app_file: app.py +pinned: false +license: mit +short_description: LLM Voice by ElevenLabs (Gradio) +tags: [webrtc, websocket, gradio, secret|TWILIO_ACCOUNT_SID, secret|TWILIO_AUTH_TOKEN, secret|GROQ_API_KEY, secret|ELEVENLABS_API_KEY] +--- + +Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference \ No newline at end of file diff --git a/demo/llm_voice_chat/app.py b/demo/llm_voice_chat/app.py new file mode 100644 index 0000000..19e9634 --- /dev/null +++ b/demo/llm_voice_chat/app.py @@ -0,0 +1,101 @@ +import os +import time + +import gradio as gr +import numpy as np +from dotenv import load_dotenv +from elevenlabs import ElevenLabs +from fastapi import FastAPI +from fastrtc import ( + AdditionalOutputs, + ReplyOnPause, + Stream, + WebRTCError, + get_stt_model, + get_twilio_turn_credentials, +) +from gradio.utils import get_space +from groq import Groq +from numpy.typing import NDArray + +load_dotenv() +groq_client = Groq() +tts_client = ElevenLabs(api_key=os.getenv("ELEVENLABS_API_KEY")) +stt_model = get_stt_model() + + +# See "Talk to Claude" in Cookbook for an example of how to keep +# track of the chat history. +def response( + audio: tuple[int, NDArray[np.int16 | np.float32]], + chatbot: list[dict] | None = None, +): + try: + chatbot = chatbot or [] + messages = [{"role": d["role"], "content": d["content"]} for d in chatbot] + start = time.time() + text = stt_model.stt(audio) + print("transcription", time.time() - start) + print("prompt", text) + chatbot.append({"role": "user", "content": text}) + yield AdditionalOutputs(chatbot) + messages.append({"role": "user", "content": text}) + response_text = ( + groq_client.chat.completions.create( + model="llama-3.1-8b-instant", + max_tokens=512, + messages=messages, # type: ignore + ) + .choices[0] + .message.content + ) + + chatbot.append({"role": "assistant", "content": response_text}) + + for chunk in tts_client.text_to_speech.convert_as_stream( + text=response_text, # type: ignore + voice_id="JBFqnCBsd6RMkjVDRZzb", + model_id="eleven_multilingual_v2", + output_format="pcm_24000", + ): + audio_array = np.frombuffer(chunk, dtype=np.int16).reshape(1, -1) + yield (24000, audio_array) + yield AdditionalOutputs(chatbot) + except Exception: + import traceback + + traceback.print_exc() + raise WebRTCError(traceback.format_exc()) + + +chatbot = gr.Chatbot(type="messages") +stream = Stream( + modality="audio", + mode="send-receive", + handler=ReplyOnPause(response, input_sample_rate=16000), + additional_outputs_handler=lambda a, b: b, + additional_inputs=[chatbot], + additional_outputs=[chatbot], + rtc_configuration=get_twilio_turn_credentials() if get_space() else None, + concurrency_limit=5 if get_space() else None, + time_limit=90 if get_space() else None, + ui_args={"title": "LLM Voice Chat (Powered by Groq, ElevenLabs, and WebRTC ⚡️)"}, +) + +# Mount the STREAM UI to the FastAPI app +# Because I don't want to build the UI manually +app = FastAPI() +app = gr.mount_gradio_app(app, stream.ui, path="/") + + +if __name__ == "__main__": + import os + + os.environ["GRADIO_SSR_MODE"] = "false" + + if (mode := os.getenv("MODE")) == "UI": + stream.ui.launch(server_port=7860) + elif mode == "PHONE": + stream.fastphone(host="0.0.0.0", port=7860) + else: + stream.ui.launch(server_port=7860) diff --git a/demo/llm_voice_chat/requirements.txt b/demo/llm_voice_chat/requirements.txt new file mode 100644 index 0000000..c83b692 --- /dev/null +++ b/demo/llm_voice_chat/requirements.txt @@ -0,0 +1,6 @@ +fastrtc[stopword] +python-dotenv +openai +twilio +groq +elevenlabs diff --git a/demo/object_detection/README.md b/demo/object_detection/README.md new file mode 100644 index 0000000..22f9552 --- /dev/null +++ b/demo/object_detection/README.md @@ -0,0 +1,15 @@ +--- +title: Object Detection +emoji: 📸 +colorFrom: purple +colorTo: red +sdk: gradio +sdk_version: 5.16.0 +app_file: app.py +pinned: false +license: mit +short_description: Use YOLOv10 to detect objects in real-time +tags: [webrtc, websocket, gradio, secret|TWILIO_ACCOUNT_SID, secret|TWILIO_AUTH_TOKEN] +--- + +Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference \ No newline at end of file diff --git a/demo/object_detection/app.py b/demo/object_detection/app.py new file mode 100644 index 0000000..419a766 --- /dev/null +++ b/demo/object_detection/app.py @@ -0,0 +1,83 @@ +import json +from pathlib import Path + +import cv2 +import gradio as gr +from fastapi import FastAPI +from fastapi.responses import HTMLResponse +from fastrtc import Stream, WebRTCError, get_twilio_turn_credentials +from gradio.utils import get_space +from huggingface_hub import hf_hub_download +from pydantic import BaseModel, Field + +try: + from demo.object_detection.inference import YOLOv10 +except (ImportError, ModuleNotFoundError): + from inference import YOLOv10 + + +cur_dir = Path(__file__).parent + +model_file = hf_hub_download( + repo_id="onnx-community/yolov10n", filename="onnx/model.onnx" +) + +model = YOLOv10(model_file) + + +def detection(image, conf_threshold=0.3): + try: + image = cv2.resize(image, (model.input_width, model.input_height)) + print("conf_threshold", conf_threshold) + new_image = model.detect_objects(image, conf_threshold) + return cv2.resize(new_image, (500, 500)) + except Exception as e: + import traceback + + traceback.print_exc() + raise WebRTCError(str(e)) + + +stream = Stream( + handler=detection, + modality="video", + mode="send-receive", + additional_inputs=[gr.Slider(minimum=0, maximum=1, step=0.01, value=0.3)], + rtc_configuration=get_twilio_turn_credentials() if get_space() else None, + concurrency_limit=2 if get_space() else None, +) + +app = FastAPI() + +stream.mount(app) + + +@app.get("/") +async def _(): + rtc_config = get_twilio_turn_credentials() if get_space() else None + html_content = open(cur_dir / "index.html").read() + html_content = html_content.replace("__RTC_CONFIGURATION__", json.dumps(rtc_config)) + return HTMLResponse(content=html_content) + + +class InputData(BaseModel): + webrtc_id: str + conf_threshold: float = Field(ge=0, le=1) + + +@app.post("/input_hook") +async def _(data: InputData): + stream.set_input(data.webrtc_id, data.conf_threshold) + + +if __name__ == "__main__": + import os + + if (mode := os.getenv("MODE")) == "UI": + stream.ui.launch(server_port=7860) + elif mode == "PHONE": + stream.fastphone(host="0.0.0.0", port=7860) + else: + import uvicorn + + uvicorn.run(app, host="0.0.0.0", port=7860) diff --git a/demo/object_detection/index.html b/demo/object_detection/index.html new file mode 100644 index 0000000..daad1f5 --- /dev/null +++ b/demo/object_detection/index.html @@ -0,0 +1,340 @@ + + + + + + + Object Detection + + + + + +
+
+

Real-time Object Detection

+

Using YOLOv10 to detect objects in your webcam feed

+
+ +
+
+
+ + +
+ +
+
+ + + + + \ No newline at end of file diff --git a/demo/inference.py b/demo/object_detection/inference.py similarity index 94% rename from demo/inference.py rename to demo/object_detection/inference.py index 7b3fdff..fd2e424 100644 --- a/demo/inference.py +++ b/demo/object_detection/inference.py @@ -3,7 +3,11 @@ import time import cv2 import numpy as np import onnxruntime -from utils import draw_detections + +try: + from demo.object_detection.utils import draw_detections +except (ImportError, ModuleNotFoundError): + from utils import draw_detections class YOLOv10: @@ -51,7 +55,7 @@ class YOLOv10: self.output_names, {self.input_names[0]: input_tensor} ) - print(f"Inference time: {(time.perf_counter() - start)*1000:.2f} ms") + print(f"Inference time: {(time.perf_counter() - start) * 1000:.2f} ms") ( boxes, scores, @@ -71,7 +75,7 @@ class YOLOv10: return [], [], [] # Get the class with the highest confidence - class_ids = np.argmax(predictions[:, 4:], axis=1) + class_ids = predictions[:, 5].astype(int) # Get bounding boxes for each object boxes = self.extract_boxes(predictions) diff --git a/demo/object_detection/requirements.txt b/demo/object_detection/requirements.txt new file mode 100644 index 0000000..b3fbe78 --- /dev/null +++ b/demo/object_detection/requirements.txt @@ -0,0 +1,4 @@ +fastrtc +opencv-python +twilio +onnxruntime-gpu \ No newline at end of file diff --git a/demo/utils.py b/demo/object_detection/utils.py similarity index 97% rename from demo/utils.py rename to demo/object_detection/utils.py index 04dadeb..896f6c7 100644 --- a/demo/utils.py +++ b/demo/object_detection/utils.py @@ -170,11 +170,11 @@ def draw_detections(image, boxes, scores, class_ids, mask_alpha=0.3): for class_id, box, score in zip(class_ids, boxes, scores): color = colors[class_id] - draw_box(det_img, box, color) + draw_box(det_img, box, color) # type: ignore label = class_names[class_id] caption = f"{label} {int(score * 100)}%" - draw_text(det_img, caption, box, color, font_size, text_thickness) + draw_text(det_img, caption, box, color, font_size, text_thickness) # type: ignore return det_img @@ -232,6 +232,6 @@ def draw_masks( x1, y1, x2, y2 = box.astype(int) # Draw fill rectangle in mask image - cv2.rectangle(mask_img, (x1, y1), (x2, y2), color, -1) + cv2.rectangle(mask_img, (x1, y1), (x2, y2), color, -1) # type: ignore return cv2.addWeighted(mask_img, mask_alpha, image, 1 - mask_alpha, 0) diff --git a/demo/old_app.py b/demo/old_app.py deleted file mode 100644 index fe697b0..0000000 --- a/demo/old_app.py +++ /dev/null @@ -1,74 +0,0 @@ -import os - -import cv2 -import gradio as gr -from gradio_webrtc import WebRTC -from huggingface_hub import hf_hub_download -from inference import YOLOv10 -from twilio.rest import Client - -model_file = hf_hub_download( - repo_id="onnx-community/yolov10n", filename="onnx/model.onnx" -) - -model = YOLOv10(model_file) - -account_sid = os.environ.get("TWILIO_ACCOUNT_SID") -auth_token = os.environ.get("TWILIO_AUTH_TOKEN") - -if account_sid and auth_token: - client = Client(account_sid, auth_token) - - token = client.tokens.create() - - rtc_configuration = { - "iceServers": token.ice_servers, - "iceTransportPolicy": "relay", - } -else: - rtc_configuration = None - - -def detection(frame, conf_threshold=0.3): - frame = cv2.flip(frame, 0) - return cv2.cvtColor(frame, cv2.COLOR_BGR2RGB) - - -css = """.my-group {max-width: 600px !important; max-height: 600 !important;} - .my-column {display: flex !important; justify-content: center !important; align-items: center !important};""" - - -with gr.Blocks(css=css) as demo: - gr.HTML( - """ -

- YOLOv10 Webcam Stream (Powered by WebRTC ⚡️) -

- """ - ) - gr.HTML( - """ -

- arXiv | github -

- """ - ) - with gr.Column(elem_classes=["my-column"]): - with gr.Group(elem_classes=["my-group"]): - image = WebRTC(label="Stream", rtc_configuration=rtc_configuration) - conf_threshold = gr.Slider( - label="Confidence Threshold", - minimum=0.0, - maximum=1.0, - step=0.05, - value=0.30, - ) - number = gr.Number() - - image.stream( - fn=detection, inputs=[image, conf_threshold], outputs=[image], time_limit=10 - ) - image.on_additional_outputs(lambda n: n, outputs=[number]) - -if __name__ == "__main__": - demo.launch() diff --git a/demo/phonic_chat/README.md b/demo/phonic_chat/README.md new file mode 100644 index 0000000..86e347b --- /dev/null +++ b/demo/phonic_chat/README.md @@ -0,0 +1,16 @@ +--- +title: Phonic AI Chat +emoji: 🎙️ +colorFrom: purple +colorTo: red +sdk: gradio +sdk_version: 5.16.0 +app_file: app.py +pinned: false +license: mit +short_description: Talk to Phonic AI's speech-to-speech model +tags: [webrtc, websocket, gradio, secret|TWILIO_ACCOUNT_SID, secret|TWILIO_AUTH_TOKEN, secret|PHONIC_API_KEY] +python_version: 3.11 +--- + +Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference \ No newline at end of file diff --git a/demo/phonic_chat/app.py b/demo/phonic_chat/app.py new file mode 100644 index 0000000..6404b1d --- /dev/null +++ b/demo/phonic_chat/app.py @@ -0,0 +1,134 @@ +import subprocess + +subprocess.run(["pip", "install", "fastrtc==0.0.3.post7"]) + +import asyncio +import base64 +import os + +import gradio as gr +from gradio.utils import get_space +import numpy as np +from dotenv import load_dotenv +from fastrtc import ( + AdditionalOutputs, + AsyncStreamHandler, + Stream, + get_twilio_turn_credentials, + WebRTCError, + audio_to_float32, +) +from fastapi import FastAPI +from phonic.client import PhonicSTSClient, get_voices + +load_dotenv() + +STS_URI = "wss://api.phonic.co/v1/sts/ws" +API_KEY = os.environ["PHONIC_API_KEY"] +SAMPLE_RATE = 44_100 +voices = get_voices(API_KEY) +voice_ids = [voice["id"] for voice in voices] + + +class PhonicHandler(AsyncStreamHandler): + def __init__(self): + super().__init__(input_sample_rate=SAMPLE_RATE, output_sample_rate=SAMPLE_RATE) + self.output_queue = asyncio.Queue() + self.client = None + + def copy(self) -> AsyncStreamHandler: + return PhonicHandler() + + async def start_up(self): + await self.wait_for_args() + voice_id = self.latest_args[1] + try: + async with PhonicSTSClient(STS_URI, API_KEY) as client: + self.client = client + sts_stream = client.sts( # type: ignore + input_format="pcm_44100", + output_format="pcm_44100", + system_prompt="You are a helpful voice assistant. Respond conversationally.", + # welcome_message="Hello! I'm your voice assistant. How can I help you today?", + voice_id=voice_id, + ) + async for message in sts_stream: + message_type = message.get("type") + if message_type == "audio_chunk": + audio_b64 = message["audio"] + audio_bytes = base64.b64decode(audio_b64) + await self.output_queue.put( + (SAMPLE_RATE, np.frombuffer(audio_bytes, dtype=np.int16)) + ) + if text := message.get("text"): + msg = {"role": "assistant", "content": text} + await self.output_queue.put(AdditionalOutputs(msg)) + elif message_type == "input_text": + msg = {"role": "user", "content": message["text"]} + await self.output_queue.put(AdditionalOutputs(msg)) + except Exception as e: + raise WebRTCError(f"Error starting up: {e}") + + async def emit(self): + try: + return await self.output_queue.get() + except Exception as e: + raise WebRTCError(f"Error emitting: {e}") + + async def receive(self, frame: tuple[int, np.ndarray]) -> None: + try: + if not self.client: + return + audio_float32 = audio_to_float32(frame) + await self.client.send_audio(audio_float32) # type: ignore + except Exception as e: + raise WebRTCError(f"Error sending audio: {e}") + + async def shutdown(self): + if self.client: + await self.client._websocket.close() + return super().shutdown() + + +def add_to_chatbot(state, chatbot, message): + state.append(message) + return state, gr.skip() + + +state = gr.State(value=[]) +chatbot = gr.Chatbot(type="messages", value=[]) +stream = Stream( + handler=PhonicHandler(), + mode="send-receive", + modality="audio", + additional_inputs=[ + gr.Dropdown( + choices=voice_ids, + value="katherine", + label="Voice", + info="Select a voice from the dropdown", + ) + ], + additional_outputs=[state, chatbot], + additional_outputs_handler=add_to_chatbot, + ui_args={ + "title": "Phonic Chat (Powered by FastRTC ⚡️)", + }, + rtc_configuration=get_twilio_turn_credentials() if get_space() else None, + concurrency_limit=5 if get_space() else None, + time_limit=90 if get_space() else None, +) + +with stream.ui: + state.change(lambda s: s, inputs=state, outputs=chatbot) + +app = FastAPI() +stream.mount(app) + +if __name__ == "__main__": + if (mode := os.getenv("MODE")) == "UI": + stream.ui.launch(server_port=7860) + elif mode == "PHONE": + stream.fastphone(host="0.0.0.0", port=7860) + else: + stream.ui.launch(server_port=7860) diff --git a/demo/phonic_chat/requirements.txt b/demo/phonic_chat/requirements.txt new file mode 100644 index 0000000..14146ab --- /dev/null +++ b/demo/phonic_chat/requirements.txt @@ -0,0 +1,74 @@ +# This file was autogenerated by uv via the following command: +# uv pip compile requirements.in -o requirements.txt +aiohappyeyeballs==2.4.6 + # via aiohttp +aiohttp==3.11.12 + # via + # aiohttp-retry + # twilio +aiohttp-retry==2.9.1 + # via twilio +aiosignal==1.3.2 + # via aiohttp +attrs==25.1.0 + # via aiohttp +certifi==2025.1.31 + # via requests +cffi==1.17.1 + # via sounddevice +charset-normalizer==3.4.1 + # via requests +fastrtc==0.0.1 + # via -r requirements.in +frozenlist==1.5.0 + # via + # aiohttp + # aiosignal +idna==3.10 + # via + # requests + # yarl +isort==6.0.0 + # via phonic-python +loguru==0.7.3 + # via phonic-python +multidict==6.1.0 + # via + # aiohttp + # yarl +numpy==2.2.3 + # via + # phonic-python + # scipy +phonic-python==0.1.3 + # via -r requirements.in +propcache==0.3.0 + # via + # aiohttp + # yarl +pycparser==2.22 + # via cffi +pyjwt==2.10.1 + # via twilio +python-dotenv==1.0.1 + # via + # -r requirements.in + # phonic-python +requests==2.32.3 + # via + # phonic-python + # twilio +scipy==1.15.2 + # via phonic-python +sounddevice==0.5.1 + # via phonic-python +twilio==9.4.6 + # via -r requirements.in +typing-extensions==4.12.2 + # via phonic-python +urllib3==2.3.0 + # via requests +websockets==15.0 + # via phonic-python +yarl==1.18.3 + # via aiohttp diff --git a/demo/requirements.txt b/demo/requirements.txt deleted file mode 100644 index b831040..0000000 --- a/demo/requirements.txt +++ /dev/null @@ -1,6 +0,0 @@ -safetensors==0.4.3 -opencv-python -twilio -https://huggingface.co/datasets/freddyaboulton/bucket/resolve/main/gradio-5.0.0b3-py3-none-any.whl -https://huggingface.co/datasets/freddyaboulton/bucket/resolve/main/gradio_webrtc-0.0.1-py3-none-any.whl -onnxruntime-gpu \ No newline at end of file diff --git a/demo/space.py b/demo/space.py deleted file mode 100644 index 7b7bc06..0000000 --- a/demo/space.py +++ /dev/null @@ -1,321 +0,0 @@ -import os - -import gradio as gr - -_docs = { - "WebRTC": { - "description": "Stream audio/video with WebRTC", - "members": { - "__init__": { - "rtc_configuration": { - "type": "dict[str, Any] | None", - "default": "None", - "description": "The configration dictionary to pass to the RTCPeerConnection constructor. If None, the default configuration is used.", - }, - "height": { - "type": "int | str | None", - "default": "None", - "description": "The height of the component, specified in pixels if a number is passed, or in CSS units if a string is passed. This has no effect on the preprocessed video file, but will affect the displayed video.", - }, - "width": { - "type": "int | str | None", - "default": "None", - "description": "The width of the component, specified in pixels if a number is passed, or in CSS units if a string is passed. This has no effect on the preprocessed video file, but will affect the displayed video.", - }, - "label": { - "type": "str | None", - "default": "None", - "description": "the label for this component. Appears above the component and is also used as the header if there are a table of examples for this component. If None and used in a `gr.Interface`, the label will be the name of the parameter this component is assigned to.", - }, - "show_label": { - "type": "bool | None", - "default": "None", - "description": "if True, will display label.", - }, - "container": { - "type": "bool", - "default": "True", - "description": "if True, will place the component in a container - providing some extra padding around the border.", - }, - "scale": { - "type": "int | None", - "default": "None", - "description": "relative size compared to adjacent Components. For example if Components A and B are in a Row, and A has scale=2, and B has scale=1, A will be twice as wide as B. Should be an integer. scale applies in Rows, and to top-level Components in Blocks where fill_height=True.", - }, - "min_width": { - "type": "int", - "default": "160", - "description": "minimum pixel width, will wrap if not sufficient screen space to satisfy this value. If a certain scale value results in this Component being narrower than min_width, the min_width parameter will be respected first.", - }, - "interactive": { - "type": "bool | None", - "default": "None", - "description": "if True, will allow users to upload a video; if False, can only be used to display videos. If not provided, this is inferred based on whether the component is used as an input or output.", - }, - "visible": { - "type": "bool", - "default": "True", - "description": "if False, component will be hidden.", - }, - "elem_id": { - "type": "str | None", - "default": "None", - "description": "an optional string that is assigned as the id of this component in the HTML DOM. Can be used for targeting CSS styles.", - }, - "elem_classes": { - "type": "list[str] | str | None", - "default": "None", - "description": "an optional list of strings that are assigned as the classes of this component in the HTML DOM. Can be used for targeting CSS styles.", - }, - "render": { - "type": "bool", - "default": "True", - "description": "if False, component will not render be rendered in the Blocks context. Should be used if the intention is to assign event listeners now but render the component later.", - }, - "key": { - "type": "int | str | None", - "default": "None", - "description": "if assigned, will be used to assume identity across a re-render. Components that have the same key across a re-render will have their value preserved.", - }, - "mirror_webcam": { - "type": "bool", - "default": "True", - "description": "if True webcam will be mirrored. Default is True.", - }, - }, - "events": {"tick": {"type": None, "default": None, "description": ""}}, - }, - "__meta__": {"additional_interfaces": {}, "user_fn_refs": {"WebRTC": []}}, - } -} - - -abs_path = os.path.join(os.path.dirname(__file__), "css.css") - -with gr.Blocks( - css_paths=abs_path, - theme=gr.themes.Default( - font_mono=[ - gr.themes.GoogleFont("Inconsolata"), - "monospace", - ], - ), -) as demo: - gr.Markdown( - """ -

Gradio WebRTC ⚡️

- -
-Static Badge -Static Badge -
-""", - elem_classes=["md-custom"], - header_links=True, - ) - gr.Markdown( - """ -## Installation - -```bash -pip install gradio_webrtc -``` - -## Examples: -1. [Object Detection from Webcam with YOLOv10](https://huggingface.co/spaces/freddyaboulton/webrtc-yolov10n) 📷 -2. [Streaming Object Detection from Video with RT-DETR](https://huggingface.co/spaces/freddyaboulton/rt-detr-object-detection-webrtc) 🎥 -3. [Text-to-Speech](https://huggingface.co/spaces/freddyaboulton/parler-tts-streaming-webrtc) 🗣️ - -## Usage - -The WebRTC component supports the following three use cases: -1. Streaming video from the user webcam to the server and back -2. Streaming Video from the server to the client -3. Streaming Audio from the server to the client - -Streaming Audio from client to the server and back (conversational AI) is not supported yet. - - -## Streaming Video from the User Webcam to the Server and Back - -```python -import gradio as gr -from gradio_webrtc import WebRTC - - -def detection(image, conf_threshold=0.3): - ... your detection code here ... - - -with gr.Blocks() as demo: - image = WebRTC(label="Stream", mode="send-receive", modality="video") - conf_threshold = gr.Slider( - label="Confidence Threshold", - minimum=0.0, - maximum=1.0, - step=0.05, - value=0.30, - ) - image.stream( - fn=detection, - inputs=[image, conf_threshold], - outputs=[image], time_limit=10 - ) - -if __name__ == "__main__": - demo.launch() - -``` -* Set the `mode` parameter to `send-receive` and `modality` to "video". -* The `stream` event's `fn` parameter is a function that receives the next frame from the webcam -as a **numpy array** and returns the processed frame also as a **numpy array**. -* Numpy arrays are in (height, width, 3) format where the color channels are in RGB format. -* The `inputs` parameter should be a list where the first element is the WebRTC component. The only output allowed is the WebRTC component. -* The `time_limit` parameter is the maximum time in seconds the video stream will run. If the time limit is reached, the video stream will stop. - -## Streaming Video from the User Webcam to the Server and Back - -```python -import gradio as gr -from gradio_webrtc import WebRTC -import cv2 - -def generation(): - url = "https://download.tsi.telecom-paristech.fr/gpac/dataset/dash/uhd/mux_sources/hevcds_720p30_2M.mp4" - cap = cv2.VideoCapture(url) - iterating = True - while iterating: - iterating, frame = cap.read() - yield frame - -with gr.Blocks() as demo: - output_video = WebRTC(label="Video Stream", mode="receive", modality="video") - button = gr.Button("Start", variant="primary") - output_video.stream( - fn=generation, inputs=None, outputs=[output_video], - trigger=button.click - ) - -if __name__ == "__main__": - demo.launch() -``` - -* Set the "mode" parameter to "receive" and "modality" to "video". -* The `stream` event's `fn` parameter is a generator function that yields the next frame from the video as a **numpy array**. -* The only output allowed is the WebRTC component. -* The `trigger` parameter the gradio event that will trigger the webrtc connection. In this case, the button click event. - -## Streaming Audio from the Server to the Client - -```python -import gradio as gr -from pydub import AudioSegment - -def generation(num_steps): - for _ in range(num_steps): - segment = AudioSegment.from_file("/Users/freddy/sources/gradio/demo/audio_debugger/cantina.wav") - yield (segment.frame_rate, np.array(segment.get_array_of_samples()).reshape(1, -1)) - -with gr.Blocks() as demo: - audio = WebRTC(label="Stream", mode="receive", modality="audio") - num_steps = gr.Slider( - label="Number of Steps", - minimum=1, - maximum=10, - step=1, - value=5, - ) - button = gr.Button("Generate") - - audio.stream( - fn=generation, inputs=[num_steps], outputs=[audio], - trigger=button.click - ) -``` - -* Set the "mode" parameter to "receive" and "modality" to "audio". -* The `stream` event's `fn` parameter is a generator function that yields the next audio segment as a tuple of (frame_rate, audio_samples). -* The numpy array should be of shape (1, num_samples). -* The `outputs` parameter should be a list with the WebRTC component as the only element. - -## Deployment - -When deploying in a cloud environment (like Hugging Face Spaces, EC2, etc), you need to set up a TURN server to relay the WebRTC traffic. -The easiest way to do this is to use a service like Twilio. - -```python -from twilio.rest import Client -import os - -account_sid = os.environ.get("TWILIO_ACCOUNT_SID") -auth_token = os.environ.get("TWILIO_AUTH_TOKEN") - -client = Client(account_sid, auth_token) - -token = client.tokens.create() - -rtc_configuration = { - "iceServers": token.ice_servers, - "iceTransportPolicy": "relay", -} - -with gr.Blocks() as demo: - ... - rtc = WebRTC(rtc_configuration=rtc_configuration, ...) - ... -``` -""", - elem_classes=["md-custom"], - header_links=True, - ) - - gr.Markdown( - """ -## -""", - elem_classes=["md-custom"], - header_links=True, - ) - - gr.ParamViewer(value=_docs["WebRTC"]["members"]["__init__"], linkify=[]) - - demo.load( - None, - js=r"""function() { - const refs = {}; - const user_fn_refs = { - WebRTC: [], }; - requestAnimationFrame(() => { - - Object.entries(user_fn_refs).forEach(([key, refs]) => { - if (refs.length > 0) { - const el = document.querySelector(`.${key}-user-fn`); - if (!el) return; - refs.forEach(ref => { - el.innerHTML = el.innerHTML.replace( - new RegExp("\\b"+ref+"\\b", "g"), - `${ref}` - ); - }) - } - }) - - Object.entries(refs).forEach(([key, refs]) => { - if (refs.length > 0) { - const el = document.querySelector(`.${key}`); - if (!el) return; - refs.forEach(ref => { - el.innerHTML = el.innerHTML.replace( - new RegExp("\\b"+ref+"\\b", "g"), - `${ref}` - ); - }) - } - }) - }) -} - -""", - ) - -demo.launch() diff --git a/demo/stream_whisper.py b/demo/stream_whisper.py deleted file mode 100644 index 0da2614..0000000 --- a/demo/stream_whisper.py +++ /dev/null @@ -1,53 +0,0 @@ -import tempfile - -import gradio as gr -import numpy as np -from gradio_webrtc import AdditionalOutputs, ReplyOnPause, WebRTC -from openai import OpenAI -from pydub import AudioSegment - -from dotenv import load_dotenv - -load_dotenv() - - -client = OpenAI() - - -def transcribe(audio: tuple[int, np.ndarray], transcript: list[dict]): - print("audio", audio) - segment = AudioSegment( - audio[1].tobytes(), - frame_rate=audio[0], - sample_width=audio[1].dtype.itemsize, - channels=1, - ) - - transcript.append({"role": "user", "content": gr.Audio((audio[0], audio[1].squeeze()))}) - - with tempfile.NamedTemporaryFile(suffix=".mp3") as temp_audio: - segment.export(temp_audio.name, format="mp3") - next_chunk = client.audio.transcriptions.create( - model="whisper-1", file=open(temp_audio.name, "rb") - ).text - transcript.append({"role": "assistant", "content": next_chunk}) - yield AdditionalOutputs(transcript) - - -with gr.Blocks() as demo: - with gr.Row(): - with gr.Column(): - audio = WebRTC( - label="Stream", - mode="send", - modality="audio", - ) - with gr.Column(): - transcript = gr.Chatbot(label="transcript", type="messages") - - audio.stream(ReplyOnPause(transcribe), inputs=[audio, transcript], outputs=[audio], - time_limit=30) - audio.on_additional_outputs(lambda s: s, outputs=transcript) - -if __name__ == "__main__": - demo.launch() diff --git a/demo/talk_to_claude/README.md b/demo/talk_to_claude/README.md new file mode 100644 index 0000000..195950b --- /dev/null +++ b/demo/talk_to_claude/README.md @@ -0,0 +1,15 @@ +--- +title: Talk to Claude +emoji: 👨‍🦰 +colorFrom: purple +colorTo: red +sdk: gradio +sdk_version: 5.16.0 +app_file: app.py +pinned: false +license: mit +short_description: Talk to Anthropic's Claude +tags: [webrtc, websocket, gradio, secret|TWILIO_ACCOUNT_SID, secret|TWILIO_AUTH_TOKEN, secret|GROQ_API_KEY, secret|ANTHROPIC_API_KEY, secret|ELEVENLABS_API_KEY] +--- + +Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference \ No newline at end of file diff --git a/demo/talk_to_claude/app.py b/demo/talk_to_claude/app.py new file mode 100644 index 0000000..13abde0 --- /dev/null +++ b/demo/talk_to_claude/app.py @@ -0,0 +1,140 @@ +import json +import os +import time +from pathlib import Path + +import anthropic +import gradio as gr +import numpy as np +from dotenv import load_dotenv +from elevenlabs import ElevenLabs +from fastapi import FastAPI +from fastapi.responses import HTMLResponse, StreamingResponse +from fastrtc import ( + AdditionalOutputs, + ReplyOnPause, + Stream, + WebRTCError, + get_tts_model, + get_twilio_turn_credentials, +) +from fastrtc.utils import audio_to_bytes +from gradio.utils import get_space +from groq import Groq +from pydantic import BaseModel + +load_dotenv() + +groq_client = Groq() +claude_client = anthropic.Anthropic() +tts_client = ElevenLabs(api_key=os.environ["ELEVENLABS_API_KEY"]) + +curr_dir = Path(__file__).parent + +tts_model = get_tts_model() + + +def response( + audio: tuple[int, np.ndarray], + chatbot: list[dict] | None = None, +): + try: + chatbot = chatbot or [] + messages = [{"role": d["role"], "content": d["content"]} for d in chatbot] + prompt = groq_client.audio.transcriptions.create( + file=("audio-file.mp3", audio_to_bytes(audio)), + model="whisper-large-v3-turbo", + response_format="verbose_json", + ).text + + print("prompt", prompt) + chatbot.append({"role": "user", "content": prompt}) + yield AdditionalOutputs(chatbot) + messages.append({"role": "user", "content": prompt}) + response = claude_client.messages.create( + model="claude-3-5-haiku-20241022", + max_tokens=512, + messages=messages, # type: ignore + ) + response_text = " ".join( + block.text # type: ignore + for block in response.content + if getattr(block, "type", None) == "text" + ) + chatbot.append({"role": "assistant", "content": response_text}) + + start = time.time() + + print("starting tts", start) + for i, chunk in enumerate(tts_model.stream_tts_sync(response_text)): + print("chunk", i, time.time() - start) + yield chunk + print("finished tts", time.time() - start) + yield AdditionalOutputs(chatbot) + except Exception as e: + raise WebRTCError(str(e)) + + +chatbot = gr.Chatbot(type="messages") +stream = Stream( + modality="audio", + mode="send-receive", + handler=ReplyOnPause(response), + additional_outputs_handler=lambda a, b: b, + additional_inputs=[chatbot], + additional_outputs=[chatbot], + rtc_configuration=get_twilio_turn_credentials() if get_space() else None, + concurrency_limit=5 if get_space() else None, + time_limit=90 if get_space() else None, +) + + +class Message(BaseModel): + role: str + content: str + + +class InputData(BaseModel): + webrtc_id: str + chatbot: list[Message] + + +app = FastAPI() +stream.mount(app) + + +@app.get("/") +async def _(): + rtc_config = get_twilio_turn_credentials() if get_space() else None + html_content = (curr_dir / "index.html").read_text() + html_content = html_content.replace("__RTC_CONFIGURATION__", json.dumps(rtc_config)) + return HTMLResponse(content=html_content, status_code=200) + + +@app.post("/input_hook") +async def _(body: InputData): + stream.set_input(body.webrtc_id, body.model_dump()["chatbot"]) + return {"status": "ok"} + + +@app.get("/outputs") +def _(webrtc_id: str): + async def output_stream(): + async for output in stream.output_stream(webrtc_id): + chatbot = output.args[0] + yield f"event: output\ndata: {json.dumps(chatbot[-1])}\n\n" + + return StreamingResponse(output_stream(), media_type="text/event-stream") + + +if __name__ == "__main__": + import os + + if (mode := os.getenv("MODE")) == "UI": + stream.ui.launch(server_port=7860, server_name="0.0.0.0") + elif mode == "PHONE": + stream.fastphone(host="0.0.0.0", port=7860) + else: + import uvicorn + + uvicorn.run(app, host="0.0.0.0", port=7860) diff --git a/demo/talk_to_claude/index.html b/demo/talk_to_claude/index.html new file mode 100644 index 0000000..c76e74b --- /dev/null +++ b/demo/talk_to_claude/index.html @@ -0,0 +1,546 @@ + + + + + + + RetroChat Audio + + + + + +
+
+
+
+ +
+
+
+
+
+
+
+
+
+
+
+ +
+
+ +
+
+ + + + + + \ No newline at end of file diff --git a/demo/talk_to_claude/requirements.txt b/demo/talk_to_claude/requirements.txt new file mode 100644 index 0000000..3912bed --- /dev/null +++ b/demo/talk_to_claude/requirements.txt @@ -0,0 +1,6 @@ +fastrtc[vad, tts] +elevenlabs +groq +anthropic +twilio +python-dotenv diff --git a/demo/talk_to_gemini/README.md b/demo/talk_to_gemini/README.md new file mode 100644 index 0000000..b6fbbcc --- /dev/null +++ b/demo/talk_to_gemini/README.md @@ -0,0 +1,15 @@ +--- +title: Talk to Gemini +emoji: ♊️ +colorFrom: purple +colorTo: red +sdk: gradio +sdk_version: 5.16.0 +app_file: app.py +pinned: false +license: mit +short_description: Talk to Gemini using Google's multimodal API +tags: [webrtc, websocket, gradio, secret|TWILIO_ACCOUNT_SID, secret|TWILIO_AUTH_TOKEN, secret|GEMINI_API_KEY] +--- + +Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference \ No newline at end of file diff --git a/demo/talk_to_gemini/README_gradio.md b/demo/talk_to_gemini/README_gradio.md new file mode 100644 index 0000000..a224a1a --- /dev/null +++ b/demo/talk_to_gemini/README_gradio.md @@ -0,0 +1,15 @@ +--- +title: Talk to Gemini (Gradio UI) +emoji: ♊️ +colorFrom: purple +colorTo: red +sdk: gradio +sdk_version: 5.16.0 +app_file: app.py +pinned: false +license: mit +short_description: Talk to Gemini (Gradio UI) +tags: [webrtc, websocket, gradio, secret|TWILIO_ACCOUNT_SID, secret|TWILIO_AUTH_TOKEN, secret|GEMINI_API_KEY] +--- + +Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference \ No newline at end of file diff --git a/demo/talk_to_gemini/app.py b/demo/talk_to_gemini/app.py new file mode 100644 index 0000000..23e1da7 --- /dev/null +++ b/demo/talk_to_gemini/app.py @@ -0,0 +1,187 @@ +import asyncio +import base64 +import json +import os +import pathlib +from typing import AsyncGenerator, Literal + +import gradio as gr +import numpy as np +from dotenv import load_dotenv +from fastapi import FastAPI +from fastapi.responses import HTMLResponse +from fastrtc import ( + AsyncStreamHandler, + Stream, + WebRTCError, + get_twilio_turn_credentials, +) +from google import genai +from google.genai.types import ( + LiveConnectConfig, + PrebuiltVoiceConfig, + SpeechConfig, + VoiceConfig, +) +from gradio.utils import get_space +from pydantic import BaseModel + +current_dir = pathlib.Path(__file__).parent + +load_dotenv() + + +def encode_audio(data: np.ndarray) -> str: + """Encode Audio data to send to the server""" + return base64.b64encode(data.tobytes()).decode("UTF-8") + + +class GeminiHandler(AsyncStreamHandler): + """Handler for the Gemini API""" + + def __init__( + self, + expected_layout: Literal["mono"] = "mono", + output_sample_rate: int = 24000, + output_frame_size: int = 480, + ) -> None: + super().__init__( + expected_layout, + output_sample_rate, + output_frame_size, + input_sample_rate=16000, + ) + self.input_queue: asyncio.Queue = asyncio.Queue() + self.output_queue: asyncio.Queue = asyncio.Queue() + self.quit: asyncio.Event = asyncio.Event() + + def copy(self) -> "GeminiHandler": + return GeminiHandler( + expected_layout="mono", + output_sample_rate=self.output_sample_rate, + output_frame_size=self.output_frame_size, + ) + + async def start_up(self): + if not self.phone_mode: + await self.wait_for_args() + api_key, voice_name = self.latest_args[1:] + else: + api_key, voice_name = None, "Puck" + try: + client = genai.Client( + api_key=api_key or os.getenv("GEMINI_API_KEY"), + http_options={"api_version": "v1alpha"}, + ) + except Exception as e: + raise WebRTCError(str(e)) + config = LiveConnectConfig( + response_modalities=["AUDIO"], # type: ignore + speech_config=SpeechConfig( + voice_config=VoiceConfig( + prebuilt_voice_config=PrebuiltVoiceConfig( + voice_name=voice_name, + ) + ) + ), + ) + try: + async with client.aio.live.connect( + model="gemini-2.0-flash-exp", config=config + ) as session: + async for audio in session.start_stream( + stream=self.stream(), mime_type="audio/pcm" + ): + if audio.data: + array = np.frombuffer(audio.data, dtype=np.int16) + self.output_queue.put_nowait(array) + except Exception as e: + raise WebRTCError(str(e)) + + async def stream(self) -> AsyncGenerator[bytes, None]: + while not self.quit.is_set(): + try: + audio = await asyncio.wait_for(self.input_queue.get(), 0.1) + yield audio + except (asyncio.TimeoutError, TimeoutError): + pass + + async def receive(self, frame: tuple[int, np.ndarray]) -> None: + _, array = frame + array = array.squeeze() + audio_message = encode_audio(array) + self.input_queue.put_nowait(audio_message) + + async def emit(self) -> tuple[int, np.ndarray]: + array = await self.output_queue.get() + return (self.output_sample_rate, array) + + def shutdown(self) -> None: + self.quit.set() + self.args_set.clear() + + +stream = Stream( + modality="audio", + mode="send-receive", + handler=GeminiHandler(), + rtc_configuration=get_twilio_turn_credentials() if get_space() else None, + concurrency_limit=5 if get_space() else None, + time_limit=90 if get_space() else None, + additional_inputs=[ + gr.Textbox( + label="API Key", + type="password", + value=os.getenv("GEMINI_API_KEY") if not get_space() else "", + ), + gr.Dropdown( + label="Voice", + choices=[ + "Puck", + "Charon", + "Kore", + "Fenrir", + "Aoede", + ], + value="Puck", + ), + ], +) + + +class InputData(BaseModel): + webrtc_id: str + voice_name: str + api_key: str + + +app = FastAPI() + +stream.mount(app) + + +@app.post("/input_hook") +async def _(body: InputData): + stream.set_input(body.webrtc_id, body.api_key, body.voice_name) + return {"status": "ok"} + + +@app.get("/") +async def index(): + rtc_config = get_twilio_turn_credentials() if get_space() else None + html_content = (current_dir / "index.html").read_text() + html_content = html_content.replace("__RTC_CONFIGURATION__", json.dumps(rtc_config)) + return HTMLResponse(content=html_content) + + +if __name__ == "__main__": + import os + + if (mode := os.getenv("MODE")) == "UI": + stream.ui.launch(server_port=7860) + elif mode == "PHONE": + stream.fastphone(host="0.0.0.0", port=7860) + else: + import uvicorn + + uvicorn.run(app, host="0.0.0.0", port=7860) diff --git a/demo/talk_to_gemini/index.html b/demo/talk_to_gemini/index.html new file mode 100644 index 0000000..a6ae8be --- /dev/null +++ b/demo/talk_to_gemini/index.html @@ -0,0 +1,452 @@ + + + + + + + Gemini Voice Chat + + + + + + +
+
+

Gemini Voice Chat

+

Speak with Gemini using real-time audio streaming

+

+ Get a Gemini API key + here +

+
+
+
+
+ + +
+
+ + +
+
+ +
+
+ +
+
+ + +
+ + + + + + + \ No newline at end of file diff --git a/demo/talk_to_gemini/requirements.txt b/demo/talk_to_gemini/requirements.txt new file mode 100644 index 0000000..e8cbeb2 --- /dev/null +++ b/demo/talk_to_gemini/requirements.txt @@ -0,0 +1,4 @@ +fastrtc +python-dotenv +google-genai +twilio diff --git a/demo/talk_to_openai/README.md b/demo/talk_to_openai/README.md new file mode 100644 index 0000000..7efde7d --- /dev/null +++ b/demo/talk_to_openai/README.md @@ -0,0 +1,15 @@ +--- +title: Talk to OpenAI +emoji: 🗣️ +colorFrom: purple +colorTo: red +sdk: gradio +sdk_version: 5.16.0 +app_file: app.py +pinned: false +license: mit +short_description: Talk to OpenAI using their multimodal API +tags: [webrtc, websocket, gradio, secret|TWILIO_ACCOUNT_SID, secret|TWILIO_AUTH_TOKEN, secret|OPENAI_API_KEY] +--- + +Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference \ No newline at end of file diff --git a/demo/talk_to_openai/README_gradio.md b/demo/talk_to_openai/README_gradio.md new file mode 100644 index 0000000..f495327 --- /dev/null +++ b/demo/talk_to_openai/README_gradio.md @@ -0,0 +1,15 @@ +--- +title: Talk to OpenAI (Gradio UI) +emoji: 🗣️ +colorFrom: purple +colorTo: red +sdk: gradio +sdk_version: 5.16.0 +app_file: app.py +pinned: false +license: mit +short_description: Talk to OpenAI (Gradio UI) +tags: [webrtc, websocket, gradio, secret|TWILIO_ACCOUNT_SID, secret|TWILIO_AUTH_TOKEN, secret|OPENAI_API_KEY] +--- + +Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference \ No newline at end of file diff --git a/demo/talk_to_openai/app.py b/demo/talk_to_openai/app.py new file mode 100644 index 0000000..23e336b --- /dev/null +++ b/demo/talk_to_openai/app.py @@ -0,0 +1,160 @@ +import asyncio +import base64 +import json +from pathlib import Path + +import gradio as gr +import numpy as np +import openai +from dotenv import load_dotenv +from fastapi import FastAPI +from fastapi.responses import HTMLResponse, StreamingResponse +from fastrtc import ( + AdditionalOutputs, + AsyncStreamHandler, + Stream, + WebRTCError, + get_twilio_turn_credentials, +) +from gradio.utils import get_space +from openai.types.beta.realtime import ResponseAudioTranscriptDoneEvent + +load_dotenv() + +cur_dir = Path(__file__).parent + +SAMPLE_RATE = 24000 + + +class OpenAIHandler(AsyncStreamHandler): + def __init__( + self, + ) -> None: + super().__init__( + expected_layout="mono", + output_sample_rate=SAMPLE_RATE, + output_frame_size=480, + input_sample_rate=SAMPLE_RATE, + ) + self.connection = None + self.output_queue = asyncio.Queue() + + def copy(self): + return OpenAIHandler() + + async def start_up( + self, + ): + """Connect to realtime API. Run forever in separate thread to keep connection open.""" + self.client = openai.AsyncOpenAI() + try: + async with self.client.beta.realtime.connect( + model="gpt-4o-mini-realtime-preview-2024-12-17" + ) as conn: + await conn.session.update( + session={"turn_detection": {"type": "server_vad"}} + ) + self.connection = conn + async for event in self.connection: + if event.type == "response.audio_transcript.done": + await self.output_queue.put(AdditionalOutputs(event)) + if event.type == "response.audio.delta": + await self.output_queue.put( + ( + self.output_sample_rate, + np.frombuffer( + base64.b64decode(event.delta), dtype=np.int16 + ).reshape(1, -1), + ), + ) + except Exception: + import traceback + + traceback.print_exc() + raise WebRTCError(str(traceback.format_exc())) + + async def receive(self, frame: tuple[int, np.ndarray]) -> None: + if not self.connection: + return + try: + _, array = frame + array = array.squeeze() + audio_message = base64.b64encode(array.tobytes()).decode("utf-8") + await self.connection.input_audio_buffer.append(audio=audio_message) # type: ignore + except Exception as e: + # print traceback + print(f"Error in receive: {str(e)}") + import traceback + + traceback.print_exc() + raise WebRTCError(str(traceback.format_exc())) + + async def emit(self) -> tuple[int, np.ndarray] | AdditionalOutputs | None: + return await self.output_queue.get() + + def reset_state(self): + """Reset connection state for new recording session""" + self.connection = None + self.args_set.clear() + + async def shutdown(self) -> None: + if self.connection: + await self.connection.close() + self.reset_state() + + +def update_chatbot(chatbot: list[dict], response: ResponseAudioTranscriptDoneEvent): + chatbot.append({"role": "assistant", "content": response.transcript}) + return chatbot + + +chatbot = gr.Chatbot(type="messages") +latest_message = gr.Textbox(type="text", visible=False) +stream = Stream( + OpenAIHandler(), + mode="send-receive", + modality="audio", + additional_inputs=[chatbot], + additional_outputs=[chatbot], + additional_outputs_handler=update_chatbot, + rtc_configuration=get_twilio_turn_credentials() if get_space() else None, + concurrency_limit=5 if get_space() else None, + time_limit=90 if get_space() else None, +) + +app = FastAPI() + +stream.mount(app) + + +@app.get("/") +async def _(): + rtc_config = get_twilio_turn_credentials() if get_space() else None + html_content = (cur_dir / "index.html").read_text() + html_content = html_content.replace("__RTC_CONFIGURATION__", json.dumps(rtc_config)) + return HTMLResponse(content=html_content) + + +@app.get("/outputs") +def _(webrtc_id: str): + async def output_stream(): + import json + + async for output in stream.output_stream(webrtc_id): + s = json.dumps({"role": "assistant", "content": output.args[0].transcript}) + yield f"event: output\ndata: {s}\n\n" + + return StreamingResponse(output_stream(), media_type="text/event-stream") + + +if __name__ == "__main__": + import os + + if (mode := os.getenv("MODE")) == "UI": + stream.ui.launch(server_port=7860) + elif mode == "PHONE": + stream.fastphone(host="0.0.0.0", port=7860) + else: + import uvicorn + + uvicorn.run(app, host="0.0.0.0", port=7860) diff --git a/demo/talk_to_openai/index.html b/demo/talk_to_openai/index.html new file mode 100644 index 0000000..8ba7876 --- /dev/null +++ b/demo/talk_to_openai/index.html @@ -0,0 +1,404 @@ + + + + + + + OpenAI Real-Time Chat + + + + + +
+
+ +
+
+
+
+ +
+
+ + + + + + \ No newline at end of file diff --git a/demo/talk_to_openai/requirements.txt b/demo/talk_to_openai/requirements.txt new file mode 100644 index 0000000..0480de2 --- /dev/null +++ b/demo/talk_to_openai/requirements.txt @@ -0,0 +1,4 @@ +fastrtc[vad] +openai +twilio +python-dotenv \ No newline at end of file diff --git a/demo/talk_to_sambanova/README.md b/demo/talk_to_sambanova/README.md new file mode 100644 index 0000000..62c88c4 --- /dev/null +++ b/demo/talk_to_sambanova/README.md @@ -0,0 +1,15 @@ +--- +title: Talk to Sambanova +emoji: 💻 +colorFrom: purple +colorTo: red +sdk: gradio +sdk_version: 5.16.0 +app_file: app.py +pinned: false +license: mit +short_description: Llama 3.2 - SambaNova API +tags: [webrtc, websocket, gradio, secret|TWILIO_ACCOUNT_SID, secret|TWILIO_AUTH_TOKEN, secret|SAMBANOVA_API_KEY] +--- + +Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference \ No newline at end of file diff --git a/demo/talk_to_sambanova/README_gradio.md b/demo/talk_to_sambanova/README_gradio.md new file mode 100644 index 0000000..866848c --- /dev/null +++ b/demo/talk_to_sambanova/README_gradio.md @@ -0,0 +1,15 @@ +--- +title: Talk to Sambanova (Gradio) +emoji: 💻 +colorFrom: purple +colorTo: red +sdk: gradio +sdk_version: 5.16.0 +app_file: app.py +pinned: false +license: mit +short_description: Llama 3.2 - SambaNova API (Gradio) +tags: [webrtc, websocket, gradio, secret|TWILIO_ACCOUNT_SID, secret|TWILIO_AUTH_TOKEN, secret|SAMBANOVA_API_KEY] +--- + +Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference \ No newline at end of file diff --git a/demo/talk_to_sambanova/app.py b/demo/talk_to_sambanova/app.py new file mode 100644 index 0000000..603e453 --- /dev/null +++ b/demo/talk_to_sambanova/app.py @@ -0,0 +1,152 @@ +import base64 +import json +import os +from pathlib import Path + +import gradio as gr +import numpy as np +import openai +from dotenv import load_dotenv +from fastapi import FastAPI +from fastapi.responses import HTMLResponse, StreamingResponse +from fastrtc import ( + AdditionalOutputs, + ReplyOnPause, + Stream, + WebRTCError, + get_stt_model, + get_twilio_turn_credentials, +) +from gradio.utils import get_space +from pydantic import BaseModel + +load_dotenv() + +curr_dir = Path(__file__).parent + + +client = openai.OpenAI( + api_key=os.environ.get("SAMBANOVA_API_KEY"), + base_url="https://api.sambanova.ai/v1", +) +stt_model = get_stt_model() + + +def response( + audio: tuple[int, np.ndarray], + gradio_chatbot: list[dict] | None = None, + conversation_state: list[dict] | None = None, +): + gradio_chatbot = gradio_chatbot or [] + conversation_state = conversation_state or [] + + text = stt_model.stt(audio) + sample_rate, array = audio + gradio_chatbot.append( + {"role": "user", "content": gr.Audio((sample_rate, array.squeeze()))} + ) + yield AdditionalOutputs(gradio_chatbot, conversation_state) + + conversation_state.append({"role": "user", "content": text}) + + try: + request = client.chat.completions.create( + model="Meta-Llama-3.2-3B-Instruct", + messages=conversation_state, # type: ignore + temperature=0.1, + top_p=0.1, + ) + response = {"role": "assistant", "content": request.choices[0].message.content} + + except Exception: + import traceback + + traceback.print_exc() + raise WebRTCError(traceback.format_exc()) + + conversation_state.append(response) + gradio_chatbot.append(response) + + yield AdditionalOutputs(gradio_chatbot, conversation_state) + + +chatbot = gr.Chatbot(type="messages", value=[]) +state = gr.State(value=[]) +stream = Stream( + ReplyOnPause( + response, # type: ignore + input_sample_rate=16000, + ), + mode="send", + modality="audio", + additional_inputs=[chatbot, state], + additional_outputs=[chatbot, state], + additional_outputs_handler=lambda *a: (a[2], a[3]), + concurrency_limit=20 if get_space() else None, + rtc_configuration=get_twilio_turn_credentials() if get_space() else None, +) + +app = FastAPI() +stream.mount(app) + + +class Message(BaseModel): + role: str + content: str + + +class InputData(BaseModel): + webrtc_id: str + chatbot: list[Message] + state: list[Message] + + +@app.get("/") +async def _(): + rtc_config = get_twilio_turn_credentials() if get_space() else None + html_content = (curr_dir / "index.html").read_text() + html_content = html_content.replace("__RTC_CONFIGURATION__", json.dumps(rtc_config)) + return HTMLResponse(content=html_content) + + +@app.post("/input_hook") +async def _(data: InputData): + body = data.model_dump() + stream.set_input(data.webrtc_id, body["chatbot"], body["state"]) + + +def audio_to_base64(file_path): + audio_format = "wav" + with open(file_path, "rb") as audio_file: + encoded_audio = base64.b64encode(audio_file.read()).decode("utf-8") + return f"data:audio/{audio_format};base64,{encoded_audio}" + + +@app.get("/outputs") +async def _(webrtc_id: str): + async def output_stream(): + async for output in stream.output_stream(webrtc_id): + chatbot = output.args[0] + state = output.args[1] + data = { + "message": state[-1], + "audio": audio_to_base64(chatbot[-1]["content"].value["path"]) + if chatbot[-1]["role"] == "user" + else None, + } + yield f"event: output\ndata: {json.dumps(data)}\n\n" + + return StreamingResponse(output_stream(), media_type="text/event-stream") + + +if __name__ == "__main__": + import os + + if (mode := os.getenv("MODE")) == "UI": + stream.ui.launch(server_port=7860) + elif mode == "PHONE": + raise ValueError("Phone mode not supported") + else: + import uvicorn + + uvicorn.run(app, host="0.0.0.0", port=7860) diff --git a/demo/talk_to_sambanova/index.html b/demo/talk_to_sambanova/index.html new file mode 100644 index 0000000..d4e206a --- /dev/null +++ b/demo/talk_to_sambanova/index.html @@ -0,0 +1,487 @@ + + + + + + + Talk to Sambanova + + + + + +
+
+ +
+
+
+
+
+
+
+
+
+
+
+ +
+
+ + + + + + \ No newline at end of file diff --git a/demo/talk_to_sambanova/requirements.txt b/demo/talk_to_sambanova/requirements.txt new file mode 100644 index 0000000..36f0d00 --- /dev/null +++ b/demo/talk_to_sambanova/requirements.txt @@ -0,0 +1,4 @@ +fastrtc[vad, stt] +python-dotenv +openai +twilio \ No newline at end of file diff --git a/demo/video_out.py b/demo/video_out.py deleted file mode 100644 index e244b51..0000000 --- a/demo/video_out.py +++ /dev/null @@ -1,65 +0,0 @@ -import os - -import cv2 -import gradio as gr -from gradio_webrtc import WebRTC -from twilio.rest import Client - -account_sid = os.environ.get("TWILIO_ACCOUNT_SID") -auth_token = os.environ.get("TWILIO_AUTH_TOKEN") - -if account_sid and auth_token: - client = Client(account_sid, auth_token) - - token = client.tokens.create() - - rtc_configuration = { - "iceServers": token.ice_servers, - "iceTransportPolicy": "relay", - } -else: - rtc_configuration = None - - -def generation(input_video): - cap = cv2.VideoCapture(input_video) - - iterating = True - - while iterating: - iterating, frame = cap.read() - - # flip frame vertically - frame = cv2.flip(frame, 0) - display_frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB) - yield display_frame - - -with gr.Blocks() as demo: - gr.HTML( - """ -

- Video Streaming (Powered by WebRTC ⚡️) -

- """ - ) - with gr.Row(): - with gr.Column(): - input_video = gr.Video(sources="upload") - with gr.Column(): - output_video = WebRTC( - label="Video Stream", - rtc_configuration=rtc_configuration, - mode="receive", - modality="video", - ) - output_video.stream( - fn=generation, - inputs=[input_video], - outputs=[output_video], - trigger=input_video.upload, - ) - - -if __name__ == "__main__": - demo.launch() diff --git a/demo/video_out_stream.py b/demo/video_out_stream.py deleted file mode 100644 index a5689ca..0000000 --- a/demo/video_out_stream.py +++ /dev/null @@ -1,54 +0,0 @@ -import os - -import cv2 -import gradio as gr -from gradio_webrtc import WebRTC -from twilio.rest import Client - -account_sid = os.environ.get("TWILIO_ACCOUNT_SID") -auth_token = os.environ.get("TWILIO_AUTH_TOKEN") - -if account_sid and auth_token: - client = Client(account_sid, auth_token) - - token = client.tokens.create() - - rtc_configuration = { - "iceServers": token.ice_servers, - "iceTransportPolicy": "relay", - } -else: - rtc_configuration = None - - -def generation(): - url = "https://download.tsi.telecom-paristech.fr/gpac/dataset/dash/uhd/mux_sources/hevcds_720p30_2M.mp4" - cap = cv2.VideoCapture(url) - iterating = True - while iterating: - iterating, frame = cap.read() - yield frame - - -with gr.Blocks() as demo: - gr.HTML( - """ -

- Video Streaming (Powered by WebRTC ⚡️) -

- """ - ) - output_video = WebRTC( - label="Video Stream", - rtc_configuration=rtc_configuration, - mode="receive", - modality="video", - ) - button = gr.Button("Start", variant="primary") - output_video.stream( - fn=generation, inputs=None, outputs=[output_video], trigger=button.click - ) - - -if __name__ == "__main__": - demo.launch() diff --git a/demo/video_send_output.py b/demo/video_send_output.py deleted file mode 100644 index 51357eb..0000000 --- a/demo/video_send_output.py +++ /dev/null @@ -1,102 +0,0 @@ -import logging -import os -import random - -import cv2 -import gradio as gr -from gradio_webrtc import AdditionalOutputs, WebRTC -from huggingface_hub import hf_hub_download -from inference import YOLOv10 -from twilio.rest import Client - -# Configure the root logger to WARNING to suppress debug messages from other libraries -logging.basicConfig(level=logging.WARNING) - -# Create a console handler -console_handler = logging.FileHandler("gradio_webrtc.log") -console_handler.setLevel(logging.DEBUG) - -# Create a formatter -formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s") -console_handler.setFormatter(formatter) - -# Configure the logger for your specific library -logger = logging.getLogger("gradio_webrtc") -logger.setLevel(logging.DEBUG) -logger.addHandler(console_handler) - - -model_file = hf_hub_download( - repo_id="onnx-community/yolov10n", filename="onnx/model.onnx" -) - -model = YOLOv10(model_file) - -account_sid = os.environ.get("TWILIO_ACCOUNT_SID") -auth_token = os.environ.get("TWILIO_AUTH_TOKEN") - -if account_sid and auth_token: - client = Client(account_sid, auth_token) - - token = client.tokens.create() - - rtc_configuration = { - "iceServers": token.ice_servers, - "iceTransportPolicy": "relay", - } -else: - rtc_configuration = None - - -def detection(image, conf_threshold=0.3): - image = cv2.resize(image, (model.input_width, model.input_height)) - new_image = model.detect_objects(image, conf_threshold) - return cv2.resize(new_image, (500, 500)) - - -css = """.my-group {max-width: 600px !important; max-height: 600 !important;} - .my-column {display: flex !important; justify-content: center !important; align-items: center !important};""" - -with gr.Blocks(css=css) as demo: - gr.HTML( - """ -

- YOLOv10 Webcam Stream (Powered by WebRTC ⚡️) -

- """ - ) - gr.HTML( - """ -

- arXiv | github -

- """ - ) - with gr.Column(elem_classes=["my-column"]): - with gr.Group(elem_classes=["my-group"]): - image = WebRTC( - label="Stream", - rtc_configuration=rtc_configuration, - mode="send-receive", - modality="video", - track_constraints={ - "width": {"exact": 800}, - "height": {"exact": 600}, - "aspectRatio": {"exact": 1.33333}, - }, - rtp_params={"degradationPreference": "maintain-resolution"}, - ) - conf_threshold = gr.Slider( - label="Confidence Threshold", - minimum=0.0, - maximum=1.0, - step=0.05, - value=0.30, - ) - number = gr.Number() - - image.stream( - fn=detection, inputs=[image, conf_threshold], outputs=[image], time_limit=90 - ) - -demo.launch() diff --git a/demo/webrtc_vs_websocket/README.md b/demo/webrtc_vs_websocket/README.md new file mode 100644 index 0000000..60a8b10 --- /dev/null +++ b/demo/webrtc_vs_websocket/README.md @@ -0,0 +1,15 @@ +--- +title: Webrtc Vs Websocket +emoji: 🧪 +colorFrom: purple +colorTo: red +sdk: gradio +sdk_version: 5.16.0 +app_file: app.py +pinned: false +license: mit +short_description: Compare Round Trip Times between WebRTC and Websockets +tags: [webrtc, websocket, gradio, secret|TWILIO_ACCOUNT_SID, secret|TWILIO_AUTH_TOKEN, secret|ELEVENLABS_API_KEY, secret|GROQ_API_KEY, secret|ANTHROPIC_API_KEY] +--- + +Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference \ No newline at end of file diff --git a/demo/webrtc_vs_websocket/app.py b/demo/webrtc_vs_websocket/app.py new file mode 100644 index 0000000..36e4e5b --- /dev/null +++ b/demo/webrtc_vs_websocket/app.py @@ -0,0 +1,147 @@ +import json +import logging +import os +from pathlib import Path + +import anthropic +import gradio as gr +import numpy as np +from dotenv import load_dotenv +from elevenlabs import ElevenLabs +from fastapi import FastAPI +from fastapi.responses import HTMLResponse, StreamingResponse +from fastrtc import AdditionalOutputs, ReplyOnPause, Stream, get_twilio_turn_credentials +from fastrtc.utils import aggregate_bytes_to_16bit, audio_to_bytes +from gradio.utils import get_space +from groq import Groq +from pydantic import BaseModel + +# Configure the root logger to WARNING to suppress debug messages from other libraries +logging.basicConfig(level=logging.WARNING) + +# Create a console handler +console_handler = logging.FileHandler("gradio_webrtc.log") +console_handler.setLevel(logging.DEBUG) + +# Create a formatter +formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s") +console_handler.setFormatter(formatter) + +# Configure the logger for your specific library +logger = logging.getLogger("fastrtc") +logger.setLevel(logging.DEBUG) +logger.addHandler(console_handler) + + +load_dotenv() + +groq_client = Groq() +claude_client = anthropic.Anthropic() +tts_client = ElevenLabs(api_key=os.environ["ELEVENLABS_API_KEY"]) + +curr_dir = Path(__file__).parent + + +def response( + audio: tuple[int, np.ndarray], + chatbot: list[dict] | None = None, +): + chatbot = chatbot or [] + messages = [{"role": d["role"], "content": d["content"]} for d in chatbot] + prompt = groq_client.audio.transcriptions.create( + file=("audio-file.mp3", audio_to_bytes(audio)), + model="whisper-large-v3-turbo", + response_format="verbose_json", + ).text + print("prompt", prompt) + chatbot.append({"role": "user", "content": prompt}) + messages.append({"role": "user", "content": prompt}) + response = claude_client.messages.create( + model="claude-3-5-haiku-20241022", + max_tokens=512, + messages=messages, # type: ignore + ) + response_text = " ".join( + block.text # type: ignore + for block in response.content + if getattr(block, "type", None) == "text" + ) + chatbot.append({"role": "assistant", "content": response_text}) + yield AdditionalOutputs(chatbot) + iterator = tts_client.text_to_speech.convert_as_stream( + text=response_text, + voice_id="JBFqnCBsd6RMkjVDRZzb", + model_id="eleven_multilingual_v2", + output_format="pcm_24000", + ) + for chunk in aggregate_bytes_to_16bit(iterator): + audio_array = np.frombuffer(chunk, dtype=np.int16).reshape(1, -1) + yield (24000, audio_array, "mono") + + +chatbot = gr.Chatbot(type="messages") +stream = Stream( + modality="audio", + mode="send-receive", + handler=ReplyOnPause(response), + additional_outputs_handler=lambda a, b: b, + additional_inputs=[chatbot], + additional_outputs=[chatbot], + rtc_configuration=get_twilio_turn_credentials() if get_space() else None, + concurrency_limit=20 if get_space() else None, +) + + +class Message(BaseModel): + role: str + content: str + + +class InputData(BaseModel): + webrtc_id: str + chatbot: list[Message] + + +app = FastAPI() + +stream.mount(app) + + +@app.get("/") +async def _(): + rtc_config = get_twilio_turn_credentials() if get_space() else None + html_content = (curr_dir / "index.html").read_text() + html_content = html_content.replace("__RTC_CONFIGURATION__", json.dumps(rtc_config)) + return HTMLResponse(content=html_content, status_code=200) + + +@app.post("/input_hook") +async def _(body: InputData): + stream.set_input(body.webrtc_id, body.model_dump()["chatbot"]) + return {"status": "ok"} + + +@app.get("/outputs") +def _(webrtc_id: str): + print("outputs", webrtc_id) + + async def output_stream(): + async for output in stream.output_stream(webrtc_id): + chatbot = output.args[0] + yield f"event: output\ndata: {json.dumps(chatbot[-2])}\n\n" + yield f"event: output\ndata: {json.dumps(chatbot[-1])}\n\n" + + return StreamingResponse(output_stream(), media_type="text/event-stream") + + +if __name__ == "__main__": + import os + + if (mode := os.getenv("MODE")) == "UI": + stream.ui.launch(server_port=7860, server_name="0.0.0.0") + elif mode == "PHONE": + stream.fastphone(host="0.0.0.0", port=7860) + else: + import uvicorn + + uvicorn.run(app, host="0.0.0.0", port=7860) diff --git a/demo/webrtc_vs_websocket/index.html b/demo/webrtc_vs_websocket/index.html new file mode 100644 index 0000000..cbc72b0 --- /dev/null +++ b/demo/webrtc_vs_websocket/index.html @@ -0,0 +1,630 @@ + + + + + + + WebRTC vs WebSocket Benchmark + + + + + + + +
+ This page compares the WebRTC Round-Trip-Time calculated from getStats() to the time taken to + process a ping/pong response pattern over websockets. It may not be a gold standard benchmark. Both WebRTC and + Websockets have their merits/advantages which is why FastRTC supports both. Artifacts in the WebSocket playback + audio are due to gaps in my frontend processing code and not FastRTC web server. +
+ +
+
+

WebRTC Connection

+
+
+
RTT (Round Trip Time): -
+
+ +
+ +
+

WebSocket Connection

+
+
+
RTT (Round Trip Time): 0
+
+ +
+
+ + + + +
+ + + + + \ No newline at end of file diff --git a/demo/webrtc_vs_websocket/requirements.txt b/demo/webrtc_vs_websocket/requirements.txt new file mode 100644 index 0000000..dd9323b --- /dev/null +++ b/demo/webrtc_vs_websocket/requirements.txt @@ -0,0 +1,6 @@ +fastrtc[vad] +elevenlabs +groq +anthropic +twilio +python-dotenv diff --git a/demo/whisper_realtime/README.md b/demo/whisper_realtime/README.md new file mode 100644 index 0000000..ec7e6cc --- /dev/null +++ b/demo/whisper_realtime/README.md @@ -0,0 +1,15 @@ +--- +title: Whisper Realtime Transcription +emoji: 👂 +colorFrom: purple +colorTo: red +sdk: gradio +sdk_version: 5.16.0 +app_file: app.py +pinned: false +license: mit +short_description: Transcribe audio in realtime with Whisper +tags: [webrtc, websocket, gradio, secret|TWILIO_ACCOUNT_SID, secret|TWILIO_AUTH_TOKEN, secret|GROQ_API_KEY] +--- + +Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference \ No newline at end of file diff --git a/demo/whisper_realtime/README_gradio.md b/demo/whisper_realtime/README_gradio.md new file mode 100644 index 0000000..37b8d2f --- /dev/null +++ b/demo/whisper_realtime/README_gradio.md @@ -0,0 +1,22 @@ +--- +app_file: app.py +colorFrom: purple +colorTo: red +emoji: 👂 +license: mit +pinned: false +sdk: gradio +sdk_version: 5.16.0 +short_description: Transcribe audio in realtime - Gradio UI version +tags: +- webrtc +- websocket +- gradio +- secret|TWILIO_ACCOUNT_SID +- secret|TWILIO_AUTH_TOKEN +- secret|GROQ_API_KEY +title: Whisper Realtime Transcription (Gradio UI) +--- + + +Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference \ No newline at end of file diff --git a/demo/whisper_realtime/app.py b/demo/whisper_realtime/app.py new file mode 100644 index 0000000..445df75 --- /dev/null +++ b/demo/whisper_realtime/app.py @@ -0,0 +1,86 @@ +import json +from pathlib import Path + +import gradio as gr +import numpy as np +from dotenv import load_dotenv +from fastapi import FastAPI +from fastapi.responses import HTMLResponse, StreamingResponse +from fastrtc import ( + AdditionalOutputs, + ReplyOnPause, + Stream, + WebRTCError, + audio_to_bytes, + get_twilio_turn_credentials, +) +from gradio.utils import get_space +from groq import AsyncClient + +cur_dir = Path(__file__).parent + +load_dotenv() + + +groq_client = AsyncClient() + + +async def transcribe(audio: tuple[int, np.ndarray]): + try: + transcript = await groq_client.audio.transcriptions.create( + file=("audio-file.mp3", audio_to_bytes(audio)), + model="whisper-large-v3-turbo", + response_format="verbose_json", + ) + yield AdditionalOutputs(transcript.text) + except Exception as e: + raise WebRTCError(str(e)) + + +stream = Stream( + ReplyOnPause(transcribe), + modality="audio", + mode="send", + additional_outputs=[ + gr.Textbox(label="Transcript"), + ], + additional_outputs_handler=lambda a, b: a + " " + b, + rtc_configuration=get_twilio_turn_credentials() if get_space() else None, + concurrency_limit=5 if get_space() else None, + time_limit=90 if get_space() else None, +) + +app = FastAPI() + +stream.mount(app) + + +@app.get("/transcript") +def _(webrtc_id: str): + async def output_stream(): + async for output in stream.output_stream(webrtc_id): + transcript = output.args[0] + yield f"event: output\ndata: {transcript}\n\n" + + return StreamingResponse(output_stream(), media_type="text/event-stream") + + +@app.get("/") +def index(): + rtc_config = get_twilio_turn_credentials() if get_space() else None + html_content = (cur_dir / "index.html").read_text() + html_content = html_content.replace("__RTC_CONFIGURATION__", json.dumps(rtc_config)) + return HTMLResponse(content=html_content) + + +if __name__ == "__main__": + import os + + if (mode := os.getenv("MODE")) == "UI": + stream.ui.launch(server_port=7860, server_name="0.0.0.0") + elif mode == "PHONE": + stream.fastphone(host="0.0.0.0", port=7860) + else: + import uvicorn + + uvicorn.run(app, host="0.0.0.0", port=7860) diff --git a/demo/whisper_realtime/index.html b/demo/whisper_realtime/index.html new file mode 100644 index 0000000..72789df --- /dev/null +++ b/demo/whisper_realtime/index.html @@ -0,0 +1,424 @@ + + + + + + + Real-time Whisper Transcription + + + + + +
+
+

Real-time Transcription

+

Powered by Groq and FastRTC

+
+ +
+
+
+ +
+
+ + + + + \ No newline at end of file diff --git a/demo/whisper_realtime/requirements.txt b/demo/whisper_realtime/requirements.txt new file mode 100644 index 0000000..5ade2d4 --- /dev/null +++ b/demo/whisper_realtime/requirements.txt @@ -0,0 +1,4 @@ +fastrtc[vad] +groq +python-dotenv +twilio \ No newline at end of file diff --git a/docs/advanced-configuration.md b/docs/advanced-configuration.md index 4dae09b..4f918ce 100644 --- a/docs/advanced-configuration.md +++ b/docs/advanced-configuration.md @@ -1,5 +1,9 @@ + +Any of the parameters for the `Stream` class can be passed to the [`WebRTC`](../userguide/gradio) component directly. + ## Track Constraints + You can specify the `track_constraints` parameter to control how the data is streamed to the server. The full documentation on track constraints is [here](https://developer.mozilla.org/en-US/docs/Web/API/MediaTrackConstraints#constraints). For example, you can control the size of the frames captured from the webcam like so: @@ -10,21 +14,22 @@ track_constraints = { "height": {"exact": 500}, "frameRate": {"ideal": 30}, } -webrtc = WebRTC(track_constraints=track_constraints, - modality="video", - mode="send-receive") +webrtc = Stream( + handler=..., + track_constraints=track_constraints, + modality="video", + mode="send-receive") ``` !!! warning - WebRTC may not enforce your constaints. For example, it may rescale your video - (while keeping the same resolution) in order to maintain the desired (or reach a better) frame rate. If you - really want to enforce height, width and resolution constraints, use the `rtp_params` parameter as set `"degradationPreference": "maintain-resolution"`. + WebRTC may not enforce your constraints. For example, it may rescale your video + (while keeping the same resolution) in order to maintain the desired frame rate (or reach a better one). If you really want to enforce height, width and resolution constraints, use the `rtp_params` parameter as set `"degradationPreference": "maintain-resolution"`. ```python - image = WebRTC( - label="Stream", + image = Stream( + modality="video", mode="send", track_constraints=track_constraints, rtp_params={"degradationPreference": "maintain-resolution"} @@ -36,7 +41,8 @@ webrtc = WebRTC(track_constraints=track_constraints, You can configure how the connection is created on the client by passing an `rtc_configuration` parameter to the `WebRTC` component constructor. See the list of available arguments [here](https://developer.mozilla.org/en-US/docs/Web/API/RTCPeerConnection/RTCPeerConnection#configuration). -When deploying on a remote server, an `rtc_configuration` parameter must be passed in. See [Deployment](/deployment). +!!! warning +When deploying on a remote server, the `rtc_configuration` parameter must be passed in. See [Deployment](../deployment). ## Reply on Pause Voice-Activity-Detection @@ -50,19 +56,18 @@ The `ReplyOnPause` class runs a Voice Activity Detection (VAD) algorithm to dete The following parameters control this argument: ```python -from gradio_webrtc import AlgoOptions, ReplyOnPause, WebRTC +from fastrtc import AlgoOptions, ReplyOnPause, Stream options = AlgoOptions(audio_chunk_duration=0.6, # (1) started_talking_threshold=0.2, # (2) speech_threshold=0.1, # (3) ) -with gr.Blocks as demo: - audio = WebRTC(...) - audio.stream(ReplyOnPause(..., algo_options=algo_options) - ) - -demo.launch() +Stream( + handler=ReplyOnPause(..., algo_options=algo_options), + modality="audio", + mode="send-receive" +) ``` 1. This is the length (in seconds) of audio chunks. @@ -75,14 +80,13 @@ demo.launch() You can configure the sampling rate of the audio passed to the `ReplyOnPause` or `StreamHandler` instance with the `input_sampling_rate` parameter. The current default is `48000` ```python -from gradio_webrtc import ReplyOnPause, WebRTC +from fastrtc import ReplyOnPause, Stream -with gr.Blocks as demo: - audio = WebRTC(...) - audio.stream(ReplyOnPause(..., input_sampling_rate=24000) - ) - -demo.launch() +stream = Stream( + handler=ReplyOnPause(..., input_sampling_rate=24000), + modality="audio", + mode="send-receive" +) ``` @@ -94,14 +98,13 @@ with the `output_sample_rate` and `output_frame_size` parameters. The following code (which uses the default values of these parameters), states that each output chunk will be a frame of 960 samples at a frame rate of `24,000` hz. So it will correspond to `0.04` seconds. ```python -from gradio_webrtc import ReplyOnPause, WebRTC +from fastrtc import ReplyOnPause, Stream -with gr.Blocks as demo: - audio = WebRTC(...) - audio.stream(ReplyOnPause(..., output_sample_rate=24000, output_frame_size=960) - ) - -demo.launch() +stream = Stream( + handler=ReplyOnPause(..., output_sample_rate=24000, output_frame_size=960), + modality="audio", + mode="send-receive" +) ``` !!! tip @@ -117,6 +120,10 @@ Pass any local path or url to an image (svg, png, jpeg) to the components `icon` You can control the button color and pulse color with `icon_button_color` and `pulse_color` parameters. They can take any valid css color. +!!! warning + + The `icon` parameter is only supported in the `WebRTC` component. + === "Code" ``` python audio = WebRTC( @@ -148,6 +155,10 @@ You can control the button color and pulse color with `icon_button_color` and `p You can supply a `button_labels` dictionary to change the text displayed in the `Start`, `Stop` and `Waiting` buttons that are displayed in the UI. The keys must be `"start"`, `"stop"`, and `"waiting"`. +!!! warning + + The `button_labels` parameter is only supported in the `WebRTC` component. + ``` python webrtc = WebRTC( label="Video Chat", diff --git a/docs/bolt.svg b/docs/bolt.svg deleted file mode 100644 index f3a0046..0000000 --- a/docs/bolt.svg +++ /dev/null @@ -1 +0,0 @@ - \ No newline at end of file diff --git a/docs/cookbook.md b/docs/cookbook.md index 0cb7910..7519cd0 100644 --- a/docs/cookbook.md +++ b/docs/cookbook.md @@ -1,6 +1,65 @@ + + + + +A collection of applications built with FastRTC. Click on the tags below to find the app you're looking for! + + +
+ + + + + + + + + + + + + +
+ + + +
- :speaking_head:{ .lg .middle }:eyes:{ .lg .middle } __Gemini Audio Video Chat__ +{: data-tags="audio,video,real-time-api"} --- @@ -8,35 +67,130 @@ - [:octicons-arrow-right-24: Demo](https://huggingface.co/spaces/freddyaboulton/gemini-audio-video-chat) + [:octicons-arrow-right-24: Demo](https://huggingface.co/spaces/fastrtc/gemini-audio-video) + + [:octicons-arrow-right-24: Gradio UI](https://huggingface.co/spaces/fastrtc/gemini-audio-video) - [:octicons-code-16: Code](https://huggingface.co/spaces/freddyaboulton/gemini-audio-video-chat/blob/main/app.py) + [:octicons-code-16: Code](https://huggingface.co/spaces/fastrtc/gemini-audio-video) - :speaking_head:{ .lg .middle } __Google Gemini Real Time Voice API__ +{: data-tags="audio,real-time-api,voice-chat"} --- Talk to Gemini in real time using Google's voice API. - + - [:octicons-arrow-right-24: Demo](https://huggingface.co/spaces/freddyaboulton/gemini-voice) - - [:octicons-code-16: Code](https://huggingface.co/spaces/freddyaboulton/gemini-voice/blob/main/app.py) + [:octicons-arrow-right-24: Demo](https://huggingface.co/spaces/fastrtc/talk-to-gemini) + + [:octicons-arrow-right-24: Gradio UI](https://huggingface.co/spaces/fastrtc/talk-to-gemini-gradio) + + [:octicons-code-16: Code](https://huggingface.co/spaces/fastrtc/talk-to-gemini/blob/main/app.py) - :speaking_head:{ .lg .middle } __OpenAI Real Time Voice API__ +{: data-tags="audio,real-time-api,voice-chat"} --- Talk to ChatGPT in real time using OpenAI's voice API. - + - [:octicons-arrow-right-24: Demo](https://huggingface.co/spaces/freddyaboulton/openai-realtime-voice) + [:octicons-arrow-right-24: Demo](https://huggingface.co/spaces/fastrtc/talk-to-openai) + + [:octicons-arrow-right-24: Gradio UI](https://huggingface.co/spaces/fastrtc/talk-to-openai-gradio) - [:octicons-code-16: Code](https://huggingface.co/spaces/freddyaboulton/openai-realtime-voice/blob/main/app.py) + [:octicons-code-16: Code](https://huggingface.co/spaces/fastrtc/talk-to-openai/blob/main/app.py) + +- :robot:{ .lg .middle } __Hello Computer__ +{: data-tags="llm,stopword,sambanova"} + + --- + + Say computer before asking your question! + + + [:octicons-arrow-right-24: Demo](https://huggingface.co/spaces/fastrtc/hello-computer) + + [:octicons-arrow-right-24: Gradio UI](https://huggingface.co/spaces/fastrtc/hello-computer-gradio) + + [:octicons-code-16: Code](https://huggingface.co/spaces/fastrtc/hello-computer/blob/main/app.py) + +- :robot:{ .lg .middle } __Llama Code Editor__ +{: data-tags="audio,llm,code-generation,groq,stopword"} + + --- + + Create and edit HTML pages with just your voice! Powered by Groq! + + + + [:octicons-arrow-right-24: Demo](https://huggingface.co/spaces/fastrtc/llama-code-editor) + + [:octicons-code-16: Code](https://huggingface.co/spaces/fastrtc/llama-code-editor/blob/main/app.py) + +- :speaking_head:{ .lg .middle } __Talk to Claude__ +{: data-tags="audio,llm,voice-chat"} + + --- + + Use the Anthropic and Play.Ht APIs to have an audio conversation with Claude. + + + + [:octicons-arrow-right-24: Demo](https://huggingface.co/spaces/fastrtc/talk-to-claude) + + [:octicons-arrow-right-24: Gradio UI](https://huggingface.co/spaces/fastrtc/talk-to-claude-gradio) + + [:octicons-code-16: Code](https://huggingface.co/spaces/fastrtc/talk-to-claude/blob/main/app.py) + +- :musical_note:{ .lg .middle } __LLM Voice Chat__ +{: data-tags="audio,llm,voice-chat,groq,elevenlabs"} + + --- + + Talk to an LLM with ElevenLabs! + + + + [:octicons-arrow-right-24: Demo](https://huggingface.co/spaces/fastrtc/llm-voice-chat) + + [:octicons-arrow-right-24: Gradio UI](https://huggingface.co/spaces/fastrtc/llm-voice-chat-gradio) + + [:octicons-code-16: Code](https://huggingface.co/spaces/fastrtc/llm-voice-chat/blob/main/app.py) + +- :musical_note:{ .lg .middle } __Whisper Transcription__ +{: data-tags="audio,transcription,groq"} + + --- + + Have whisper transcribe your speech in real time! + + + + [:octicons-arrow-right-24: Demo](https://huggingface.co/spaces/fastrtc/whisper-realtime) + + [:octicons-arrow-right-24: Gradio UI](https://huggingface.co/spaces/fastrtc/whisper-realtime-gradio) + + [:octicons-code-16: Code](https://huggingface.co/spaces/fastrtc/whisper-realtime/blob/main/app.py) + +- :robot:{ .lg .middle } __Talk to Sambanova__ +{: data-tags="llm,stopword,sambanova"} + + --- + + Talk to Llama 3.2 with the SambaNova API. + + + [:octicons-arrow-right-24: Demo](https://huggingface.co/spaces/fastrtc/talk-to-sambanova) + + [:octicons-arrow-right-24: Gradio UI](https://huggingface.co/spaces/fastrtc/talk-to-sambanova-gradio) + + [:octicons-code-16: Code](https://huggingface.co/spaces/fastrtc/talk-to-sambanova/blob/main/app.py) - :speaking_head:{ .lg .middle } __Hello Llama: Stop Word Detection__ +{: data-tags="audio,llm,code-generation,stopword,sambanova"} --- @@ -49,19 +203,9 @@ [:octicons-code-16: Code](https://huggingface.co/spaces/freddyaboulton/hey-llama-code-editor/blob/main/app.py) -- :robot:{ .lg .middle } __Llama Code Editor__ - - --- - - Create and edit HTML pages with just your voice! Powered by SambaNova systems. - - - - [:octicons-arrow-right-24: Demo](https://huggingface.co/spaces/freddyaboulton/llama-code-editor) - - [:octicons-code-16: Code](https://huggingface.co/spaces/freddyaboulton/llama-code-editor/blob/main/app.py) - :speaking_head:{ .lg .middle } __Audio Input/Output with mini-omni2__ +{: data-tags="audio,llm,voice-chat"} --- @@ -73,19 +217,8 @@ [:octicons-code-16: Code](https://huggingface.co/spaces/freddyaboulton/mini-omni2-webrtc/blob/main/app.py) -- :speaking_head:{ .lg .middle } __Talk to Claude__ - - --- - - Use the Anthropic and Play.Ht APIs to have an audio conversation with Claude. - - - - [:octicons-arrow-right-24: Demo](https://huggingface.co/spaces/freddyaboulton/talk-to-claude) - - [:octicons-code-16: Code](https://huggingface.co/spaces/freddyaboulton/talk-to-claude/blob/main/app.py) - - :speaking_head:{ .lg .middle } __Kyutai Moshi__ +{: data-tags="audio,llm,voice-chat,kyutai"} --- @@ -98,6 +231,7 @@ [:octicons-code-16: Code](https://huggingface.co/spaces/freddyaboulton/talk-to-moshi/blob/main/app.py) - :speaking_head:{ .lg .middle } __Talk to Ultravox__ +{: data-tags="audio,llm,voice-chat"} --- @@ -111,6 +245,7 @@ - :speaking_head:{ .lg .middle } __Talk to Llama 3.2 3b__ +{: data-tags="audio,llm,voice-chat"} --- @@ -124,6 +259,7 @@ - :robot:{ .lg .middle } __Talk to Qwen2-Audio__ +{: data-tags="audio,llm,voice-chat"} --- @@ -137,18 +273,20 @@ - :camera:{ .lg .middle } __Yolov10 Object Detection__ +{: data-tags="video,computer-vision"} --- Run the Yolov10 model on a user webcam stream in real time! - + - [:octicons-arrow-right-24: Demo](https://huggingface.co/spaces/freddyaboulton/webrtc-yolov10n) + [:octicons-arrow-right-24: Demo](https://huggingface.co/spaces/fastrtc/object-detection) - [:octicons-code-16: Code](https://huggingface.co/spaces/freddyaboulton/webrtc-yolov10n/blob/main/app.py) + [:octicons-code-16: Code](https://huggingface.co/spaces/fastrtc/object-detection/blob/main/app.py) - :camera:{ .lg .middle } __Video Object Detection with RT-DETR__ +{: data-tags="video,computer-vision"} --- @@ -159,6 +297,7 @@ [:octicons-code-16: Code](https://huggingface.co/spaces/freddyaboulton/rt-detr-object-detection-webrtc/blob/main/app.py) - :speaker:{ .lg .middle } __Text-to-Speech with Parler__ +{: data-tags="audio"} --- diff --git a/docs/deployment.md b/docs/deployment.md index 2ab85b9..0db5c44 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -1,9 +1,12 @@ -When deploying in a cloud environment (like Hugging Face Spaces, EC2, etc), you need to set up a TURN server to relay the WebRTC traffic. +When deploying in a cloud environment (like Hugging Face Spaces, EC2, etc), you need to set up a TURN server to relay the WebRTC traffic. This guide will cover the different options for setting up a TURN server. + +!!! tip + The `rtc_configuration` parameter of the `Stream` class also be passed to the [`WebRTC`](../userguide/gradio) component directly if you're building a standalone gradio app. ## Community Server Hugging Face graciously provides a TURN server for the community. -In order to use it, you need to first create a Hugging Face account by going to the [huggingface.co](https://huggingface.co/). +In order to use it, you need to first create a Hugging Face account by going to [huggingface.co](https://huggingface.co/). Then navigate to this [space](https://huggingface.co/spaces/freddyaboulton/turn-server-login) and follow the instructions on the page. You just have to click the "Log in" button and then the "Sign Up" button. @@ -12,17 +15,18 @@ Then navigate to this [space](https://huggingface.co/spaces/freddyaboulton/turn- Then you can use the `get_hf_turn_credentials` helper to get your credentials: ```python -from gradio_webrtc import get_hf_turn_credentials, WebRTC +from fastrtc import get_hf_turn_credentials, Stream # Pass a valid access token for your Hugging Face account # or set the HF_TOKEN environment variable credentials = get_hf_turn_credentials(token=None) -with gr.Blcocks() as demo: - webrtc = WebRTC(rtc_configuration=credentials) - ... - -demo.launch() +Stream( + handler=..., + rtc_configuration=credentials, + modality="audio", + mode="send-receive" +) ``` !!! warning @@ -38,6 +42,7 @@ The easiest way to do this is to use a service like Twilio. Create a **free** [account](https://login.twilio.com/u/signup) and the install the `twilio` package with pip (`pip install twilio`). You can then connect from the WebRTC component like so: ```python +from fastrtc import Stream from twilio.rest import Client import os @@ -53,13 +58,15 @@ rtc_configuration = { "iceTransportPolicy": "relay", } -with gr.Blocks() as demo: - ... - rtc = WebRTC(rtc_configuration=rtc_configuration, ...) - ... +Stream( + handler=..., + rtc_configuration=rtc_configuration, + modality="audio", + mode="send-receive" +) ``` -!!! tip "Automatic Login" +!!! tip "Automatic login" You can log in automatically with the `get_twilio_turn_credentials` helper @@ -148,8 +155,7 @@ The `server-info.json` file will have the server's public IP and public DNS: Finally, you can connect to your EC2 server from the gradio WebRTC component via the `rtc_configuration` argument: ```python -import gradio as gr -from gradio_webrtc import WebRTC +from fastrtc import Stream rtc_configuration = { "iceServers": [ { @@ -159,7 +165,10 @@ rtc_configuration = { }, ] } - -with gr.Blocks() as demo: - webrtc = WebRTC(rtc_configuration=rtc_configuration) +Stream( + handler=..., + rtc_configuration=rtc_configuration, + modality="audio", + mode="send-receive" +) ``` \ No newline at end of file diff --git a/docs/faq.md b/docs/faq.md index f3a69f6..38ec5d3 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -1,34 +1,37 @@ ## Demo does not work when deploying to the cloud -Make sure you are using a TURN server. See [deployment](/deployment). +Make sure you are using a TURN server. See [deployment](../deployment). ## Recorded input audio sounds muffled during output audio playback -By default, the microphone is [configured](https://github.com/freddyaboulton/gradio-webrtc/blob/903f1f70bd586f638ad3b5a3940c7a8ec70ad1f5/backend/gradio_webrtc/webrtc.py#L575) to do echoCancellation. +By default, the microphone is [configured](https://github.com/freddyaboulton/gradio-webrtc/blob/903f1f70bd586f638ad3b5a3940c7a8ec70ad1f5/backend/gradio_webrtc/webrtc.py#L575) to do echo cancellation. This is what's causing the recorded audio to sound muffled when the streamed audio starts playing. -You can disable this via the `track_constraints` (see [advanced configuration](./advanced-configuration])) with the following code: +You can disable this via the `track_constraints` (see [Advanced Configuration](../advanced-configuration)) with the following code: ```python - audio = WebRTC( - label="Stream", - track_constraints={ - "echoCancellation": False, - "noiseSuppression": {"exact": True}, - "autoGainControl": {"exact": True}, - "sampleRate": {"ideal": 24000}, - "sampleSize": {"ideal": 16}, - "channelCount": {"exact": 1}, - }, - rtc_configuration=None, - mode="send-receive", - modality="audio", - ) +stream = Stream( + track_constraints={ + "echoCancellation": False, + "noiseSuppression": {"exact": True}, + "autoGainControl": {"exact": True}, + "sampleRate": {"ideal": 24000}, + "sampleSize": {"ideal": 16}, + "channelCount": {"exact": 1}, + }, + rtc_configuration=None, + mode="send-receive", + modality="audio", +) ``` ## How to raise errors in the UI You can raise `WebRTCError` in order for an error message to show up in the user's screen. This is similar to how `gr.Error` works. +!!! warning + + The `WebRTCError` class is only supported in the `WebRTC` component. + Here is a simple example: ```python diff --git a/docs/fastrtc_logo.png b/docs/fastrtc_logo.png new file mode 100644 index 0000000..680bd75 Binary files /dev/null and b/docs/fastrtc_logo.png differ diff --git a/docs/fastrtc_logo_small.png b/docs/fastrtc_logo_small.png new file mode 100644 index 0000000..51faff2 Binary files /dev/null and b/docs/fastrtc_logo_small.png differ diff --git a/docs/index.md b/docs/index.md index 92923e7..ef9a9f0 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,30 +1,209 @@ -

Gradio WebRTC ⚡️

+
+

FastRTC

+ FastRTC Logo +
-Static Badge -Static Badge +Static Badge +Static Badge

-Stream video and audio in real time with Gradio using WebRTC. +The Real-Time Communication Library for Python.

+Turn any python function into a real-time audio and video stream over WebRTC or WebSockets. ## Installation ```bash -pip install gradio_webrtc +pip install fastrtc ``` -to use built-in pause detection (see [ReplyOnPause](/user-guide/#reply-on-pause)), install the `vad` extra: +to use built-in pause detection (see [ReplyOnPause](userguide/audio/#reply-on-pause)), and text to speech (see [Text To Speech](userguide/audio/#text-to-speech)), install the `vad` and `tts` extras: ```bash -pip install gradio_webrtc[vad] +pip install fastrtc[vad, tts] ``` -For stop word detection (see [ReplyOnStopWords](/user-guide/#reply-on-stopwords)), install the `stopword` extra: -```bash -pip install gradio_webrtc[stopword] -``` +## Quickstart + +Import the [Stream](userguide/streams) class and pass in a [handler](userguide/streams/#handlers). +The `Stream` has three main methods: + +- `.ui.launch()`: Launch a built-in UI for easily testing and sharing your stream. Built with [Gradio](https://www.gradio.app/). +- `.fastphone()`: Get a free temporary phone number to call into your stream. Hugging Face token required. +- `.mount(app)`: Mount the stream on a [FastAPI](https://fastapi.tiangolo.com/) app. Perfect for integrating with your already existing production system. + + +=== "Echo Audio" + + ```python + from fastrtc import Stream, ReplyOnPause + import numpy as np + + def echo(audio: tuple[int, np.ndarray]): + # The function will be passed the audio until the user pauses + # Implement any iterator that yields audio + # See "LLM Voice Chat" for a more complete example + yield audio + + stream = Stream( + handler=ReplyOnPause(detection), + modality="audio", + mode="send-receive", + ) + ``` + +=== "LLM Voice Chat" + + ```py + from fastrtc import ( + ReplyOnPause, AdditionalOutputs, Stream, + audio_to_bytes, aggregate_bytes_to_16bit + ) + import gradio as gr + from groq import Groq + import anthropic + from elevenlabs import ElevenLabs + + groq_client = Groq() + claude_client = anthropic.Anthropic() + tts_client = ElevenLabs() + + + # See "Talk to Claude" in Cookbook for an example of how to keep + # track of the chat history. + def response( + audio: tuple[int, np.ndarray], + ): + prompt = groq_client.audio.transcriptions.create( + file=("audio-file.mp3", audio_to_bytes(audio)), + model="whisper-large-v3-turbo", + response_format="verbose_json", + ).text + response = claude_client.messages.create( + model="claude-3-5-haiku-20241022", + max_tokens=512, + messages=[{"role": "user", "content": prompt}], + ) + response_text = " ".join( + block.text + for block in response.content + if getattr(block, "type", None) == "text" + ) + iterator = tts_client.text_to_speech.convert_as_stream( + text=response_text, + voice_id="JBFqnCBsd6RMkjVDRZzb", + model_id="eleven_multilingual_v2", + output_format="pcm_24000" + + ) + for chunk in aggregate_bytes_to_16bit(iterator): + audio_array = np.frombuffer(chunk, dtype=np.int16).reshape(1, -1) + yield (24000, audio_array) + + stream = Stream( + modality="audio", + mode="send-receive", + handler=ReplyOnPause(response), + ) + ``` + +=== "Webcam Stream" + + ```python + from fastrtc import Stream + import numpy as np + + + def flip_vertically(image): + return np.flip(image, axis=0) + + + stream = Stream( + handler=flip_vertically, + modality="video", + mode="send-receive", + ) + ``` + +=== "Object Detection" + + ```python + from fastrtc import Stream + import gradio as gr + import cv2 + from huggingface_hub import hf_hub_download + from .inference import YOLOv10 + + model_file = hf_hub_download( + repo_id="onnx-community/yolov10n", filename="onnx/model.onnx" + ) + + # git clone https://huggingface.co/spaces/fastrtc/object-detection + # for YOLOv10 implementation + model = YOLOv10(model_file) + + def detection(image, conf_threshold=0.3): + image = cv2.resize(image, (model.input_width, model.input_height)) + new_image = model.detect_objects(image, conf_threshold) + return cv2.resize(new_image, (500, 500)) + + stream = Stream( + handler=detection, + modality="video", + mode="send-receive", + additional_inputs=[ + gr.Slider(minimum=0, maximum=1, step=0.01, value=0.3) + ] + ) + ``` + +Run: +=== "UI" + + ```py + stream.ui.launch() + ``` + +=== "Telephone" + + ```py + stream.fastphone() + ``` + +=== "FastAPI" + + ```py + app = FastAPI() + stream.mount(app) + + # Optional: Add routes + @app.get("/") + async def _(): + return HTMLResponse(content=open("index.html").read()) + + # uvicorn app:app --host 0.0.0.0 --port 8000 + ``` + +Learn more about the [Stream](userguide/streams) in the user guide. +## Key Features + +:speaking_head:{ .lg } Automatic Voice Detection and Turn Taking built-in, only worry about the logic for responding to the user. + +:material-laptop:{ .lg } Automatic UI - Use the `.ui.launch()` method to launch the webRTC-enabled built-in Gradio UI. + +:material-lightning-bolt:{ .lg } Automatic WebRTC Support - Use the `.mount(app)` method to mount the stream on a FastAPI app and get a webRTC endpoint for your own frontend! + +:simple-webstorm:{ .lg } Websocket Support - Use the `.mount(app)` method to mount the stream on a FastAPI app and get a websocket endpoint for your own frontend! + +:telephone:{ .lg } Automatic Telephone Support - Use the `fastphone()` method of the stream to launch the application and get a free temporary phone number! + +:robot:{ .lg } Completely customizable backend - A `Stream` can easily be mounted on a FastAPI app so you can easily extend it to fit your production application. See the [Talk To Claude](https://huggingface.co/spaces/fastrtc/talk-to-claude) demo for an example on how to serve a custom JS frontend. + ## Examples -See the [cookbook](/cookbook) \ No newline at end of file +See the [cookbook](/cookbook). \ No newline at end of file diff --git a/docs/stylesheets/extra.css b/docs/stylesheets/extra.css new file mode 100644 index 0000000..a64f0fc --- /dev/null +++ b/docs/stylesheets/extra.css @@ -0,0 +1,63 @@ +:root { + --white: #ffffff; + --galaxy: #393931; + --space: #2d2d2a; + --rock: #2d2d2a; + --cosmic: #ffdd00c5; + --radiate: #d6cec0; + --sun: #ffac2f; + --neutron: #F7F5F6; + --supernova: #ffdd00; + --asteroid: #d6cec0; +} + +[data-md-color-scheme="fastrtc-dark"] { + --md-default-bg-color: var(--galaxy); + --md-default-fg-color: var(--white); + --md-default-fg-color--light: var(--white); + --md-default-fg-color--lighter: var(--white); + --md-primary-fg-color: var(--space); + --md-primary-bg-color: var(--white); + --md-accent-fg-color: var(--sun); + + --md-typeset-color: var(--white); + --md-typeset-a-color: var(--supernova); + --md-typeset-mark-color: var(--sun); + + --md-code-fg-color: var(--white); + --md-code-bg-color: var(--rock); + + --md-code-hl-comment-color: var(--asteroid); + --md-code-hl-punctuation-color: var(--supernova); + --md-code-hl-generic-color: var(--supernova); + --md-code-hl-variable-color: var(--white); + --md-code-hl-string-color: var(--radiate); + --md-code-hl-keyword-color: var(--supernova); + --md-code-hl-operator-color: var(--supernova); + --md-code-hl-number-color: var(--radiate); + --md-code-hl-special-color: var(--supernova); + --md-code-hl-function-color: var(--neutron); + --md-code-hl-constant-color: var(--radiate); + --md-code-hl-name-color: var(--md-code-fg-color); + + --md-typeset-del-color: hsla(6, 90%, 60%, 0.15); + --md-typeset-ins-color: hsla(150, 90%, 44%, 0.15); + + --md-typeset-table-color: hsla(0, 0%, 100%, 0.12); + --md-typeset-table-color--light: hsla(0, 0%, 100%, 0.035); +} + +[data-md-color-scheme="fastrtc-dark"] div.admonition { + color: var(--md-code-fg-color); + background-color: var(--galaxy); +} + + +[data-md-color-scheme="fastrtc-dark"] .grid.cards>ul>li { + border-color: var(--rock); + border-width: thick; +} + +[data-md-color-scheme="fastrtc-dark"] .grid.cards>ul>li>hr { + border-color: var(--rock); +} \ No newline at end of file diff --git a/docs/userguide/api.md b/docs/userguide/api.md new file mode 100644 index 0000000..c7fe3aa --- /dev/null +++ b/docs/userguide/api.md @@ -0,0 +1,459 @@ +# Connecting via API + +Before continuing, select the `modality`, `mode` of your `Stream` and whether you're using `WebRTC` or `WebSocket`s. + +
+
+ + +
+
+ + +
+
+ + +
+ +
+ +### Sample Code +
+ +### Message Format + +Over both WebRTC and WebSocket, the server can send messages of the following format: + +```json +{ + "type": `send_input` | `fetch_output` | `stopword` | `error` | `warning` | `log`, + "data": string | object +} +``` + +- `send_input`: Send any input data for the handler to the server. See [`Additional Inputs`](#additional-inputs) for more details. +- `fetch_output`: An instance of [`AdditionalOutputs`](#additional-outputs) is sent to the server. +- `stopword`: The stopword has been detected. See [`ReplyOnStopWords`](../audio/#reply-on-stopwords) for more details. +- `error`: An error occurred. The `data` will be a string containing the error message. +- `warning`: A warning occurred. The `data` will be a string containing the warning message. +- `log`: A log message. The `data` will be a string containing the log message. + +The `ReplyOnPause` handler can also send the following `log` messages. + +```json +{ + "type": "log", + "data": "pause_detected" | "response_starting" +} +``` + +!!! tip + When using WebRTC, the messages will be encoded as strings, so parse as JSON before using. + +### Additional Inputs + +When the `send_input` message is received, update the inputs of your handler however you like by using the `set_input` method of the `Stream` object. + +A common pattern is to use a `POST` request to send the updated data. The first argument to the `set_input` method is the `webrtc_id` of the handler. + +```python +from pydantic import BaseModel, Field + +class InputData(BaseModel): + webrtc_id: str + conf_threshold: float = Field(ge=0, le=1) + + +@app.post("/input_hook") +async def _(data: InputData): + stream.set_input(data.webrtc_id, data.conf_threshold) +``` + +The updated data will be passed to the handler on the **next** call. + +### Additional Outputs + +The `fetch_output` message is sent to the client whenever an instance of [`AdditionalOutputs`](../streams/#additional-outputs) is available. You can access the latest output data by calling the `fetch_latest_output` method of the `Stream` object. + +However, rather than fetching each output manually, a common pattern is to fetch the entire stream of output data by calling the `output_stream` method. + +Here is an example: +```python +from fastapi.responses import StreamingResponse + +@app.get("/updates") +async def stream_updates(webrtc_id: str): + async def output_stream(): + async for output in stream.output_stream(webrtc_id): + # Output is the AdditionalOutputs instance + # Be sure to serialize it however you would like + yield f"data: {output.args[0]}\n\n" + + return StreamingResponse( + output_stream(), + media_type="text/event-stream" + ) +``` + +### Handling Errors + +When connecting via `WebRTC`, the server will respond to the `/webrtc/offer` route with a JSON response. If there are too many connections, the server will respond with a 429 error. + +```json +{ + "status": "failed", + "meta": { + "error": "concurrency_limit_reached", + "limit": 10 + } +``` + +Over `WebSocket`, the server will send the same message before closing the connection. + + + + + + diff --git a/docs/userguide/audio-video.md b/docs/userguide/audio-video.md new file mode 100644 index 0000000..e602a50 --- /dev/null +++ b/docs/userguide/audio-video.md @@ -0,0 +1,27 @@ +# Audio-Video Streaming + +You can simultaneously stream audio and video using `AudioVideoStreamHandler` or `AsyncAudioVideoStreamHandler`. +They are identical to the audio `StreamHandlers` with the addition of `video_receive` and `video_emit` methods which take and return a `numpy` array, respectively. + +Here is an example of the video handling functions for connecting with the Gemini multimodal API. In this case, we simply reflect the webcam feed back to the user but every second we'll send the latest webcam frame (and an additional image component) to the Gemini server. + +Please see the "Gemini Audio Video Chat" example in the [cookbook](../../cookbook) for the complete code. + +``` python title="Async Gemini Video Handling" + +async def video_receive(self, frame: np.ndarray): + """Send video frames to the server""" + if self.session: + # send image every 1 second + # otherwise we flood the API + if time.time() - self.last_frame_time > 1: + self.last_frame_time = time.time() + await self.session.send(encode_image(frame)) + if self.latest_args[2] is not None: + await self.session.send(encode_image(self.latest_args[2])) + self.video_queue.put_nowait(frame) + +async def video_emit(self) -> VideoEmitType: + """Return video frames to the client""" + return await self.video_queue.get() +``` \ No newline at end of file diff --git a/docs/userguide/audio.md b/docs/userguide/audio.md new file mode 100644 index 0000000..797ce33 --- /dev/null +++ b/docs/userguide/audio.md @@ -0,0 +1,343 @@ +# Audio Streaming + +## Reply On Pause + +Typically, you want to run a python function whenever a user has stopped speaking. This can be done by wrapping a python generator with the `ReplyOnPause` class and passing it to the `handler` argument of the `Stream` object. + +=== "Code" + ```python + from fastrtc import ReplyOnPause, Stream + + def response(audio: tuple[int, np.ndarray]): # (1) + sample_rate, audio_array = audio + # Generate response + for audio_chunk in generate_response(sample_rate, audio_array): + yield (sample_rate, audio_chunk) # (2) + + stream = Stream( + handler=ReplyOnPause(response), + modality="audio", + mode="send-receive" + ) + ``` + + 1. The python generator will receive the **entire** audio up until the user stopped. It will be a tuple of the form (sampling_rate, numpy array of audio). The array will have a shape of (1, num_samples). You can also pass in additional input components. + + 2. The generator must yield audio chunks as a tuple of (sampling_rate, numpy audio array). Each numpy audio array must have a shape of (1, num_samples). + +=== "Notes" + 1. The python generator will receive the **entire** audio up until the user stopped. It will be a tuple of the form (sampling_rate, numpy array of audio). The array will have a shape of (1, num_samples). You can also pass in additional input components. + + 2. The generator must yield audio chunks as a tuple of (sampling_rate, numpy audio array). Each numpy audio array must have a shape of (1, num_samples). + + +The `ReplyOnPause` class will handle the voice detection and turn taking logic automatically! + +!!! warning "Argument Order" + + The first argument to the function must be the audio + +!!! tip "Parameters" + You can customize the voice detection parameters by passing in `algo_options` and `model_options` to the `ReplyOnPause` class. + ```python + from fastrtc import AlgoOptions, SileroVadOptions + + stream = Stream( + handler=ReplyOnPause( + response, + algo_options=AlgoOptions( + audio_chunk_duration=0.6, + started_talking_threshold=0.2, + speech_threshold=0.1 + ), + model_options=SileroVadOptions( + threshold=0.5, + min_speech_duration_ms=250, + min_silence_duration_ms=100 + ) + ) + ) + ``` + +## Reply On Stopwords + +You can configure your AI model to run whenever a set of "stop words" are detected, like "Hey Siri" or "computer", with the `ReplyOnStopWords` class. + +The API is similar to `ReplyOnPause` with the addition of a `stop_words` parameter. + +=== "Code" + ``` py + from fastrtc import Stream, ReplyOnStopWords + + def response(audio: tuple[int, np.ndarray]): + """This function must yield audio frames""" + ... + for numpy_array in generated_audio: + yield (sampling_rate, numpy_array, "mono") + + stream = Stream( + handler=ReplyOnStopWords(generate, + input_sample_rate=16000, + stop_words=["computer"]), # (1) + modality="audio", + mode="send-receive" + ) + ``` + + 1. The `stop_words` can be single words or pairs of words. Be sure to include common misspellings of your word for more robust detection, e.g. "llama", "lamma". In my experience, it's best to use two very distinct words like "ok computer" or "hello iris". + +=== "Notes" + 1. The `stop_words` can be single words or pairs of words. Be sure to include common misspellings of your word for more robust detection, e.g. "llama", "lamma". In my experience, it's best to use two very distinct words like "ok computer" or "hello iris". + +!!! tip "Extra Dependencies" + The `ReplyOnStopWords` class requires the the `stopword` extra. Run `pip install fastrtc[stopword]` to install it. + +!!! warning "English Only" + The `ReplyOnStopWords` class is currently only supported for English. + +## Stream Handler + +`ReplyOnPause` and `ReplyOnStopWords` are implementations of a `StreamHandler`. The `StreamHandler` is a low-level abstraction that gives you arbitrary control over how the input audio stream and output audio stream are created. The following example echos back the user audio. + +=== "Code" + ``` py + import gradio as gr + from gradio_webrtc import WebRTC, StreamHandler + from queue import Queue + + class EchoHandler(StreamHandler): + def __init__(self) -> None: + super().__init__() + self.queue = Queue() + + def receive(self, frame: tuple[int, np.ndarray]) -> None: # (1) + self.queue.put(frame) + + def emit(self) -> None: # (2) + return self.queue.get() + + def copy(self) -> StreamHandler: + return EchoHandler() + + def shutdown(self) -> None: # (3) + pass + + def start_up(self) -> None: # (4) + pass + + stream = Stream( + handler=EchoHandler(), + modality="audio", + mode="send-receive" + ) + ``` + + 1. The `StreamHandler` class implements three methods: `receive`, `emit` and `copy`. The `receive` method is called when a new frame is received from the client, and the `emit` method returns the next frame to send to the client. The `copy` method is called at the beginning of the stream to ensure each user has a unique stream handler. + 2. The `emit` method SHOULD NOT block. If a frame is not ready to be sent, the method should return `None`. + 3. The `shutdown` method is called when the stream is closed. It should be used to clean up any resources. + 4. The `start_up` method is called when the stream is first created. It should be used to initialize any resources. See [Talk To OpenAI](https://huggingface.co/spaces/fastrtc/talk-to-openai-gradio) or [Talk To Gemini](https://huggingface.co/spaces/fastrtc/talk-to-gemini-gradio) for an example of a `StreamHandler` that uses the `start_up` method to connect to an API. +=== "Notes" + 1. The `StreamHandler` class implements three methods: `receive`, `emit` and `copy`. The `receive` method is called when a new frame is received from the client, and the `emit` method returns the next frame to send to the client. The `copy` method is called at the beginning of the stream to ensure each user has a unique stream handler. + 2. The `emit` method SHOULD NOT block. If a frame is not ready to be sent, the method should return `None`. + 3. The `shutdown` method is called when the stream is closed. It should be used to clean up any resources. + 4. The `start_up` method is called when the stream is first created. It should be used to initialize any resources. See [Talk To OpenAI](https://huggingface.co/spaces/fastrtc/talk-to-openai-gradio) or [Talk To Gemini](https://huggingface.co/spaces/fastrtc/talk-to-gemini-gradio) for an example of a `StreamHandler` that uses the `start_up` method to connect to an API. + +!!! tip + See this [Talk To Gemini](https://huggingface.co/spaces/fastrtc/talk-to-gemini-gradio) for a complete example of a more complex stream handler. + +## Async Stream Handlers + +It is also possible to create asynchronous stream handlers. This is very convenient for accessing async APIs from major LLM developers, like Google and OpenAI. The main difference is that `receive`, `emit`, and `start_up` are now defined with `async def`. + +Here is a complete example of using `AsyncStreamHandler` for using the Google Gemini real time API: + +=== "Code" + ``` py + + from fastrtc import AsyncStreamHandler + import asyncio + import base64 + import os + import google.generativeai as genai + from google.generativeai.types import ( + LiveConnectConfig, SpeechConfig, + VoiceConfig, PrebuiltVoiceConfig + ) + + class GeminiHandler(AsyncStreamHandler): + """Handler for the Gemini API""" + + def __init__( + self, + expected_layout: Literal["mono"] = "mono", + output_sample_rate: int = 24000, + output_frame_size: int = 480, + ) -> None: + super().__init__( + expected_layout, + output_sample_rate, + output_frame_size, + input_sample_rate=16000, + ) + self.input_queue: asyncio.Queue = asyncio.Queue() + self.output_queue: asyncio.Queue = asyncio.Queue() + self.quit: asyncio.Event = asyncio.Event() + + def copy(self) -> "GeminiHandler": + return GeminiHandler( + expected_layout="mono", + output_sample_rate=self.output_sample_rate, + output_frame_size=self.output_frame_size, + ) + + async def start_up(self): + await self.wait_for_args() + api_key, voice_name = self.latest_args[1:] + client = genai.Client( + api_key=api_key or os.getenv("GEMINI_API_KEY"), + http_options={"api_version": "v1alpha"}, + ) + config = LiveConnectConfig( + response_modalities=["AUDIO"], # type: ignore + speech_config=SpeechConfig( + voice_config=VoiceConfig( + prebuilt_voice_config=PrebuiltVoiceConfig( + voice_name=voice_name, + ) + ) + ), + ) + async with client.aio.live.connect( + model="gemini-2.0-flash-exp", config=config + ) as session: + async for audio in session.start_stream( + stream=self.stream(), mime_type="audio/pcm" + ): + if audio.data: + array = np.frombuffer(audio.data, dtype=np.int16) + self.output_queue.put_nowait(array) + + async def stream(self) -> AsyncGenerator[bytes, None]: + while not self.quit.is_set(): + try: + audio = await asyncio.wait_for(self.input_queue.get(), 0.1) + yield audio + except (asyncio.TimeoutError, TimeoutError): + pass + + async def receive(self, frame: tuple[int, np.ndarray]) -> None: + _, array = frame + array = array.squeeze() + audio_message = encode_audio(array) + self.input_queue.put_nowait(audio_message) + + async def emit(self) -> tuple[int, np.ndarray]: + array = await self.output_queue.get() + return (self.output_sample_rate, array) + + def shutdown(self) -> None: + self.quit.set() + self.args_set.clear() + ``` + +## Text To Speech + +You can use an on-device text to speech model if you have the `tts` extra installed. +Import the `get_tts_model` function and call it with the model name you want to use. +At the moment, the only model supported is `kokoro`. + +The `get_tts_model` function returns an object with three methods: + +- `tts`: Synchronous text to speech. +- `stream_tts_sync`: Synchronous text to speech streaming. +- `stream_tts`: Asynchronous text to speech streaming. + +```python +from fastrtc import get_tts_model + +model = get_tts_model(model="kokoro") + +for audio in model.stream_tts_sync("Hello, world!"): + yield audio + +async for audio in model.stream_tts("Hello, world!"): + yield audio + +audio = model.tts("Hello, world!") +``` + +!!! tip + You can customize the audio by passing in an instace of `KokoroTTSOptions` to the method. + See [here](https://huggingface.co/hexgrad/Kokoro-82M/blob/main/VOICES.md) for a list of available voices. + ```python + from fastrtc import KokoroTTSOptions, get_tts_model + + model = get_tts_model(model="kokoro") + + options = KokoroTTSOptions( + voice="af_heart", + speed=1.0, + lang="en-us" + ) + + audio = model.tts("Hello, world!", options=options) + ``` + +## Speech To Text + +You can use an on-device speech to text model if you have the `stt` or `stopword` extra installed. +Import the `get_stt_model` function and call it with the model name you want to use. +At the moment, the only models supported are `moonshine/base` and `moonshine/tiny`. + +The `get_stt_model` function returns an object with the following method: + +- `stt`: Synchronous speech to text. + +```python +from fastrtc import get_stt_model + +model = get_stt_model(model="moonshine/base") + +audio = (16000, np.random.randint(-32768, 32768, size=(1, 16000))) +text = model.stt(audio) +``` + +!!! tip "Example" + See [LLM Voice Chat](https://huggingface.co/spaces/fastrtc/llm-voice-chat) for an example of using the `stt` method in a `ReplyOnPause` handler. + +!!! warning "English Only" + The `stt` model is currently only supported for English. + +## Requesting Inputs + +In `ReplyOnPause` and `ReplyOnStopWords`, any additional input data is automatically passed to your generator. For `StreamHandler`s, you must manually request the input data from the client. + +You can do this by calling `await self.wait_for_args()` (for `AsyncStreamHandler`s) in either the `emit` or `receive` methods. For a `StreamHandler`, you can call `self.wait_for_args_sync()`. + + +We can access the value of this component via the `latest_args` property of the `StreamHandler`. The `latest_args` is a list storing each of the values. The 0th index is the dummy string `__webrtc_value__`. + +## Telephone Integration + +In order for your handler to work over the phone, you must make sure that your handler is not expecting any additional input data besides the audio. + +If you call `await self.wait_for_args()` your stream will wait forever for the additional input data. + +The stream handlers have a `phone_mode` property that is set to `True` if the stream is running over the phone. You can use this property to determine if you should wait for additional input data. + +```python +def emit(self): + if self.phone_mode: + self.latest_args = [None] + else: + await self.wait_for_args() +``` + +### `ReplyOnPause` + +The generator you pass to `ReplyOnPause` must have default arguments for all arguments except audio. + +If you yield `AdditionalOutputs`, they will be passed in as the input arguments to the generator the next time it is called. + +!!! tip + See [Talk To Claude](https://huggingface.co/spaces/fastrtc/talk-to-claude) for an example of a `ReplyOnPause` handler that is compatible with telephone usage. Notice how the input chatbot history is yielded as an `AdditionalOutput` on each invocation. \ No newline at end of file diff --git a/docs/user-guide.md b/docs/userguide/gradio.md similarity index 55% rename from docs/user-guide.md rename to docs/userguide/gradio.md index e2cd946..685053e 100644 --- a/docs/user-guide.md +++ b/docs/userguide/gradio.md @@ -1,19 +1,17 @@ -# User Guide +# Gradio Component -To get started with WebRTC streams, all that's needed is to import the `WebRTC` component from this package and implement its `stream` event. +The automatic gradio UI is a great way to test your stream. However, you may want to customize the UI to your liking or simply build a standalone Gradio application. -This page will show how to do so with simple code examples. -For complete implementations of common tasks, see the [cookbook](/cookbook). +## The WebRTC Component -## Audio Streaming +To build a standalone Gradio application, you can use the `WebRTC` component and implement the `stream` event. +Similarly to the `Stream` object, you must set the `mode` and `modality` arguments and pass in a `handler`. -### Reply on Pause +Below are some common examples of how to use the `WebRTC` component. -Typically, you want to run an AI model that generates audio when the user has stopped speaking. This can be done by wrapping a python generator with the `ReplyOnPause` class -and passing it to the `stream` event of the `WebRTC` component. -=== "Code" - ``` py title="ReplyonPause" +=== "Reply On Pause" + ``` py import gradio as gr from gradio_webrtc import WebRTC, ReplyOnPause @@ -54,122 +52,9 @@ and passing it to the `stream` event of the `WebRTC` component. 4. The `WebRTC` component must be the first input and output component. 5. Set a `time_limit` to control how long a conversation will last. If the `concurrency_count` is 1 (default), only one conversation will be handled at a time. -=== "Notes" - 1. The python generator will receive the **entire** audio up until the user stopped. It will be a tuple of the form (sampling_rate, numpy array of audio). The array will have a shape of (1, num_samples). You can also pass in additional input components. - 2. The generator must yield audio chunks as a tuple of (sampling_rate, numpy audio arrays). Each numpy audio array must have a shape of (1, num_samples). - - 3. The `mode` and `modality` arguments must be set to `"send-receive"` and `"audio"`. - - 4. The `WebRTC` component must be the first input and output component. - - 5. Set a `time_limit` to control how long a conversation will last. If the `concurrency_count` is 1 (default), only one conversation will be handled at a time. - - -### Reply On Stopwords - -You can configure your AI model to run whenever a set of "stop words" are detected, like "Hey Siri" or "computer", with the `ReplyOnStopWords` class. - -The API is similar to `ReplyOnPause` with the addition of a `stop_words` parameter. - -=== "Code" - ``` py title="ReplyonPause" - import gradio as gr - from gradio_webrtc import WebRTC, ReplyOnPause - - def response(audio: tuple[int, np.ndarray]): - """This function must yield audio frames""" - ... - for numpy_array in generated_audio: - yield (sampling_rate, numpy_array, "mono") - - - with gr.Blocks() as demo: - gr.HTML( - """ -

- Chat (Powered by WebRTC ⚡️) -

- """ - ) - with gr.Column(): - with gr.Group(): - audio = WebRTC( - mode="send", - modality="audio", - ) - webrtc.stream(ReplyOnStopWords(generate, - input_sample_rate=16000, - stop_words=["computer"]), # (1) - inputs=[webrtc, history, code], - outputs=[webrtc], time_limit=90, - concurrency_limit=10) - - demo.launch() - ``` - - 1. The `stop_words` can be single words or pairs of words. Be sure to include common misspellings of your word for more robust detection, e.g. "llama", "lamma". In my experience, it's best to use two very distinct words like "ok computer" or "hello iris". - -=== "Notes" - 1. The `stop_words` can be single words or pairs of words. Be sure to include common misspellings of your word for more robust detection, e.g. "llama", "lamma". In my experience, it's best to use two very distinct words like "ok computer" or "hello iris". - -### Stream Handler - -`ReplyOnPause` is an implementation of a `StreamHandler`. The `StreamHandler` is a low-level -abstraction that gives you arbitrary control over how the input audio stream and output audio stream are created. The following example echos back the user audio. - -=== "Code" - ``` py title="Stream Handler" - import gradio as gr - from gradio_webrtc import WebRTC, StreamHandler - from queue import Queue - - class EchoHandler(StreamHandler): - def __init__(self) -> None: - super().__init__() - self.queue = Queue() - - def receive(self, frame: tuple[int, np.ndarray]) -> None: # (1) - self.queue.put(frame) - - def emit(self) -> None: # (2) - return self.queue.get() - - def copy(self) -> StreamHandler: - return EchoHandler() - - - with gr.Blocks() as demo: - with gr.Column(): - with gr.Group(): - audio = WebRTC( - mode="send-receive", - modality="audio", - ) - - audio.stream(fn=EchoHandler(), - inputs=[audio], outputs=[audio], - time_limit=15) - - demo.launch() - ``` - - 1. The `StreamHandler` class implements three methods: `receive`, `emit` and `copy`. The `receive` method is called when a new frame is received from the client, and the `emit` method returns the next frame to send to the client. The `copy` method is called at the beginning of the stream to ensure each user has a unique stream handler. - 2. The `emit` method SHOULD NOT block. If a frame is not ready to be sent, the method should return `None`. - -=== "Notes" - 1. The `StreamHandler` class implements three methods: `receive`, `emit` and `copy`. The `receive` method is called when a new frame is received from the client, and the `emit` method returns the next frame to send to the client. The `copy` method is called at the beginning of the stream to ensure each user has a unique stream handler. - 2. The `emit` method SHOULD NOT block. If a frame is not ready to be sent, the method should return `None`. - - -### Async Stream Handlers - -It is also possible to create asynchronous stream handlers. This is very convenient for accessing async APIs from major LLM developers, like Google and OpenAI. The main difference is that `receive` and `emit` are now defined with `async def`. - -Here is a complete example of using `AsyncStreamHandler` for using the Google Gemini real time API: - -=== "Code" - ``` py title="AsyncStreamHandler" +=== "AsyncStreamHandler" + ``` py import asyncio import base64 @@ -292,26 +177,11 @@ Here is a complete example of using `AsyncStreamHandler` for using the Google Ge lambda: (gr.update(visible=False), gr.update(visible=True)), None, [api_key_row, row], - ) - - demo.launch() + ) ``` +=== "Server-To-Client Audio" -### Accessing Other Component Values from a StreamHandler - -In the gemini demo above, you'll notice that we have the user input their google API key. This is stored in a `gr.Textbox` parameter. -We can access the value of this component via the `latest_args` prop of the `StreamHandler`. The `latest_args` is a list storing the values of each component in the WebRTC `stream` event `inputs` parameter. The value of the `WebRTC` component is the 0th index and it's always the dummy string `__webrtc_value__`. - -In order to fetch the latest value from the user however, we `await self.wait_for_args()`. In a synchronous `StreamHandler`, we would call `self.wait_for_args_sync()`. - - -### Server-To-Client Only - -To stream only from the server to the client, implement a python generator and pass it to the component's `stream` event. The stream event must also specify a `trigger` corresponding to a UI interaction that starts the stream. In this case, it's a button click. - -=== "Code" - - ``` py title="Server-To-CLient" + ``` py import gradio as gr from gradio_webrtc import WebRTC from pydub import AudioSegment @@ -334,21 +204,13 @@ To stream only from the server to the client, implement a python generator and p trigger=button.click # (2) ) ``` - - 1. Set `mode="receive"` to only receive audio from the server. - 2. The `stream` event must take a `trigger` that corresponds to the gradio event that starts the stream. In this case, it's the button click. -=== "Notes" + 1. Set `mode="receive"` to only receive audio from the server. 2. The `stream` event must take a `trigger` that corresponds to the gradio event that starts the stream. In this case, it's the button click. -## Video Streaming - -### Input/Output Streaming -Set up a video Input/Output stream to continuosly receive webcam frames from the user and run an arbitrary python function to return a modified frame. - -=== "Code" +=== "Video Streaming" - ``` py title="Input/Output Streaming" + ``` py import gradio as gr from gradio_webrtc import WebRTC @@ -381,18 +243,9 @@ Set up a video Input/Output stream to continuosly receive webcam frames from the 2. The function must return a numpy array. It can take arbitrary values from other components. 3. Set the `modality="video"` and `mode="send-receive"` 4. The `inputs` parameter should be a list where the first element is the WebRTC component. The only output allowed is the WebRTC component. -=== "Notes" - 1. The webcam frame will be represented as a numpy array of shape (height, width, RGB). - 2. The function must return a numpy array. It can take arbitrary values from other components. - 3. Set the `modality="video"` and `mode="send-receive"` - 4. The `inputs` parameter should be a list where the first element is the WebRTC component. The only output allowed is the WebRTC component. -### Server-to-Client Only - -Set up a server-to-client stream to stream video from an arbitrary user interaction. - -=== "Code" - ``` py title="Server-To-Client" +=== "Server-To-Client Video" + ``` py import gradio as gr from gradio_webrtc import WebRTC import cv2 @@ -419,39 +272,9 @@ Set up a server-to-client stream to stream video from an arbitrary user interact 1. The `stream` event's `fn` parameter is a generator function that yields the next frame from the video as a **numpy array**. 2. Set `mode="receive"` to only receive audio from the server. 3. The `trigger` parameter the gradio event that will trigger the stream. In this case, the button click event. -=== "Notes" - 1. The `stream` event's `fn` parameter is a generator function that yields the next frame from the video as a **numpy array**. - 2. Set `mode="receive"` to only receive audio from the server. - 3. The `trigger` parameter the gradio event that will trigger the stream. In this case, the button click event. - -## Audio-Video Streaming - -You can simultaneously stream audio and video simultaneously to/from a server using `AudioVideoStreamHandler` or `AsyncAudioVideoStreamHandler`. -They are identical to the audio `StreamHandlers` with the addition of `video_receive` and `video_emit` methods which take and return a `numpy` array, respectively. - -Here is an example of the video handling functions for connecting with the Gemini multimodal API. In this case, we simply reflect the webcam feed back to the user but every second we'll send the latest webcam frame (and an additional image component) to the Gemini server. - -Please see the "Gemini Audio Video Chat" example in the [cookbook](/cookbook) for the complete code. - -``` python title="Async Gemini Video Handling" - -async def video_receive(self, frame: np.ndarray): - """Send video frames to the server""" - if self.session: - # send image every 1 second - # otherwise we flood the API - if time.time() - self.last_frame_time > 1: - self.last_frame_time = time.time() - await self.session.send(encode_image(frame)) - if self.latest_args[2] is not None: - await self.session.send(encode_image(self.latest_args[2])) - self.video_queue.put_nowait(frame) - -async def video_emit(self) -> VideoEmitType: - """Return video frames to the client""" - return await self.video_queue.get() -``` +!!! tip + You can configure the `time_limit` and `concurrency_limit` parameters of the `stream` event similar to the `Stream` object. ## Additional Outputs diff --git a/docs/userguide/streams.md b/docs/userguide/streams.md new file mode 100644 index 0000000..d70dba9 --- /dev/null +++ b/docs/userguide/streams.md @@ -0,0 +1,236 @@ +# Core Concepts + + +The core of FastRTC is the `Stream` object. It can be used to stream audio, video, or both. + +Here's a simple example of creating a video stream that flips the video vertically. We'll use it to explain the core concepts of the `Stream` object. Click on the plus icons to get a link to the relevant section. + +```python +from fastrtc import Stream +import gradio as gr +import numpy as np + +def detection(image): + return np.flip(image, axis=0) + +stream = Stream( + handler=detection, # (1) + modality="video", # (2) + mode="send-receive", # (3) + additional_inputs=[ + gr.Slider(minimum=0, maximum=1, step=0.01, value=0.3) # (4) + ], + additional_outputs=None, # (5) + additional_outputs_handler=None # (6) +) +``` + +1. See [Handlers](#handlers) for more information. +2. See [Modalities](#modalities) for more information. +3. See [Stream Modes](#stream-modes) for more information. +4. See [Additional Inputs](#additional-inputs) for more information. +5. See [Additional Outputs](#additional-outputs) for more information. +6. See [Additional Outputs Handler](#additional-outputs) for more information. +7. Mount the `Stream` on a `FastAPI` app with `stream.mount(app)` and you can add custom routes to it. See [Custom Routes and Frontend Integration](#custom-routes-and-frontend-integration) for more information. +8. See [Built-in Routes](#built-in-routes) for more information. + +Run: +=== "UI" + + ```py + stream.ui.launch() + ``` +=== "FastAPI" + + ```py + app = FastAPI() + stream.mount(app) + + # uvicorn app:app --host 0.0.0.0 --port 8000 + ``` + +### Stream Modes + +FastRTC supports three streaming modes: + +- `send-receive`: Bidirectional streaming (default) +- `send`: Client-to-server only +- `receive`: Server-to-client only + +### Modalities + +FastRTC supports three modalities: + +- `video`: Video streaming +- `audio`: Audio streaming +- `audio-video`: Combined audio and video streaming + +### Handlers + +The `handler` argument is the main argument of the `Stream` object. A handler should be a function or a class that inherits from `StreamHandler` or `AsyncStreamHandler` depending on the modality and mode. + + +| Modality | send-receive | send | receive | +|----------|--------------|------|----------| +| video | Function that takes a video frame and returns a new video frame | Function that takes a video frame and returns a new frame | Function that takes a video frame and returns a new frame | +| audio | `StreamHandler` or `AsyncStreamHandler` subclass | `StreamHandler` or `AsyncStreamHandler` subclass | Generator yielding audio frames | +| audio-video | `AudioVideoStreamHandler` or `AsyncAudioVideoStreamHandler` subclass | Not Supported Yet | Not Supported Yet | + + +## Methods + +The `Stream` has three main methods: + +- `.ui.launch()`: Launch a built-in UI for easily testing and sharing your stream. Built with [Gradio](https://www.gradio.app/). You can change the UI by setting the `ui` property of the `Stream` object. Also see the [Gradio guide](../gradio.md) for building Gradio apss with fastrtc. +- `.fastphone()`: Get a free temporary phone number to call into your stream. Hugging Face token required. +- `.mount(app)`: Mount the stream on a [FastAPI](https://fastapi.tiangolo.com/) app. Perfect for integrating with your already existing production system or for building a custom UI. + +!!! warning + Websocket docs are only available for audio streams. Telephone docs are only available for audio streams in `send-receive` mode. + +## Additional Inputs + +You can add additional inputs to your stream using the `additional_inputs` argument. These inputs will be displayed in the generated Gradio UI and they will be passed to the handler as additional arguments. + +!!! tip + For audio `StreamHandlers`, please read the special [note](../audio#requesting-inputs) on requesting inputs. + +In the automatic gradio UI, these inputs will be the same python type corresponding to the Gradio component. In our case, we used a `gr.Slider` as the additional input, so it will be passed as a float. See the [Gradio documentation](https://www.gradio.app/docs/gradio) for a complete list of components and their corresponding types. + +### Input Hooks + +Outside of the gradio UI, you are free to update the inputs however you like by using the `set_input` method of the `Stream` object. + +A common pattern is to use a `POST` request to send the updated data. + +```python +from pydantic import BaseModel, Field +from fastapi import FastAPI + +class InputData(BaseModel): + webrtc_id: str + conf_threshold: float = Field(ge=0, le=1) + +app = FastAPI() +stream.mount(app) + +@app.post("/input_hook") +async def _(data: InputData): + stream.set_input(data.webrtc_id, data.conf_threshold) +``` + +The updated data will be passed to the handler on the **next** call. + + +## Additional Outputs + +You can return additional output from the handler by returning an instance of `AdditionalOutputs` from the handler. +Let's modify our previous example to also return the number of detections in the frame. + +```python +from fastrtc import Stream, AdditionalOutputs +import gradio as gr + +def detection(image, conf_threshold=0.3): + processed_frame, n_objects = process_frame(image, conf_threshold) + return processed_frame, AdditionalOutputs(n_objects) + +stream = Stream( + handler=detection, + modality="video", + mode="send-receive", + additional_inputs=[ + gr.Slider(minimum=0, maximum=1, step=0.01, value=0.3) + ], + additional_outputs=[gr.Number()], # (5) + additional_outputs_handler=lambda component, n_objects: n_objects +) +``` + +We added a `gr.Number()` to the additional outputs and we provided an `additional_outputs_handler`. + +The `additional_outputs_handler` is **only** needed for the gradio UI. It is a function that takes the current state of the `component` and the instance of `AdditionalOutputs` and returns the updated state of the `component`. In our case, we want to update the `gr.Number()` with the number of detections. + +!!! tip + Since the webRTC is very low latency, you probably don't want to return an additional output on each frame. + +### Output Hooks + +Outside of the gradio UI, you are free to access the output data however you like by calling the `output_stream` method of the `Stream` object. + +A common pattern is to use a `GET` request to get a stream of the output data. + +```python +from fastapi.responses import StreamingResponse + +@app.get("/updates") +async def stream_updates(webrtc_id: str): + async def output_stream(): + async for output in stream.output_stream(webrtc_id): + # Output is the AdditionalOutputs instance + # Be sure to serialize it however you would like + yield f"data: {output.args[0]}\n\n" + + return StreamingResponse( + output_stream(), + media_type="text/event-stream" + ) +``` + +## Custom Routes and Frontend Integration + +You can add custom routes for serving your own frontend or handling additional functionality once you have mounted the stream on a FastAPI app. + +```python +from fastapi.responses import HTMLResponse +from fastapi import FastAPI +from fastrtc import Stream + +stream = Stream(...) + +app = FastAPI() +stream.mount(app) + +# Serve a custom frontend +@app.get("/") +async def serve_frontend(): + return HTMLResponse(content=open("index.html").read()) + +``` + +## Telephone Integration + +FastRTC provides built-in telephone support through the `fastphone()` method: + +```python +# Launch with a temporary phone number +stream.fastphone( + # Optional: If None, will use the default token in your machine or read from the HF_TOKEN environment variable + token="your_hf_token", + host="127.0.0.1", + port=8000 +) +``` + +This will print out a phone number along with your temporary code you can use to connect to the stream. You are limited to **10 minutes** of calls per calendar month. + +!!! warning + + See this [section](../audio#telephone-integration) on making sure your stream handler is compatible for telephone usage. + +!!! tip + + If you don't have a HF token, you can get one [here](https://huggingface.co/settings/tokens). + +## Concurrency + +1. You can limit the number of concurrent connections by setting the `concurrency_limit` argument. +2. You can limit the amount of time (in seconds) a connection can stay open by setting the `time_limit` argument. + +```python +stream = Stream( + handler=handler, + concurrency_limit=10, + time_limit=3600 +) +``` \ No newline at end of file diff --git a/docs/userguide/video.md b/docs/userguide/video.md new file mode 100644 index 0000000..a05d10a --- /dev/null +++ b/docs/userguide/video.md @@ -0,0 +1,57 @@ +# Video Streaming + +## Input/Output Streaming + +We already saw this example in the [Quickstart](../../#quickstart) and the [Core Concepts](../streams) section. + +=== "Code" + + ``` py title="Input/Output Streaming" + from fastrtc import Stream + import gradio as gr + + def detection(image, conf_threshold=0.3): # (1) + processed_frame = process_frame(image, conf_threshold) + return processed_frame # (2) + + stream = Stream( + handler=detection, + modality="video", + mode="send-receive", # (3) + additional_inputs=[ + gr.Slider(minimum=0, maximum=1, step=0.01, value=0.3) + ], + ) + ``` + + 1. The webcam frame will be represented as a numpy array of shape (height, width, RGB). + 2. The function must return a numpy array. It can take arbitrary values from other components. + 3. Set the `modality="video"` and `mode="send-receive"` +=== "Notes" + 1. The webcam frame will be represented as a numpy array of shape (height, width, RGB). + 2. The function must return a numpy array. It can take arbitrary values from other components. + 3. Set the `modality="video"` and `mode="send-receive"` + +## Server-to-Client Only + +In this case, we stream from the server to the client so we will write a generator function that yields the next frame from the video (as a numpy array) +and set the `mode="receive"` in the `WebRTC` component. + +=== "Code" + ``` py title="Server-To-Client" + from fastrtc import Stream + + def generation(): + url = "https://download.tsi.telecom-paristech.fr/gpac/dataset/dash/uhd/mux_sources/hevcds_720p30_2M.mp4" + cap = cv2.VideoCapture(url) + iterating = True + while iterating: + iterating, frame = cap.read() + yield frame + + stream = Stream( + handler=generation, + modality="video", + mode="receive" + ) + ``` diff --git a/docs/userguide/webrtc_docs.md b/docs/userguide/webrtc_docs.md new file mode 100644 index 0000000..7270b43 --- /dev/null +++ b/docs/userguide/webrtc_docs.md @@ -0,0 +1,160 @@ +# FastRTC Docs + +## Connecting + +To connect to the server, you need to create a new RTCPeerConnection object and call the `setupWebRTC` function below. +{% if mode in ["send-receive", "receive"] %} +This code snippet assumes there is an html element with an id of `{{ modality }}_output_component_id` where the output will be displayed. It should be {{ "a `