mirror of
https://github.com/HumanAIGC-Engineering/gradio-webrtc.git
synced 2026-02-04 17:39:23 +08:00
Add API Reference and llms.txt (#256)
* stream api reference * docs * Add code * Add code * code
This commit is contained in:
Freddy Boulton
GitHub
326
docs/reference/reply_on_pause.md
Normal file
326
docs/reference/reply_on_pause.md
Normal file
@@ -0,0 +1,326 @@
|
||||
## `ReplyOnPause` Class
|
||||
|
||||
```python
|
||||
ReplyOnPause(
|
||||
fn: ReplyFnGenerator,
|
||||
startup_fn: Callable | None = None,
|
||||
algo_options: AlgoOptions | None = None,
|
||||
model_options: ModelOptions | None = None,
|
||||
can_interrupt: bool = True,
|
||||
expected_layout: Literal["mono", "stereo"] = "mono",
|
||||
output_sample_rate: int = 24000,
|
||||
output_frame_size: int | None = None, # Deprecated
|
||||
input_sample_rate: int = 48000,
|
||||
model: PauseDetectionModel | None = None,
|
||||
)
|
||||
```
|
||||
|
||||
A stream handler that processes incoming audio, detects pauses, and triggers a reply function (`fn`) when a pause is detected.
|
||||
|
||||
This handler accumulates audio chunks, uses a Voice Activity Detection (VAD) model to determine speech segments, and identifies pauses based on configurable thresholds. Once a pause is detected after speech has started, it calls the provided generator function `fn` with the accumulated audio.
|
||||
|
||||
It can optionally run a `startup_fn` at the beginning and supports interruption of the reply function if new audio arrives.
|
||||
|
||||
### Methods
|
||||
|
||||
#### `__init__`
|
||||
|
||||
```python
|
||||
__init__(
|
||||
fn: ReplyFnGenerator,
|
||||
startup_fn: Callable | None = None,
|
||||
algo_options: AlgoOptions | None = None,
|
||||
model_options: ModelOptions | None = None,
|
||||
can_interrupt: bool = True,
|
||||
expected_layout: Literal["mono", "stereo"] = "mono",
|
||||
output_sample_rate: int = 24000,
|
||||
output_frame_size: int | None = None, # Deprecated
|
||||
input_sample_rate: int = 48000,
|
||||
model: PauseDetectionModel | None = None,
|
||||
)
|
||||
```
|
||||
|
||||
Initializes the ReplyOnPause handler.
|
||||
|
||||
**Args:**
|
||||
|
||||
| Name | Type | Description |
|
||||
| :------------------- | :---------------------------------- | :------------------------------------------------------------------------------------------------------- |
|
||||
| `fn` | `ReplyFnGenerator` | The generator function to execute upon pause detection. It receives `(sample_rate, audio_array)` and optionally `*args`. |
|
||||
| `startup_fn` | `Callable | None` | An optional function to run once at the beginning. |
|
||||
| `algo_options` | `AlgoOptions | None` | Options for the pause detection algorithm. |
|
||||
| `model_options` | `ModelOptions | None` | Options for the VAD model. |
|
||||
| `can_interrupt` | `bool` | If True, incoming audio during `fn` execution will stop the generator and process the new audio. |
|
||||
| `expected_layout` | `Literal["mono", "stereo"]` | Expected input audio layout ('mono' or 'stereo'). |
|
||||
| `output_sample_rate` | `int` | The sample rate expected for audio yielded by `fn`. |
|
||||
| `output_frame_size` | `int | None` | Deprecated. |
|
||||
| `input_sample_rate` | `int` | The expected sample rate of incoming audio. |
|
||||
| `model` | `PauseDetectionModel | None` | An optional pre-initialized VAD model instance. |
|
||||
|
||||
---
|
||||
|
||||
#### `receive`
|
||||
|
||||
```python
|
||||
receive(frame: tuple[int, np.ndarray]) -> None
|
||||
```
|
||||
|
||||
Receives an audio frame from the stream. Processes the audio frame using `process_audio`. If a pause is detected, it sets the event. If interruption is enabled and a reply is ongoing, it closes the current generator and clears the processing queue.
|
||||
|
||||
**Args:**
|
||||
|
||||
| Name | Type | Description |
|
||||
| :------ | :--------------------- | :---------------------------------------------------------------- |
|
||||
| `frame` | `tuple[int, np.ndarray]` | A tuple containing the sample rate and the audio frame data. |
|
||||
|
||||
---
|
||||
|
||||
#### `emit`
|
||||
|
||||
```python
|
||||
emit() -> EmitType | None
|
||||
```
|
||||
|
||||
Produces the next output chunk from the reply generator (`fn`).
|
||||
|
||||
This method is called repeatedly after a pause is detected (event is set). If the generator is not already running, it initializes it by calling `fn` with the accumulated audio and any required additional arguments. It then yields the next item from the generator. Handles both sync and async generators. Resets the state upon generator completion or error.
|
||||
|
||||
**Returns:**
|
||||
|
||||
| Type | Description |
|
||||
| :--------------- | :------------------------------------------------------------------------------- |
|
||||
| `EmitType | None` | The next output item from the generator, or None if no pause event has occurred or the generator is exhausted. |
|
||||
|
||||
**Raises:**
|
||||
|
||||
* **`Exception`**: Re-raises exceptions occurring within the `fn` generator.
|
||||
|
||||
---
|
||||
|
||||
|
||||
#### `start_up`
|
||||
|
||||
```python
|
||||
start_up()
|
||||
```
|
||||
|
||||
Executes the startup function `startup_fn` if provided. Waits for additional arguments if needed before calling `startup_fn`.
|
||||
|
||||
---
|
||||
|
||||
#### `copy`
|
||||
|
||||
```python
|
||||
copy() -> ReplyOnPause
|
||||
```
|
||||
|
||||
Creates a new instance of ReplyOnPause with the same configuration.
|
||||
|
||||
**Returns:**
|
||||
|
||||
| Type | Description |
|
||||
| :------------- | :---------------------------------------------------- |
|
||||
| `ReplyOnPause` | A new `ReplyOnPause` instance with identical settings. |
|
||||
|
||||
---
|
||||
|
||||
#### `determine_pause`
|
||||
|
||||
```python
|
||||
determine_pause(audio: np.ndarray, sampling_rate: int, state: AppState) -> bool
|
||||
```
|
||||
|
||||
Analyzes an audio chunk to detect if a significant pause occurred after speech.
|
||||
|
||||
Uses the VAD model to measure speech duration within the chunk. Updates the application state (`state`) regarding whether talking has started and accumulates speech segments.
|
||||
|
||||
**Args:**
|
||||
|
||||
| Name | Type | Description |
|
||||
| :-------------- | :----------- | :-------------------------------------- |
|
||||
| `audio` | `np.ndarray` | The numpy array containing the audio chunk. |
|
||||
| `sampling_rate` | `int` | The sample rate of the audio chunk. |
|
||||
| `state` | `AppState` | The current application state. |
|
||||
|
||||
**Returns:**
|
||||
|
||||
| Type | Description |
|
||||
| :----- | :------------------------------------------------------------------------------------------------------ |
|
||||
| `bool` | True if a pause satisfying the configured thresholds is detected after speech has started, False otherwise. |
|
||||
|
||||
---
|
||||
|
||||
#### `process_audio`
|
||||
|
||||
```python
|
||||
process_audio(audio: tuple[int, np.ndarray], state: AppState) -> None
|
||||
```
|
||||
|
||||
Processes an incoming audio frame. Appends the frame to the buffer, runs pause detection on the buffer, and updates the application state.
|
||||
|
||||
**Args:**
|
||||
|
||||
| Name | Type | Description |
|
||||
| :------ | :--------------------- | :---------------------------------------------------------------- |
|
||||
| `audio` | `tuple[int, np.ndarray]` | A tuple containing the sample rate and the audio frame data. |
|
||||
| `state` | `AppState` | The current application state to update. |
|
||||
|
||||
---
|
||||
|
||||
#### `reset`
|
||||
|
||||
```python
|
||||
reset()
|
||||
```
|
||||
|
||||
Resets the handler state to its initial condition. Clears accumulated audio, resets state flags, closes any active generator, and clears the event flag.
|
||||
|
||||
---
|
||||
|
||||
#### `trigger_response`
|
||||
|
||||
```python
|
||||
trigger_response()
|
||||
```
|
||||
|
||||
Manually triggers the response generation process. Sets the event flag, effectively simulating a pause detection. Initializes the stream buffer if it's empty.
|
||||
|
||||
---
|
||||
|
||||
|
||||
|
||||
## `ReplyOnStopWords` Class
|
||||
|
||||
```python
|
||||
ReplyOnStopWords(
|
||||
fn: ReplyFnGenerator,
|
||||
stop_words: list[str],
|
||||
startup_fn: Callable | None = None,
|
||||
algo_options: AlgoOptions | None = None,
|
||||
model_options: ModelOptions | None = None,
|
||||
can_interrupt: bool = True,
|
||||
expected_layout: Literal["mono", "stereo"] = "mono",
|
||||
output_sample_rate: int = 24000,
|
||||
output_frame_size: int | None = None, # Deprecated
|
||||
input_sample_rate: int = 48000,
|
||||
model: PauseDetectionModel | None = None,
|
||||
)
|
||||
```
|
||||
|
||||
A stream handler that extends `ReplyOnPause` to trigger based on stop words followed by a pause.
|
||||
|
||||
This handler listens to the incoming audio stream, performs Speech-to-Text (STT) to detect predefined stop words. Once a stop word is detected, it waits for a subsequent pause in speech (using the VAD model) before triggering the reply function (`fn`) with the audio recorded *after* the stop word.
|
||||
|
||||
### Methods
|
||||
|
||||
#### `__init__`
|
||||
|
||||
```python
|
||||
__init__(
|
||||
fn: ReplyFnGenerator,
|
||||
stop_words: list[str],
|
||||
startup_fn: Callable | None = None,
|
||||
algo_options: AlgoOptions | None = None,
|
||||
model_options: ModelOptions | None = None,
|
||||
can_interrupt: bool = True,
|
||||
expected_layout: Literal["mono", "stereo"] = "mono",
|
||||
output_sample_rate: int = 24000,
|
||||
output_frame_size: int | None = None, # Deprecated
|
||||
input_sample_rate: int = 48000,
|
||||
model: PauseDetectionModel | None = None,
|
||||
)
|
||||
```
|
||||
|
||||
Initializes the ReplyOnStopWords handler.
|
||||
|
||||
**Args:**
|
||||
|
||||
*(Inherits Args from `ReplyOnPause.__init__`)*
|
||||
|
||||
| Name | Type | Description |
|
||||
| :----------- | :---------- | :------------------------------------------------------------------------------------------------------- |
|
||||
| `stop_words` | `list[str]` | A list of strings (words or phrases) to listen for. Detection is case-insensitive and ignores punctuation. |
|
||||
|
||||
---
|
||||
|
||||
#### `stop_word_detected`
|
||||
|
||||
```python
|
||||
stop_word_detected(text: str) -> bool
|
||||
```
|
||||
|
||||
Checks if any of the configured stop words are present in the text. Performs a case-insensitive search, treating multi-word stop phrases correctly and ignoring basic punctuation.
|
||||
|
||||
**Args:**
|
||||
|
||||
| Name | Type | Description |
|
||||
| :----- | :----- | :----------------------------------- |
|
||||
| `text` | `str` | The text transcribed from the audio. |
|
||||
|
||||
**Returns:**
|
||||
|
||||
| Type | Description |
|
||||
| :----- | :---------------------------------------- |
|
||||
| `bool` | True if a stop word is found, False otherwise. |
|
||||
|
||||
---
|
||||
|
||||
#### `send_stopword`
|
||||
|
||||
```python
|
||||
send_stopword()
|
||||
```
|
||||
|
||||
Sends a 'stopword' message asynchronously via the communication channel (if configured).
|
||||
|
||||
---
|
||||
|
||||
#### `determine_pause`
|
||||
|
||||
```python
|
||||
determine_pause(audio: np.ndarray, sampling_rate: int, state: ReplyOnStopWordsState) -> bool
|
||||
```
|
||||
|
||||
Analyzes an audio chunk to detect stop words and subsequent pauses. Overrides the `ReplyOnPause.determine_pause` method. First, it performs STT on the audio buffer to detect stop words. Once a stop word is detected (`state.stop_word_detected` is True), it then uses the VAD model to detect a pause in the audio *following* the stop word.
|
||||
|
||||
**Args:**
|
||||
|
||||
| Name | Type | Description |
|
||||
| :-------------- | :---------------------- | :--------------------------------------------------- |
|
||||
| `audio` | `np.ndarray` | The numpy array containing the audio chunk. |
|
||||
| `sampling_rate` | `int` | The sample rate of the audio chunk. |
|
||||
| `state` | `ReplyOnStopWordsState` | The current application state (ReplyOnStopWordsState). |
|
||||
|
||||
**Returns:**
|
||||
|
||||
| Type | Description |
|
||||
| :----- | :------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `bool` | True if a stop word has been detected and a subsequent pause satisfying the configured thresholds is detected, False otherwise. |
|
||||
|
||||
---
|
||||
|
||||
#### `reset`
|
||||
|
||||
```python
|
||||
reset()
|
||||
```
|
||||
|
||||
Resets the handler state to its initial condition. Clears accumulated audio, resets state flags (including stop word state), closes any active generator, and clears the event flag.
|
||||
|
||||
---
|
||||
|
||||
#### `copy`
|
||||
|
||||
```python
|
||||
copy() -> ReplyOnStopWords
|
||||
```
|
||||
|
||||
Creates a new instance of ReplyOnStopWords with the same configuration.
|
||||
|
||||
**Returns:**
|
||||
|
||||
| Type | Description |
|
||||
| :----------------- | :------------------------------------------------------- |
|
||||
| `ReplyOnStopWords` | A new `ReplyOnStopWords` instance with identical settings. |
|
||||
|
||||
*(Inherits other public methods like `start_up`, `process_audio`, `receive`, `trigger_response`, `async_iterate`, `emit` from `ReplyOnPause`)*
|
||||
195
docs/reference/stream.md
Normal file
195
docs/reference/stream.md
Normal file
@@ -0,0 +1,195 @@
|
||||
# `Stream` Class
|
||||
|
||||
```python
|
||||
Stream(
|
||||
handler: HandlerType,
|
||||
*,
|
||||
additional_outputs_handler: Callable | None = None,
|
||||
mode: Literal["send-receive", "receive", "send"] = "send-receive",
|
||||
modality: Literal["video", "audio", "audio-video"] = "video",
|
||||
concurrency_limit: int | None | Literal["default"] = "default",
|
||||
time_limit: float | None = None,
|
||||
allow_extra_tracks: bool = False,
|
||||
rtp_params: dict[str, Any] | None = None,
|
||||
rtc_configuration: dict[str, Any] | None = None,
|
||||
track_constraints: dict[str, Any] | None = None,
|
||||
additional_inputs: list[Component] | None = None,
|
||||
additional_outputs: list[Component] | None = None,
|
||||
ui_args: UIArgs | None = None
|
||||
)
|
||||
```
|
||||
|
||||
Define an audio or video stream with a built-in UI, mountable on a FastAPI app.
|
||||
|
||||
This class encapsulates the logic for handling real-time communication (WebRTC) streams, including setting up peer connections, managing tracks, generating a Gradio user interface, and integrating with FastAPI for API endpoints. It supports different modes (send, receive, send-receive) and modalities (audio, video, audio-video), and can optionally handle additional Gradio input/output components alongside the stream. It also provides functionality for telephone integration via the FastPhone service.
|
||||
|
||||
## Attributes
|
||||
|
||||
| Name | Type | Description |
|
||||
| :----------------------------- | :-------------------------------------------- | :----------------------------------------------------------------------- |
|
||||
| `mode` | `Literal["send-receive", "receive", "send"]` | The direction of the stream. |
|
||||
| `modality` | `Literal["video", "audio", "audio-video"]` | The type of media stream. |
|
||||
| `rtp_params` | `dict[str, Any] \| None` | Parameters for RTP encoding. |
|
||||
| `event_handler` | `HandlerType` | The main function to process stream data. |
|
||||
| `concurrency_limit` | `int` | The maximum number of concurrent connections allowed. |
|
||||
| `time_limit` | `float \| None` | Time limit in seconds for the event handler execution. |
|
||||
| `allow_extra_tracks` | `bool` | Whether to allow extra tracks beyond the specified modality. |
|
||||
| `additional_output_components` | `list[Component] \| None` | Extra Gradio output components. |
|
||||
| `additional_input_components` | `list[Component] \| None` | Extra Gradio input components. |
|
||||
| `additional_outputs_handler` | `Callable \| None` | Handler for additional outputs. |
|
||||
| `track_constraints` | `dict[str, Any] \| None` | Constraints for media tracks (e.g., resolution). |
|
||||
| `webrtc_component` | `WebRTC` | The underlying Gradio WebRTC component instance. |
|
||||
| `rtc_configuration` | `dict[str, Any] \| None` | Configuration for the RTCPeerConnection (e.g., ICE servers). |
|
||||
| `_ui` | `Blocks` | The Gradio Blocks UI instance. |
|
||||
|
||||
## Methods
|
||||
|
||||
### `mount`
|
||||
|
||||
```python
|
||||
mount(app: FastAPI, path: str = "")
|
||||
```
|
||||
|
||||
Mount the stream's API endpoints onto a FastAPI application.
|
||||
|
||||
This method adds the necessary routes (`/webrtc/offer`, `/telephone/handler`, `/telephone/incoming`, `/websocket/offer`) to the provided FastAPI app, prefixed with the optional `path`. It also injects a startup message into the app's lifespan.
|
||||
|
||||
**Args:**
|
||||
|
||||
| Name | Type | Description |
|
||||
| :----- | :-------- | :----------------------------------------------- |
|
||||
| `app` | `FastAPI` | The FastAPI application instance. |
|
||||
| `path` | `str` | An optional URL prefix for the mounted routes. |
|
||||
|
||||
---
|
||||
|
||||
### `fastphone`
|
||||
|
||||
```python
|
||||
fastphone(
|
||||
token: str | None = None,
|
||||
host: str = "127.0.0.1",
|
||||
port: int = 8000,
|
||||
**kwargs
|
||||
)
|
||||
```
|
||||
|
||||
Launch the FastPhone service for telephone integration.
|
||||
|
||||
Starts a local FastAPI server, mounts the stream, creates a public tunnel (using Gradio's tunneling), registers the tunnel URL with the FastPhone backend service, and prints the assigned phone number and access code. This allows users to call the phone number and interact with the stream handler.
|
||||
|
||||
**Args:**
|
||||
|
||||
| Name | Type | Description |
|
||||
| :------- | :-------------- | :--------------------------------------------------------------------------------------------------------- |
|
||||
| `token` | `str \| None` | Optional Hugging Face Hub token for authentication with the FastPhone service. If None, attempts to find one automatically. |
|
||||
| `host` | `str` | The local host address to bind the server to. |
|
||||
| `port` | `int` | The local port to bind the server to. |
|
||||
| `**kwargs` | | Additional keyword arguments passed to `uvicorn.run`. |
|
||||
|
||||
**Raises:**
|
||||
|
||||
* **`httpx.HTTPStatusError`**: If registration with the FastPhone service fails.
|
||||
* **`RuntimeError`**: If running in Colab/Spaces without `rtc_configuration`.
|
||||
|
||||
### `offer`
|
||||
|
||||
```python
|
||||
async offer(body: Body)
|
||||
```
|
||||
|
||||
Handle an incoming WebRTC offer via HTTP POST.
|
||||
|
||||
Processes the SDP offer and ICE candidates from the client to establish a WebRTC connection.
|
||||
|
||||
**Args:**
|
||||
|
||||
| Name | Type | Description |
|
||||
| :----- | :----- | :------------------------------------------------------------------------------------------------------ |
|
||||
| `body` | `Body` | A Pydantic model containing the SDP offer, optional ICE candidate, type ('offer'), and a unique WebRTC ID. |
|
||||
|
||||
**Returns:**
|
||||
|
||||
* A dictionary containing the SDP answer generated by the server.
|
||||
|
||||
---
|
||||
|
||||
### `handle_incoming_call`
|
||||
|
||||
```python
|
||||
async handle_incoming_call(request: Request)
|
||||
```
|
||||
|
||||
Handle incoming telephone calls (e.g., via Twilio).
|
||||
|
||||
Generates TwiML instructions to connect the incoming call to the WebSocket handler (`/telephone/handler`) for audio streaming.
|
||||
|
||||
**Args:**
|
||||
|
||||
| Name | Type | Description |
|
||||
| :-------- | :-------- | :----------------------------------------------------------- |
|
||||
| `request` | `Request` | The FastAPI Request object for the incoming call webhook. |
|
||||
|
||||
**Returns:**
|
||||
|
||||
* An `HTMLResponse` containing the TwiML instructions as XML.
|
||||
|
||||
---
|
||||
|
||||
### `telephone_handler`
|
||||
|
||||
```python
|
||||
async telephone_handler(websocket: WebSocket)
|
||||
```
|
||||
|
||||
The websocket endpoint for streaming audio over Twilio phone.
|
||||
|
||||
**Args:**
|
||||
|
||||
| Name | Type | Description |
|
||||
| :---------- | :---------- | :-------------------------------------- |
|
||||
| `websocket` | `WebSocket` | The incoming WebSocket connection object. |
|
||||
|
||||
---
|
||||
|
||||
### `websocket_offer`
|
||||
|
||||
```python
|
||||
async websocket_offer(websocket: WebSocket)
|
||||
```
|
||||
|
||||
Establish a Websocket connection to the Stream..
|
||||
|
||||
**Args:**
|
||||
|
||||
| Name | Type | Description |
|
||||
| :---------- | :---------- | :-------------------------------------- |
|
||||
| `websocket` | `WebSocket` | The incoming WebSocket connection object. |
|
||||
|
||||
## Properties
|
||||
|
||||
### `ui`
|
||||
|
||||
```python
|
||||
@property
|
||||
ui() -> Blocks
|
||||
```
|
||||
|
||||
Get the Gradio Blocks UI instance associated with this stream.
|
||||
|
||||
**Returns:**
|
||||
|
||||
* The `gradio.Blocks` UI instance.
|
||||
|
||||
```python
|
||||
@ui.setter
|
||||
ui(blocks: Blocks)
|
||||
```
|
||||
|
||||
Set a custom Gradio Blocks UI for this stream.
|
||||
|
||||
**Args:**
|
||||
|
||||
| Name | Type | Description |
|
||||
| :------- | :------- | :----------------------------------------------- |
|
||||
| `blocks` | `Blocks` | The `gradio.Blocks` instance to use as the UI. |
|
||||
420
docs/reference/stream_handlers.md
Normal file
420
docs/reference/stream_handlers.md
Normal file
@@ -0,0 +1,420 @@
|
||||
# Stream Handlers
|
||||
|
||||
These abstract base classes define the core interfaces for handling audio and video streams within FastRTC. Concrete handlers like `ReplyOnPause` inherit from these.
|
||||
|
||||
## `StreamHandlerBase` Class
|
||||
|
||||
```python
|
||||
StreamHandlerBase(
|
||||
expected_layout: Literal["mono", "stereo"] = "mono",
|
||||
output_sample_rate: int = 24000,
|
||||
output_frame_size: int | None = None, # Deprecated
|
||||
input_sample_rate: int = 48000,
|
||||
)
|
||||
```
|
||||
|
||||
Base class for handling media streams in FastRTC.
|
||||
|
||||
Provides common attributes and methods for managing stream state, communication channels, and basic configuration. This class is intended to be subclassed by concrete stream handlers like `StreamHandler` or `AsyncStreamHandler`.
|
||||
|
||||
### Attributes
|
||||
|
||||
| Name | Type | Description |
|
||||
| :------------------- | :---------------------------- | :----------------------------------------------------------------------- |
|
||||
| `expected_layout` | `Literal["mono", "stereo"]` | The expected channel layout of the input audio ('mono' or 'stereo'). |
|
||||
| `output_sample_rate` | `int` | The target sample rate for the output audio. |
|
||||
| `output_frame_size` | `int` | The desired number of samples per output audio frame. |
|
||||
| `input_sample_rate` | `int` | The expected sample rate of the input audio. |
|
||||
| `channel` | `DataChannel \| None` | The WebRTC data channel for communication. |
|
||||
| `channel_set` | `asyncio.Event` | Event indicating if the data channel is set. |
|
||||
| `args_set` | `asyncio.Event` | Event indicating if additional arguments are set. |
|
||||
| `latest_args` | `str \| list[Any]` | Stores the latest arguments received. |
|
||||
| `loop` | `asyncio.AbstractEventLoop` | The asyncio event loop. |
|
||||
| `phone_mode` | `bool` | Flag indicating if operating in telephone mode. |
|
||||
|
||||
### Methods
|
||||
|
||||
#### `__init__`
|
||||
|
||||
```python
|
||||
__init__(
|
||||
expected_layout: Literal["mono", "stereo"] = "mono",
|
||||
output_sample_rate: int = 24000,
|
||||
output_frame_size: int | None = None, # Deprecated
|
||||
input_sample_rate: int = 48000,
|
||||
)
|
||||
```
|
||||
|
||||
Initializes the StreamHandlerBase.
|
||||
|
||||
**Args:**
|
||||
|
||||
| Name | Type | Description |
|
||||
| :------------------- | :-------------------------- | :------------------------------------------------------------------- |
|
||||
| `expected_layout` | `Literal["mono", "stereo"]` | Expected input audio layout ('mono' or 'stereo'). |
|
||||
| `output_sample_rate` | `int` | Target output audio sample rate. |
|
||||
| `output_frame_size` | `int \| None` | Deprecated. Frame size is now derived from sample rate. |
|
||||
| `input_sample_rate` | `int` | Expected input audio sample rate. |
|
||||
|
||||
---
|
||||
|
||||
#### `clear_queue`
|
||||
|
||||
```python
|
||||
clear_queue()
|
||||
```
|
||||
|
||||
Clears the internal processing queue via the registered callback.
|
||||
|
||||
---
|
||||
|
||||
#### `send_message`
|
||||
|
||||
```python
|
||||
async send_message(msg: str)
|
||||
```
|
||||
|
||||
Asynchronously sends a message over the data channel.
|
||||
|
||||
**Args:**
|
||||
|
||||
| Name | Type | Description |
|
||||
| :---- | :----- | :------------------------ |
|
||||
| `msg` | `str` | The string message to send. |
|
||||
|
||||
---
|
||||
|
||||
#### `send_message_sync`
|
||||
|
||||
```python
|
||||
send_message_sync(msg: str)
|
||||
```
|
||||
|
||||
Synchronously sends a message over the data channel. Runs the async `send_message` in the event loop and waits for completion.
|
||||
|
||||
**Args:**
|
||||
|
||||
| Name | Type | Description |
|
||||
| :---- | :----- | :------------------------ |
|
||||
| `msg` | `str` | The string message to send. |
|
||||
|
||||
---
|
||||
|
||||
|
||||
#### `reset`
|
||||
|
||||
```python
|
||||
reset()
|
||||
```
|
||||
|
||||
Resets the argument set event.
|
||||
|
||||
---
|
||||
|
||||
#### `shutdown`
|
||||
|
||||
```python
|
||||
shutdown()
|
||||
```
|
||||
|
||||
Placeholder for shutdown logic. Subclasses can override.
|
||||
|
||||
---
|
||||
|
||||
## `StreamHandler` Class
|
||||
|
||||
```python
|
||||
StreamHandler(
|
||||
expected_layout: Literal["mono", "stereo"] = "mono",
|
||||
output_sample_rate: int = 24000,
|
||||
output_frame_size: int | None = None, # Deprecated
|
||||
input_sample_rate: int = 48000,
|
||||
)
|
||||
```
|
||||
|
||||
Abstract base class for synchronous stream handlers.
|
||||
|
||||
Inherits from `StreamHandlerBase` and defines the core synchronous interface for processing audio streams. Subclasses must implement `receive`, `emit`, and `copy`.
|
||||
|
||||
*(Inherits Attributes and Methods from `StreamHandlerBase`)*
|
||||
|
||||
### Abstract Methods
|
||||
|
||||
#### `receive`
|
||||
|
||||
```python
|
||||
@abstractmethod
|
||||
receive(frame: tuple[int, npt.NDArray[np.int16]]) -> None
|
||||
```
|
||||
|
||||
Process an incoming audio frame synchronously.
|
||||
|
||||
**Args:**
|
||||
|
||||
| Name | Type | Description |
|
||||
| :------ | :------------------------------------ | :----------------------------------------------------------------------- |
|
||||
| `frame` | `tuple[int, npt.NDArray[np.int16]]` | A tuple containing the sample rate (int) and the audio data as a numpy array (int16). |
|
||||
|
||||
---
|
||||
|
||||
#### `emit`
|
||||
|
||||
```python
|
||||
@abstractmethod
|
||||
emit() -> EmitType
|
||||
```
|
||||
|
||||
Produce the next output chunk synchronously. This method is called repeatedly to generate the output to be sent back over the stream.
|
||||
|
||||
**Returns:**
|
||||
|
||||
| Type | Description |
|
||||
| :--------- | :------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `EmitType` | An output item conforming to `EmitType`, which could be audio data, additional outputs, control signals (like `CloseStream`), or None. |
|
||||
|
||||
---
|
||||
|
||||
#### `copy`
|
||||
|
||||
```python
|
||||
@abstractmethod
|
||||
copy() -> StreamHandler
|
||||
```
|
||||
|
||||
Create a copy of this synchronous stream handler instance. Used to create a new handler for each connection.
|
||||
|
||||
**Returns:**
|
||||
|
||||
| Type | Description |
|
||||
| :-------------- | :----------------------------------------------------------- |
|
||||
| `StreamHandler` | A new instance of the concrete StreamHandler subclass. |
|
||||
|
||||
---
|
||||
|
||||
#### `start_up`
|
||||
|
||||
```python
|
||||
start_up()
|
||||
```
|
||||
|
||||
Optional synchronous startup logic.
|
||||
|
||||
---
|
||||
|
||||
## `AsyncStreamHandler` Class
|
||||
|
||||
```python
|
||||
AsyncStreamHandler(
|
||||
expected_layout: Literal["mono", "stereo"] = "mono",
|
||||
output_sample_rate: int = 24000,
|
||||
output_frame_size: int | None = None, # Deprecated
|
||||
input_sample_rate: int = 48000,
|
||||
)
|
||||
```
|
||||
|
||||
Abstract base class for asynchronous stream handlers.
|
||||
|
||||
Inherits from `StreamHandlerBase` and defines the core asynchronous interface using coroutines (`async def`) for processing audio streams. Subclasses must implement `receive`, `emit`, and `copy`. The `start_up` method must also be a coroutine.
|
||||
|
||||
*(Inherits Attributes and Methods from `StreamHandlerBase`)*
|
||||
|
||||
### Abstract Methods
|
||||
|
||||
#### `receive`
|
||||
|
||||
```python
|
||||
@abstractmethod
|
||||
async receive(frame: tuple[int, npt.NDArray[np.int16]]) -> None
|
||||
```
|
||||
|
||||
Process an incoming audio frame asynchronously.
|
||||
|
||||
**Args:**
|
||||
|
||||
| Name | Type | Description |
|
||||
| :------ | :------------------------------------ | :----------------------------------------------------------------------- |
|
||||
| `frame` | `tuple[int, npt.NDArray[np.int16]]` | A tuple containing the sample rate (int) and the audio data as a numpy array (int16). |
|
||||
|
||||
---
|
||||
|
||||
#### `emit`
|
||||
|
||||
```python
|
||||
@abstractmethod
|
||||
async emit() -> EmitType
|
||||
```
|
||||
|
||||
Produce the next output chunk asynchronously. This coroutine is called to generate the output to be sent back over the stream.
|
||||
|
||||
**Returns:**
|
||||
|
||||
| Type | Description |
|
||||
| :--------- | :------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `EmitType` | An output item conforming to `EmitType`, which could be audio data, additional outputs, control signals (like `CloseStream`), or None. |
|
||||
|
||||
---
|
||||
|
||||
#### `copy`
|
||||
|
||||
```python
|
||||
@abstractmethod
|
||||
copy() -> AsyncStreamHandler
|
||||
```
|
||||
|
||||
Create a copy of this asynchronous stream handler instance. Used to create a new handler for each connection.
|
||||
|
||||
**Returns:**
|
||||
|
||||
| Type | Description |
|
||||
| :------------------- | :--------------------------------------------------------------- |
|
||||
| `AsyncStreamHandler` | A new instance of the concrete AsyncStreamHandler subclass. |
|
||||
|
||||
---
|
||||
|
||||
#### `start_up`
|
||||
|
||||
```python
|
||||
async start_up()
|
||||
```
|
||||
|
||||
Optional asynchronous startup logic. Must be a coroutine (`async def`).
|
||||
|
||||
---
|
||||
|
||||
## `AudioVideoStreamHandler` Class
|
||||
|
||||
```python
|
||||
AudioVideoStreamHandler(
|
||||
expected_layout: Literal["mono", "stereo"] = "mono",
|
||||
output_sample_rate: int = 24000,
|
||||
output_frame_size: int | None = None, # Deprecated
|
||||
input_sample_rate: int = 48000,
|
||||
)
|
||||
```
|
||||
|
||||
Abstract base class for synchronous handlers processing both audio and video.
|
||||
|
||||
Inherits from `StreamHandler` (synchronous audio) and adds abstract methods for handling video frames synchronously. Subclasses must implement the audio methods (`receive`, `emit`) and the video methods (`video_receive`, `video_emit`), as well as `copy`.
|
||||
|
||||
*(Inherits Attributes and Methods from `StreamHandler`)*
|
||||
|
||||
### Abstract Methods
|
||||
|
||||
#### `video_receive`
|
||||
|
||||
```python
|
||||
@abstractmethod
|
||||
video_receive(frame: VideoFrame) -> None
|
||||
```
|
||||
|
||||
Process an incoming video frame synchronously.
|
||||
|
||||
**Args:**
|
||||
|
||||
| Name | Type | Description |
|
||||
| :------ | :----------- | :---------------------------- |
|
||||
| `frame` | `VideoFrame` | The incoming aiortc `VideoFrame`. |
|
||||
|
||||
---
|
||||
|
||||
#### `video_emit`
|
||||
|
||||
```python
|
||||
@abstractmethod
|
||||
video_emit() -> VideoEmitType
|
||||
```
|
||||
|
||||
Produce the next output video frame synchronously.
|
||||
|
||||
**Returns:**
|
||||
|
||||
| Type | Description |
|
||||
| :-------------- | :------------------------------------------------------------------------------------------------------- |
|
||||
| `VideoEmitType` | An output item conforming to `VideoEmitType`, typically a numpy array representing the video frame, or None. |
|
||||
|
||||
---
|
||||
|
||||
#### `copy`
|
||||
|
||||
```python
|
||||
@abstractmethod
|
||||
copy() -> AudioVideoStreamHandler
|
||||
```
|
||||
|
||||
Create a copy of this audio-video stream handler instance.
|
||||
|
||||
**Returns:**
|
||||
|
||||
| Type | Description |
|
||||
| :---------------------- | :------------------------------------------------------------------- |
|
||||
| `AudioVideoStreamHandler` | A new instance of the concrete AudioVideoStreamHandler subclass. |
|
||||
|
||||
---
|
||||
|
||||
## `AsyncAudioVideoStreamHandler` Class
|
||||
|
||||
```python
|
||||
AsyncAudioVideoStreamHandler(
|
||||
expected_layout: Literal["mono", "stereo"] = "mono",
|
||||
output_sample_rate: int = 24000,
|
||||
output_frame_size: int | None = None, # Deprecated
|
||||
input_sample_rate: int = 48000,
|
||||
)
|
||||
```
|
||||
|
||||
Abstract base class for asynchronous handlers processing both audio and video.
|
||||
|
||||
Inherits from `AsyncStreamHandler` (asynchronous audio) and adds abstract coroutines for handling video frames asynchronously. Subclasses must implement the async audio methods (`receive`, `emit`, `start_up`) and the async video methods (`video_receive`, `video_emit`), as well as `copy`.
|
||||
|
||||
*(Inherits Attributes and Methods from `AsyncStreamHandler`)*
|
||||
|
||||
### Abstract Methods
|
||||
|
||||
#### `video_receive`
|
||||
|
||||
```python
|
||||
@abstractmethod
|
||||
async video_receive(frame: npt.NDArray[np.float32]) -> None
|
||||
```
|
||||
|
||||
Process an incoming video frame asynchronously.
|
||||
|
||||
**Args:**
|
||||
|
||||
| Name | Type | Description |
|
||||
| :------ | :----------------------- | :------------------------------------------------------------------------------------------------------- |
|
||||
| `frame` | `npt.NDArray[np.float32]` | The video frame data as a numpy array (float32). Note: The type hint differs from the synchronous version. |
|
||||
|
||||
---
|
||||
|
||||
#### `video_emit`
|
||||
|
||||
```python
|
||||
@abstractmethod
|
||||
async video_emit() -> VideoEmitType
|
||||
```
|
||||
|
||||
Produce the next output video frame asynchronously.
|
||||
|
||||
**Returns:**
|
||||
|
||||
| Type | Description |
|
||||
| :-------------- | :------------------------------------------------------------------------------------------------------- |
|
||||
| `VideoEmitType` | An output item conforming to `VideoEmitType`, typically a numpy array representing the video frame, or None. |
|
||||
|
||||
---
|
||||
|
||||
#### `copy`
|
||||
|
||||
```python
|
||||
@abstractmethod
|
||||
copy() -> AsyncAudioVideoStreamHandler
|
||||
```
|
||||
|
||||
Create a copy of this asynchronous audio-video stream handler instance.
|
||||
|
||||
**Returns:**
|
||||
|
||||
| Type | Description |
|
||||
| :--------------------------- | :----------------------------------------------------------------------- |
|
||||
| `AsyncAudioVideoStreamHandler` | A new instance of the concrete AsyncAudioVideoStreamHandler subclass. |
|
||||
123
docs/reference/utils.md
Normal file
123
docs/reference/utils.md
Normal file
@@ -0,0 +1,123 @@
|
||||
# Utils
|
||||
|
||||
## `audio_to_bytes`
|
||||
|
||||
Convert an audio tuple containing sample rate and numpy array data into bytes.
|
||||
Useful for sending data to external APIs from `ReplyOnPause` handler.
|
||||
|
||||
Parameters
|
||||
```
|
||||
audio : tuple[int, np.ndarray]
|
||||
A tuple containing:
|
||||
- sample_rate (int): The audio sample rate in Hz
|
||||
- data (np.ndarray): The audio data as a numpy array
|
||||
```
|
||||
|
||||
Returns
|
||||
```
|
||||
bytes
|
||||
The audio data encoded as bytes, suitable for transmission or storage
|
||||
```
|
||||
|
||||
Example
|
||||
```python
|
||||
>>> sample_rate = 44100
|
||||
>>> audio_data = np.array([0.1, -0.2, 0.3]) # Example audio samples
|
||||
>>> audio_tuple = (sample_rate, audio_data)
|
||||
>>> audio_bytes = audio_to_bytes(audio_tuple)
|
||||
```
|
||||
|
||||
## `audio_to_file`
|
||||
|
||||
Save an audio tuple containing sample rate and numpy array data to a file.
|
||||
|
||||
Parameters
|
||||
```
|
||||
audio : tuple[int, np.ndarray]
|
||||
A tuple containing:
|
||||
- sample_rate (int): The audio sample rate in Hz
|
||||
- data (np.ndarray): The audio data as a numpy array
|
||||
```
|
||||
Returns
|
||||
```
|
||||
str
|
||||
The path to the saved audio file
|
||||
```
|
||||
Example
|
||||
```
|
||||
```python
|
||||
>>> sample_rate = 44100
|
||||
>>> audio_data = np.array([0.1, -0.2, 0.3]) # Example audio samples
|
||||
>>> audio_tuple = (sample_rate, audio_data)
|
||||
>>> file_path = audio_to_file(audio_tuple)
|
||||
>>> print(f"Audio saved to: {file_path}")
|
||||
```
|
||||
|
||||
## `aggregate_bytes_to_16bit`
|
||||
Aggregate bytes to 16-bit audio samples.
|
||||
|
||||
This function takes an iterator of chunks and aggregates them into 16-bit audio samples.
|
||||
It handles incomplete samples and combines them with the next chunk.
|
||||
|
||||
Parameters
|
||||
```
|
||||
chunks_iterator : Iterator[bytes]
|
||||
An iterator of byte chunks to aggregate
|
||||
```
|
||||
Returns
|
||||
```
|
||||
Iterator[NDArray[np.int16]]
|
||||
An iterator of 16-bit audio samples
|
||||
```
|
||||
Example
|
||||
```python
|
||||
>>> chunks_iterator = [b'\x00\x01', b'\x02\x03', b'\x04\x05']
|
||||
>>> for chunk in aggregate_bytes_to_16bit(chunks_iterator):
|
||||
>>> print(chunk)
|
||||
```
|
||||
|
||||
## `async_aggregate_bytes_to_16bit`
|
||||
|
||||
Aggregate bytes to 16-bit audio samples asynchronously.
|
||||
|
||||
Parameters
|
||||
```
|
||||
chunks_iterator : Iterator[bytes]
|
||||
An iterator of byte chunks to aggregate
|
||||
```
|
||||
Returns
|
||||
```
|
||||
Iterator[NDArray[np.int16]]
|
||||
An iterator of 16-bit audio samples
|
||||
```
|
||||
Example
|
||||
```python
|
||||
>>> chunks_iterator = [b'\x00\x01', b'\x02\x03', b'\x04\x05']
|
||||
>>> for chunk in async_aggregate_bytes_to_16bit(chunks_iterator):
|
||||
>>> print(chunk)
|
||||
```
|
||||
|
||||
## `wait_for_item`
|
||||
|
||||
Wait for an item from an asyncio.Queue with a timeout.
|
||||
|
||||
Parameters
|
||||
```
|
||||
queue : asyncio.Queue
|
||||
The queue to wait for an item from
|
||||
timeout : float
|
||||
The timeout in seconds
|
||||
```
|
||||
Returns
|
||||
```
|
||||
Any
|
||||
The item from the queue or None if the timeout is reached
|
||||
```
|
||||
|
||||
Example
|
||||
```python
|
||||
>>> queue = asyncio.Queue()
|
||||
>>> queue.put_nowait(1)
|
||||
>>> item = await wait_for_item(queue)
|
||||
>>> print(item)
|
||||
```
|
||||
Reference in New Issue
Block a user