mirror of
https://github.com/HumanAIGC-Engineering/gradio-webrtc.git
synced 2026-02-04 09:29:23 +08:00
gs对话接入
本次代码评审新增并完善了gs视频聊天功能，包括前后端接口定义、状态管理及UI组件实现，并引入了新的依赖库以支持更多互动特性。 Link: https://code.alibaba-inc.com/xr-paas/gradio_webrtc/codereview/21273476 * 更新python 部分 * 合并videochat前端部分 * Merge branch 'feature/update-fastrtc-0.0.19' of http://gitlab.alibaba-inc.com/xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19 * 替换audiowave * 导入路径修改 * 合并websocket mode逻辑 * feat: gaussian avatar chat * 增加其他渲染的入参 * feat: ws连接和使用 * Merge branch 'feature/update-fastrtc-0.0.19' of http://gitlab.alibaba-inc.com/xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19 * 右边距离超出容器宽度，则向左移动 * 配置传递 * Merge branch 'feature/update-fastrtc-0.0.19' of gitlab.alibaba-inc.com:xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19 * 高斯包异常 * 同步webrtc_utils * 更新webrtc_utils * 兼容on_chat_datachannel * 修复设备名称列表没有正常显示的问题 * copy 传递 webrtc_id * Merge branch 'feature/update-fastrtc-0.0.19' of gitlab.alibaba-inc.com:xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19 * 保证webrtc 完成后再进行websocket连接 * feat: 音频表情数据接入 * dist 上传 * canvas 隐藏 * feat: 高斯文件下载进度透出 * Merge branch 'feature/update-fastrtc-0.0.19' of http://gitlab.alibaba-inc.com/xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19 * 修改无法获取权限问题 * Merge branch 'feature/update-fastrtc-0.0.19' of gitlab.alibaba-inc.com:xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19 * 先获取权限再获取设备 * fix: gs资源下载完成前不处理ws数据 * fix: merge * 话术调整 * Merge branch 'feature/update-fastrtc-0.0.19' of gitlab.alibaba-inc.com:xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19 * 修复设备切换后重新对话，又切换回默认设备的问题 * Merge branch 'feature/update-fastrtc-0.0.19' of http://gitlab.alibaba-inc.com/xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19 * 更新localvideo 尺寸 * Merge branch 'feature/update-fastrtc-0.0.19' of gitlab.alibaba-inc.com:xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19 * 不能默认default * 修改音频权限问题 * 更新打包结果 * fix: 对话按钮状态跟gs资源挂钩，删除无用代码 * fix: merge * feat: gs渲染模块从npm包引入 * fix * 新增对话记录 * Merge branch 'feature/update-fastrtc-0.0.19' of http://gitlab.alibaba-inc.com/xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19 * 样式修改 * 更新包 * fix: gs数字人初始化位置和静音 * 对话记录滚到底部 * 至少100%高度 * Merge branch 'feature/update-fastrtc-0.0.19' of gitlab.alibaba-inc.com:xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19 * 略微上移文本框 * 开始连接时清空对话记录 * fix: update gs render npm * Merge branch 'feature/update-fastrtc-0.0.19' of http://gitlab.alibaba-inc.com/xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19 * 逻辑保证 * Merge branch 'feature/update-fastrtc-0.0.19' of gitlab.alibaba-inc.com:xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19 * feat: 音频初始化配置是否静音 * actionsbar在有字幕时调整位置 * Merge branch 'feature/update-fastrtc-0.0.19' of http://gitlab.alibaba-inc.com/xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19 * 样式优化 * feat: 增加readme * fix: 资源图片 * fix: docs * fix: update gs render sdk * fix: gs模式下画面位置计算 * fix: update readme * 设备判断，太窄处理 * Merge branch 'feature/update-fastrtc-0.0.19' of gitlab.alibaba-inc.com:xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19 * 是否有权限和是否有设备分开 * feat: gs 下载和加载钩子函数分离 * Merge branch 'feature/update-fastrtc-0.0.19' of http://gitlab.alibaba-inc.com/xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19 * fix: update gs render sdk * 替换 * dist * 上传文件 * del
This commit is contained in:
neil.xh
9
.gitignore
vendored
9
.gitignore
vendored
@@ -1,5 +1,6 @@
|
||||
.eggs/
|
||||
dist/
|
||||
dist/*
|
||||
|
||||
*.pyc
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
@@ -17,4 +18,8 @@ demo/scratch
|
||||
.DS_Store
|
||||
test/
|
||||
.venv*
|
||||
.env
|
||||
.env
|
||||
|
||||
!dist/fastrtc-0.0.19.dev0-py3-none-any.whl
|
||||
backend/fastrtc/templates/*
|
||||
frontend/package-lock.json
|
||||
426
README.md
426
README.md
@@ -7,311 +7,191 @@
|
||||
|
||||
<div style="display: flex; flex-direction: row; justify-content: center">
|
||||
<img style="display: block; padding-right: 5px; height: 20px;" alt="Static Badge" src="https://img.shields.io/pypi/v/fastrtc">
|
||||
<a href="https://github.com/freddyaboulton/fastrtc" target="_blank"><img alt="Static Badge" src="https://img.shields.io/badge/github-white?logo=github&logoColor=black"></a>
|
||||
<a href="https://github.com/freddyaboulton/fastrtc" target="_blank"><img alt="Static Badge" src="https://img.shields.io/badge/github-white?logo=github&logoColor=black"></a><a href="https://fastrtc.org//" target="_blank"><img alt="Static Badge" src="https://img.shields.io/badge/Docs-ffcf40"></a>
|
||||
</div>
|
||||
</div>
|
||||
<div align="center">
|
||||
<strong>中文|<a href="./README_EN.md">English</a></strong>
|
||||
</div>
|
||||
|
||||
<h3 style='text-align: center'>
|
||||
The Real-Time Communication Library for Python.
|
||||
</h3>
|
||||
本仓库是从原有的 fastrtc 仓库 fork 而来，主要增加了`video_chat`作为允许的入参，并默认开启，这个模式和原有的`modality="audio-video"`且`mode="send-receive"`的行为保持一致，但重写了 UI 部分，增加了更多的交互能力(更多的麦克风操作，同时展示本地视频信息），其视觉表现如下图。
|
||||
|
||||
如果手动将`video_chat`参数设置为`False`，则其用法与原仓库保持一致 [https://github.com/freddyaboulton/fastrtc/](https://github.com/freddyaboulton/fastrtc)
|
||||
|
||||
|
||||
|
||||
|
||||
## Configuration
|
||||
|参数|默认值|说明|
|
||||
|---|---|---|
|
||||
|video_chat|True|是否开启视频聊天功能|
|
||||
|avatar_type|''|端渲染数字人的类型，目前只支持'gs'|
|
||||
|avatar_ws_route|''|端渲染websocket连接路径|
|
||||
|avatar_assets_path|''|端渲染数字人的模型资源地址|
|
||||
|
||||
Turn any python function into a real-time audio and video stream over WebRTC or WebSockets.
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
pip install fastrtc
|
||||
gradio cc install
|
||||
gradio cc build --no-generate-docs
|
||||
```
|
||||
|
||||
to use built-in pause detection (see [ReplyOnPause](https://fastrtc.org/userguide/audio/#reply-on-pause)), and text to speech (see [Text To Speech](https://fastrtc.org/userguide/audio/#text-to-speech)), install the `vad` and `tts` extras:
|
||||
|
||||
```bash
|
||||
pip install "fastrtc[vad, tts]"
|
||||
pip install dist/fastrtc-0.0.19.dev0-py3-none-any.whl
|
||||
```
|
||||
|
||||
## Key Features
|
||||
|
||||
- 🗣️ Automatic Voice Detection and Turn Taking built-in, only worry about the logic for responding to the user.
|
||||
- 💻 Automatic UI - Use the `.ui.launch()` method to launch the webRTC-enabled built-in Gradio UI.
|
||||
- 🔌 Automatic WebRTC Support - Use the `.mount(app)` method to mount the stream on a FastAPI app and get a webRTC endpoint for your own frontend!
|
||||
- ⚡️ Websocket Support - Use the `.mount(app)` method to mount the stream on a FastAPI app and get a websocket endpoint for your own frontend!
|
||||
- 📞 Automatic Telephone Support - Use the `fastphone()` method of the stream to launch the application and get a free temporary phone number!
|
||||
- 🤖 Completely customizable backend - A `Stream` can easily be mounted on a FastAPI app so you can easily extend it to fit your production application. See the [Talk To Claude](https://huggingface.co/spaces/fastrtc/talk-to-claude) demo for an example on how to serve a custom JS frontend.
|
||||
|
||||
## Docs
|
||||
|
||||
[https://fastrtc.org](https://fastrtc.org)
|
||||
|
||||
## Examples
|
||||
See the [Cookbook](https://fastrtc.org/cookbook/) for examples of how to use the library.
|
||||
|
||||
<table>
|
||||
<tr>
|
||||
<td width="50%">
|
||||
<h3>🗣️👀 Gemini Audio Video Chat</h3>
|
||||
<p>Stream BOTH your webcam video and audio feeds to Google Gemini. You can also upload images to augment your conversation!</p>
|
||||
<video width="100%" src="https://github.com/user-attachments/assets/9636dc97-4fee-46bb-abb8-b92e69c08c71" controls></video>
|
||||
<p>
|
||||
<a href="https://huggingface.co/spaces/freddyaboulton/gemini-audio-video-chat">Demo</a> |
|
||||
<a href="https://huggingface.co/spaces/freddyaboulton/gemini-audio-video-chat/blob/main/app.py">Code</a>
|
||||
</p>
|
||||
</td>
|
||||
<td width="50%">
|
||||
<h3>🗣️ Google Gemini Real Time Voice API</h3>
|
||||
<p>Talk to Gemini in real time using Google's voice API.</p>
|
||||
<video width="100%" src="https://github.com/user-attachments/assets/ea6d18cb-8589-422b-9bba-56332d9f61de" controls></video>
|
||||
<p>
|
||||
<a href="https://huggingface.co/spaces/fastrtc/talk-to-gemini">Demo</a> |
|
||||
<a href="https://huggingface.co/spaces/fastrtc/talk-to-gemini/blob/main/app.py">Code</a>
|
||||
</p>
|
||||
</td>
|
||||
</tr>
|
||||
|
||||
<tr>
|
||||
<td width="50%">
|
||||
<h3>🗣️ OpenAI Real Time Voice API</h3>
|
||||
<p>Talk to ChatGPT in real time using OpenAI's voice API.</p>
|
||||
<video width="100%" src="https://github.com/user-attachments/assets/178bdadc-f17b-461a-8d26-e915c632ff80" controls></video>
|
||||
<p>
|
||||
<a href="https://huggingface.co/spaces/fastrtc/talk-to-openai">Demo</a> |
|
||||
<a href="https://huggingface.co/spaces/fastrtc/talk-to-openai/blob/main/app.py">Code</a>
|
||||
</p>
|
||||
</td>
|
||||
<td width="50%">
|
||||
<h3>🤖 Hello Computer</h3>
|
||||
<p>Say computer before asking your question!</p>
|
||||
<video width="100%" src="https://github.com/user-attachments/assets/afb2a3ef-c1ab-4cfb-872d-578f895a10d5" controls></video>
|
||||
<p>
|
||||
<a href="https://huggingface.co/spaces/fastrtc/hello-computer">Demo</a> |
|
||||
<a href="https://huggingface.co/spaces/fastrtc/hello-computer/blob/main/app.py">Code</a>
|
||||
</p>
|
||||
</td>
|
||||
</tr>
|
||||
|
||||
<tr>
|
||||
<td width="50%">
|
||||
<h3>🤖 Llama Code Editor</h3>
|
||||
<p>Create and edit HTML pages with just your voice! Powered by SambaNova systems.</p>
|
||||
<video width="100%" src="https://github.com/user-attachments/assets/98523cf3-dac8-4127-9649-d91a997e3ef5" controls></video>
|
||||
<p>
|
||||
<a href="https://huggingface.co/spaces/fastrtc/llama-code-editor">Demo</a> |
|
||||
<a href="https://huggingface.co/spaces/fastrtc/llama-code-editor/blob/main/app.py">Code</a>
|
||||
</p>
|
||||
</td>
|
||||
<td width="50%">
|
||||
<h3>🗣️ Talk to Claude</h3>
|
||||
<p>Use the Anthropic and Play.Ht APIs to have an audio conversation with Claude.</p>
|
||||
<video width="100%" src="https://github.com/user-attachments/assets/fb6ef07f-3ccd-444a-997b-9bc9bdc035d3" controls></video>
|
||||
<p>
|
||||
<a href="https://huggingface.co/spaces/fastrtc/talk-to-claude">Demo</a> |
|
||||
<a href="https://huggingface.co/spaces/fastrtc/talk-to-claude/blob/main/app.py">Code</a>
|
||||
</p>
|
||||
</td>
|
||||
</tr>
|
||||
|
||||
<tr>
|
||||
<td width="50%">
|
||||
<h3>🎵 Whisper Transcription</h3>
|
||||
<p>Have whisper transcribe your speech in real time!</p>
|
||||
<video width="100%" src="https://github.com/user-attachments/assets/87603053-acdc-4c8a-810f-f618c49caafb" controls></video>
|
||||
<p>
|
||||
<a href="https://huggingface.co/spaces/fastrtc/whisper-realtime">Demo</a> |
|
||||
<a href="https://huggingface.co/spaces/fastrtc/whisper-realtime/blob/main/app.py">Code</a>
|
||||
</p>
|
||||
</td>
|
||||
<td width="50%">
|
||||
<h3>📷 Yolov10 Object Detection</h3>
|
||||
<p>Run the Yolov10 model on a user webcam stream in real time!</p>
|
||||
<video width="100%" src="https://github.com/user-attachments/assets/f82feb74-a071-4e81-9110-a01989447ceb" controls></video>
|
||||
<p>
|
||||
<a href="https://huggingface.co/spaces/fastrtc/object-detection">Demo</a> |
|
||||
<a href="https://huggingface.co/spaces/fastrtc/object-detection/blob/main/app.py">Code</a>
|
||||
</p>
|
||||
</td>
|
||||
</tr>
|
||||
|
||||
<tr>
|
||||
<td width="50%">
|
||||
<h3>🗣️ Kyutai Moshi</h3>
|
||||
<p>Kyutai's moshi is a novel speech-to-speech model for modeling human conversations.</p>
|
||||
<video width="100%" src="https://github.com/user-attachments/assets/becc7a13-9e89-4a19-9df2-5fb1467a0137" controls></video>
|
||||
<p>
|
||||
<a href="https://huggingface.co/spaces/freddyaboulton/talk-to-moshi">Demo</a> |
|
||||
<a href="https://huggingface.co/spaces/freddyaboulton/talk-to-moshi/blob/main/app.py">Code</a>
|
||||
</p>
|
||||
</td>
|
||||
<td width="50%">
|
||||
<h3>🗣️ Hello Llama: Stop Word Detection</h3>
|
||||
<p>A code editor built with Llama 3.3 70b that is triggered by the phrase "Hello Llama". Build a Siri-like coding assistant in 100 lines of code!</p>
|
||||
<video width="100%" src="https://github.com/user-attachments/assets/3e10cb15-ff1b-4b17-b141-ff0ad852e613" controls></video>
|
||||
<p>
|
||||
<a href="https://huggingface.co/spaces/freddyaboulton/hey-llama-code-editor">Demo</a> |
|
||||
<a href="https://huggingface.co/spaces/freddyaboulton/hey-llama-code-editor/blob/main/app.py">Code</a>
|
||||
</p>
|
||||
</td>
|
||||
</tr>
|
||||
</table>
|
||||
|
||||
## Usage
|
||||
|
||||
This is an shortened version of the official [usage guide](https://freddyaboulton.github.io/gradio-webrtc/user-guide/).
|
||||
|
||||
- `.ui.launch()`: Launch a built-in UI for easily testing and sharing your stream. Built with [Gradio](https://www.gradio.app/).
|
||||
- `.fastphone()`: Get a free temporary phone number to call into your stream. Hugging Face token required.
|
||||
- `.mount(app)`: Mount the stream on a [FastAPI](https://fastapi.tiangolo.com/) app. Perfect for integrating with your already existing production system.
|
||||
|
||||
|
||||
## Quickstart
|
||||
|
||||
### Echo Audio
|
||||
使用时需要一个 handler 作为组件的入参，并实现类似以下代码：
|
||||
|
||||
```python
|
||||
from fastrtc import Stream, ReplyOnPause
|
||||
import numpy as np
|
||||
import asyncio
|
||||
import base64
|
||||
from io import BytesIO
|
||||
|
||||
def echo(audio: tuple[int, np.ndarray]):
|
||||
# The function will be passed the audio until the user pauses
|
||||
# Implement any iterator that yields audio
|
||||
# See "LLM Voice Chat" for a more complete example
|
||||
yield audio
|
||||
|
||||
stream = Stream(
|
||||
handler=ReplyOnPause(echo),
|
||||
modality="audio",
|
||||
mode="send-receive",
|
||||
)
|
||||
```
|
||||
|
||||
### LLM Voice Chat
|
||||
|
||||
```py
|
||||
from fastrtc import (
|
||||
ReplyOnPause, AdditionalOutputs, Stream,
|
||||
audio_to_bytes, aggregate_bytes_to_16bit
|
||||
)
|
||||
import gradio as gr
|
||||
from groq import Groq
|
||||
import anthropic
|
||||
from elevenlabs import ElevenLabs
|
||||
|
||||
groq_client = Groq()
|
||||
claude_client = anthropic.Anthropic()
|
||||
tts_client = ElevenLabs()
|
||||
|
||||
|
||||
# See "Talk to Claude" in Cookbook for an example of how to keep
|
||||
# track of the chat history.
|
||||
def response(
|
||||
audio: tuple[int, np.ndarray],
|
||||
):
|
||||
prompt = groq_client.audio.transcriptions.create(
|
||||
file=("audio-file.mp3", audio_to_bytes(audio)),
|
||||
model="whisper-large-v3-turbo",
|
||||
response_format="verbose_json",
|
||||
).text
|
||||
response = claude_client.messages.create(
|
||||
model="claude-3-5-haiku-20241022",
|
||||
max_tokens=512,
|
||||
messages=[{"role": "user", "content": prompt}],
|
||||
)
|
||||
response_text = " ".join(
|
||||
block.text
|
||||
for block in response.content
|
||||
if getattr(block, "type", None) == "text"
|
||||
)
|
||||
iterator = tts_client.text_to_speech.convert_as_stream(
|
||||
text=response_text,
|
||||
voice_id="JBFqnCBsd6RMkjVDRZzb",
|
||||
model_id="eleven_multilingual_v2",
|
||||
output_format="pcm_24000"
|
||||
|
||||
)
|
||||
for chunk in aggregate_bytes_to_16bit(iterator):
|
||||
audio_array = np.frombuffer(chunk, dtype=np.int16).reshape(1, -1)
|
||||
yield (24000, audio_array)
|
||||
|
||||
stream = Stream(
|
||||
modality="audio",
|
||||
mode="send-receive",
|
||||
handler=ReplyOnPause(response),
|
||||
)
|
||||
```
|
||||
|
||||
### Webcam Stream
|
||||
|
||||
```python
|
||||
from fastrtc import Stream
|
||||
import numpy as np
|
||||
|
||||
|
||||
def flip_vertically(image):
|
||||
return np.flip(image, axis=0)
|
||||
|
||||
|
||||
stream = Stream(
|
||||
handler=flip_vertically,
|
||||
modality="video",
|
||||
mode="send-receive",
|
||||
from gradio_webrtc import (
|
||||
AsyncAudioVideoStreamHandler,
|
||||
WebRTC,
|
||||
VideoEmitType,
|
||||
AudioEmitType,
|
||||
)
|
||||
from PIL import Image
|
||||
|
||||
|
||||
def encode_audio(data: np.ndarray) -> dict:
|
||||
"""Encode Audio data to send to the server"""
|
||||
return {"mime_type": "audio/pcm", "data": base64.b64encode(data.tobytes()).decode("UTF-8")}
|
||||
|
||||
|
||||
def encode_image(data: np.ndarray) -> dict:
|
||||
with BytesIO() as output_bytes:
|
||||
pil_image = Image.fromarray(data)
|
||||
pil_image.save(output_bytes, "JPEG")
|
||||
bytes_data = output_bytes.getvalue()
|
||||
base64_str = str(base64.b64encode(bytes_data), "utf-8")
|
||||
return {"mime_type": "image/jpeg", "data": base64_str}
|
||||
|
||||
|
||||
class VideoChatHandler(AsyncAudioVideoStreamHandler):
|
||||
def __init__(
|
||||
self, expected_layout="mono", output_sample_rate=24000, output_frame_size=480
|
||||
) -> None:
|
||||
super().__init__(
|
||||
expected_layout,
|
||||
output_sample_rate,
|
||||
output_frame_size,
|
||||
input_sample_rate=24000,
|
||||
)
|
||||
self.audio_queue = asyncio.Queue()
|
||||
self.video_queue = asyncio.Queue()
|
||||
self.quit = asyncio.Event()
|
||||
self.session = None
|
||||
self.last_frame_time = 0
|
||||
|
||||
def copy(self) -> "VideoChatHandler":
|
||||
return VideoChatHandler(
|
||||
expected_layout=self.expected_layout,
|
||||
output_sample_rate=self.output_sample_rate,
|
||||
output_frame_size=self.output_frame_size,
|
||||
)
|
||||
|
||||
#处理客户端上传的视频数据
|
||||
async def video_receive(self, frame: np.ndarray):
|
||||
newFrame = np.array(frame)
|
||||
newFrame[0:, :, 0] = 255 - newFrame[0:, :, 0]
|
||||
self.video_queue.put_nowait(newFrame)
|
||||
|
||||
#准备服务端下发的视频数据
|
||||
async def video_emit(self) -> VideoEmitType:
|
||||
return await self.video_queue.get()
|
||||
|
||||
#处理客户端上传的音频数据
|
||||
async def receive(self, frame: tuple[int, np.ndarray]) -> None:
|
||||
frame_size, array = frame
|
||||
self.audio_queue.put_nowait(array)
|
||||
|
||||
#准备服务端下发的音频数据
|
||||
async def emit(self) -> AudioEmitType:
|
||||
if not self.args_set.is_set():
|
||||
await self.wait_for_args()
|
||||
array = await self.audio_queue.get()
|
||||
return (self.output_sample_rate, array)
|
||||
|
||||
def shutdown(self) -> None:
|
||||
self.quit.set()
|
||||
self.connection = None
|
||||
self.args_set.clear()
|
||||
self.quit.clear()
|
||||
|
||||
|
||||
|
||||
css = """
|
||||
footer {
|
||||
display: none !important;
|
||||
}
|
||||
"""
|
||||
|
||||
with gr.Blocks(css=css) as demo:
|
||||
webrtc = WebRTC(
|
||||
label="Video Chat",
|
||||
modality="audio-video",
|
||||
mode="send-receive",
|
||||
video_chat=True,
|
||||
elem_id="video-source",
|
||||
)
|
||||
webrtc.stream(
|
||||
VideoChatHandler(),
|
||||
inputs=[webrtc],
|
||||
outputs=[webrtc],
|
||||
time_limit=150,
|
||||
concurrency_limit=2,
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
demo.launch()
|
||||
|
||||
```
|
||||
|
||||
### Object Detection
|
||||
## Deployment
|
||||
|
||||
在云环境中部署（例如 huggingface，EC2 等）时，您需要设置转向服务器以中继 WEBRTC 流量。
|
||||
最简单的方法是使用 Twilio 之类的服务。国内部署需要寻找适合的替代方案。
|
||||
|
||||
```python
|
||||
from fastrtc import Stream
|
||||
import gradio as gr
|
||||
import cv2
|
||||
from huggingface_hub import hf_hub_download
|
||||
from .inference import YOLOv10
|
||||
from twilio.rest import Client
|
||||
import os
|
||||
|
||||
model_file = hf_hub_download(
|
||||
repo_id="onnx-community/yolov10n", filename="onnx/model.onnx"
|
||||
)
|
||||
account_sid = os.environ.get("TWILIO_ACCOUNT_SID")
|
||||
auth_token = os.environ.get("TWILIO_AUTH_TOKEN")
|
||||
|
||||
# git clone https://huggingface.co/spaces/fastrtc/object-detection
|
||||
# for YOLOv10 implementation
|
||||
model = YOLOv10(model_file)
|
||||
client = Client(account_sid, auth_token)
|
||||
|
||||
def detection(image, conf_threshold=0.3):
|
||||
image = cv2.resize(image, (model.input_width, model.input_height))
|
||||
new_image = model.detect_objects(image, conf_threshold)
|
||||
return cv2.resize(new_image, (500, 500))
|
||||
token = client.tokens.create()
|
||||
|
||||
stream = Stream(
|
||||
handler=detection,
|
||||
modality="video",
|
||||
mode="send-receive",
|
||||
additional_inputs=[
|
||||
gr.Slider(minimum=0, maximum=1, step=0.01, value=0.3)
|
||||
]
|
||||
)
|
||||
rtc_configuration = {
|
||||
"iceServers": token.ice_servers,
|
||||
"iceTransportPolicy": "relay",
|
||||
}
|
||||
|
||||
with gr.Blocks() as demo:
|
||||
...
|
||||
rtc = WebRTC(rtc_configuration=rtc_configuration, ...)
|
||||
...
|
||||
```
|
||||
|
||||
## Running the Stream
|
||||
## Contributors
|
||||
|
||||
Run:
|
||||
|
||||
### Gradio
|
||||
|
||||
```py
|
||||
stream.ui.launch()
|
||||
```
|
||||
|
||||
### Telephone (Audio Only)
|
||||
|
||||
```py
|
||||
stream.fastphone()
|
||||
```
|
||||
|
||||
### FastAPI
|
||||
|
||||
```py
|
||||
app = FastAPI()
|
||||
stream.mount(app)
|
||||
|
||||
# Optional: Add routes
|
||||
@app.get("/")
|
||||
async def _():
|
||||
return HTMLResponse(content=open("index.html").read())
|
||||
|
||||
# uvicorn app:app --host 0.0.0.0 --port 8000
|
||||
```
|
||||
[csxh47](https://github.com/xhup)
|
||||
[bingochaos](https://github.com/bingochaos)
|
||||
[sudowind](https://github.com/sudowind)
|
||||
[emililykimura](https://github.com/emililykimura)
|
||||
[Tony](https://github.com/raidios)
|
||||
[Cheng Gang](https://github.com/lovepope)
|
||||
190
README_EN.md
Normal file
190
README_EN.md
Normal file
@@ -0,0 +1,190 @@
|
||||
<h1 style='text-align: center; margin-bottom: 1rem'> Gradio WebRTC ⚡️ </h1>
|
||||
|
||||
<div style="display: flex; flex-direction: row; justify-content: center">
|
||||
<img style="display: block; padding-right: 5px; height: 20px;" alt="Static Badge" src="https://img.shields.io/pypi/v/gradio_webrtc">
|
||||
<a href="https://github.com/freddyaboulton/fastrtc" target="_blank"><img alt="Static Badge" style="display: block; padding-right: 5px; height: 20px;" src="https://img.shields.io/badge/github-white?logo=github&logoColor=black"></a>
|
||||
<a href="https://fastrtc.org//" target="_blank"><img alt="Static Badge" src="https://img.shields.io/badge/Docs-ffcf40"></a>
|
||||
</div>
|
||||
<div align="center">
|
||||
<strong><a href="./README.md">中文</a>|English</strong>
|
||||
</div>
|
||||
This repository is forked from the original gradio_webrtc repository, primarily adding `video_chat` as an allowed parameter to be enabled by default. This mode is consistent with the behavior of the original `modality="audio-video"` and `mode="send-receive"`, but the UI has been rewritten to include more interactive capabilities (more microphone controls, and the ability to display local video information). The visual presentation is shown below.
|
||||
|
||||
If `video_chat` is manually set to `False`, its usage is consistent with the original repository https://github.com/freddyaboulton/fastrtc/
|
||||
|
||||
|
||||
|
||||
|
||||
## Configuration
|
||||
|parameter|default|describe|
|
||||
|---|---|---|
|
||||
|video_chat|True|enable video chat|
|
||||
|avatar_type|''|local avatar type, only supports 'gs' now|
|
||||
|avatar_ws_route|''|websocket connection path for local avatar|
|
||||
|avatar_assets_path|''|local avatar assets path|
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
gradio cc install
|
||||
gradio cc build --no-generate-docs
|
||||
```
|
||||
|
||||
```bash
|
||||
pip install dist/fastrtc-0.0.19.dev0-py3-none-any.whl
|
||||
```
|
||||
|
||||
## Docs
|
||||
|
||||
[https://fastrtc.org](https://fastrtc.org)
|
||||
|
||||
## Examples
|
||||
|
||||
When using it, you need a handler as the entry parameter of the component and implement code similar to the following:
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
import base64
|
||||
from io import BytesIO
|
||||
|
||||
import gradio as gr
|
||||
import numpy as np
|
||||
from gradio_webrtc import (
|
||||
AsyncAudioVideoStreamHandler,
|
||||
WebRTC,
|
||||
VideoEmitType,
|
||||
AudioEmitType,
|
||||
)
|
||||
from PIL import Image
|
||||
|
||||
|
||||
def encode_audio(data: np.ndarray) -> dict:
|
||||
"""Encode Audio data to send to the server"""
|
||||
return {"mime_type": "audio/pcm", "data": base64.b64encode(data.tobytes()).decode("UTF-8")}
|
||||
|
||||
|
||||
def encode_image(data: np.ndarray) -> dict:
|
||||
with BytesIO() as output_bytes:
|
||||
pil_image = Image.fromarray(data)
|
||||
pil_image.save(output_bytes, "JPEG")
|
||||
bytes_data = output_bytes.getvalue()
|
||||
base64_str = str(base64.b64encode(bytes_data), "utf-8")
|
||||
return {"mime_type": "image/jpeg", "data": base64_str}
|
||||
|
||||
|
||||
class VideoChatHandler(AsyncAudioVideoStreamHandler):
|
||||
def __init__(
|
||||
self, expected_layout="mono", output_sample_rate=24000, output_frame_size=480
|
||||
) -> None:
|
||||
super().__init__(
|
||||
expected_layout,
|
||||
output_sample_rate,
|
||||
output_frame_size,
|
||||
input_sample_rate=24000,
|
||||
)
|
||||
self.audio_queue = asyncio.Queue()
|
||||
self.video_queue = asyncio.Queue()
|
||||
self.quit = asyncio.Event()
|
||||
self.session = None
|
||||
self.last_frame_time = 0
|
||||
|
||||
def copy(self) -> "VideoChatHandler":
|
||||
return VideoChatHandler(
|
||||
expected_layout=self.expected_layout,
|
||||
output_sample_rate=self.output_sample_rate,
|
||||
output_frame_size=self.output_frame_size,
|
||||
)
|
||||
|
||||
#Process video data uploaded by the client
|
||||
async def video_receive(self, frame: np.ndarray):
|
||||
newFrame = np.array(frame)
|
||||
newFrame[0:, :, 0] = 255 - newFrame[0:, :, 0]
|
||||
self.video_queue.put_nowait(newFrame)
|
||||
|
||||
#Prepare the video data sent by the server
|
||||
async def video_emit(self) -> VideoEmitType:
|
||||
return await self.video_queue.get()
|
||||
|
||||
#Process audio data uploaded by the client
|
||||
async def receive(self, frame: tuple[int, np.ndarray]) -> None:
|
||||
frame_size, array = frame
|
||||
self.audio_queue.put_nowait(array)
|
||||
|
||||
#Prepare the audio data sent by the server
|
||||
async def emit(self) -> AudioEmitType:
|
||||
if not self.args_set.is_set():
|
||||
await self.wait_for_args()
|
||||
array = await self.audio_queue.get()
|
||||
return (self.output_sample_rate, array)
|
||||
|
||||
def shutdown(self) -> None:
|
||||
self.quit.set()
|
||||
self.connection = None
|
||||
self.args_set.clear()
|
||||
self.quit.clear()
|
||||
|
||||
|
||||
|
||||
css = """
|
||||
footer {
|
||||
display: none !important;
|
||||
}
|
||||
"""
|
||||
|
||||
with gr.Blocks(css=css) as demo:
|
||||
webrtc = WebRTC(
|
||||
label="Video Chat",
|
||||
modality="audio-video",
|
||||
mode="send-receive",
|
||||
video_chat=True,
|
||||
elem_id="video-source",
|
||||
)
|
||||
webrtc.stream(
|
||||
VideoChatHandler(),
|
||||
inputs=[webrtc],
|
||||
outputs=[webrtc],
|
||||
time_limit=150,
|
||||
concurrency_limit=2,
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
demo.launch()
|
||||
|
||||
```
|
||||
|
||||
## Deployment
|
||||
|
||||
When deploying in a cloud environment (like Hugging Face Spaces, EC2, etc), you need to set up a TURN server to relay the WebRTC traffic.
|
||||
The easiest way to do this is to use a service like Twilio.
|
||||
|
||||
```python
|
||||
from twilio.rest import Client
|
||||
import os
|
||||
|
||||
account_sid = os.environ.get("TWILIO_ACCOUNT_SID")
|
||||
auth_token = os.environ.get("TWILIO_AUTH_TOKEN")
|
||||
|
||||
client = Client(account_sid, auth_token)
|
||||
|
||||
token = client.tokens.create()
|
||||
|
||||
rtc_configuration = {
|
||||
"iceServers": token.ice_servers,
|
||||
"iceTransportPolicy": "relay",
|
||||
}
|
||||
|
||||
with gr.Blocks() as demo:
|
||||
...
|
||||
rtc = WebRTC(rtc_configuration=rtc_configuration, ...)
|
||||
...
|
||||
```
|
||||
|
||||
## Contributors
|
||||
|
||||
[csxh47](https://github.com/xhup)
|
||||
[bingochaos](https://github.com/bingochaos)
|
||||
[sudowind](https://github.com/sudowind)
|
||||
[emililykimura](https://github.com/emililykimura)
|
||||
[Tony](https://github.com/raidios)
|
||||
[Cheng Gang](https://github.com/lovepope)
|
||||
317
README_FASTRTC.md
Normal file
317
README_FASTRTC.md
Normal file
@@ -0,0 +1,317 @@
|
||||
<div style='text-align: center; margin-bottom: 1rem; display: flex; justify-content: center; align-items: center;'>
|
||||
<h1 style='color: white; margin: 0;'>FastRTC</h1>
|
||||
<img src='https://huggingface.co/datasets/freddyaboulton/bucket/resolve/main/fastrtc_logo_small.png'
|
||||
alt="FastRTC Logo"
|
||||
style="margin-right: 10px;">
|
||||
</div>
|
||||
|
||||
<div style="display: flex; flex-direction: row; justify-content: center">
|
||||
<img style="display: block; padding-right: 5px; height: 20px;" alt="Static Badge" src="https://img.shields.io/pypi/v/fastrtc">
|
||||
<a href="https://github.com/freddyaboulton/fastrtc" target="_blank"><img alt="Static Badge" src="https://img.shields.io/badge/github-white?logo=github&logoColor=black"></a>
|
||||
</div>
|
||||
|
||||
<h3 style='text-align: center'>
|
||||
The Real-Time Communication Library for Python.
|
||||
</h3>
|
||||
|
||||
Turn any python function into a real-time audio and video stream over WebRTC or WebSockets.
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
pip install fastrtc
|
||||
```
|
||||
|
||||
to use built-in pause detection (see [ReplyOnPause](https://fastrtc.org/userguide/audio/#reply-on-pause)), and text to speech (see [Text To Speech](https://fastrtc.org/userguide/audio/#text-to-speech)), install the `vad` and `tts` extras:
|
||||
|
||||
```bash
|
||||
pip install "fastrtc[vad, tts]"
|
||||
```
|
||||
|
||||
## Key Features
|
||||
|
||||
- 🗣️ Automatic Voice Detection and Turn Taking built-in, only worry about the logic for responding to the user.
|
||||
- 💻 Automatic UI - Use the `.ui.launch()` method to launch the webRTC-enabled built-in Gradio UI.
|
||||
- 🔌 Automatic WebRTC Support - Use the `.mount(app)` method to mount the stream on a FastAPI app and get a webRTC endpoint for your own frontend!
|
||||
- ⚡️ Websocket Support - Use the `.mount(app)` method to mount the stream on a FastAPI app and get a websocket endpoint for your own frontend!
|
||||
- 📞 Automatic Telephone Support - Use the `fastphone()` method of the stream to launch the application and get a free temporary phone number!
|
||||
- 🤖 Completely customizable backend - A `Stream` can easily be mounted on a FastAPI app so you can easily extend it to fit your production application. See the [Talk To Claude](https://huggingface.co/spaces/fastrtc/talk-to-claude) demo for an example on how to serve a custom JS frontend.
|
||||
|
||||
## Docs
|
||||
|
||||
[https://fastrtc.org](https://fastrtc.org)
|
||||
|
||||
## Examples
|
||||
See the [Cookbook](https://fastrtc.org/cookbook/) for examples of how to use the library.
|
||||
|
||||
<table>
|
||||
<tr>
|
||||
<td width="50%">
|
||||
<h3>🗣️👀 Gemini Audio Video Chat</h3>
|
||||
<p>Stream BOTH your webcam video and audio feeds to Google Gemini. You can also upload images to augment your conversation!</p>
|
||||
<video width="100%" src="https://github.com/user-attachments/assets/9636dc97-4fee-46bb-abb8-b92e69c08c71" controls></video>
|
||||
<p>
|
||||
<a href="https://huggingface.co/spaces/freddyaboulton/gemini-audio-video-chat">Demo</a> |
|
||||
<a href="https://huggingface.co/spaces/freddyaboulton/gemini-audio-video-chat/blob/main/app.py">Code</a>
|
||||
</p>
|
||||
</td>
|
||||
<td width="50%">
|
||||
<h3>🗣️ Google Gemini Real Time Voice API</h3>
|
||||
<p>Talk to Gemini in real time using Google's voice API.</p>
|
||||
<video width="100%" src="https://github.com/user-attachments/assets/ea6d18cb-8589-422b-9bba-56332d9f61de" controls></video>
|
||||
<p>
|
||||
<a href="https://huggingface.co/spaces/fastrtc/talk-to-gemini">Demo</a> |
|
||||
<a href="https://huggingface.co/spaces/fastrtc/talk-to-gemini/blob/main/app.py">Code</a>
|
||||
</p>
|
||||
</td>
|
||||
</tr>
|
||||
|
||||
<tr>
|
||||
<td width="50%">
|
||||
<h3>🗣️ OpenAI Real Time Voice API</h3>
|
||||
<p>Talk to ChatGPT in real time using OpenAI's voice API.</p>
|
||||
<video width="100%" src="https://github.com/user-attachments/assets/178bdadc-f17b-461a-8d26-e915c632ff80" controls></video>
|
||||
<p>
|
||||
<a href="https://huggingface.co/spaces/fastrtc/talk-to-openai">Demo</a> |
|
||||
<a href="https://huggingface.co/spaces/fastrtc/talk-to-openai/blob/main/app.py">Code</a>
|
||||
</p>
|
||||
</td>
|
||||
<td width="50%">
|
||||
<h3>🤖 Hello Computer</h3>
|
||||
<p>Say computer before asking your question!</p>
|
||||
<video width="100%" src="https://github.com/user-attachments/assets/afb2a3ef-c1ab-4cfb-872d-578f895a10d5" controls></video>
|
||||
<p>
|
||||
<a href="https://huggingface.co/spaces/fastrtc/hello-computer">Demo</a> |
|
||||
<a href="https://huggingface.co/spaces/fastrtc/hello-computer/blob/main/app.py">Code</a>
|
||||
</p>
|
||||
</td>
|
||||
</tr>
|
||||
|
||||
<tr>
|
||||
<td width="50%">
|
||||
<h3>🤖 Llama Code Editor</h3>
|
||||
<p>Create and edit HTML pages with just your voice! Powered by SambaNova systems.</p>
|
||||
<video width="100%" src="https://github.com/user-attachments/assets/98523cf3-dac8-4127-9649-d91a997e3ef5" controls></video>
|
||||
<p>
|
||||
<a href="https://huggingface.co/spaces/fastrtc/llama-code-editor">Demo</a> |
|
||||
<a href="https://huggingface.co/spaces/fastrtc/llama-code-editor/blob/main/app.py">Code</a>
|
||||
</p>
|
||||
</td>
|
||||
<td width="50%">
|
||||
<h3>🗣️ Talk to Claude</h3>
|
||||
<p>Use the Anthropic and Play.Ht APIs to have an audio conversation with Claude.</p>
|
||||
<video width="100%" src="https://github.com/user-attachments/assets/fb6ef07f-3ccd-444a-997b-9bc9bdc035d3" controls></video>
|
||||
<p>
|
||||
<a href="https://huggingface.co/spaces/fastrtc/talk-to-claude">Demo</a> |
|
||||
<a href="https://huggingface.co/spaces/fastrtc/talk-to-claude/blob/main/app.py">Code</a>
|
||||
</p>
|
||||
</td>
|
||||
</tr>
|
||||
|
||||
<tr>
|
||||
<td width="50%">
|
||||
<h3>🎵 Whisper Transcription</h3>
|
||||
<p>Have whisper transcribe your speech in real time!</p>
|
||||
<video width="100%" src="https://github.com/user-attachments/assets/87603053-acdc-4c8a-810f-f618c49caafb" controls></video>
|
||||
<p>
|
||||
<a href="https://huggingface.co/spaces/fastrtc/whisper-realtime">Demo</a> |
|
||||
<a href="https://huggingface.co/spaces/fastrtc/whisper-realtime/blob/main/app.py">Code</a>
|
||||
</p>
|
||||
</td>
|
||||
<td width="50%">
|
||||
<h3>📷 Yolov10 Object Detection</h3>
|
||||
<p>Run the Yolov10 model on a user webcam stream in real time!</p>
|
||||
<video width="100%" src="https://github.com/user-attachments/assets/f82feb74-a071-4e81-9110-a01989447ceb" controls></video>
|
||||
<p>
|
||||
<a href="https://huggingface.co/spaces/fastrtc/object-detection">Demo</a> |
|
||||
<a href="https://huggingface.co/spaces/fastrtc/object-detection/blob/main/app.py">Code</a>
|
||||
</p>
|
||||
</td>
|
||||
</tr>
|
||||
|
||||
<tr>
|
||||
<td width="50%">
|
||||
<h3>🗣️ Kyutai Moshi</h3>
|
||||
<p>Kyutai's moshi is a novel speech-to-speech model for modeling human conversations.</p>
|
||||
<video width="100%" src="https://github.com/user-attachments/assets/becc7a13-9e89-4a19-9df2-5fb1467a0137" controls></video>
|
||||
<p>
|
||||
<a href="https://huggingface.co/spaces/freddyaboulton/talk-to-moshi">Demo</a> |
|
||||
<a href="https://huggingface.co/spaces/freddyaboulton/talk-to-moshi/blob/main/app.py">Code</a>
|
||||
</p>
|
||||
</td>
|
||||
<td width="50%">
|
||||
<h3>🗣️ Hello Llama: Stop Word Detection</h3>
|
||||
<p>A code editor built with Llama 3.3 70b that is triggered by the phrase "Hello Llama". Build a Siri-like coding assistant in 100 lines of code!</p>
|
||||
<video width="100%" src="https://github.com/user-attachments/assets/3e10cb15-ff1b-4b17-b141-ff0ad852e613" controls></video>
|
||||
<p>
|
||||
<a href="https://huggingface.co/spaces/freddyaboulton/hey-llama-code-editor">Demo</a> |
|
||||
<a href="https://huggingface.co/spaces/freddyaboulton/hey-llama-code-editor/blob/main/app.py">Code</a>
|
||||
</p>
|
||||
</td>
|
||||
</tr>
|
||||
</table>
|
||||
|
||||
## Usage
|
||||
|
||||
This is an shortened version of the official [usage guide](https://freddyaboulton.github.io/gradio-webrtc/user-guide/).
|
||||
|
||||
- `.ui.launch()`: Launch a built-in UI for easily testing and sharing your stream. Built with [Gradio](https://www.gradio.app/).
|
||||
- `.fastphone()`: Get a free temporary phone number to call into your stream. Hugging Face token required.
|
||||
- `.mount(app)`: Mount the stream on a [FastAPI](https://fastapi.tiangolo.com/) app. Perfect for integrating with your already existing production system.
|
||||
|
||||
|
||||
## Quickstart
|
||||
|
||||
### Echo Audio
|
||||
|
||||
```python
|
||||
from fastrtc import Stream, ReplyOnPause
|
||||
import numpy as np
|
||||
|
||||
def echo(audio: tuple[int, np.ndarray]):
|
||||
# The function will be passed the audio until the user pauses
|
||||
# Implement any iterator that yields audio
|
||||
# See "LLM Voice Chat" for a more complete example
|
||||
yield audio
|
||||
|
||||
stream = Stream(
|
||||
handler=ReplyOnPause(echo),
|
||||
modality="audio",
|
||||
mode="send-receive",
|
||||
)
|
||||
```
|
||||
|
||||
### LLM Voice Chat
|
||||
|
||||
```py
|
||||
from fastrtc import (
|
||||
ReplyOnPause, AdditionalOutputs, Stream,
|
||||
audio_to_bytes, aggregate_bytes_to_16bit
|
||||
)
|
||||
import gradio as gr
|
||||
from groq import Groq
|
||||
import anthropic
|
||||
from elevenlabs import ElevenLabs
|
||||
|
||||
groq_client = Groq()
|
||||
claude_client = anthropic.Anthropic()
|
||||
tts_client = ElevenLabs()
|
||||
|
||||
|
||||
# See "Talk to Claude" in Cookbook for an example of how to keep
|
||||
# track of the chat history.
|
||||
def response(
|
||||
audio: tuple[int, np.ndarray],
|
||||
):
|
||||
prompt = groq_client.audio.transcriptions.create(
|
||||
file=("audio-file.mp3", audio_to_bytes(audio)),
|
||||
model="whisper-large-v3-turbo",
|
||||
response_format="verbose_json",
|
||||
).text
|
||||
response = claude_client.messages.create(
|
||||
model="claude-3-5-haiku-20241022",
|
||||
max_tokens=512,
|
||||
messages=[{"role": "user", "content": prompt}],
|
||||
)
|
||||
response_text = " ".join(
|
||||
block.text
|
||||
for block in response.content
|
||||
if getattr(block, "type", None) == "text"
|
||||
)
|
||||
iterator = tts_client.text_to_speech.convert_as_stream(
|
||||
text=response_text,
|
||||
voice_id="JBFqnCBsd6RMkjVDRZzb",
|
||||
model_id="eleven_multilingual_v2",
|
||||
output_format="pcm_24000"
|
||||
|
||||
)
|
||||
for chunk in aggregate_bytes_to_16bit(iterator):
|
||||
audio_array = np.frombuffer(chunk, dtype=np.int16).reshape(1, -1)
|
||||
yield (24000, audio_array)
|
||||
|
||||
stream = Stream(
|
||||
modality="audio",
|
||||
mode="send-receive",
|
||||
handler=ReplyOnPause(response),
|
||||
)
|
||||
```
|
||||
|
||||
### Webcam Stream
|
||||
|
||||
```python
|
||||
from fastrtc import Stream
|
||||
import numpy as np
|
||||
|
||||
|
||||
def flip_vertically(image):
|
||||
return np.flip(image, axis=0)
|
||||
|
||||
|
||||
stream = Stream(
|
||||
handler=flip_vertically,
|
||||
modality="video",
|
||||
mode="send-receive",
|
||||
)
|
||||
```
|
||||
|
||||
### Object Detection
|
||||
|
||||
```python
|
||||
from fastrtc import Stream
|
||||
import gradio as gr
|
||||
import cv2
|
||||
from huggingface_hub import hf_hub_download
|
||||
from .inference import YOLOv10
|
||||
|
||||
model_file = hf_hub_download(
|
||||
repo_id="onnx-community/yolov10n", filename="onnx/model.onnx"
|
||||
)
|
||||
|
||||
# git clone https://huggingface.co/spaces/fastrtc/object-detection
|
||||
# for YOLOv10 implementation
|
||||
model = YOLOv10(model_file)
|
||||
|
||||
def detection(image, conf_threshold=0.3):
|
||||
image = cv2.resize(image, (model.input_width, model.input_height))
|
||||
new_image = model.detect_objects(image, conf_threshold)
|
||||
return cv2.resize(new_image, (500, 500))
|
||||
|
||||
stream = Stream(
|
||||
handler=detection,
|
||||
modality="video",
|
||||
mode="send-receive",
|
||||
additional_inputs=[
|
||||
gr.Slider(minimum=0, maximum=1, step=0.01, value=0.3)
|
||||
]
|
||||
)
|
||||
```
|
||||
|
||||
## Running the Stream
|
||||
|
||||
Run:
|
||||
|
||||
### Gradio
|
||||
|
||||
```py
|
||||
stream.ui.launch()
|
||||
```
|
||||
|
||||
### Telephone (Audio Only)
|
||||
|
||||
```py
|
||||
stream.fastphone()
|
||||
```
|
||||
|
||||
### FastAPI
|
||||
|
||||
```py
|
||||
app = FastAPI()
|
||||
stream.mount(app)
|
||||
|
||||
# Optional: Add routes
|
||||
@app.get("/")
|
||||
async def _():
|
||||
return HTMLResponse(content=open("index.html").read())
|
||||
|
||||
# uvicorn app:app --host 0.0.0.0 --port 8000
|
||||
```
|
||||
@@ -1 +0,0 @@
|
||||
(function(){"use strict";const R="https://unpkg.com/@ffmpeg/core@0.12.6/dist/umd/ffmpeg-core.js";var E;(function(t){t.LOAD="LOAD",t.EXEC="EXEC",t.WRITE_FILE="WRITE_FILE",t.READ_FILE="READ_FILE",t.DELETE_FILE="DELETE_FILE",t.RENAME="RENAME",t.CREATE_DIR="CREATE_DIR",t.LIST_DIR="LIST_DIR",t.DELETE_DIR="DELETE_DIR",t.ERROR="ERROR",t.DOWNLOAD="DOWNLOAD",t.PROGRESS="PROGRESS",t.LOG="LOG",t.MOUNT="MOUNT",t.UNMOUNT="UNMOUNT"})(E||(E={}));const a=new Error("unknown message type"),f=new Error("ffmpeg is not loaded, call `await ffmpeg.load()` first"),u=new Error("failed to import ffmpeg-core.js");let r;const O=async({coreURL:t,wasmURL:n,workerURL:e})=>{const o=!r;try{t||(t=R),importScripts(t)}catch{if(t||(t=R.replace("/umd/","/esm/")),self.createFFmpegCore=(await import(t)).default,!self.createFFmpegCore)throw u}const s=t,c=n||t.replace(/.js$/g,".wasm"),b=e||t.replace(/.js$/g,".worker.js");return r=await self.createFFmpegCore({mainScriptUrlOrBlob:`${s}#${btoa(JSON.stringify({wasmURL:c,workerURL:b}))}`}),r.setLogger(i=>self.postMessage({type:E.LOG,data:i})),r.setProgress(i=>self.postMessage({type:E.PROGRESS,data:i})),o},l=({args:t,timeout:n=-1})=>{r.setTimeout(n),r.exec(...t);const e=r.ret;return r.reset(),e},m=({path:t,data:n})=>(r.FS.writeFile(t,n),!0),D=({path:t,encoding:n})=>r.FS.readFile(t,{encoding:n}),S=({path:t})=>(r.FS.unlink(t),!0),I=({oldPath:t,newPath:n})=>(r.FS.rename(t,n),!0),L=({path:t})=>(r.FS.mkdir(t),!0),N=({path:t})=>{const n=r.FS.readdir(t),e=[];for(const o of n){const s=r.FS.stat(`${t}/${o}`),c=r.FS.isDir(s.mode);e.push({name:o,isDir:c})}return e},A=({path:t})=>(r.FS.rmdir(t),!0),w=({fsType:t,options:n,mountPoint:e})=>{const o=t,s=r.FS.filesystems[o];return s?(r.FS.mount(s,n,e),!0):!1},k=({mountPoint:t})=>(r.FS.unmount(t),!0);self.onmessage=async({data:{id:t,type:n,data:e}})=>{const o=[];let s;try{if(n!==E.LOAD&&!r)throw f;switch(n){case E.LOAD:s=await O(e);break;case E.EXEC:s=l(e);break;case E.WRITE_FILE:s=m(e);break;case E.READ_FILE:s=D(e);break;case E.DELETE_FILE:s=S(e);break;case E.RENAME:s=I(e);break;case E.CREATE_DIR:s=L(e);break;case E.LIST_DIR:s=N(e);break;case E.DELETE_DIR:s=A(e);break;case E.MOUNT:s=w(e);break;case E.UNMOUNT:s=k(e);break;default:throw a}}catch(c){self.postMessage({id:t,type:E.ERROR,data:c.toString()});return}s instanceof Uint8Array&&o.push(s.buffer),self.postMessage({id:t,type:n,data:s},o)}})();
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -1 +0,0 @@
|
||||
(function(){"use strict";const R="https://unpkg.com/@ffmpeg/core@0.12.6/dist/umd/ffmpeg-core.js";var E;(function(t){t.LOAD="LOAD",t.EXEC="EXEC",t.WRITE_FILE="WRITE_FILE",t.READ_FILE="READ_FILE",t.DELETE_FILE="DELETE_FILE",t.RENAME="RENAME",t.CREATE_DIR="CREATE_DIR",t.LIST_DIR="LIST_DIR",t.DELETE_DIR="DELETE_DIR",t.ERROR="ERROR",t.DOWNLOAD="DOWNLOAD",t.PROGRESS="PROGRESS",t.LOG="LOG",t.MOUNT="MOUNT",t.UNMOUNT="UNMOUNT"})(E||(E={}));const a=new Error("unknown message type"),f=new Error("ffmpeg is not loaded, call `await ffmpeg.load()` first"),u=new Error("failed to import ffmpeg-core.js");let r;const O=async({coreURL:t,wasmURL:n,workerURL:e})=>{const o=!r;try{t||(t=R),importScripts(t)}catch{if(t||(t=R.replace("/umd/","/esm/")),self.createFFmpegCore=(await import(t)).default,!self.createFFmpegCore)throw u}const s=t,c=n||t.replace(/.js$/g,".wasm"),b=e||t.replace(/.js$/g,".worker.js");return r=await self.createFFmpegCore({mainScriptUrlOrBlob:`${s}#${btoa(JSON.stringify({wasmURL:c,workerURL:b}))}`}),r.setLogger(i=>self.postMessage({type:E.LOG,data:i})),r.setProgress(i=>self.postMessage({type:E.PROGRESS,data:i})),o},l=({args:t,timeout:n=-1})=>{r.setTimeout(n),r.exec(...t);const e=r.ret;return r.reset(),e},m=({path:t,data:n})=>(r.FS.writeFile(t,n),!0),D=({path:t,encoding:n})=>r.FS.readFile(t,{encoding:n}),S=({path:t})=>(r.FS.unlink(t),!0),I=({oldPath:t,newPath:n})=>(r.FS.rename(t,n),!0),L=({path:t})=>(r.FS.mkdir(t),!0),N=({path:t})=>{const n=r.FS.readdir(t),e=[];for(const o of n){const s=r.FS.stat(`${t}/${o}`),c=r.FS.isDir(s.mode);e.push({name:o,isDir:c})}return e},A=({path:t})=>(r.FS.rmdir(t),!0),w=({fsType:t,options:n,mountPoint:e})=>{const o=t,s=r.FS.filesystems[o];return s?(r.FS.mount(s,n,e),!0):!1},k=({mountPoint:t})=>(r.FS.unmount(t),!0);self.onmessage=async({data:{id:t,type:n,data:e}})=>{const o=[];let s;try{if(n!==E.LOAD&&!r)throw f;switch(n){case E.LOAD:s=await O(e);break;case E.EXEC:s=l(e);break;case E.WRITE_FILE:s=m(e);break;case E.READ_FILE:s=D(e);break;case E.DELETE_FILE:s=S(e);break;case E.RENAME:s=I(e);break;case E.CREATE_DIR:s=L(e);break;case E.LIST_DIR:s=N(e);break;case E.DELETE_DIR:s=A(e);break;case E.MOUNT:s=w(e);break;case E.UNMOUNT:s=k(e);break;default:throw a}}catch(c){self.postMessage({id:t,type:E.ERROR,data:c.toString()});return}s instanceof Uint8Array&&o.push(s.buffer),self.postMessage({id:t,type:n,data:s},o)}})();
|
||||
@@ -1,222 +0,0 @@
|
||||
var v;
|
||||
(function(e) {
|
||||
e.LOAD = "LOAD", e.EXEC = "EXEC", e.WRITE_FILE = "WRITE_FILE", e.READ_FILE = "READ_FILE", e.DELETE_FILE = "DELETE_FILE", e.RENAME = "RENAME", e.CREATE_DIR = "CREATE_DIR", e.LIST_DIR = "LIST_DIR", e.DELETE_DIR = "DELETE_DIR", e.ERROR = "ERROR", e.DOWNLOAD = "DOWNLOAD", e.PROGRESS = "PROGRESS", e.LOG = "LOG", e.MOUNT = "MOUNT", e.UNMOUNT = "UNMOUNT";
|
||||
})(v || (v = {}));
|
||||
const {
|
||||
SvelteComponent: X,
|
||||
append_hydration: T,
|
||||
attr: I,
|
||||
binding_callbacks: j,
|
||||
children: A,
|
||||
claim_element: N,
|
||||
claim_text: Q,
|
||||
detach: a,
|
||||
element: k,
|
||||
empty: b,
|
||||
init: z,
|
||||
insert_hydration: O,
|
||||
is_function: p,
|
||||
listen: L,
|
||||
noop: y,
|
||||
run_all: B,
|
||||
safe_not_equal: H,
|
||||
set_data: Y,
|
||||
src_url_equal: w,
|
||||
text: Z,
|
||||
toggle_class: d
|
||||
} = window.__gradio__svelte__internal;
|
||||
function S(e) {
|
||||
let l;
|
||||
function t(u, i) {
|
||||
return J;
|
||||
}
|
||||
let o = t()(e);
|
||||
return {
|
||||
c() {
|
||||
o.c(), l = b();
|
||||
},
|
||||
l(u) {
|
||||
o.l(u), l = b();
|
||||
},
|
||||
m(u, i) {
|
||||
o.m(u, i), O(u, l, i);
|
||||
},
|
||||
p(u, i) {
|
||||
o.p(u, i);
|
||||
},
|
||||
d(u) {
|
||||
u && a(l), o.d(u);
|
||||
}
|
||||
};
|
||||
}
|
||||
function J(e) {
|
||||
let l, t, n, o, u;
|
||||
return {
|
||||
c() {
|
||||
l = k("div"), t = k("video"), this.h();
|
||||
},
|
||||
l(i) {
|
||||
l = N(i, "DIV", { class: !0 });
|
||||
var c = A(l);
|
||||
t = N(c, "VIDEO", { src: !0 }), A(t).forEach(a), c.forEach(a), this.h();
|
||||
},
|
||||
h() {
|
||||
var i;
|
||||
w(t.src, n = /*value*/
|
||||
(i = e[2]) == null ? void 0 : i.video.url) || I(t, "src", n), I(l, "class", "container svelte-1uoo7dd"), d(
|
||||
l,
|
||||
"table",
|
||||
/*type*/
|
||||
e[0] === "table"
|
||||
), d(
|
||||
l,
|
||||
"gallery",
|
||||
/*type*/
|
||||
e[0] === "gallery"
|
||||
), d(
|
||||
l,
|
||||
"selected",
|
||||
/*selected*/
|
||||
e[1]
|
||||
);
|
||||
},
|
||||
m(i, c) {
|
||||
O(i, l, c), T(l, t), e[6](t), o || (u = [
|
||||
L(
|
||||
t,
|
||||
"loadeddata",
|
||||
/*init*/
|
||||
e[4]
|
||||
),
|
||||
L(t, "mouseover", function() {
|
||||
p(
|
||||
/*video*/
|
||||
e[3].play.bind(
|
||||
/*video*/
|
||||
e[3]
|
||||
)
|
||||
) && e[3].play.bind(
|
||||
/*video*/
|
||||
e[3]
|
||||
).apply(this, arguments);
|
||||
}),
|
||||
L(t, "mouseout", function() {
|
||||
p(
|
||||
/*video*/
|
||||
e[3].pause.bind(
|
||||
/*video*/
|
||||
e[3]
|
||||
)
|
||||
) && e[3].pause.bind(
|
||||
/*video*/
|
||||
e[3]
|
||||
).apply(this, arguments);
|
||||
})
|
||||
], o = !0);
|
||||
},
|
||||
p(i, c) {
|
||||
var _;
|
||||
e = i, c & /*value*/
|
||||
4 && !w(t.src, n = /*value*/
|
||||
(_ = e[2]) == null ? void 0 : _.video.url) && I(t, "src", n), c & /*type*/
|
||||
1 && d(
|
||||
l,
|
||||
"table",
|
||||
/*type*/
|
||||
e[0] === "table"
|
||||
), c & /*type*/
|
||||
1 && d(
|
||||
l,
|
||||
"gallery",
|
||||
/*type*/
|
||||
e[0] === "gallery"
|
||||
), c & /*selected*/
|
||||
2 && d(
|
||||
l,
|
||||
"selected",
|
||||
/*selected*/
|
||||
e[1]
|
||||
);
|
||||
},
|
||||
d(i) {
|
||||
i && a(l), e[6](null), o = !1, B(u);
|
||||
}
|
||||
};
|
||||
}
|
||||
function K(e) {
|
||||
let l, t = (
|
||||
/*value*/
|
||||
e[2] && S(e)
|
||||
);
|
||||
return {
|
||||
c() {
|
||||
t && t.c(), l = b();
|
||||
},
|
||||
l(n) {
|
||||
t && t.l(n), l = b();
|
||||
},
|
||||
m(n, o) {
|
||||
t && t.m(n, o), O(n, l, o);
|
||||
},
|
||||
p(n, [o]) {
|
||||
/*value*/
|
||||
n[2] ? t ? t.p(n, o) : (t = S(n), t.c(), t.m(l.parentNode, l)) : t && (t.d(1), t = null);
|
||||
},
|
||||
i: y,
|
||||
o: y,
|
||||
d(n) {
|
||||
n && a(l), t && t.d(n);
|
||||
}
|
||||
};
|
||||
}
|
||||
function P(e, l, t) {
|
||||
var n = this && this.__awaiter || function(f, G, s, R) {
|
||||
function W(E) {
|
||||
return E instanceof s ? E : new s(function(m) {
|
||||
m(E);
|
||||
});
|
||||
}
|
||||
return new (s || (s = Promise))(function(E, m) {
|
||||
function q(r) {
|
||||
try {
|
||||
h(R.next(r));
|
||||
} catch (D) {
|
||||
m(D);
|
||||
}
|
||||
}
|
||||
function V(r) {
|
||||
try {
|
||||
h(R.throw(r));
|
||||
} catch (D) {
|
||||
m(D);
|
||||
}
|
||||
}
|
||||
function h(r) {
|
||||
r.done ? E(r.value) : W(r.value).then(q, V);
|
||||
}
|
||||
h((R = R.apply(f, G || [])).next());
|
||||
});
|
||||
};
|
||||
let { type: o } = l, { selected: u = !1 } = l, { value: i } = l, { loop: c } = l, _;
|
||||
function U() {
|
||||
return n(this, void 0, void 0, function* () {
|
||||
t(3, _.muted = !0, _), t(3, _.playsInline = !0, _), t(3, _.controls = !1, _), _.setAttribute("muted", ""), yield _.play(), _.pause();
|
||||
});
|
||||
}
|
||||
function C(f) {
|
||||
j[f ? "unshift" : "push"](() => {
|
||||
_ = f, t(3, _);
|
||||
});
|
||||
}
|
||||
return e.$$set = (f) => {
|
||||
"type" in f && t(0, o = f.type), "selected" in f && t(1, u = f.selected), "value" in f && t(2, i = f.value), "loop" in f && t(5, c = f.loop);
|
||||
}, [o, u, i, _, U, c, C];
|
||||
}
|
||||
class M extends X {
|
||||
constructor(l) {
|
||||
super(), z(this, l, P, K, H, { type: 0, selected: 1, value: 2, loop: 5 });
|
||||
}
|
||||
}
|
||||
export {
|
||||
M as default
|
||||
};
|
||||
@@ -1 +0,0 @@
|
||||
.container.svelte-1uoo7dd{flex:none;max-width:none}.container.svelte-1uoo7dd video{width:var(--size-full);height:var(--size-full);object-fit:cover}.container.svelte-1uoo7dd:hover,.container.selected.svelte-1uoo7dd{border-color:var(--border-color-accent)}.container.table.svelte-1uoo7dd{margin:0 auto;border:2px solid var(--border-color-primary);border-radius:var(--radius-lg);overflow:hidden;width:var(--size-20);height:var(--size-20);object-fit:cover}.container.gallery.svelte-1uoo7dd{height:var(--size-20);max-height:var(--size-20);object-fit:cover}
|
||||
@@ -360,6 +360,10 @@ class StreamHandlerBase(ABC):
|
||||
)
|
||||
yield from self._resampler.resample(frame)
|
||||
|
||||
class StreamHandlerFactory(ABC):
|
||||
@abstractmethod
|
||||
def create(id:str)-> StreamHandlerBase:
|
||||
pass
|
||||
|
||||
EmitType: TypeAlias = (
|
||||
tuple[int, npt.NDArray[np.int16 | np.float32]]
|
||||
@@ -381,7 +385,7 @@ class StreamHandler(StreamHandlerBase):
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def copy(self) -> StreamHandler:
|
||||
def copy(self, **kwargs) -> StreamHandler:
|
||||
pass
|
||||
|
||||
def start_up(self):
|
||||
@@ -398,12 +402,15 @@ class AsyncStreamHandler(StreamHandlerBase):
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def copy(self) -> AsyncStreamHandler:
|
||||
def copy(self, **kwargs) -> AsyncStreamHandler:
|
||||
pass
|
||||
|
||||
async def start_up(self):
|
||||
pass
|
||||
|
||||
async def on_chat_datachannel(self, message: dict,channel):
|
||||
pass
|
||||
|
||||
|
||||
StreamHandlerImpl = StreamHandler | AsyncStreamHandler
|
||||
|
||||
@@ -418,7 +425,7 @@ class AudioVideoStreamHandler(StreamHandler):
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def copy(self) -> AudioVideoStreamHandler:
|
||||
def copy(self, **kwargs) -> AudioVideoStreamHandler:
|
||||
pass
|
||||
|
||||
|
||||
@@ -432,7 +439,7 @@ class AsyncAudioVideoStreamHandler(AsyncStreamHandler):
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def copy(self) -> AsyncAudioVideoStreamHandler:
|
||||
def copy(self, **kwargs) -> AsyncAudioVideoStreamHandler:
|
||||
pass
|
||||
|
||||
|
||||
|
||||
@@ -10,6 +10,7 @@ from typing import (
|
||||
Concatenate,
|
||||
Iterable,
|
||||
Literal,
|
||||
Optional,
|
||||
ParamSpec,
|
||||
Sequence,
|
||||
TypeVar,
|
||||
@@ -84,12 +85,18 @@ class WebRTC(Component, WebRTCConnectionMixin):
|
||||
time_limit: float | None = None,
|
||||
mode: Literal["send-receive", "receive", "send"] = "send-receive",
|
||||
modality: Literal["video", "audio", "audio-video"] = "video",
|
||||
video_chat: bool = True,
|
||||
rtp_params: dict[str, Any] | None = None,
|
||||
icon: str | None = None,
|
||||
icon_button_color: str | None = None,
|
||||
pulse_color: str | None = None,
|
||||
icon_radius: int | None = None,
|
||||
button_labels: dict | None = None,
|
||||
|
||||
#video_chat = True 后生效
|
||||
avatar_type: Optional['gs'] = None,
|
||||
avatar_ws_route: str | None = None,
|
||||
avatar_assets_path: str | None = None
|
||||
):
|
||||
"""
|
||||
Parameters:
|
||||
@@ -123,6 +130,13 @@ class WebRTC(Component, WebRTCConnectionMixin):
|
||||
button_labels: Text to display on the audio or video start, stop, waiting buttons. Dict with keys "start", "stop", "waiting" mapping to the text to display on the buttons.
|
||||
icon_radius: Border radius of the icon button expressed as a percentage of the button size. Default is 50%
|
||||
"""
|
||||
self.video_chat = video_chat
|
||||
if video_chat is True:
|
||||
mode = 'send-receive'
|
||||
modality = 'audio-video'
|
||||
self.avatar_type = avatar_type
|
||||
self.avatar_ws_route = avatar_ws_route
|
||||
self.avatar_assets_path = avatar_assets_path
|
||||
WebRTCConnectionMixin.__init__(self)
|
||||
self.time_limit = time_limit
|
||||
self.height = height
|
||||
|
||||
@@ -4,6 +4,7 @@ from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import inspect
|
||||
import json
|
||||
import logging
|
||||
from collections import defaultdict
|
||||
from collections.abc import Callable
|
||||
@@ -30,6 +31,7 @@ from fastrtc.tracks import (
|
||||
ServerToClientAudio,
|
||||
ServerToClientVideo,
|
||||
StreamHandlerBase,
|
||||
StreamHandlerFactory,
|
||||
StreamHandlerImpl,
|
||||
VideoCallback,
|
||||
VideoEventHandler,
|
||||
@@ -246,7 +248,7 @@ class WebRTCConnectionMixin:
|
||||
self.pcs[body["webrtc_id"]] = pc
|
||||
|
||||
if isinstance(self.event_handler, StreamHandlerBase):
|
||||
handler = self.event_handler.copy()
|
||||
handler = self.event_handler.copy(webrtc_id=body['webrtc_id'])
|
||||
handler.emit = webrtc_error_handler(handler.emit) # type: ignore
|
||||
handler.receive = webrtc_error_handler(handler.receive) # type: ignore
|
||||
handler.start_up = webrtc_error_handler(handler.start_up) # type: ignore
|
||||
@@ -255,6 +257,9 @@ class WebRTCConnectionMixin:
|
||||
handler.video_receive = webrtc_error_handler(handler.video_receive) # type: ignore
|
||||
if hasattr(handler, "video_emit"):
|
||||
handler.video_emit = webrtc_error_handler(handler.video_emit) # type: ignore
|
||||
if hasattr(handler, "on_pc_connected"):
|
||||
handler.on_pc_connected(body["webrtc_id"])
|
||||
|
||||
elif isinstance(self.event_handler, VideoStreamHandler):
|
||||
self.event_handler.callable = cast(
|
||||
VideoEventHandler, webrtc_error_handler(self.event_handler.callable)
|
||||
@@ -265,6 +270,7 @@ class WebRTCConnectionMixin:
|
||||
|
||||
self.handlers[body["webrtc_id"]] = handler
|
||||
|
||||
|
||||
@pc.on("iceconnectionstatechange")
|
||||
async def on_iceconnectionstatechange():
|
||||
logger.debug("ICE connection state change %s", pc.iceConnectionState)
|
||||
@@ -393,6 +399,23 @@ class WebRTCConnectionMixin:
|
||||
def _(message):
|
||||
logger.debug(f"Received message: {message}")
|
||||
if channel.readyState == "open":
|
||||
def parse_json_safely(str: str):
|
||||
try:
|
||||
result = json.loads(str)
|
||||
return result, None
|
||||
except json.JSONDecodeError as e:
|
||||
# print(f"JSON解析错误: {e.msg}")
|
||||
return None, e
|
||||
msg_dict,error = parse_json_safely(message)
|
||||
if(error is None and msg_dict['type'] in ['chat','stop_chat', 'init']):
|
||||
msg_dict = cast(dict, json.loads(message))
|
||||
handler = self.handlers[body["webrtc_id"]]
|
||||
if inspect.iscoroutinefunction(handler.on_chat_datachannel):
|
||||
asyncio.create_task(
|
||||
handler.on_chat_datachannel(msg_dict,channel))
|
||||
else:
|
||||
handler.on_chat_datachannel(msg_dict,channel)
|
||||
else:
|
||||
channel.send(
|
||||
create_message("log", data=f"Server received: {message}")
|
||||
)
|
||||
|
||||
198
demo/video_chat/app.py
Normal file
198
demo/video_chat/app.py
Normal file
@@ -0,0 +1,198 @@
|
||||
import asyncio
|
||||
import base64
|
||||
from io import BytesIO
|
||||
import json
|
||||
import math
|
||||
import queue
|
||||
import time
|
||||
from typing import TypedDict
|
||||
import uuid
|
||||
import threading
|
||||
|
||||
# from fastrtc.utils import Message
|
||||
from fastapi import FastAPI, WebSocket, WebSocketDisconnect
|
||||
import gradio as gr
|
||||
import numpy as np
|
||||
from fastrtc import (
|
||||
AsyncAudioVideoStreamHandler,
|
||||
WebRTC,
|
||||
VideoEmitType,
|
||||
AudioEmitType,
|
||||
)
|
||||
from PIL import Image
|
||||
import uvicorn
|
||||
|
||||
|
||||
class Message(TypedDict):
|
||||
type: str
|
||||
data: any
|
||||
|
||||
def encode_audio(data: np.ndarray) -> dict:
|
||||
"""Encode Audio data to send to the server"""
|
||||
return {"mime_type": "audio/pcm", "data": base64.b64encode(data.tobytes()).decode("UTF-8")}
|
||||
|
||||
|
||||
def encode_image(data: np.ndarray) -> dict:
|
||||
with BytesIO() as output_bytes:
|
||||
pil_image = Image.fromarray(data)
|
||||
pil_image.save(output_bytes, "JPEG")
|
||||
bytes_data = output_bytes.getvalue()
|
||||
base64_str = str(base64.b64encode(bytes_data), "utf-8")
|
||||
return {"mime_type": "image/jpeg", "data": base64_str}
|
||||
|
||||
frame_queue = queue.Queue(maxsize=100)
|
||||
|
||||
class VideoChatHandler(AsyncAudioVideoStreamHandler):
|
||||
def __init__(
|
||||
self, expected_layout="mono", output_sample_rate=24000, output_frame_size=480
|
||||
) -> None:
|
||||
super().__init__(
|
||||
expected_layout,
|
||||
output_sample_rate,
|
||||
output_frame_size,
|
||||
input_sample_rate=24000,
|
||||
)
|
||||
self.audio_queue = asyncio.Queue()
|
||||
self.video_queue = frame_queue
|
||||
self.quit = asyncio.Event()
|
||||
self.session = None
|
||||
self.last_frame_time = 0
|
||||
|
||||
def copy(self, **kwargs) -> "VideoChatHandler":
|
||||
return VideoChatHandler(
|
||||
expected_layout=self.expected_layout,
|
||||
output_sample_rate=self.output_sample_rate,
|
||||
output_frame_size=self.output_frame_size,
|
||||
)
|
||||
|
||||
def on_pc_connected(self, webrtc_id):
|
||||
print(webrtc_id)
|
||||
|
||||
chat_id = ''
|
||||
async def on_chat_datachannel(self,message: Message,channel):
|
||||
# 返回
|
||||
# {"type":"chat",id:"标识属于同一段话", "message":"Hello, world!"}
|
||||
# {"type":"avatar_end"} 表示本次对话结束
|
||||
if message['type'] == 'stop_chat':
|
||||
self.chat_id = ''
|
||||
channel.send(json.dumps({'type':'avatar_end'}))
|
||||
else:
|
||||
id = uuid.uuid4().hex
|
||||
self.chat_id = id
|
||||
data = message["data"]
|
||||
halfLen = math.floor(data.__len__()/2)
|
||||
channel.send(json.dumps({"type":"chat","id":id,"message":data[:halfLen]}))
|
||||
await asyncio.sleep(5)
|
||||
if self.chat_id == id:
|
||||
channel.send(json.dumps({"type":"chat","id":id,"message":data[halfLen:]}))
|
||||
channel.send(json.dumps({'type':'avatar_end'}))
|
||||
|
||||
async def video_receive(self, frame: np.ndarray):
|
||||
# if self.session:
|
||||
# # send image every 1 second
|
||||
# if time.time() - self.last_frame_time > 1:
|
||||
# self.last_frame_time = time.time()
|
||||
# await self.session.send(encode_image(frame))
|
||||
# if self.latest_args[2] is not None:
|
||||
# await self.session.send(encode_image(self.latest_args[2]))
|
||||
# print(frame.shape)
|
||||
newFrame = np.array(frame)
|
||||
newFrame[0:, :, 0] = 255 - newFrame[0:, :, 0]
|
||||
# self.video_queue.put_nowait(newFrame)
|
||||
|
||||
async def video_emit(self) -> VideoEmitType:
|
||||
# print('123123',frame_queue.qsize())
|
||||
return frame_queue.get()
|
||||
|
||||
async def receive(self, frame: tuple[int, np.ndarray]) -> None:
|
||||
frame_size, array = frame
|
||||
self.audio_queue.put_nowait(array)
|
||||
|
||||
async def emit(self) -> AudioEmitType:
|
||||
if not self.args_set.is_set():
|
||||
await self.wait_for_args()
|
||||
array = await self.audio_queue.get()
|
||||
return (self.output_sample_rate, array)
|
||||
|
||||
def shutdown(self) -> None:
|
||||
self.quit.set()
|
||||
self.connection = None
|
||||
self.args_set.clear()
|
||||
self.quit.clear()
|
||||
|
||||
|
||||
css = """
|
||||
footer {
|
||||
display: none !important;
|
||||
}
|
||||
"""
|
||||
|
||||
with gr.Blocks(css=css) as demo:
|
||||
webrtc = WebRTC(
|
||||
label="Video Chat",
|
||||
modality="audio-video",
|
||||
mode="send-receive",
|
||||
video_chat=True,
|
||||
avatar_type="",
|
||||
avatar_assets_path="",
|
||||
avatar_ws_route='/ws',
|
||||
elem_id="video-source",
|
||||
track_constraints={
|
||||
"video": {
|
||||
"facingMode": "user",
|
||||
"width": {"ideal": 500},
|
||||
"height": {"ideal": 500},
|
||||
"frameRate": {"ideal": 30},
|
||||
},
|
||||
"audio": {
|
||||
"echoCancellation": True,
|
||||
"noiseSuppression": {"exact": True},
|
||||
"autoGainControl": {"exact": False},
|
||||
"sampleRate": {"ideal": 24000},
|
||||
"sampleSize": {"ideal": 16},
|
||||
"channelCount": {"exact": 1},
|
||||
},
|
||||
}
|
||||
)
|
||||
handler = VideoChatHandler()
|
||||
webrtc.stream(
|
||||
handler,
|
||||
inputs=[webrtc],
|
||||
outputs=[webrtc],
|
||||
time_limit=1500,
|
||||
concurrency_limit=2,
|
||||
)
|
||||
# 线程函数：随机生成 numpy 帧
|
||||
def generate_frames(width=480, height=960, channels=3):
|
||||
while True:
|
||||
try:
|
||||
# 随机生成一个 RGB 图像帧
|
||||
frame = np.random.randint(188, 256, (height, width, channels), dtype=np.uint8)
|
||||
|
||||
# 将帧放入队列
|
||||
frame_queue.put(frame)
|
||||
# print("生成一帧数据，形状:", frame.shape, frame_queue.qsize())
|
||||
|
||||
# 模拟实时性：避免过度消耗 CPU
|
||||
time.sleep(0.03) # 每秒约生成 30 帧
|
||||
except Exception as e:
|
||||
print(f"生成帧时出错: {e}")
|
||||
break
|
||||
thread = threading.Thread(target=generate_frames, daemon=True)
|
||||
thread.start()
|
||||
|
||||
@demo.app.websocket("/ws")
|
||||
async def on_websocket(websocket: WebSocket):
|
||||
await websocket.accept()
|
||||
while True:
|
||||
try:
|
||||
data = await websocket.receive_text()
|
||||
await websocket.send_text(f"Message text was: {data}")
|
||||
except WebSocketDisconnect:
|
||||
break
|
||||
|
||||
if __name__ == "__main__":
|
||||
demo.launch()
|
||||
|
||||
|
||||
|
||||
BIN
dist/fastrtc-0.0.19.dev0-py3-none-any.whl
vendored
Normal file
BIN
dist/fastrtc-0.0.19.dev0-py3-none-any.whl
vendored
Normal file
Binary file not shown.
BIN
docs/image.png
Normal file
BIN
docs/image.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 367 KiB |
BIN
docs/image2.png
Normal file
BIN
docs/image2.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 938 KiB |
@@ -8,7 +8,12 @@
|
||||
import StaticVideo from "./shared/StaticVideo.svelte";
|
||||
import StaticAudio from "./shared/StaticAudio.svelte";
|
||||
import InteractiveAudio from "./shared/InteractiveAudio.svelte";
|
||||
import VideoChat from './shared/VideoChat/index.svelte'
|
||||
|
||||
export let video_chat = false;
|
||||
export let avatar_type: string = ""
|
||||
export let avatar_ws_route: string = ""
|
||||
export let avatar_assets_path: string = ""
|
||||
export let elem_id = "";
|
||||
export let elem_classes: string[] = [];
|
||||
export let visible = true;
|
||||
@@ -81,6 +86,44 @@
|
||||
let dragging = false;
|
||||
</script>
|
||||
|
||||
{#if video_chat}
|
||||
<Block
|
||||
{visible}
|
||||
variant={"solid"}
|
||||
border_mode={dragging ? "focus" : "base"}
|
||||
padding={false}
|
||||
{elem_id}
|
||||
{elem_classes}
|
||||
{height}
|
||||
{width}
|
||||
{container}
|
||||
{scale}
|
||||
{min_width}
|
||||
allow_overflow={false}>
|
||||
<VideoChat {server} bind:webrtc_id={value}
|
||||
on:clear={() => gradio.dispatch("clear")}
|
||||
on:play={() => gradio.dispatch("play")}
|
||||
on:pause={() => gradio.dispatch("pause")}
|
||||
on:upload={() => gradio.dispatch("upload")}
|
||||
on:stop={() => gradio.dispatch("stop")}
|
||||
on:end={() => gradio.dispatch("end")}
|
||||
on:start_recording={() => gradio.dispatch("start_recording")}
|
||||
on:stop_recording={() => gradio.dispatch("stop_recording")}
|
||||
on:tick={() => gradio.dispatch("tick")}
|
||||
on:error={({ detail }) => gradio.dispatch("error", detail)}
|
||||
i18n={gradio.i18n}
|
||||
stream_handler={(...args) => gradio.client.stream(...args)}
|
||||
{avatar_type}
|
||||
{avatar_ws_route}
|
||||
{avatar_assets_path}
|
||||
{track_constraints}
|
||||
{height}
|
||||
{on_change_cb} {rtc_configuration}
|
||||
on:tick={() => gradio.dispatch("tick")}
|
||||
on:error={({ detail }) => gradio.dispatch("error", detail)}></VideoChat>
|
||||
</Block>
|
||||
|
||||
{:else}
|
||||
<Block
|
||||
{visible}
|
||||
variant={"solid"}
|
||||
@@ -189,3 +232,4 @@
|
||||
/>
|
||||
{/if}
|
||||
</Block>
|
||||
{/if}
|
||||
313
frontend/package-lock.json
generated
313
frontend/package-lock.json
generated
@@ -20,11 +20,18 @@
|
||||
"@gradio/upload": "0.13.3",
|
||||
"@gradio/utils": "0.7.0",
|
||||
"@gradio/wasm": "0.14.2",
|
||||
"base64-js": "^1.5.1",
|
||||
"buffer": "^6.0.3",
|
||||
"eventemitter3": "^5.0.1",
|
||||
"gaussian-splat-renderer-for-lam": "^0.0.6",
|
||||
"hls.js": "^1.5.16",
|
||||
"mrmime": "^2.0.0"
|
||||
"mrmime": "^2.0.0",
|
||||
"p-queue": "^8.0.1",
|
||||
"python-struct": "^1.1.3"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@gradio/preview": "0.12.0",
|
||||
"less": "^4.2.2",
|
||||
"prettier": "^3.3.3",
|
||||
"prettier-plugin-svelte": "^3.3.3"
|
||||
},
|
||||
@@ -1816,6 +1823,26 @@
|
||||
"integrity": "sha512-3oSeUO0TMV67hN1AmbXsK4yaqU7tjiHlbxRDZOpH0KW9+CeX4bRAaX0Anxt0tx2MrpRpWwQaPwIlISEJhYU5Pw==",
|
||||
"dev": true
|
||||
},
|
||||
"node_modules/base64-js": {
|
||||
"version": "1.5.1",
|
||||
"resolved": "https://registry.anpm.alibaba-inc.com/base64-js/-/base64-js-1.5.1.tgz",
|
||||
"integrity": "sha512-AKpaYlHn8t4SVbOHCy+b5+KKgvR4vrsD8vbvrbiQJps7fKDTkjkDry6ji0rUJjC0kzbNePLwzxq8iypo41qeWA==",
|
||||
"funding": [
|
||||
{
|
||||
"type": "github",
|
||||
"url": "https://github.com/sponsors/feross"
|
||||
},
|
||||
{
|
||||
"type": "patreon",
|
||||
"url": "https://www.patreon.com/feross"
|
||||
},
|
||||
{
|
||||
"type": "consulting",
|
||||
"url": "https://feross.org/support"
|
||||
}
|
||||
],
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/brace-expansion": {
|
||||
"version": "1.1.11",
|
||||
"resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-1.1.11.tgz",
|
||||
@@ -1838,6 +1865,30 @@
|
||||
"node": ">=8"
|
||||
}
|
||||
},
|
||||
"node_modules/buffer": {
|
||||
"version": "6.0.3",
|
||||
"resolved": "https://registry.anpm.alibaba-inc.com/buffer/-/buffer-6.0.3.tgz",
|
||||
"integrity": "sha512-FTiCpNxtwiZZHEZbcbTIcZjERVICn9yq/pDFkTl95/AxzD1naBctN7YO68riM/gLSDY7sdrMby8hofADYuuqOA==",
|
||||
"funding": [
|
||||
{
|
||||
"type": "github",
|
||||
"url": "https://github.com/sponsors/feross"
|
||||
},
|
||||
{
|
||||
"type": "patreon",
|
||||
"url": "https://www.patreon.com/feross"
|
||||
},
|
||||
{
|
||||
"type": "consulting",
|
||||
"url": "https://feross.org/support"
|
||||
}
|
||||
],
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"base64-js": "^1.3.1",
|
||||
"ieee754": "^1.2.1"
|
||||
}
|
||||
},
|
||||
"node_modules/buffer-crc32": {
|
||||
"version": "1.0.0",
|
||||
"resolved": "https://registry.npmjs.org/buffer-crc32/-/buffer-crc32-1.0.0.tgz",
|
||||
@@ -2072,6 +2123,19 @@
|
||||
"node": ">= 0.6"
|
||||
}
|
||||
},
|
||||
"node_modules/copy-anything": {
|
||||
"version": "2.0.6",
|
||||
"resolved": "https://registry.anpm.alibaba-inc.com/copy-anything/-/copy-anything-2.0.6.tgz",
|
||||
"integrity": "sha512-1j20GZTsvKNkc4BY3NpMOM8tt///wY3FpIzozTOFO2ffuZcV61nojHXVKIy3WM+7ADCy5FVhdZYHYDdgTU0yJw==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"is-what": "^3.14.1"
|
||||
},
|
||||
"funding": {
|
||||
"url": "https://github.com/sponsors/mesqueeb"
|
||||
}
|
||||
},
|
||||
"node_modules/cropperjs": {
|
||||
"version": "1.6.2",
|
||||
"resolved": "https://registry.npmjs.org/cropperjs/-/cropperjs-1.6.2.tgz",
|
||||
@@ -2358,6 +2422,20 @@
|
||||
"url": "https://github.com/fb55/entities?sponsor=1"
|
||||
}
|
||||
},
|
||||
"node_modules/errno": {
|
||||
"version": "0.1.8",
|
||||
"resolved": "https://registry.anpm.alibaba-inc.com/errno/-/errno-0.1.8.tgz",
|
||||
"integrity": "sha512-dJ6oBr5SQ1VSd9qkk7ByRgb/1SH4JZjCHSW/mr63/QcXO9zLVxvJ6Oy13nio03rxpSnVDDjFor75SjVeZWPW/A==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"dependencies": {
|
||||
"prr": "~1.0.1"
|
||||
},
|
||||
"bin": {
|
||||
"errno": "cli.js"
|
||||
}
|
||||
},
|
||||
"node_modules/es-define-property": {
|
||||
"version": "1.0.0",
|
||||
"resolved": "https://registry.npmjs.org/es-define-property/-/es-define-property-1.0.0.tgz",
|
||||
@@ -2836,6 +2914,12 @@
|
||||
"es5-ext": "~0.10.14"
|
||||
}
|
||||
},
|
||||
"node_modules/eventemitter3": {
|
||||
"version": "5.0.1",
|
||||
"resolved": "https://registry.anpm.alibaba-inc.com/eventemitter3/-/eventemitter3-5.0.1.tgz",
|
||||
"integrity": "sha512-GWkBvjiSZK87ELrYOSESUYeVIc9mvLLf/nXalMOS5dYrgZq9o5OVkbZAVM06CVxYsCwH9BDZFPlQTlPA1j4ahA==",
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/eventsource": {
|
||||
"version": "2.0.2",
|
||||
"resolved": "https://registry.npmjs.org/eventsource/-/eventsource-2.0.2.tgz",
|
||||
@@ -2927,6 +3011,15 @@
|
||||
"url": "https://github.com/sponsors/ljharb"
|
||||
}
|
||||
},
|
||||
"node_modules/gaussian-splat-renderer-for-lam": {
|
||||
"version": "0.0.6",
|
||||
"resolved": "https://registry.anpm.alibaba-inc.com/gaussian-splat-renderer-for-lam/-/gaussian-splat-renderer-for-lam-0.0.6.tgz",
|
||||
"integrity": "sha512-DujMYTW4hD9ZzEWMvCgm20HfkiMjYuA/b89Y7lbp7F1wOQvdtdC5pR63Ku1Ab7f9dvqS8cNwvw/adIlV0jkK4A==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"three": "^0.173.0"
|
||||
}
|
||||
},
|
||||
"node_modules/get-caller-file": {
|
||||
"version": "2.0.5",
|
||||
"resolved": "https://registry.npmjs.org/get-caller-file/-/get-caller-file-2.0.5.tgz",
|
||||
@@ -3161,6 +3254,40 @@
|
||||
"node": ">=0.10.0"
|
||||
}
|
||||
},
|
||||
"node_modules/ieee754": {
|
||||
"version": "1.2.1",
|
||||
"resolved": "https://registry.anpm.alibaba-inc.com/ieee754/-/ieee754-1.2.1.tgz",
|
||||
"integrity": "sha512-dcyqhDvX1C46lXZcVqCpK+FtMRQVdIMN6/Df5js2zouUsqG7I6sFxitIC+7KYK29KdXOLHdu9zL4sFnoVQnqaA==",
|
||||
"funding": [
|
||||
{
|
||||
"type": "github",
|
||||
"url": "https://github.com/sponsors/feross"
|
||||
},
|
||||
{
|
||||
"type": "patreon",
|
||||
"url": "https://www.patreon.com/feross"
|
||||
},
|
||||
{
|
||||
"type": "consulting",
|
||||
"url": "https://feross.org/support"
|
||||
}
|
||||
],
|
||||
"license": "BSD-3-Clause"
|
||||
},
|
||||
"node_modules/image-size": {
|
||||
"version": "0.5.5",
|
||||
"resolved": "https://registry.anpm.alibaba-inc.com/image-size/-/image-size-0.5.5.tgz",
|
||||
"integrity": "sha512-6TDAlDPZxUFCv+fuOkIoXT/V/f3Qbq8e37p+YOiYrUv3v9cc3/6x78VdfPgFVaB9dZYeLUfKgHRebpkm/oP2VQ==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"bin": {
|
||||
"image-size": "bin/image-size.js"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=0.10.0"
|
||||
}
|
||||
},
|
||||
"node_modules/immutable": {
|
||||
"version": "4.3.7",
|
||||
"resolved": "https://registry.npmjs.org/immutable/-/immutable-4.3.7.tgz",
|
||||
@@ -3305,6 +3432,13 @@
|
||||
"url": "https://github.com/sponsors/ljharb"
|
||||
}
|
||||
},
|
||||
"node_modules/is-what": {
|
||||
"version": "3.14.1",
|
||||
"resolved": "https://registry.anpm.alibaba-inc.com/is-what/-/is-what-3.14.1.tgz",
|
||||
"integrity": "sha512-sNxgpk9793nzSs7bA6JQJGeIuRBQhAaNGG77kzYQgMkrID+lS6SlK07K5LaptscDlSaIgH+GPFzf+d75FVxozA==",
|
||||
"dev": true,
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/isexe": {
|
||||
"version": "3.1.1",
|
||||
"resolved": "https://registry.npmjs.org/isexe/-/isexe-3.1.1.tgz",
|
||||
@@ -3426,6 +3560,44 @@
|
||||
"resolved": "https://registry.npmjs.org/lazy-brush/-/lazy-brush-1.0.1.tgz",
|
||||
"integrity": "sha512-xT/iSClTVi7vLoF8dCWTBhCuOWqsLXCMPa6ucVmVAk6hyNCM5JeS1NLhXqIrJktUg+caEYKlqSOUU4u3cpXzKg=="
|
||||
},
|
||||
"node_modules/less": {
|
||||
"version": "4.3.0",
|
||||
"resolved": "https://registry.anpm.alibaba-inc.com/less/-/less-4.3.0.tgz",
|
||||
"integrity": "sha512-X9RyH9fvemArzfdP8Pi3irr7lor2Ok4rOttDXBhlwDg+wKQsXOXgHWduAJE1EsF7JJx0w0bcO6BC6tCKKYnXKA==",
|
||||
"dev": true,
|
||||
"license": "Apache-2.0",
|
||||
"dependencies": {
|
||||
"copy-anything": "^2.0.1",
|
||||
"parse-node-version": "^1.0.1",
|
||||
"tslib": "^2.3.0"
|
||||
},
|
||||
"bin": {
|
||||
"lessc": "bin/lessc"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=14"
|
||||
},
|
||||
"optionalDependencies": {
|
||||
"errno": "^0.1.1",
|
||||
"graceful-fs": "^4.1.2",
|
||||
"image-size": "~0.5.0",
|
||||
"make-dir": "^2.1.0",
|
||||
"mime": "^1.4.1",
|
||||
"needle": "^3.1.0",
|
||||
"source-map": "~0.6.0"
|
||||
}
|
||||
},
|
||||
"node_modules/less/node_modules/source-map": {
|
||||
"version": "0.6.1",
|
||||
"resolved": "https://registry.anpm.alibaba-inc.com/source-map/-/source-map-0.6.1.tgz",
|
||||
"integrity": "sha512-UjgapumWlbMhkBgzT7Ykc5YXUT46F0iKu8SGXq0bcwP5dz/h0Plj6enJqjz1Zbq2l5WaqYnrVbwWOWMyF3F47g==",
|
||||
"dev": true,
|
||||
"license": "BSD-3-Clause",
|
||||
"optional": true,
|
||||
"engines": {
|
||||
"node": ">=0.10.0"
|
||||
}
|
||||
},
|
||||
"node_modules/lightningcss": {
|
||||
"version": "1.27.0",
|
||||
"resolved": "https://registry.npmjs.org/lightningcss/-/lightningcss-1.27.0.tgz",
|
||||
@@ -3665,6 +3837,12 @@
|
||||
"resolved": "https://registry.npmjs.org/locate-character/-/locate-character-3.0.0.tgz",
|
||||
"integrity": "sha512-SW13ws7BjaeJ6p7Q6CO2nchbYEc3X3J6WrmTTDto7yMPqVSZTUyY5Tjbid+Ab8gLnATtygYtiDIJGQRRn2ZOiA=="
|
||||
},
|
||||
"node_modules/long": {
|
||||
"version": "4.0.0",
|
||||
"resolved": "https://registry.anpm.alibaba-inc.com/long/-/long-4.0.0.tgz",
|
||||
"integrity": "sha512-XsP+KhQif4bjX1kbuSiySJFNAehNxgLb6hPRGJ9QsUr8ajHkuXGdrHmFUTUUXhDwVX2R5bY4JNZEwbUiMhV+MA==",
|
||||
"license": "Apache-2.0"
|
||||
},
|
||||
"node_modules/lru-cache": {
|
||||
"version": "10.4.3",
|
||||
"resolved": "https://registry.npmjs.org/lru-cache/-/lru-cache-10.4.3.tgz",
|
||||
@@ -3687,6 +3865,32 @@
|
||||
"@jridgewell/sourcemap-codec": "^1.5.0"
|
||||
}
|
||||
},
|
||||
"node_modules/make-dir": {
|
||||
"version": "2.1.0",
|
||||
"resolved": "https://registry.anpm.alibaba-inc.com/make-dir/-/make-dir-2.1.0.tgz",
|
||||
"integrity": "sha512-LS9X+dc8KLxXCb8dni79fLIIUA5VyZoyjSMCwTluaXA0o27cCK0bhXkpgw+sTXVpPy/lSO57ilRixqk0vDmtRA==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"dependencies": {
|
||||
"pify": "^4.0.1",
|
||||
"semver": "^5.6.0"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=6"
|
||||
}
|
||||
},
|
||||
"node_modules/make-dir/node_modules/semver": {
|
||||
"version": "5.7.2",
|
||||
"resolved": "https://registry.anpm.alibaba-inc.com/semver/-/semver-5.7.2.tgz",
|
||||
"integrity": "sha512-cBznnQ9KjJqU67B52RMC65CMarK2600WFnbkcaiwWq3xy/5haFJlshgnpjovMVJ+Hff49d8GEn0b87C5pDQ10g==",
|
||||
"dev": true,
|
||||
"license": "ISC",
|
||||
"optional": true,
|
||||
"bin": {
|
||||
"semver": "bin/semver"
|
||||
}
|
||||
},
|
||||
"node_modules/marked": {
|
||||
"version": "12.0.2",
|
||||
"resolved": "https://registry.npmjs.org/marked/-/marked-12.0.2.tgz",
|
||||
@@ -3753,6 +3957,20 @@
|
||||
"node": ">=8.6"
|
||||
}
|
||||
},
|
||||
"node_modules/mime": {
|
||||
"version": "1.6.0",
|
||||
"resolved": "https://registry.anpm.alibaba-inc.com/mime/-/mime-1.6.0.tgz",
|
||||
"integrity": "sha512-x0Vn8spI+wuJ1O6S7gnbaQg8Pxh4NNHb7KSINmEWKiPE4RKOplvijn+NkmYmmRgP68mc70j2EbeTFRsrswaQeg==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"bin": {
|
||||
"mime": "cli.js"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=4"
|
||||
}
|
||||
},
|
||||
"node_modules/mime-db": {
|
||||
"version": "1.52.0",
|
||||
"resolved": "https://registry.npmjs.org/mime-db/-/mime-db-1.52.0.tgz",
|
||||
@@ -3922,6 +4140,24 @@
|
||||
"node": "^10 || ^12 || ^13.7 || ^14 || >=15.0.1"
|
||||
}
|
||||
},
|
||||
"node_modules/needle": {
|
||||
"version": "3.3.1",
|
||||
"resolved": "https://registry.anpm.alibaba-inc.com/needle/-/needle-3.3.1.tgz",
|
||||
"integrity": "sha512-6k0YULvhpw+RoLNiQCRKOl09Rv1dPLr8hHnVjHqdolKwDrdNyk+Hmrthi4lIGPPz3r39dLx0hsF5s40sZ3Us4Q==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"dependencies": {
|
||||
"iconv-lite": "^0.6.3",
|
||||
"sax": "^1.2.4"
|
||||
},
|
||||
"bin": {
|
||||
"needle": "bin/needle"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">= 4.4.x"
|
||||
}
|
||||
},
|
||||
"node_modules/next-tick": {
|
||||
"version": "1.1.0",
|
||||
"resolved": "https://registry.npmjs.org/next-tick/-/next-tick-1.1.0.tgz",
|
||||
@@ -3971,12 +4207,50 @@
|
||||
"resolved": "https://registry.npmjs.org/outvariant/-/outvariant-1.4.3.tgz",
|
||||
"integrity": "sha512-+Sl2UErvtsoajRDKCE5/dBz4DIvHXQQnAxtQTF04OJxY0+DyZXSo5P5Bb7XYWOh81syohlYL24hbDwxedPUJCA=="
|
||||
},
|
||||
"node_modules/p-queue": {
|
||||
"version": "8.1.0",
|
||||
"resolved": "https://registry.anpm.alibaba-inc.com/p-queue/-/p-queue-8.1.0.tgz",
|
||||
"integrity": "sha512-mxLDbbGIBEXTJL0zEx8JIylaj3xQ7Z/7eEVjcF9fJX4DBiH9oqe+oahYnlKKxm0Ci9TlWTyhSHgygxMxjIB2jw==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"eventemitter3": "^5.0.1",
|
||||
"p-timeout": "^6.1.2"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=18"
|
||||
},
|
||||
"funding": {
|
||||
"url": "https://github.com/sponsors/sindresorhus"
|
||||
}
|
||||
},
|
||||
"node_modules/p-timeout": {
|
||||
"version": "6.1.4",
|
||||
"resolved": "https://registry.anpm.alibaba-inc.com/p-timeout/-/p-timeout-6.1.4.tgz",
|
||||
"integrity": "sha512-MyIV3ZA/PmyBN/ud8vV9XzwTrNtR4jFrObymZYnZqMmW0zA8Z17vnT0rBgFE/TlohB+YCHqXMgZzb3Csp49vqg==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=14.16"
|
||||
},
|
||||
"funding": {
|
||||
"url": "https://github.com/sponsors/sindresorhus"
|
||||
}
|
||||
},
|
||||
"node_modules/package-json-from-dist": {
|
||||
"version": "1.0.1",
|
||||
"resolved": "https://registry.npmjs.org/package-json-from-dist/-/package-json-from-dist-1.0.1.tgz",
|
||||
"integrity": "sha512-UEZIS3/by4OC8vL3P2dTXRETpebLI2NiI5vIrjaD/5UtrkFX/tNbwjTSRAGC/+7CAo2pIcBaRgWmcBBHcsaCIw==",
|
||||
"dev": true
|
||||
},
|
||||
"node_modules/parse-node-version": {
|
||||
"version": "1.0.1",
|
||||
"resolved": "https://registry.anpm.alibaba-inc.com/parse-node-version/-/parse-node-version-1.0.1.tgz",
|
||||
"integrity": "sha512-3YHlOa/JgH6Mnpr05jP9eDG254US9ek25LyIxZlDItp2iJtwyaXQb57lBYLdT3MowkUFYEV2XXNAYIPlESvJlA==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">= 0.10"
|
||||
}
|
||||
},
|
||||
"node_modules/parse-srcset": {
|
||||
"version": "1.0.2",
|
||||
"resolved": "https://registry.npmjs.org/parse-srcset/-/parse-srcset-1.0.2.tgz",
|
||||
@@ -4078,6 +4352,17 @@
|
||||
"url": "https://github.com/sponsors/jonschlinkert"
|
||||
}
|
||||
},
|
||||
"node_modules/pify": {
|
||||
"version": "4.0.1",
|
||||
"resolved": "https://registry.anpm.alibaba-inc.com/pify/-/pify-4.0.1.tgz",
|
||||
"integrity": "sha512-uB80kBFb/tfd68bVleG9T5GGsGPjJrLAUpR5PZIrhBnIaRTQRjqdJSsIKkOP6OAIFbj7GOrcudc5pNjZ+geV2g==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"engines": {
|
||||
"node": ">=6"
|
||||
}
|
||||
},
|
||||
"node_modules/pirates": {
|
||||
"version": "4.0.6",
|
||||
"resolved": "https://registry.npmjs.org/pirates/-/pirates-4.0.6.tgz",
|
||||
@@ -4158,6 +4443,14 @@
|
||||
"asap": "~2.0.3"
|
||||
}
|
||||
},
|
||||
"node_modules/prr": {
|
||||
"version": "1.0.1",
|
||||
"resolved": "https://registry.anpm.alibaba-inc.com/prr/-/prr-1.0.1.tgz",
|
||||
"integrity": "sha512-yPw4Sng1gWghHQWj0B3ZggWUm4qVbPwPFcRG8KyxiU7J2OHFSoEHKS+EZ3fv5l1t9CyCiop6l/ZYeWbrgoQejw==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"optional": true
|
||||
},
|
||||
"node_modules/psl": {
|
||||
"version": "1.9.0",
|
||||
"resolved": "https://registry.npmjs.org/psl/-/psl-1.9.0.tgz",
|
||||
@@ -4306,6 +4599,15 @@
|
||||
"node": ">=18.0.0"
|
||||
}
|
||||
},
|
||||
"node_modules/python-struct": {
|
||||
"version": "1.1.3",
|
||||
"resolved": "https://registry.anpm.alibaba-inc.com/python-struct/-/python-struct-1.1.3.tgz",
|
||||
"integrity": "sha512-UsI/mNvk25jRpGKYI38Nfbv84z48oiIWwG67DLVvjRhy8B/0aIK+5Ju5WOHgw/o9rnEmbAS00v4rgKFQeC332Q==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"long": "^4.0.0"
|
||||
}
|
||||
},
|
||||
"node_modules/querystringify": {
|
||||
"version": "2.2.0",
|
||||
"resolved": "https://registry.npmjs.org/querystringify/-/querystringify-2.2.0.tgz",
|
||||
@@ -4476,8 +4778,7 @@
|
||||
"resolved": "https://registry.npmjs.org/sax/-/sax-1.2.4.tgz",
|
||||
"integrity": "sha512-NqVDv9TpANUjFm0N8uM5GxL36UgKi9/atZw+x7YFnQ8ckwFGKrl4xX4yWtrey3UJm5nP1kUbnYgLopqWNSRhWw==",
|
||||
"dev": true,
|
||||
"optional": true,
|
||||
"peer": true
|
||||
"optional": true
|
||||
},
|
||||
"node_modules/saxes": {
|
||||
"version": "6.0.0",
|
||||
@@ -5074,6 +5375,12 @@
|
||||
"node": ">=0.8"
|
||||
}
|
||||
},
|
||||
"node_modules/three": {
|
||||
"version": "0.173.0",
|
||||
"resolved": "https://registry.npmjs.org/three/-/three-0.173.0.tgz",
|
||||
"integrity": "sha512-AUwVmViIEUgBwxJJ7stnF0NkPpZxx1aZ6WiAbQ/Qq61h6I9UR4grXtZDmO8mnlaNORhHnIBlXJ1uBxILEKuVyw==",
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/timers-ext": {
|
||||
"version": "0.1.8",
|
||||
"resolved": "https://registry.npmjs.org/timers-ext/-/timers-ext-0.1.8.tgz",
|
||||
|
||||
@@ -18,11 +18,18 @@
|
||||
"@gradio/upload": "0.13.3",
|
||||
"@gradio/utils": "0.7.0",
|
||||
"@gradio/wasm": "0.14.2",
|
||||
"base64-js": "^1.5.1",
|
||||
"buffer": "^6.0.3",
|
||||
"eventemitter3": "^5.0.1",
|
||||
"gaussian-splat-renderer-for-lam": "^0.0.6",
|
||||
"hls.js": "^1.5.16",
|
||||
"mrmime": "^2.0.0"
|
||||
"mrmime": "^2.0.0",
|
||||
"p-queue": "^8.0.1",
|
||||
"python-struct": "^1.1.3"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@gradio/preview": "0.12.0",
|
||||
"less": "^4.2.2",
|
||||
"prettier": "^3.3.3",
|
||||
"prettier-plugin-svelte": "^3.3.3"
|
||||
},
|
||||
|
||||
BIN
frontend/shared/VideoChat/background.png
Normal file
BIN
frontend/shared/VideoChat/background.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 155 KiB |
28
frontend/shared/VideoChat/binary_utils.ts
Normal file
28
frontend/shared/VideoChat/binary_utils.ts
Normal file
@@ -0,0 +1,28 @@
|
||||
import base64js from "base64-js";
|
||||
import { Buffer } from "buffer";
|
||||
import PythonStruct from "python-struct";
|
||||
|
||||
export const unpack = async function (blob: Blob, str = "<II") {
|
||||
const unpackBuffer = await blob.slice(4, 12).arrayBuffer();
|
||||
const [jsonSize, binSize] = PythonStruct.unpack(
|
||||
str,
|
||||
Buffer.from(unpackBuffer),
|
||||
) as number[];
|
||||
const jsonBlob = await blob.slice(12, 12 + jsonSize).text();
|
||||
const parsedData = JSON.parse(jsonBlob);
|
||||
return {
|
||||
parsedData,
|
||||
jsonSize,
|
||||
binSize,
|
||||
};
|
||||
};
|
||||
export const mergeBlob = (strArray: string[], target: Uint8Array) => {
|
||||
let offset = 0;
|
||||
strArray.forEach((str) => {
|
||||
const byteArray = base64js.toByteArray(str);
|
||||
target.set(byteArray, offset);
|
||||
offset += byteArray.byteLength;
|
||||
});
|
||||
const blob = new Blob([target]);
|
||||
return blob;
|
||||
};
|
||||
225
frontend/shared/VideoChat/components/AudioWave.svelte
Normal file
225
frontend/shared/VideoChat/components/AudioWave.svelte
Normal file
@@ -0,0 +1,225 @@
|
||||
<script lang="ts">
|
||||
import { onDestroy } from "svelte";
|
||||
import type { ComponentType } from "svelte";
|
||||
|
||||
import PulsingIcon from "../../PulsingIcon.svelte";
|
||||
|
||||
export let numBars = 16;
|
||||
export let stream_state: "open" | "closed" | "waiting" = "closed";
|
||||
export let audio_source_callback: () => MediaStream;
|
||||
export let icon: string | undefined | ComponentType = undefined;
|
||||
export let icon_button_color: string = "var(--color-accent)";
|
||||
export let pulse_color: string = "var(--color-accent)";
|
||||
export let wave_color: string = "var(--color-accent)";
|
||||
|
||||
let audioContext: AudioContext;
|
||||
let analyser: AnalyserNode;
|
||||
let dataArray: Uint8Array;
|
||||
let animationId: number;
|
||||
export let pulseScale = 1;
|
||||
|
||||
$: containerWidth = icon
|
||||
? "128px"
|
||||
: `calc((var(--boxSize) + var(--gutter)) * ${numBars} + 80px)`;
|
||||
|
||||
$: if (stream_state === "open") setupAudioContext();
|
||||
|
||||
onDestroy(() => {
|
||||
if (animationId) {
|
||||
cancelAnimationFrame(animationId);
|
||||
}
|
||||
if (audioContext) {
|
||||
audioContext.close();
|
||||
}
|
||||
});
|
||||
|
||||
function setupAudioContext() {
|
||||
// @ts-ignore
|
||||
audioContext = new (window.AudioContext || window.webkitAudioContext)();
|
||||
analyser = audioContext.createAnalyser();
|
||||
const streamSource = audio_source_callback()
|
||||
if(!streamSource)return
|
||||
const source = audioContext.createMediaStreamSource(
|
||||
streamSource,
|
||||
);
|
||||
|
||||
source.connect(analyser);
|
||||
|
||||
analyser.fftSize = 64;
|
||||
analyser.smoothingTimeConstant = 0.8;
|
||||
dataArray = new Uint8Array(analyser.frequencyBinCount);
|
||||
|
||||
updateVisualization();
|
||||
}
|
||||
|
||||
function updateVisualization() {
|
||||
analyser.getByteFrequencyData(dataArray);
|
||||
|
||||
// Update bars
|
||||
const bars = document.querySelectorAll('.gradio-webrtc-waveContainer .gradio-webrtc-box');
|
||||
for (let i = 0; i < bars.length; i++) {
|
||||
const barHeight = (dataArray[transformIndex(i)] / 255);
|
||||
bars[i].style.transform = `scaleY(${Math.max(0.1, barHeight)})`;
|
||||
bars[i].style.background = wave_color;
|
||||
bars[i].style.opacity = 0.5;
|
||||
}
|
||||
|
||||
animationId = requestAnimationFrame(updateVisualization);
|
||||
}
|
||||
|
||||
// 声波高度从两侧向中间收拢
|
||||
function transformIndex(index: number): number {
|
||||
const mapping = [0, 2, 4, 6, 8, 10, 12, 14, 15, 13, 11, 9, 7, 5, 3, 1];
|
||||
if (index < 0 || index >= mapping.length) {
|
||||
throw new Error("Index must be between 0 and 15");
|
||||
}
|
||||
return mapping[index];
|
||||
}
|
||||
</script>
|
||||
|
||||
<div class="gradio-webrtc-waveContainer">
|
||||
{#if icon && !pending}
|
||||
<div class="gradio-webrtc-icon-container">
|
||||
<div
|
||||
class="gradio-webrtc-icon"
|
||||
style:transform={`scale(${pulseScale})`}
|
||||
style:background={icon_button_color}
|
||||
>
|
||||
<PulsingIcon
|
||||
{stream_state}
|
||||
{pulse_color}
|
||||
{icon}
|
||||
{icon_button_color}
|
||||
{icon_radius}
|
||||
{audio_source_callback}
|
||||
/>
|
||||
</div>
|
||||
</div>
|
||||
{:else}
|
||||
<div class="gradio-webrtc-boxContainer" style:width={containerWidth}>
|
||||
{#each Array(numBars/2) as _}
|
||||
<div class="gradio-webrtc-box"></div>
|
||||
{/each}
|
||||
<div class="split-container"></div>
|
||||
{#each Array(numBars/2) as _}
|
||||
<div class="gradio-webrtc-box"></div>
|
||||
{/each}
|
||||
</div>
|
||||
{/if}
|
||||
</div>
|
||||
|
||||
<style>
|
||||
.gradio-webrtc-waveContainer {
|
||||
position: relative;
|
||||
display: flex;
|
||||
min-height: 100px;
|
||||
max-height: 128px;
|
||||
justify-content: center;
|
||||
align-items: center;
|
||||
}
|
||||
|
||||
.gradio-webrtc-boxContainer {
|
||||
display: flex;
|
||||
justify-content: space-between;
|
||||
height: 64px;
|
||||
--boxSize: 4px;
|
||||
--gutter: 4px;
|
||||
|
||||
}
|
||||
.split-container {
|
||||
width: 80px;
|
||||
}
|
||||
.gradio-webrtc-box {
|
||||
height: 100%;
|
||||
width: var(--boxSize);
|
||||
background: var(--color-accent);
|
||||
border-radius: 8px;
|
||||
transition: transform 0.05s ease;
|
||||
}
|
||||
|
||||
.gradio-webrtc-icon-container {
|
||||
position: relative;
|
||||
width: 128px;
|
||||
height: 128px;
|
||||
display: flex;
|
||||
justify-content: center;
|
||||
align-items: center;
|
||||
}
|
||||
|
||||
.gradio-webrtc-icon {
|
||||
position: relative;
|
||||
width: 48px;
|
||||
height: 48px;
|
||||
border-radius: 50%;
|
||||
transition: transform 0.1s ease;
|
||||
display: flex;
|
||||
justify-content: center;
|
||||
align-items: center;
|
||||
z-index: 2;
|
||||
}
|
||||
|
||||
.icon-image {
|
||||
width: 32px;
|
||||
height: 32px;
|
||||
object-fit: contain;
|
||||
filter: brightness(0) invert(1);
|
||||
}
|
||||
|
||||
.pulse-ring {
|
||||
position: absolute;
|
||||
top: 50%;
|
||||
left: 50%;
|
||||
transform: translate(-50%, -50%);
|
||||
width: 48px;
|
||||
height: 48px;
|
||||
border-radius: 50%;
|
||||
animation: pulse 2s cubic-bezier(0.4, 0, 0.6, 1) infinite;
|
||||
opacity: 0.5;
|
||||
}
|
||||
|
||||
@keyframes pulse {
|
||||
0% {
|
||||
transform: translate(-50%, -50%) scale(1);
|
||||
opacity: 0.5;
|
||||
}
|
||||
100% {
|
||||
transform: translate(-50%, -50%) scale(var(--max-scale, 3));
|
||||
opacity: 0;
|
||||
}
|
||||
}
|
||||
|
||||
.dots {
|
||||
display: flex;
|
||||
gap: 8px;
|
||||
align-items: center;
|
||||
height: 64px;
|
||||
}
|
||||
|
||||
.dot {
|
||||
width: 12px;
|
||||
height: 12px;
|
||||
border-radius: 50%;
|
||||
opacity: 0.5;
|
||||
animation: pulse 1.5s infinite;
|
||||
}
|
||||
|
||||
.dot:nth-child(2) {
|
||||
animation-delay: 0.2s;
|
||||
}
|
||||
|
||||
.dot:nth-child(3) {
|
||||
animation-delay: 0.4s;
|
||||
}
|
||||
|
||||
@keyframes pulse {
|
||||
0%,
|
||||
100% {
|
||||
opacity: 0.4;
|
||||
transform: scale(1);
|
||||
}
|
||||
50% {
|
||||
opacity: 1;
|
||||
transform: scale(1.1);
|
||||
}
|
||||
}
|
||||
</style>
|
||||
105
frontend/shared/VideoChat/components/ChatBtn.svelte
Normal file
105
frontend/shared/VideoChat/components/ChatBtn.svelte
Normal file
@@ -0,0 +1,105 @@
|
||||
<script lang="ts">
|
||||
import { Spinner } from "@gradio/icons";
|
||||
import AudioWave from "./AudioWave.svelte";
|
||||
import { createEventDispatcher } from 'svelte';
|
||||
|
||||
const dispatch = createEventDispatcher();
|
||||
|
||||
export let stream_state;
|
||||
export let onStartChat
|
||||
export let audio_source_callback
|
||||
export let wave_color
|
||||
export let assetLoaded = true
|
||||
</script>
|
||||
|
||||
<div class="player-controls">
|
||||
<!-- svelte-ignore a11y-click-events-have-key-events -->
|
||||
<!-- svelte-ignore a11y-no-static-element-interactions -->
|
||||
<div
|
||||
class="chat-btn"
|
||||
class:start-chat={stream_state === "closed"}
|
||||
class:stop-chat={stream_state === "open" && assetLoaded === true}
|
||||
on:click={onStartChat}
|
||||
>
|
||||
{#if stream_state === "closed"}
|
||||
<span>点击开始对话</span>
|
||||
{:else if stream_state === "waiting" || assetLoaded === false}
|
||||
<div class="waiting-icon-text">
|
||||
<div class="icon" title="spinner">
|
||||
<Spinner />
|
||||
</div>
|
||||
<span>等待中</span>
|
||||
</div>
|
||||
{:else}
|
||||
<div class="stop-chat-inner"></div>
|
||||
{/if}
|
||||
</div>
|
||||
{#if stream_state === "open" && assetLoaded === true}
|
||||
<div class="input-audio-wave">
|
||||
<AudioWave {audio_source_callback} {stream_state} {wave_color} />
|
||||
</div>
|
||||
{/if}
|
||||
</div>
|
||||
|
||||
<style lang="less">
|
||||
.player-controls {
|
||||
height: 15%;
|
||||
position: relative;
|
||||
display: flex;
|
||||
justify-content: center;
|
||||
align-items: center;
|
||||
min-height: 84px;
|
||||
|
||||
.chat-btn {
|
||||
height: 64px;
|
||||
width: 296px;
|
||||
display: flex;
|
||||
justify-content: center;
|
||||
align-items: center;
|
||||
border-radius: 999px;
|
||||
opacity: 1;
|
||||
background: linear-gradient(180deg, #7873f6 0%, #524de1 100%);
|
||||
transition: all 0.3s;
|
||||
z-index: 2;
|
||||
cursor: pointer;
|
||||
}
|
||||
.start-chat {
|
||||
font-size: 16px;
|
||||
font-weight: 500;
|
||||
text-align: center;
|
||||
color: #ffffff;
|
||||
}
|
||||
.waiting-icon-text {
|
||||
width: 80px;
|
||||
align-items: center;
|
||||
font-size: 16px;
|
||||
font-weight: 500;
|
||||
color: #ffffff;
|
||||
margin: 0 var(--spacing-sm);
|
||||
display: flex;
|
||||
justify-content: space-evenly;
|
||||
gap: var(--size-1);
|
||||
.icon {
|
||||
width: 25px;
|
||||
height: 25px;
|
||||
fill: #ffffff;
|
||||
stroke: #ffffff;
|
||||
color: #ffffff;
|
||||
}
|
||||
}
|
||||
|
||||
.stop-chat {
|
||||
width: 64px;
|
||||
.stop-chat-inner {
|
||||
width: 25px;
|
||||
height: 25px;
|
||||
border-radius: 6.25px;
|
||||
background: #fafafa;
|
||||
}
|
||||
}
|
||||
|
||||
.input-audio-wave {
|
||||
position: absolute;
|
||||
}
|
||||
}
|
||||
</style>
|
||||
181
frontend/shared/VideoChat/components/ChatInput.svelte
Normal file
181
frontend/shared/VideoChat/components/ChatInput.svelte
Normal file
@@ -0,0 +1,181 @@
|
||||
<script lang="ts">
|
||||
import { IconFont, Send, Stop } from "../icons";
|
||||
import { insertStringAt } from "../utils";
|
||||
|
||||
export let replying;
|
||||
export let onSend;
|
||||
export let onStop;
|
||||
export let onInterrupt;
|
||||
|
||||
let inputHeight = 24;
|
||||
let rowsDivRef: HTMLDivElement;
|
||||
let chatInputRef: HTMLTextAreaElement;
|
||||
let inputValue = "";
|
||||
function on_chat_input_keydown(event: KeyboardEvent) {
|
||||
if (event.key === "Enter") {
|
||||
if (event.altKey) {
|
||||
chatInputRef.value = insertStringAt(
|
||||
chatInputRef.value,
|
||||
"\n",
|
||||
chatInputRef.selectionStart,
|
||||
);
|
||||
chatInputRef.dispatchEvent(new InputEvent("input"));
|
||||
} else {
|
||||
event.preventDefault();
|
||||
on_send();
|
||||
}
|
||||
}
|
||||
}
|
||||
async function on_send() {
|
||||
await onSend(chatInputRef.value);
|
||||
chatInputRef.value = "";
|
||||
}
|
||||
function on_chat_input(event: InputEvent) {
|
||||
if (rowsDivRef) {
|
||||
rowsDivRef.textContent = (event.target as any).value.replace(
|
||||
/\n$/,
|
||||
"\n\n",
|
||||
);
|
||||
inputHeight = rowsDivRef.offsetHeight;
|
||||
}
|
||||
}
|
||||
</script>
|
||||
|
||||
<div class="chat-input-container">
|
||||
<!-- svelte-ignore a11y-click-events-have-key-events -->
|
||||
<!-- svelte-ignore a11y-no-static-element-interactions -->
|
||||
<div class="stop-chat-btn" on:click={onStop}></div>
|
||||
|
||||
<div class="chat-input-inner">
|
||||
<div class="chat-input-wrapper">
|
||||
<textarea
|
||||
class="chat-input"
|
||||
bind:this={chatInputRef}
|
||||
on:keydown={on_chat_input_keydown}
|
||||
on:input={on_chat_input}
|
||||
style={`height:${inputHeight}px`}
|
||||
/>
|
||||
<div class="rowsDiv" bind:this={rowsDivRef}>{inputValue}</div>
|
||||
</div>
|
||||
{#if replying}
|
||||
<button class="interrupt-btn" on:click={onInterrupt}></button>
|
||||
{:else}
|
||||
<button class="send-btn" on:click={on_send}>
|
||||
<IconFont icon={Send} color={"#fff"} ></IconFont>
|
||||
</button>
|
||||
{/if}
|
||||
<div class="chat-tip">Texts are ignored during responding.</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<style lang="less">
|
||||
.chat-input-container {
|
||||
height: 15%;
|
||||
position: relative;
|
||||
display: flex;
|
||||
justify-content: center;
|
||||
align-items: center;
|
||||
min-height: 84px;
|
||||
// padding: 0 12px;
|
||||
|
||||
.chat-input-inner {
|
||||
position: relative;
|
||||
padding: 0 12px;
|
||||
background-color: #fff;
|
||||
height: 64px;
|
||||
flex: 1;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
border: 1px solid #e8eaf2;
|
||||
border-radius: 12px;
|
||||
border-radius: 20px;
|
||||
box-shadow:
|
||||
0 12px 24px -16px rgba(54, 54, 73, 0.04),
|
||||
0 12px 40px 0 rgba(51, 51, 71, 0.08),
|
||||
0 0 1px 0 rgba(44, 44, 54, 0.02);
|
||||
.chat-tip {
|
||||
position: absolute;
|
||||
top: 100%;
|
||||
color: #cecece;
|
||||
}
|
||||
.chat-input::placeholder {
|
||||
font-size: 12px;
|
||||
}
|
||||
.chat-input-wrapper {
|
||||
flex: 1;
|
||||
position: relative;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
.chat-input {
|
||||
width: 100%;
|
||||
border: none;
|
||||
outline: none;
|
||||
color: #26244c;
|
||||
font-size: 16px;
|
||||
font-weight: 400;
|
||||
resize: none;
|
||||
padding: 0;
|
||||
margin: 8px 0;
|
||||
line-height: 24px;
|
||||
max-height: 48px;
|
||||
min-height: 24px;
|
||||
}
|
||||
.rowsDiv {
|
||||
position: absolute;
|
||||
left: 0;
|
||||
right: 0;
|
||||
z-index: -1;
|
||||
visibility: hidden;
|
||||
font-size: 16px;
|
||||
font-weight: 400;
|
||||
line-height: 24px;
|
||||
white-space: pre-wrap;
|
||||
word-wrap: break-word;
|
||||
}
|
||||
}
|
||||
|
||||
.send-btn,.interrupt-btn {
|
||||
flex: 0 0 auto;
|
||||
background: #615ced;
|
||||
border-radius: 20px;
|
||||
height: 28px;
|
||||
width: 28px;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
margin-left: 16px;
|
||||
cursor: pointer;
|
||||
}
|
||||
.interrupt-btn{
|
||||
&::after {
|
||||
content: " ";
|
||||
width: 12px;
|
||||
height: 12px;
|
||||
border-radius: 2px;
|
||||
background: #fafafa;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
.stop-chat-btn {
|
||||
cursor: pointer;
|
||||
margin-right: 12px;
|
||||
height: 28px;
|
||||
width: 28px;
|
||||
display: flex;
|
||||
justify-content: center;
|
||||
align-items: center;
|
||||
border-radius: 999px;
|
||||
opacity: 1;
|
||||
background: linear-gradient(180deg, #7873f6 0%, #524de1 100%);
|
||||
|
||||
&::after {
|
||||
content: " ";
|
||||
width: 12px;
|
||||
height: 12px;
|
||||
border-radius: 2px;
|
||||
background: #fafafa;
|
||||
}
|
||||
}
|
||||
}
|
||||
</style>
|
||||
34
frontend/shared/VideoChat/components/ChatMessage.svelte
Normal file
34
frontend/shared/VideoChat/components/ChatMessage.svelte
Normal file
@@ -0,0 +1,34 @@
|
||||
<script lang="ts">
|
||||
export let message;
|
||||
export let style = "";
|
||||
export let role = "";
|
||||
|
||||
$: classnames = `answer-message-container ${role}`
|
||||
</script>
|
||||
|
||||
<div class={classnames} {style}>
|
||||
<div class="answer-message-text">
|
||||
{message}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<style lang="less">
|
||||
.answer-message-container {
|
||||
padding: 6px 12px;
|
||||
background: rgba(255, 255, 255, 0.8);
|
||||
border-radius: 12px;
|
||||
color: #26244c;
|
||||
|
||||
&.human {
|
||||
background: #dddddd99;
|
||||
// margin-left: 20px;
|
||||
margin-right: 0;
|
||||
}
|
||||
|
||||
&.avatar {
|
||||
background: #9189fa;
|
||||
color: #ffffff;
|
||||
// margin-right: 20px;
|
||||
}
|
||||
}
|
||||
</style>
|
||||
65
frontend/shared/VideoChat/components/ChatRecords.svelte
Normal file
65
frontend/shared/VideoChat/components/ChatRecords.svelte
Normal file
@@ -0,0 +1,65 @@
|
||||
<script lang="ts">
|
||||
import { tick } from "svelte";
|
||||
import ChatMessage from "./ChatMessage.svelte";
|
||||
|
||||
export let chatRecords;
|
||||
|
||||
let containerRef: HTMLElement
|
||||
$: if(chatRecords){
|
||||
tick().then(() => {
|
||||
scrollToBottom()
|
||||
})
|
||||
}
|
||||
function scrollToBottom() {
|
||||
// console.log("🚀 ~ scrollToBottom ~ scrollToBottom:")
|
||||
if(containerRef){
|
||||
containerRef.scrollTop = containerRef.scrollHeight;
|
||||
}
|
||||
}
|
||||
|
||||
export const expose = { scrollToBottom };
|
||||
</script>
|
||||
|
||||
<div class="chat-records" bind:this={containerRef}>
|
||||
<div class="chat-records-inner">
|
||||
{#each chatRecords as item, i (item.id)}
|
||||
<div class={`chat-message ${item.role}`}>
|
||||
<ChatMessage message={item.message} role={item.role}></ChatMessage>
|
||||
</div>
|
||||
{/each}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<style lang="less">
|
||||
.chat-records{
|
||||
width: 100%;
|
||||
height: 100%;
|
||||
overflow-y: auto;
|
||||
&::-webkit-scrollbar {
|
||||
display: none;
|
||||
}
|
||||
}
|
||||
.chat-records-inner {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
align-items: flex-end;
|
||||
justify-content: end;
|
||||
width: 100%;
|
||||
// height: 100%;
|
||||
height: auto;
|
||||
min-height: 100%;
|
||||
.chat-message {
|
||||
margin-bottom: 12px;
|
||||
max-width: 80%;
|
||||
&.human {
|
||||
align-self: flex-end;
|
||||
}
|
||||
&.avatar {
|
||||
align-self: flex-start;
|
||||
}
|
||||
&:last-child {
|
||||
margin-bottom: 0;
|
||||
}
|
||||
}
|
||||
}
|
||||
</style>
|
||||
177
frontend/shared/VideoChat/gaussianAvatar.ts
Normal file
177
frontend/shared/VideoChat/gaussianAvatar.ts
Normal file
@@ -0,0 +1,177 @@
|
||||
import EventEmitter from "eventemitter3";
|
||||
import { Player } from "./helpers/player";
|
||||
import { Processor } from "./helpers/processor";
|
||||
import { type WS } from "./helpers/ws.js";
|
||||
import { EventTypes, PlayerEventTypes } from "./interface/eventType";
|
||||
import { TYVoiceChatState } from "./interface/voiceChat";
|
||||
// import * as GaussianSplats3D from "./gaussian-splats-3d.module.js";
|
||||
import * as GaussianSplats3D from "gaussian-splat-renderer-for-lam";
|
||||
import { WsEventTypes } from "./interface/eventType";
|
||||
|
||||
interface GaussianOptions {
|
||||
container: HTMLDivElement
|
||||
assetsPath: string
|
||||
ws: WS,
|
||||
downloadProgress?: (percent: number) => void;
|
||||
loadProgress?: (percent: number) => void;
|
||||
}
|
||||
|
||||
export class GaussianAvatar extends EventEmitter {
|
||||
private _avatarDivEle: HTMLDivElement;
|
||||
private _assetsPath = "";
|
||||
private _ws: WS;
|
||||
private _downloadProgress: (percent: number) => void;
|
||||
private _loadProgress: (percent: number) => void;
|
||||
private _loadPercent = 0;
|
||||
private _downloadPercent = 0;
|
||||
private _processor: Processor;
|
||||
private _renderer: any;
|
||||
private _audioMute = false;
|
||||
public curState = TYVoiceChatState.Idle;
|
||||
constructor(options: GaussianOptions) {
|
||||
const { container, assetsPath, ws, downloadProgress, loadProgress } = options
|
||||
super();
|
||||
this._avatarDivEle = container;
|
||||
this._assetsPath = assetsPath;
|
||||
this._ws = ws;
|
||||
if (downloadProgress) {
|
||||
this._downloadProgress = (percent: number) => {
|
||||
this._downloadPercent = percent;
|
||||
downloadProgress(percent)
|
||||
};
|
||||
} else {
|
||||
this._downloadProgress = (percent: number) => {
|
||||
this._downloadPercent = percent;
|
||||
};
|
||||
}
|
||||
if(loadProgress) {
|
||||
this._loadProgress = (percent: number) => {
|
||||
this._loadPercent = percent;
|
||||
loadProgress(percent)
|
||||
};
|
||||
} else {
|
||||
this._loadProgress = (percent: number) => {
|
||||
this._loadPercent = percent;
|
||||
};
|
||||
}
|
||||
this._init();
|
||||
}
|
||||
private _init() {
|
||||
if (!this._avatarDivEle || !this._assetsPath || !this._ws) {
|
||||
throw new Error(
|
||||
"Lack of necessary initialization parameters for gaussian render",
|
||||
);
|
||||
}
|
||||
this._processor = new Processor(this);
|
||||
this._bindEventTypes();
|
||||
}
|
||||
public start() {
|
||||
this.getData();
|
||||
this.render();
|
||||
}
|
||||
|
||||
public async getData() {
|
||||
this._ws.on(WsEventTypes.WS_MESSAGE, (data: Blob) => {
|
||||
if (this._downloadPercent < 1 || this._loadPercent < 1) { // 本地数字人未加载完成前，不处理数据
|
||||
return
|
||||
}
|
||||
|
||||
this.emit(EventTypes.MessageReceived, this.curState);
|
||||
|
||||
this._processor.add({
|
||||
avatar_motion_data: {
|
||||
first_package: true, // 是否首包
|
||||
segment_num: 1, // 分片数量，首包存在该值
|
||||
binary_size: data.size, // 数据大小，首包存在该值
|
||||
use_binary_frame: false, // 是否使用二进制帧，首包存在该值
|
||||
},
|
||||
});
|
||||
|
||||
this._processor.add({
|
||||
avatar_motion_data: {
|
||||
first_package: false,
|
||||
motion_data_slice: data, // 数据分片，非首包存在该值
|
||||
is_audio_mute: this._audioMute, // 音频片段是否静音，非首包存在该值
|
||||
},
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
public async render() {
|
||||
this._renderer = await GaussianSplats3D.GaussianSplatRenderer.getInstance(
|
||||
this._avatarDivEle,
|
||||
this._assetsPath,
|
||||
{
|
||||
getChatState: this.getChatState.bind(this),
|
||||
getExpressionData: this.getArkitFaceFrame.bind(this),
|
||||
downloadProgress: this._downloadProgress.bind(this),
|
||||
loadProgress: this._loadProgress.bind(this),
|
||||
},
|
||||
);
|
||||
}
|
||||
public setAvatarMute(isMute: boolean) {
|
||||
this._processor.setMute(isMute);
|
||||
this._audioMute = isMute;
|
||||
}
|
||||
public getChatState() {
|
||||
return this.curState;
|
||||
}
|
||||
public getArkitFaceFrame() {
|
||||
return this._processor?.getArkitFaceFrame().arkitFace;
|
||||
}
|
||||
public interrupt(): void {
|
||||
this._ws.send("%interrupt%"); // 约定的打断标识
|
||||
this._processor?.interrupt();
|
||||
this.curState = TYVoiceChatState.Idle;
|
||||
this.emit(EventTypes.StateChanged, this.curState);
|
||||
}
|
||||
public sendSpeech(data: string | Int8Array | Uint8Array) {
|
||||
this._ws.send(data);
|
||||
this.curState = TYVoiceChatState.Listening;
|
||||
this.emit(EventTypes.StateChanged, this.curState);
|
||||
this._processor?.clear();
|
||||
}
|
||||
public exit() {
|
||||
this._renderer?.dispose();
|
||||
this.curState = TYVoiceChatState.Idle;
|
||||
this._downloadPercent = 0;
|
||||
this._loadPercent = 0;
|
||||
this._processor?.clear();
|
||||
this.removeAllListeners();
|
||||
}
|
||||
private _bindEventTypes() {
|
||||
this.on(PlayerEventTypes.Player_StartSpeaking, (player: Player) => {
|
||||
console.log('startSpeach')
|
||||
this.curState = TYVoiceChatState.Responding;
|
||||
this.emit(EventTypes.StateChanged, this.curState);
|
||||
this._ws.send(
|
||||
JSON.stringify({
|
||||
header: { name: EventTypes.StartSpeech },
|
||||
payload: {},
|
||||
}),
|
||||
);
|
||||
});
|
||||
this.on(PlayerEventTypes.Player_EndSpeaking, (player: Player) => {
|
||||
console.log('endSpeach')
|
||||
this.curState = TYVoiceChatState.Idle;
|
||||
this.emit(EventTypes.StateChanged, this.curState);
|
||||
this._ws.send(
|
||||
JSON.stringify({ header: { name: EventTypes.EndSpeech }, payload: {} }),
|
||||
);
|
||||
});
|
||||
this.on(EventTypes.ErrorReceived, (data) => {
|
||||
console.log('ErrorReceived', data)
|
||||
this.curState = TYVoiceChatState.Idle;
|
||||
this.emit(EventTypes.StateChanged, this.curState);
|
||||
this._ws.send(
|
||||
JSON.stringify({
|
||||
header: { name: EventTypes.ErrorReceived },
|
||||
payload: { ...data },
|
||||
}),
|
||||
);
|
||||
});
|
||||
this._ws.on(WsEventTypes.WS_CLOSE, () => {
|
||||
this.exit()
|
||||
})
|
||||
}
|
||||
}
|
||||
268
frontend/shared/VideoChat/helpers/player.ts
Normal file
268
frontend/shared/VideoChat/helpers/player.ts
Normal file
@@ -0,0 +1,268 @@
|
||||
import type EventEmitter from 'eventemitter3'
|
||||
import { nanoid } from 'nanoid'
|
||||
|
||||
import { PlayerEventTypes } from '../interface/eventType'
|
||||
interface IOption {
|
||||
// 传入的数据是采用多少位编码，默认16位
|
||||
channels: number
|
||||
// 缓存时间 单位 ms
|
||||
fftSize: number
|
||||
|
||||
inputCodec: 'Int8' | 'Int16' | 'Int32' | 'Float32'
|
||||
// analyserNode fftSize
|
||||
onended: (extParams?: IExtInfo) => void
|
||||
// 采样率 单位Hz
|
||||
sampleRate: number
|
||||
// 是否静音
|
||||
isMute: boolean
|
||||
}
|
||||
interface ITypedArrays {
|
||||
Float32: typeof Float32Array
|
||||
Int16: typeof Int16Array
|
||||
Int32: typeof Int32Array
|
||||
Int8: typeof Int8Array
|
||||
}
|
||||
type IExtInfo = Record<string, unknown>
|
||||
interface ISamples {
|
||||
data: Float32Array
|
||||
end_of_batch: boolean
|
||||
startTime?: number
|
||||
}
|
||||
export class Player {
|
||||
static isTypedArray(
|
||||
data: Int8Array | Int16Array | Int32Array | Float32Array
|
||||
) {
|
||||
// 检测输入的数据是否为 TypedArray 类型或 ArrayBuffer 类型
|
||||
return (
|
||||
(data.byteLength &&
|
||||
data.buffer &&
|
||||
data.buffer.constructor === ArrayBuffer) ||
|
||||
data.constructor === ArrayBuffer
|
||||
)
|
||||
}
|
||||
id = nanoid()
|
||||
analyserNode?: AnalyserNode
|
||||
audioCtx?: AudioContext
|
||||
// 是否自动播放
|
||||
autoPlay = true
|
||||
bufferSource?: AudioBufferSourceNode
|
||||
convertValue = 32768
|
||||
ee: EventEmitter
|
||||
gainNode?: GainNode
|
||||
option: IOption = {
|
||||
inputCodec: 'Int16', // 传入的数据是采用多少位编码，默认16位
|
||||
channels: 1, // 声道数
|
||||
sampleRate: 8000, // 采样率 单位Hz
|
||||
fftSize: 2048, // analyserNode fftSize
|
||||
onended: () => {}
|
||||
}
|
||||
samplesList: ISamples[] = []
|
||||
|
||||
startTime?: number
|
||||
typedArray?:
|
||||
| typeof Int8Array
|
||||
| typeof Int16Array
|
||||
| typeof Int32Array
|
||||
| typeof Float32Array
|
||||
|
||||
_firstStartRelativeTime?: number
|
||||
_firstStartAbsoluteTime?: number
|
||||
|
||||
constructor(option: IOption, ee: EventEmitter) {
|
||||
this.ee = ee
|
||||
this.init(option)
|
||||
}
|
||||
|
||||
async continue() {
|
||||
await this.audioCtx!.resume()
|
||||
}
|
||||
destroy() {
|
||||
this.samplesList = []
|
||||
this.audioCtx?.close()
|
||||
this.audioCtx = undefined
|
||||
}
|
||||
feed(audioOptions: {
|
||||
audio: Int8Array | Int16Array | Int32Array | Float32Array
|
||||
end_of_batch: boolean
|
||||
}) {
|
||||
let { audio } = audioOptions
|
||||
const { end_of_batch } = audioOptions
|
||||
if (!audio) {
|
||||
return
|
||||
}
|
||||
this._isSupported(audio)
|
||||
// 获取格式化后的buffer
|
||||
audio = this._getFormattedValue(audio)
|
||||
// 开始拷贝buffer数据
|
||||
// 新建一个Float32Array的空间
|
||||
const data = new Float32Array(audio.length)
|
||||
// 复制传入的新数据
|
||||
// 从历史buff位置开始
|
||||
data.set(audio, 0)
|
||||
// 将新的完整buff数据赋值给samples
|
||||
const samples = {
|
||||
data,
|
||||
end_of_batch
|
||||
}
|
||||
this.samplesList.push(samples)
|
||||
this.flush(samples, this.samplesList.length - 1)
|
||||
}
|
||||
flush(samples: ISamples, index: number) {
|
||||
if (!(samples && this.autoPlay && this.audioCtx)) return
|
||||
const { data, end_of_batch } = samples
|
||||
if (this.bufferSource) {
|
||||
this.bufferSource.onended = () => {}
|
||||
}
|
||||
this.bufferSource = this.audioCtx!.createBufferSource()
|
||||
if (typeof this.option.onended === 'function') {
|
||||
this.bufferSource.onended = () => {
|
||||
if (!end_of_batch && index === this.samplesList.length - 1) {
|
||||
this.ee.emit(PlayerEventTypes.Player_WaitNextAudioClip)
|
||||
}
|
||||
this.option.onended()
|
||||
}
|
||||
}
|
||||
const length = data.length / this.option.channels
|
||||
const audioBuffer = this.audioCtx!.createBuffer(
|
||||
this.option.channels,
|
||||
length,
|
||||
this.option.sampleRate
|
||||
)
|
||||
|
||||
for (let channel = 0; channel < this.option.channels; channel++) {
|
||||
const audioData = audioBuffer.getChannelData(channel)
|
||||
let offset = channel
|
||||
let decrement = 50
|
||||
for (let i = 0; i < length; i++) {
|
||||
audioData[i] = data[offset]
|
||||
/* fadein */
|
||||
if (i < 50) {
|
||||
audioData[i] = (audioData[i] * i) / 50
|
||||
}
|
||||
/* fadeout */
|
||||
if (i >= length - 51) {
|
||||
audioData[i] = (audioData[i] * decrement--) / 50
|
||||
}
|
||||
offset += this.option.channels
|
||||
}
|
||||
}
|
||||
|
||||
if (this.startTime! < this.audioCtx!.currentTime) {
|
||||
this.startTime = this.audioCtx!.currentTime
|
||||
}
|
||||
this.bufferSource.buffer = audioBuffer
|
||||
this.bufferSource.connect(this.gainNode!)
|
||||
this.bufferSource.connect(this.analyserNode!) // bufferSource连接到analyser
|
||||
this.bufferSource.start(this.startTime)
|
||||
samples.startTime = this.startTime
|
||||
if (this._firstStartAbsoluteTime === undefined) {
|
||||
this._firstStartAbsoluteTime = Date.now()
|
||||
}
|
||||
if (this._firstStartRelativeTime === undefined) {
|
||||
this._firstStartRelativeTime = this.startTime
|
||||
this.ee.emit(PlayerEventTypes.Player_StartSpeaking, this)
|
||||
}
|
||||
this.startTime! += audioBuffer.duration
|
||||
}
|
||||
init(option: IOption) {
|
||||
this.option = Object.assign(this.option, option) // 实例最终配置参数
|
||||
this.convertValue = this._getConvertValue()
|
||||
this.typedArray = this._getTypedArray()
|
||||
this.initAudioContext()
|
||||
}
|
||||
initAudioContext() {
|
||||
// 初始化音频上下文的东西
|
||||
this.audioCtx = new (window.AudioContext || window.webkitAudioContext)()
|
||||
// 控制音量的 GainNode
|
||||
// https://developer.mozilla.org/en-US/docs/Web/API/BaseAudioContext/createGain
|
||||
this.gainNode = this.audioCtx.createGain()
|
||||
this.gainNode.gain.value = this.option.isMute ? 0 : 1
|
||||
this.gainNode.connect(this.audioCtx.destination)
|
||||
this.startTime = this.audioCtx.currentTime
|
||||
this.analyserNode = this.audioCtx.createAnalyser()
|
||||
this.analyserNode.fftSize = this.option.fftSize
|
||||
}
|
||||
setMute(isMute: boolean) {
|
||||
this.gainNode!.gain.value = isMute ? 0 : 1;
|
||||
}
|
||||
async pause() {
|
||||
await this.audioCtx!.suspend()
|
||||
}
|
||||
async updateAutoPlay(value: boolean) {
|
||||
if (this.autoPlay !== value && value) {
|
||||
this.autoPlay = value
|
||||
this.samplesList.forEach((sample, index) => {
|
||||
this.flush(sample, index)
|
||||
})
|
||||
} else {
|
||||
this.autoPlay = value
|
||||
}
|
||||
}
|
||||
|
||||
volume(volume: number) {
|
||||
this.gainNode!.gain.value = volume
|
||||
}
|
||||
_getFormattedValue(data: Int8Array | Int16Array | Int32Array | Float32Array) {
|
||||
const TargetArray = this.typedArray!
|
||||
if (data.constructor === ArrayBuffer) {
|
||||
data = new TargetArray(data)
|
||||
} else {
|
||||
data = new TargetArray(data.buffer)
|
||||
}
|
||||
|
||||
const float32 = new Float32Array(data.length)
|
||||
|
||||
for (let i = 0; i < data.length; i++) {
|
||||
// buffer 缓冲区的数据，需要是IEEE754 里32位的线性PCM，范围从-1到+1
|
||||
// 所以对数据进行除法
|
||||
// 除以对应的位数范围，得到-1到+1的数据
|
||||
// float32[i] = data[i] / 0x8000;
|
||||
float32[i] = data[i] / this.convertValue
|
||||
}
|
||||
return float32
|
||||
}
|
||||
|
||||
private _isSupported(
|
||||
data: Int8Array | Int16Array | Int32Array | Float32Array
|
||||
) {
|
||||
// 数据类型是否支持
|
||||
// 目前支持 ArrayBuffer 或者 TypedArray
|
||||
if (!Player.isTypedArray(data))
|
||||
throw new Error('请传入ArrayBuffer或者任意TypedArray')
|
||||
return true
|
||||
}
|
||||
|
||||
private _getConvertValue() {
|
||||
// 根据传入的目标编码位数
|
||||
// 选定转换数据所需要的基本值
|
||||
const inputCodecs = {
|
||||
Int8: 128,
|
||||
Int16: 32768,
|
||||
Int32: 2147483648,
|
||||
Float32: 1
|
||||
}
|
||||
if (!inputCodecs[this.option.inputCodec])
|
||||
throw new Error(
|
||||
'wrong codec.please input one of these codecs:Int8,Int16,Int32,Float32'
|
||||
)
|
||||
return inputCodecs[this.option.inputCodec]
|
||||
}
|
||||
|
||||
private _getTypedArray() {
|
||||
// 根据传入的目标编码位数
|
||||
// 选定前端的所需要的保存的二进制数据格式
|
||||
// 完整TypedArray请看文档
|
||||
// https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/TypedArray
|
||||
const typedArrays = {
|
||||
Int8: Int8Array,
|
||||
Int16: Int16Array,
|
||||
Int32: Int32Array,
|
||||
Float32: Float32Array
|
||||
} as ITypedArrays
|
||||
if (!typedArrays[this.option.inputCodec])
|
||||
throw new Error(
|
||||
'wrong codec.please input one of these codecs:Int8,Int16,Int32,Float32'
|
||||
)
|
||||
return typedArrays[this.option.inputCodec]
|
||||
}
|
||||
}
|
||||
610
frontend/shared/VideoChat/helpers/processor.ts
Normal file
610
frontend/shared/VideoChat/helpers/processor.ts
Normal file
@@ -0,0 +1,610 @@
|
||||
import EventEmitter from "eventemitter3";
|
||||
import PQueue from "p-queue";
|
||||
|
||||
import { mergeBlob, unpack } from "../binary_utils";
|
||||
import {
|
||||
EventTypes,
|
||||
PlayerEventTypes,
|
||||
ProcessorEventTypes,
|
||||
} from "../interface/eventType";
|
||||
import { Player } from "./player";
|
||||
|
||||
export type IPayload = Record<string, string | number | object | Blob>;
|
||||
|
||||
interface IDataRecords {
|
||||
channel_names?: string[];
|
||||
data_id: number;
|
||||
data_offset: number;
|
||||
data_type: string;
|
||||
sample_rate: number;
|
||||
shape: number[];
|
||||
}
|
||||
interface IEvent {
|
||||
avatar_status?: string;
|
||||
event_type: string;
|
||||
speech_id: string;
|
||||
}
|
||||
interface IParsedData {
|
||||
batch_id?: number;
|
||||
batch_name?: string;
|
||||
data_records: Record<string, IDataRecords>;
|
||||
end_of_batch: boolean;
|
||||
events: IEvent[];
|
||||
}
|
||||
interface IAvatarMotionData {
|
||||
// 数据大小，首包存在该值
|
||||
binary_size: number;
|
||||
// 是否首包
|
||||
first_package: boolean;
|
||||
// 数据分片，非首包存在该值
|
||||
motion_data_slice?: Blob;
|
||||
// 分片数量，首包存在该值
|
||||
segment_num?: number;
|
||||
// 分片索引，非首包存在该值
|
||||
slice_index?: number;
|
||||
// 是否使用二进制帧，首包存在该值
|
||||
use_binary_frame?: boolean;
|
||||
// 初始化的音频是否静音
|
||||
is_audio_mute?: boolean;
|
||||
}
|
||||
|
||||
interface IAvatarMotionGroupBase {
|
||||
arkitFaceArrayBufferArray?: ArrayBuffer[];
|
||||
batch_id?: number;
|
||||
batch_name?: string;
|
||||
binSize?: number;
|
||||
jsonSize?: number;
|
||||
merged_motion_data: Uint8Array;
|
||||
motion_data_slices: Blob[];
|
||||
player?: Player;
|
||||
tts2faceArrayBufferArray?: ArrayBuffer[];
|
||||
}
|
||||
interface IAvatarMotionGroup extends IAvatarMotionGroupBase {
|
||||
binary_size: number;
|
||||
first_package: boolean;
|
||||
segment_num?: number;
|
||||
use_binary_frame?: boolean;
|
||||
}
|
||||
const InputCodecs: Record<string, "Int8" | "Int16" | "Int32" | "Float32"> = {
|
||||
int16: "Int16",
|
||||
int32: "Int32",
|
||||
float32: "Float32",
|
||||
};
|
||||
const TypedArrays: Record<
|
||||
string,
|
||||
typeof Int16Array | typeof Int32Array | typeof Float32Array
|
||||
> = {
|
||||
int16: Int16Array,
|
||||
int32: Int32Array,
|
||||
float32: Float32Array,
|
||||
};
|
||||
|
||||
export class Processor {
|
||||
private ee: EventEmitter;
|
||||
private _motionDataGroupHandlerQueue = new PQueue({
|
||||
concurrency: 1,
|
||||
});
|
||||
private _motionDataGroups: IAvatarMotionGroup[] = [];
|
||||
private _arkit_face_sample_rate?: number;
|
||||
private _arkit_face_channel_names?: string[];
|
||||
private _tts2face_sample_rate?: number;
|
||||
private _tts2face_channel_names?: string[];
|
||||
private _maxBatchId?: number;
|
||||
private _arkitFaceShape?: number;
|
||||
private _tts2FaceShape?: number;
|
||||
constructor(ee: EventEmitter) {
|
||||
this.ee = ee;
|
||||
}
|
||||
add(payload: IPayload) {
|
||||
const { avatar_motion_data } = payload;
|
||||
this._motionDataGroupHandlerQueue.add(
|
||||
async () =>
|
||||
await this._motionDataGroupHandler(
|
||||
avatar_motion_data as IAvatarMotionData,
|
||||
),
|
||||
);
|
||||
}
|
||||
clear() {
|
||||
this._motionDataGroups.forEach((group) => {
|
||||
group.player?.destroy();
|
||||
});
|
||||
this._motionDataGroups = [];
|
||||
}
|
||||
setMute(isMute: boolean) {
|
||||
this._motionDataGroups.forEach((group) => {
|
||||
group.player?.setMute(isMute);
|
||||
});
|
||||
}
|
||||
getArkitFaceFrame() {
|
||||
return {
|
||||
arkitFace: this._getArkitFaceFrame(),
|
||||
};
|
||||
}
|
||||
getLastBatchId() {
|
||||
let batch_id = undefined;
|
||||
this._motionDataGroups.forEach((group) => {
|
||||
if (group.batch_id) {
|
||||
batch_id = group.batch_id;
|
||||
}
|
||||
});
|
||||
return batch_id;
|
||||
}
|
||||
getTtt2FaceFrame() {
|
||||
return {
|
||||
tts2Face: this._getTts2FaceFrame(),
|
||||
};
|
||||
}
|
||||
|
||||
interrupt() {
|
||||
this._motionDataGroups.forEach((group) => {
|
||||
if (group.batch_id) {
|
||||
this._maxBatchId = group.batch_id;
|
||||
}
|
||||
group.player?.destroy();
|
||||
});
|
||||
this._motionDataGroups = [];
|
||||
}
|
||||
|
||||
private _getArkitFaceFrame() {
|
||||
if (!this._motionDataGroups.length) {
|
||||
return null;
|
||||
}
|
||||
const targetMotion = this._motionDataGroups.find(
|
||||
(_motion) => _motion.player,
|
||||
);
|
||||
|
||||
if (!targetMotion) {
|
||||
return null;
|
||||
}
|
||||
const { arkitFaceArrayBufferArray, player } = targetMotion!;
|
||||
if (
|
||||
player &&
|
||||
player._firstStartAbsoluteTime &&
|
||||
arkitFaceArrayBufferArray &&
|
||||
arkitFaceArrayBufferArray.length > 0 &&
|
||||
this._arkitFaceShape &&
|
||||
this._arkit_face_sample_rate
|
||||
) {
|
||||
const offsetTime = Date.now() - player._firstStartAbsoluteTime;
|
||||
let lastIndex = 0;
|
||||
let firstSampleStartTime: number;
|
||||
player.samplesList.forEach((item, index) => {
|
||||
if (
|
||||
firstSampleStartTime === undefined &&
|
||||
item.startTime !== undefined
|
||||
) {
|
||||
firstSampleStartTime = item.startTime;
|
||||
}
|
||||
if (
|
||||
item.startTime !== undefined &&
|
||||
item.startTime - firstSampleStartTime <= offsetTime / 1000
|
||||
) {
|
||||
lastIndex = index;
|
||||
}
|
||||
});
|
||||
const samples = player.samplesList[lastIndex];
|
||||
const subOffsetTime = offsetTime - samples.startTime! * 1000;
|
||||
const offset = Math.floor(
|
||||
(subOffsetTime / 1000) * this._arkit_face_sample_rate,
|
||||
);
|
||||
const arkitFaceFloat32ArrayArray = new Float32Array(
|
||||
arkitFaceArrayBufferArray[lastIndex],
|
||||
);
|
||||
const subData = arkitFaceFloat32ArrayArray?.slice(
|
||||
offset * this._arkitFaceShape,
|
||||
offset * this._arkitFaceShape + this._arkitFaceShape,
|
||||
);
|
||||
if (subData?.length) {
|
||||
const result = {};
|
||||
const channelNames = this._arkit_face_channel_names || [];
|
||||
channelNames.forEach((channelName, index) => {
|
||||
Object.assign(result, {
|
||||
[channelName]: subData[index],
|
||||
});
|
||||
});
|
||||
return result;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
private _getTts2FaceFrame() {
|
||||
if (!this._motionDataGroups.length) {
|
||||
return null;
|
||||
}
|
||||
const targetMotion = this._motionDataGroups.find(
|
||||
(_motion) => _motion.player,
|
||||
);
|
||||
if (!targetMotion) {
|
||||
return null;
|
||||
}
|
||||
const { tts2faceArrayBufferArray, player } = targetMotion!;
|
||||
if (
|
||||
player &&
|
||||
player._firstStartAbsoluteTime &&
|
||||
tts2faceArrayBufferArray &&
|
||||
tts2faceArrayBufferArray.length > 0 &&
|
||||
this._tts2FaceShape &&
|
||||
this._tts2face_sample_rate
|
||||
) {
|
||||
const offsetTime = Date.now() - player._firstStartAbsoluteTime;
|
||||
let lastIndex = 0;
|
||||
let firstSampleStartTime: number;
|
||||
player.samplesList.forEach((item, index) => {
|
||||
if (
|
||||
firstSampleStartTime === undefined &&
|
||||
item.startTime !== undefined
|
||||
) {
|
||||
firstSampleStartTime = item.startTime;
|
||||
}
|
||||
if (
|
||||
item.startTime !== undefined &&
|
||||
item.startTime - firstSampleStartTime <= offsetTime / 1000
|
||||
) {
|
||||
lastIndex = index;
|
||||
}
|
||||
});
|
||||
const samples = player.samplesList[lastIndex];
|
||||
const subOffsetTime = offsetTime - samples.startTime! * 1000;
|
||||
const offset = Math.floor(
|
||||
(subOffsetTime / 1000) * this._tts2face_sample_rate,
|
||||
);
|
||||
const arkitFaceFloat32ArrayArray = new Float32Array(
|
||||
tts2faceArrayBufferArray[lastIndex],
|
||||
);
|
||||
const subData = arkitFaceFloat32ArrayArray?.slice(
|
||||
offset * this._tts2FaceShape,
|
||||
offset * this._tts2FaceShape + this._tts2FaceShape,
|
||||
);
|
||||
if (subData?.length) {
|
||||
return subData;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
private async _motionDataGroupHandler(avatar_motion_data: IAvatarMotionData) {
|
||||
try {
|
||||
const {
|
||||
first_package,
|
||||
motion_data_slice,
|
||||
segment_num,
|
||||
binary_size,
|
||||
use_binary_frame,
|
||||
is_audio_mute
|
||||
} = avatar_motion_data;
|
||||
if (first_package) {
|
||||
const lastMotionGroup =
|
||||
this._motionDataGroups[this._motionDataGroups.length - 1];
|
||||
if (lastMotionGroup) {
|
||||
// 检测上一大片数量是否丢包
|
||||
if (
|
||||
lastMotionGroup.segment_num !==
|
||||
lastMotionGroup.motion_data_slices.length
|
||||
) {
|
||||
// 丢包触发错误
|
||||
this.ee.emit(EventTypes.ErrorReceived, 'lost data packets');
|
||||
}
|
||||
}
|
||||
this._motionDataGroups.push({
|
||||
first_package,
|
||||
binary_size,
|
||||
segment_num,
|
||||
use_binary_frame,
|
||||
motion_data_slices: [],
|
||||
merged_motion_data: new Uint8Array(binary_size),
|
||||
});
|
||||
} else {
|
||||
if (this._motionDataGroups.length === 0) {
|
||||
return;
|
||||
}
|
||||
if (!motion_data_slice) {
|
||||
return;
|
||||
}
|
||||
const lastMotionGroup =
|
||||
this._motionDataGroups[this._motionDataGroups.length - 1];
|
||||
const prevMotionGroup =
|
||||
this._motionDataGroups[this._motionDataGroups.length - 2];
|
||||
lastMotionGroup.motion_data_slices.push(motion_data_slice);
|
||||
if (
|
||||
lastMotionGroup.motion_data_slices.length ===
|
||||
lastMotionGroup.segment_num
|
||||
) {
|
||||
// 单段不分小片段的情况，不需要mergeBlob，为了兼容后续逻辑，这里直接赋值
|
||||
const blob = lastMotionGroup.motion_data_slices[0]
|
||||
// const blob = mergeBlob(
|
||||
// lastMotionGroup.motion_data_slices,
|
||||
// lastMotionGroup.merged_motion_data,
|
||||
// );
|
||||
const { parsedData, jsonSize, binSize } = await unpack(blob);
|
||||
lastMotionGroup.jsonSize = jsonSize;
|
||||
lastMotionGroup.binSize = binSize;
|
||||
const bin = blob.slice(12 + lastMotionGroup.jsonSize!);
|
||||
if (bin.size !== lastMotionGroup.binSize) {
|
||||
this.ee.emit(ProcessorEventTypes.Chat_BinsizeError);
|
||||
}
|
||||
const batchCheckResult = this._connectBatch(
|
||||
parsedData,
|
||||
lastMotionGroup,
|
||||
prevMotionGroup,
|
||||
);
|
||||
if (!batchCheckResult) {
|
||||
return;
|
||||
}
|
||||
await this._handleArkitFaceConfig(
|
||||
parsedData,
|
||||
lastMotionGroup,
|
||||
prevMotionGroup,
|
||||
bin,
|
||||
);
|
||||
// await this._handletts2faceConfig(
|
||||
// parsedData,
|
||||
// lastMotionGroup,
|
||||
// prevMotionGroup,
|
||||
// bin,
|
||||
// );
|
||||
await this._handleAudioConfig(
|
||||
parsedData,
|
||||
lastMotionGroup,
|
||||
prevMotionGroup,
|
||||
bin,
|
||||
is_audio_mute || false
|
||||
);
|
||||
this._handleEvents(parsedData);
|
||||
}
|
||||
}
|
||||
} catch (err: unknown) {
|
||||
console.error('err', err)
|
||||
this.ee.emit(EventTypes.ErrorReceived, (err as Error).message);
|
||||
}
|
||||
}
|
||||
private async _handleAudioConfig(
|
||||
parsedData: IParsedData,
|
||||
lastMotionGroup: IAvatarMotionGroup,
|
||||
prevMotionGroup: IAvatarMotionGroup,
|
||||
bin: Blob,
|
||||
isPlayerMute: boolean
|
||||
) {
|
||||
const { data_records = {}, end_of_batch } = parsedData;
|
||||
const { audio } = data_records;
|
||||
if (audio) {
|
||||
const { sample_rate, shape, data_offset, data_type } = audio;
|
||||
const inputCodec = InputCodecs[data_type];
|
||||
const targetTypedArray = TypedArrays[data_type];
|
||||
if (lastMotionGroup.player === undefined) {
|
||||
if (
|
||||
prevMotionGroup &&
|
||||
prevMotionGroup.player &&
|
||||
prevMotionGroup.batch_id === lastMotionGroup.batch_id
|
||||
) {
|
||||
lastMotionGroup.player = prevMotionGroup.player;
|
||||
} else if (sample_rate) {
|
||||
lastMotionGroup.player = new Player(
|
||||
{
|
||||
inputCodec,
|
||||
channels: 1,
|
||||
sampleRate: sample_rate,
|
||||
fftSize: 1024,
|
||||
isMute: isPlayerMute,
|
||||
onended: (option) => {
|
||||
if (!option) {
|
||||
return;
|
||||
}
|
||||
const {
|
||||
end_of_batch: innerEndOfBatch,
|
||||
lastMotionGroup: innerLastMotion,
|
||||
} = option;
|
||||
if (innerEndOfBatch) {
|
||||
const { batch_id, player } =
|
||||
innerLastMotion as IAvatarMotionGroup;
|
||||
this.ee.emit(PlayerEventTypes.Player_EndSpeaking, player);
|
||||
this._motionDataGroups = this._motionDataGroups.filter(
|
||||
(item) => item.batch_id! > batch_id!,
|
||||
);
|
||||
if (
|
||||
this._motionDataGroups.length &&
|
||||
this._motionDataGroups[0].player
|
||||
) {
|
||||
this._motionDataGroups[0].player.updateAutoPlay(true);
|
||||
} else {
|
||||
this.ee.emit(PlayerEventTypes.Player_NoLegacy);
|
||||
}
|
||||
}
|
||||
},
|
||||
},
|
||||
this.ee,
|
||||
);
|
||||
}
|
||||
if (end_of_batch) {
|
||||
const originEnded = lastMotionGroup.player!.option.onended;
|
||||
lastMotionGroup.player!.option.onended = () => {
|
||||
originEnded({
|
||||
end_of_batch,
|
||||
lastMotionGroup,
|
||||
});
|
||||
};
|
||||
}
|
||||
}
|
||||
const shapeLength = shape.reduce(
|
||||
(acc: number, cur: number) => acc * cur,
|
||||
inputCodec === "Int16" ? 2 : 4,
|
||||
);
|
||||
const audioBlobSliceStart = data_offset;
|
||||
const audioBlobSliceEnd = data_offset + shapeLength;
|
||||
const audioBlob = bin.slice(audioBlobSliceStart, audioBlobSliceEnd);
|
||||
const audioArrayBuffer = await audioBlob.arrayBuffer();
|
||||
// 如果前一段还没播放结束，后一段已接收到，那么后一段则不能自动播放
|
||||
const prevHasPlayerMotionDataGroup = this._motionDataGroups.find(
|
||||
(item) => item.player,
|
||||
);
|
||||
if (
|
||||
this._motionDataGroups.length &&
|
||||
lastMotionGroup.player &&
|
||||
prevHasPlayerMotionDataGroup &&
|
||||
prevHasPlayerMotionDataGroup.player !== lastMotionGroup.player
|
||||
) {
|
||||
lastMotionGroup.player.autoPlay = false;
|
||||
}
|
||||
if (lastMotionGroup.player) {
|
||||
lastMotionGroup.player.feed({
|
||||
audio: new targetTypedArray(audioArrayBuffer),
|
||||
end_of_batch,
|
||||
});
|
||||
}
|
||||
} else {
|
||||
// 特殊事件motion挂上这个
|
||||
if (
|
||||
prevMotionGroup &&
|
||||
prevMotionGroup.player &&
|
||||
lastMotionGroup.batch_id === prevMotionGroup.batch_id
|
||||
) {
|
||||
lastMotionGroup.player = prevMotionGroup.player;
|
||||
}
|
||||
}
|
||||
}
|
||||
private async _handleArkitFaceConfig(
|
||||
parsedData: IParsedData,
|
||||
lastMotionGroup: IAvatarMotionGroup,
|
||||
prevMotionGroup: IAvatarMotionGroup,
|
||||
bin: Blob,
|
||||
) {
|
||||
const { data_records = {} } = parsedData;
|
||||
const { arkit_face } = data_records;
|
||||
if (arkit_face) {
|
||||
const { channel_names, shape, data_offset, sample_rate } =
|
||||
arkit_face as IDataRecords;
|
||||
if (channel_names && !this._arkit_face_channel_names) {
|
||||
this._arkit_face_channel_names = channel_names;
|
||||
this._arkit_face_sample_rate = sample_rate;
|
||||
}
|
||||
if (lastMotionGroup.arkitFaceArrayBufferArray === undefined) {
|
||||
if (
|
||||
prevMotionGroup &&
|
||||
prevMotionGroup.arkitFaceArrayBufferArray &&
|
||||
prevMotionGroup.batch_id === lastMotionGroup.batch_id
|
||||
) {
|
||||
lastMotionGroup.arkitFaceArrayBufferArray =
|
||||
prevMotionGroup.arkitFaceArrayBufferArray;
|
||||
} else {
|
||||
lastMotionGroup.arkitFaceArrayBufferArray = [];
|
||||
}
|
||||
const shapeLength = shape.reduce(
|
||||
(acc: number, cur: number) => acc * cur,
|
||||
4,
|
||||
);
|
||||
this._arkitFaceShape = shape[1];
|
||||
const arkitFaceBlob = bin.slice(data_offset, data_offset + shapeLength);
|
||||
const arkitFaceArrayBuffer = await arkitFaceBlob.arrayBuffer();
|
||||
lastMotionGroup.arkitFaceArrayBufferArray.push(arkitFaceArrayBuffer);
|
||||
}
|
||||
} else {
|
||||
if (
|
||||
prevMotionGroup &&
|
||||
prevMotionGroup.arkitFaceArrayBufferArray &&
|
||||
lastMotionGroup.batch_id === prevMotionGroup.batch_id
|
||||
) {
|
||||
lastMotionGroup.arkitFaceArrayBufferArray =
|
||||
prevMotionGroup.arkitFaceArrayBufferArray;
|
||||
}
|
||||
}
|
||||
}
|
||||
private async _handletts2faceConfig(
|
||||
parsedData: IParsedData,
|
||||
lastMotionGroup: IAvatarMotionGroup,
|
||||
prevMotionGroup: IAvatarMotionGroup,
|
||||
bin: Blob,
|
||||
) {
|
||||
const { data_records = {} } = parsedData;
|
||||
const { tts2face } = data_records;
|
||||
if (tts2face) {
|
||||
const { channel_names, shape, data_offset, sample_rate } =
|
||||
tts2face as IDataRecords;
|
||||
if (channel_names && !this._tts2face_channel_names) {
|
||||
this._tts2face_channel_names = channel_names;
|
||||
this._tts2face_sample_rate = sample_rate;
|
||||
}
|
||||
if (lastMotionGroup.tts2faceArrayBufferArray === undefined) {
|
||||
if (
|
||||
prevMotionGroup &&
|
||||
prevMotionGroup.tts2faceArrayBufferArray &&
|
||||
prevMotionGroup.batch_id === lastMotionGroup.batch_id
|
||||
) {
|
||||
lastMotionGroup.tts2faceArrayBufferArray =
|
||||
prevMotionGroup.tts2faceArrayBufferArray;
|
||||
} else {
|
||||
lastMotionGroup.tts2faceArrayBufferArray = [];
|
||||
}
|
||||
const shapeLength = shape.reduce(
|
||||
(acc: number, cur: number) => acc * cur,
|
||||
4,
|
||||
);
|
||||
this._tts2FaceShape = shape[1];
|
||||
const tts2faceBlob = bin.slice(data_offset, data_offset + shapeLength);
|
||||
const tts2faceArrayBuffer = await tts2faceBlob.arrayBuffer();
|
||||
lastMotionGroup.tts2faceArrayBufferArray.push(tts2faceArrayBuffer);
|
||||
}
|
||||
} else {
|
||||
if (
|
||||
prevMotionGroup &&
|
||||
prevMotionGroup.tts2faceArrayBufferArray &&
|
||||
lastMotionGroup.batch_id === prevMotionGroup.batch_id
|
||||
) {
|
||||
lastMotionGroup.tts2faceArrayBufferArray =
|
||||
prevMotionGroup.tts2faceArrayBufferArray;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private _handleEvents(parsedData: IParsedData) {
|
||||
const { events } = parsedData;
|
||||
if (events && events.length) {
|
||||
events.forEach((e) => {
|
||||
switch (e.event_type) {
|
||||
case "interrupt_speech":
|
||||
// console.log('HandleEvents: interrupt_speech')
|
||||
break;
|
||||
case "change_status":
|
||||
// console.log('HandleEvents: change_status')
|
||||
this.ee.emit(ProcessorEventTypes.Change_Status, e);
|
||||
break;
|
||||
default:
|
||||
break;
|
||||
}
|
||||
});
|
||||
}
|
||||
}
|
||||
private _connectBatch(
|
||||
parsedData: IParsedData,
|
||||
lastMotionGroup: IAvatarMotionGroup,
|
||||
prevMotionGroup: IAvatarMotionGroup,
|
||||
) {
|
||||
let batchCheckResult = true;
|
||||
// 处理二进制batch_id
|
||||
if (parsedData.batch_id && lastMotionGroup.batch_id === undefined) {
|
||||
lastMotionGroup.batch_id = parsedData.batch_id;
|
||||
}
|
||||
// 特殊事件motion如果没有batch_id，也可挂上此batch_id
|
||||
if (
|
||||
!lastMotionGroup.batch_id &&
|
||||
prevMotionGroup &&
|
||||
prevMotionGroup.batch_id
|
||||
) {
|
||||
lastMotionGroup.batch_id = prevMotionGroup.batch_id;
|
||||
}
|
||||
// 特殊事件motion如果没有batch_name，也可挂上此batch_name
|
||||
if (parsedData.batch_name && lastMotionGroup.batch_name === undefined) {
|
||||
lastMotionGroup.batch_name = parsedData.batch_name;
|
||||
}
|
||||
// 处理打断后，如果仍接收到上一个batch的motionData, 那么重新销毁
|
||||
if (
|
||||
this._maxBatchId &&
|
||||
lastMotionGroup.batch_id &&
|
||||
lastMotionGroup.batch_id <= this._maxBatchId
|
||||
) {
|
||||
this.clear();
|
||||
batchCheckResult = false;
|
||||
}
|
||||
return batchCheckResult;
|
||||
}
|
||||
}
|
||||
40
frontend/shared/VideoChat/helpers/ws.ts
Normal file
40
frontend/shared/VideoChat/helpers/ws.ts
Normal file
@@ -0,0 +1,40 @@
|
||||
import EventEmitter from "eventemitter3";
|
||||
import { WsEventTypes } from "../interface/eventType";
|
||||
|
||||
export class WS extends EventEmitter {
|
||||
engine: WebSocket;
|
||||
|
||||
private _inited = false;
|
||||
|
||||
constructor(url: string) {
|
||||
super();
|
||||
this._init(url);
|
||||
}
|
||||
private _init(url: string) {
|
||||
if (this._inited) {
|
||||
return;
|
||||
}
|
||||
this._inited = true;
|
||||
this.engine = new WebSocket(url);
|
||||
this.engine.addEventListener("error", (event) => {
|
||||
this.emit(WsEventTypes.WS_ERROR, event);
|
||||
});
|
||||
this.engine.addEventListener("open", () => {
|
||||
this.emit(WsEventTypes.WS_OPEN);
|
||||
});
|
||||
this.engine.addEventListener("message", (event) => {
|
||||
this.emit(WsEventTypes.WS_MESSAGE, event.data);
|
||||
});
|
||||
this.engine.addEventListener("close", () => {
|
||||
this.emit(WsEventTypes.WS_CLOSE);
|
||||
});
|
||||
}
|
||||
public send(data: string | Int8Array | Uint8Array) {
|
||||
this.engine?.send(data);
|
||||
}
|
||||
public stop() {
|
||||
this.emit(WsEventTypes.WS_CLOSE);
|
||||
this._inited = false;
|
||||
this.engine?.close();
|
||||
}
|
||||
}
|
||||
55
frontend/shared/VideoChat/icons/CameraOff.svelte
Normal file
55
frontend/shared/VideoChat/icons/CameraOff.svelte
Normal file
File diff suppressed because one or more lines are too long
|
After Width: | Height: | Size: 10 KiB |
54
frontend/shared/VideoChat/icons/CameraOn.svelte
Normal file
54
frontend/shared/VideoChat/icons/CameraOn.svelte
Normal file
@@ -0,0 +1,54 @@
|
||||
<svg
|
||||
xmlns="http://www.w3.org/2000/svg"
|
||||
xmlns:xlink="http://www.w3.org/1999/xlink"
|
||||
fill="none"
|
||||
version="1.1"
|
||||
width="20"
|
||||
height="20"
|
||||
viewBox="0 0 20 20"
|
||||
><defs
|
||||
><clipPath id="master_svg0_13_279"
|
||||
><rect x="0" y="0" width="20" height="20" rx="0" /></clipPath
|
||||
><clipPath id="master_svg1_13_279/13_007"
|
||||
><rect x="0" y="0" width="20" height="20" rx="0" /></clipPath
|
||||
></defs
|
||||
><g clip-path="url(#master_svg0_13_279)"
|
||||
><g clip-path="url(#master_svg1_13_279/13_007)"
|
||||
><g
|
||||
><rect
|
||||
x="0"
|
||||
y="0"
|
||||
width="20"
|
||||
height="20"
|
||||
rx="0"
|
||||
fill="#FFFFFF"
|
||||
fill-opacity="0.009999999776482582"
|
||||
style="mix-blend-mode:passthrough"
|
||||
/></g
|
||||
><g
|
||||
><path
|
||||
d="M0.83317090625,15.8333259765625L0.83317090625,4.1666259765625Q0.83317090625,4.0845497765625,0.84918290625,4.0040509765625Q0.86519490625,3.9235519765625,0.89660490625,3.8477229765625Q0.92801390625,3.7718949765625,0.97361290625,3.7036509765625Q1.01921190625,3.6354069765625,1.07724790625,3.5773699765625Q1.13528490625,3.5193339765625,1.2035289062499999,3.4737349765625Q1.27177290625,3.4281359765625,1.34760090625,3.3967269765625Q1.42342990625,3.3653169765625,1.50392890625,3.3493049765625003Q1.58442770625,3.3332929765625,1.66650390625,3.3332929765625L14.99980390625,3.3332929765625Q15.08190390625,3.3332929765625,15.16240390625,3.3493049765625003Q15.24290390625,3.3653169765625,15.31870390625,3.3967269765625Q15.39460390625,3.4281359765625,15.46280390625,3.4737349765625Q15.53100390625,3.5193339765625,15.58910390625,3.5773699765625Q15.64710390625,3.6354069765625,15.69270390625,3.7036509765625Q15.73830390625,3.7718949765625,15.76970390625,3.8477229765625Q15.80110390625,3.9235519765625,15.81720390625,4.0040509765625Q15.83320390625,4.0845497765625,15.83320390625,4.1666259765625L15.83320390625,15.8333259765625Q15.83320390625,15.9153259765625,15.81720390625,15.9958259765625Q15.80110390625,16.0763259765625,15.76970390625,16.152225976562498Q15.73830390625,16.2280259765625,15.69270390625,16.2962259765625Q15.64710390625,16.3645259765625,15.58910390625,16.4225259765625Q15.53100390625,16.4806259765625,15.46280390625,16.5262259765625Q15.39460390625,16.5718259765625,15.31870390625,16.6032259765625Q15.24290390625,16.6346259765625,15.16240390625,16.6506259765625Q15.08190390625,16.6666259765625,14.99980390625,16.6666259765625L1.66650390625,16.6666259765625Q1.58442770625,16.6666259765625,1.50392890625,16.6506259765625Q1.42342990625,16.6346259765625,1.34760090625,16.6032259765625Q1.27177290625,16.5718259765625,1.2035289062499999,16.5262259765625Q1.13528490625,16.4806259765625,1.07724790625,16.4225259765625Q1.01921190625,16.3645259765625,0.97361290625,16.2962259765625Q0.92801390625,16.2280259765625,0.89660490625,16.152225976562498Q0.86519490625,16.0763259765625,0.84918290625,15.9958259765625Q0.83317090625,15.9153259765625,0.83317090625,15.8333259765625ZM2.49983690625,4.9999589765625L2.49983690625,14.9999259765625L14.16650390625,14.9999259765625L14.16650390625,4.9999589765625L2.49983690625,4.9999589765625Z"
|
||||
fill="#FFFFFF"
|
||||
fill-opacity="1"
|
||||
style="mix-blend-mode:passthrough"
|
||||
/></g
|
||||
><g
|
||||
><path
|
||||
d="M18.97024,14.7041040234375Q19.06538,14.5913440234375,19.11602,14.4527940234375Q19.16667,14.3142340234375,19.16667,14.1667040234375L19.16667,5.8333740234375Q19.16667,5.7512978234375,19.15065,5.6707990234375Q19.13464,5.5903000234375,19.10323,5.5144710234375Q19.07182,5.4386430234375,19.026220000000002,5.3703990234375Q18.98063,5.3021550234375,18.92259,5.2441180234375Q18.86455,5.1860820234375,18.79631,5.1404830234375Q18.72806,5.0948840234375,18.65224,5.0634750234375Q18.57641,5.0320650234375,18.49591,5.0160530234375Q18.41541,5.0000410234375,18.33333,5.0000410234375Q18.18581,5.0000410234375,18.04725,5.0506860234375Q17.90869,5.1013300234375,17.79594,5.1964640234375L14.462608,8.0089640234375Q14.393074,8.067634023437499,14.337838,8.1399240234375Q14.282601,8.2122140234375,14.244265,8.2947240234375Q14.205928,8.377224023437499,14.186297,8.4660640234375Q14.166667,8.5548940234375,14.166667,8.6458740234375L14.166667,11.3542040234375Q14.166667,11.4451840234375,14.186297,11.5340240234375Q14.205928,11.622854023437501,14.244265,11.7053640234375Q14.282601,11.7878640234375,14.337838,11.860154023437499Q14.393074,11.932444023437501,14.462608,11.9911140234375L17.79594,14.8036140234375Q17.922629999999998,14.9105140234375,18.08058,14.9607840234375Q18.23853,15.0110640234375,18.4037,14.9970640234375Q18.56887,14.9830640234375,18.71611,14.9069240234375Q18.86335,14.8307940234375,18.97024,14.7041040234375ZM17.5,12.3732440234375L17.5,7.6268340234375L15.833333,9.0330840234375L15.833333,10.9669940234375L17.5,12.3732440234375Z"
|
||||
fill-rule="evenodd"
|
||||
fill="#FFFFFF"
|
||||
fill-opacity="1"
|
||||
style="mix-blend-mode:passthrough"
|
||||
/></g
|
||||
><g
|
||||
><path
|
||||
d="M7.65749209375,7.3101989765625L10.11698609375,9.3597759765625Q10.17518609375,9.4082759765625,10.22367609375,9.4664759765625Q10.32979609375,9.5938159765625,10.37910609375,9.7520659765625Q10.42841609375,9.9103259765625,10.41340609375,10.0754059765625Q10.39839609375,10.2404859765625,10.321366093750001,10.3872559765625Q10.24432609375,10.5340259765625,10.11698609375,10.6401459765625L7.65748809375,12.6897259765625Q7.54118509375,12.7998059765625,7.39241009375,12.8590459765625Q7.24363409375,12.9182959765625,7.08349609375,12.9182959765625Q7.00125579375,12.9182959765625,6.92059609375,12.9022459765625Q6.83993609375,12.8862059765625,6.76395509375,12.8547359765625Q6.6879750937499995,12.8232559765625,6.61959509375,12.7775659765625Q6.55121509375,12.731875976562499,6.49306209375,12.673725976562501Q6.43490909375,12.6155759765625,6.38921909375,12.547195976562499Q6.34352909375,12.478815976562501,6.31205709375,12.4028359765625Q6.28058509375,12.3268559765625,6.26454009375,12.2461959765625Q6.24849609375,12.1655359765625,6.24849609375,12.0832959765625Q6.24849609375,11.9848059765625,6.27141009375,11.8890159765625Q6.29432409375,11.7932359765625,6.33889509375,11.7054159765625Q6.38346609375,11.6175859765625,6.44724609375,11.5425459765625Q6.51102709375,11.4674959765625,6.59051809375,11.4093459765625L8.28178609375,9.9999559765625L6.59051809375,8.5905679765625Q6.51102709375,8.5324209765625,6.44724609375,8.4573769765625Q6.38346509375,8.3823319765625,6.33889509375,8.2945069765625Q6.29432409375,8.2066819765625,6.27141009375,8.1108979765625Q6.24849609375,8.0151131765625,6.24849609375,7.9166259765625Q6.24849609375,7.8343856765625,6.26454009375,7.7537259765625Q6.28058509375,7.6730659765625,6.31205709375,7.5970849765625Q6.34352909375,7.5211049765624995,6.38921909375,7.4527249765625Q6.43490909375,7.3843449765625,6.49306209375,7.3261919765625Q6.55121509375,7.2680389765625,6.61959509375,7.2223489765625Q6.6879750937499995,7.1766589765625,6.76395509375,7.1451869765625Q6.83993609375,7.1137149765625,6.92059609375,7.0976699765625Q7.00125579375,7.0816259765625,7.08349609375,7.0816259765625Q7.24363509375,7.0816259765625,7.39241209375,7.1408709765625Q7.54118909375,7.2001159765625005,7.65749209375,7.3101989765625Z"
|
||||
fill-rule="evenodd"
|
||||
fill="#FFFFFF"
|
||||
fill-opacity="1"
|
||||
style="mix-blend-mode:passthrough"
|
||||
/></g
|
||||
></g
|
||||
></g
|
||||
></svg
|
||||
>
|
||||
|
After Width: | Height: | Size: 7.2 KiB |
18
frontend/shared/VideoChat/icons/Check.svelte
Normal file
18
frontend/shared/VideoChat/icons/Check.svelte
Normal file
@@ -0,0 +1,18 @@
|
||||
<svg
|
||||
xmlns="http://www.w3.org/2000/svg"
|
||||
xmlns:xlink="http://www.w3.org/1999/xlink"
|
||||
fill="none"
|
||||
version="1.1"
|
||||
width="14.000000357627869"
|
||||
height="10.000000357627869"
|
||||
viewBox="0 0 14.000000357627869 10.000000357627869"
|
||||
><g
|
||||
><path
|
||||
d="M13.802466686534881,1.1380186865348816Q13.89646668653488,1.0444176865348815,13.947366686534881,0.9218876865348815Q13.998366686534881,0.7993576865348816,13.998366686534881,0.6666666865348816Q13.998366686534881,0.6011698865348816,13.98556668653488,0.5369316865348817Q13.972766686534882,0.4726936865348816,13.947666686534882,0.4121826865348816Q13.922666686534882,0.3516706865348816,13.886266686534881,0.2972126865348816Q13.849866686534881,0.2427536865348816,13.803566686534882,0.19644068653488161Q13.757266686534882,0.15012768653488162,13.702766686534881,0.11373968653488165Q13.648366686534882,0.07735168653488156,13.587866686534882,0.052286686534881555Q13.527266686534881,0.02722268653488158,13.463066686534882,0.014444686534881623Q13.398866686534882,0.0016666865348815563,13.333366686534882,0.0016666865348815563Q13.201466686534882,0.0016666865348815563,13.079566686534882,0.051981686534881555Q12.957666686534882,0.10229768653488158,12.864266686534881,0.1953146865348816L12.863066686534882,0.19413268653488158L4.624996686534882,8.392776686534882L1.1369396865348815,4.921396686534882L1.1357636865348817,4.922586686534881Q1.0422996865348817,4.829566686534881,0.9204146865348816,4.779246686534882Q0.7985286865348816,4.728936686534881,0.6666666865348816,4.728936686534881Q0.6011698865348816,4.728936686534881,0.5369316865348817,4.741706686534882Q0.4726936865348816,4.754486686534881,0.4121826865348816,4.779556686534882Q0.3516706865348816,4.804616686534882,0.2972126865348816,4.8410066865348815Q0.2427536865348816,4.8773966865348815,0.19644068653488161,4.9237066865348815Q0.15012768653488162,4.970016686534882,0.11373968653488165,5.024476686534881Q0.07735168653488156,5.078936686534882,0.052286686534881555,5.139446686534882Q0.02722268653488158,5.199956686534882,0.014444686534881623,5.2641966865348815Q0.0016666865348815563,5.328436686534881,0.0016666865348815563,5.3939366865348815Q0.0016666865348815563,5.526626686534882,0.05259268653488158,5.649156686534882Q0.10351768653488158,5.771686686534881,0.1975696865348816,5.865286686534882L0.1963936865348816,5.866466686534881L4.1547266865348815,9.805866686534882Q4.201126686534882,9.852046686534882,4.255616686534882,9.888306686534882Q4.310106686534882,9.924576686534882,4.3706166865348814,9.949556686534882Q4.431126686534881,9.974536686534881,4.495326686534882,9.987266686534882Q4.559536686534882,9.999996686534882,4.624996686534882,9.999996686534882Q4.690456686534882,9.999996686534882,4.754666686534882,9.987266686534882Q4.818876686534882,9.974536686534881,4.879386686534882,9.949556686534882Q4.939886686534882,9.924576686534882,4.994386686534882,9.888306686534882Q5.048876686534881,9.852046686534882,5.0952766865348815,9.805866686534882L13.803566686534882,1.1392006865348816L13.802466686534881,1.1380186865348816Z"
|
||||
fill-rule="evenodd"
|
||||
fill="#E0E0FC"
|
||||
fill-opacity="1"
|
||||
style="mix-blend-mode:passthrough"
|
||||
/></g
|
||||
></svg
|
||||
>
|
||||
|
After Width: | Height: | Size: 3.1 KiB |
13
frontend/shared/VideoChat/icons/IconFont.svelte
Normal file
13
frontend/shared/VideoChat/icons/IconFont.svelte
Normal file
@@ -0,0 +1,13 @@
|
||||
<script lang="ts">
|
||||
export let color;
|
||||
export let fontSize = "16px";
|
||||
export let icon: any = undefined;
|
||||
</script>
|
||||
|
||||
<span style={`color: ${color}; font-size: ${fontSize}`}>
|
||||
{#if icon}
|
||||
<svelte:component this={icon} />
|
||||
{:else}
|
||||
<slot></slot>
|
||||
{/if}
|
||||
</span>
|
||||
60
frontend/shared/VideoChat/icons/MicOff.svelte
Normal file
60
frontend/shared/VideoChat/icons/MicOff.svelte
Normal file
@@ -0,0 +1,60 @@
|
||||
<svg
|
||||
xmlns="http://www.w3.org/2000/svg"
|
||||
xmlns:xlink="http://www.w3.org/1999/xlink"
|
||||
fill="none"
|
||||
version="1.1"
|
||||
width="20"
|
||||
height="20"
|
||||
viewBox="0 0 20 20"
|
||||
><defs
|
||||
><clipPath id="master_svg0_13_287/13_278"
|
||||
><rect x="0" y="0" width="20" height="20" rx="0" /></clipPath
|
||||
><clipPath id="master_svg1_13_287/13_278/13_040"
|
||||
><rect x="0" y="0" width="20" height="20" rx="0" /></clipPath
|
||||
></defs
|
||||
><g clip-path="url(#master_svg0_13_287/13_278)"
|
||||
><g clip-path="url(#master_svg1_13_287/13_278/13_040)"
|
||||
><g
|
||||
><path
|
||||
d="M7.34851109375,12.6516259765625Q8.44685609375,13.7499259765625,10.00016609375,13.7499259765625Q11.55346609375,13.7499259765625,12.65181609375,12.6516259765625Q13.75016609375,11.5532659765625,13.75016609375,9.9999559765625L13.75016609375,4.5832959765625Q13.75016609375,3.0299959765624997,12.65181609375,1.9316429765625Q11.55346609375,0.8332929765625,10.00016609375,0.8332919765625Q8.44685609375,0.8332929765625,7.34851109375,1.9316429765625Q6.25016309375,3.0299959765624997,6.25016309375,4.5832959765625L6.25016309375,9.9999559765625Q6.25016309375,11.5532659765625,7.34851109375,12.6516259765625ZM11.47330609375,11.4730959765625Q10.86310609375,12.0833259765625,10.00016609375,12.0833259765625Q9.13721609375,12.0833259765625,8.527026093749999,11.4730959765625Q7.91682909375,10.8629059765625,7.91682909375,9.9999559765625L7.91683009375,4.5832959765625Q7.91683009375,3.7203459765625,8.527026093749999,3.1101559765625Q9.13721609375,2.4999589765625,10.00016609375,2.4999589765625Q10.86310609375,2.4999589765625,11.47330609375,3.1101559765625Q12.08349609375,3.7203459765625,12.08349609375,4.5832959765625L12.08349609375,9.9999559765625Q12.08349609375,10.8629059765625,11.47330609375,11.4730959765625Z"
|
||||
fill-rule="evenodd"
|
||||
fill="#FFFFFF"
|
||||
fill-opacity="1"
|
||||
style="mix-blend-mode:passthrough"
|
||||
/></g
|
||||
><g
|
||||
><path
|
||||
d="M17.08315046875,9.6393233234375Q17.08502046875,9.6113801234375,17.08502046875,9.5833740234375Q17.08502046875,9.5011337234375,17.06898046875,9.4204740234375Q17.05293046875,9.3398140234375,17.02146046875,9.2638330234375Q16.98999046875,9.1878530234375,16.94430046875,9.1194730234375Q16.89861046875,9.0510930234375,16.84046046875,8.9929400234375Q16.78230046875,8.9347870234375,16.71392346875,8.8890970234375Q16.64554246875,8.8434070234375,16.56956246875,8.8119350234375Q16.49358246875,8.7804630234375,16.41292246875,8.7644180234375Q16.33226246875,8.7483740234375,16.25002246875,8.7483740234375Q16.16778146875,8.7483740234375,16.08712146875,8.7644180234375Q16.00646146875,8.7804630234375,15.93048146875,8.8119350234375Q15.85450146875,8.8434070234375,15.78612106875,8.8890970234375Q15.71774076875,8.9347870234375,15.65958806875,8.9929400234375Q15.60143546875,9.0510930234375,15.55574546875,9.1194730234375Q15.51005446875,9.1878530234375,15.47858246875,9.2638330234375Q15.44711046875,9.3398140234375,15.43106646875,9.4204740234375Q15.41502246875,9.5011337234375,15.41502246875,9.5833740234375Q15.41502246875,9.6080265234375,15.41647646875,9.6326360234375Q15.40712446875,10.7164940234375,14.98582546875,11.7046140234375Q14.89498046875,11.8831040234375,14.89498046875,12.0833740234375Q14.89498046875,12.1656140234375,14.91102446875,12.2462740234375Q14.92706946875,12.3269340234375,14.95854146875,12.4029140234375Q14.99001346875,12.4788940234375,15.03570346875,12.5472740234375Q15.08139346875,12.6156540234375,15.13954646875,12.6738040234375Q15.19769946875,12.7319640234375,15.26607946875,12.7776540234375Q15.33445946875,12.8233440234375,15.41043946875,12.8548140234375Q15.48642046875,12.8862840234375,15.56708046875,12.9023340234375Q15.64774016875,12.9183740234375,15.72998046875,12.9183740234375Q15.79409136875,12.9183740234375,15.85745046875,12.9085840234375Q15.92081046875,12.8988040234375,15.98193346875,12.8794540234375Q16.04305546875,12.8601140234375,16.10050846875,12.8316640234375Q16.15796246875,12.8032140234375,16.21039846875,12.7663240234375Q16.26283546875,12.7294440234375,16.309026468749998,12.6849840234375Q16.35521746875,12.6405240234375,16.39408046875,12.5895340234375Q16.43294246875,12.538544023437499,16.46356646875,12.4822240234375Q16.49418946875,12.4258940234375,16.51585446875,12.3655540234375Q17.07244046875,11.0643540234375,17.08315046875,9.6393233234375Z"
|
||||
fill-rule="evenodd"
|
||||
fill="#FFFFFF"
|
||||
fill-opacity="1"
|
||||
style="mix-blend-mode:passthrough"
|
||||
/></g
|
||||
><g
|
||||
><path
|
||||
d="M4.583527,9.6329521234375Q4.585,9.6081849234375,4.585,9.5833740234375Q4.585,9.5011337234375,4.568956,9.4204740234375Q4.552911,9.3398140234375,4.521439,9.2638330234375Q4.489967,9.1878530234375,4.444277,9.1194730234375Q4.398587,9.0510930234375,4.340434,8.9929400234375Q4.282281,8.9347870234375,4.213901,8.8890970234375Q4.1455210000000005,8.8434070234375,4.069541,8.8119350234375Q3.99356,8.7804630234375,3.9129,8.7644180234375Q3.8322403,8.7483740234375,3.75,8.7483740234375Q3.6677597,8.7483740234375,3.5871,8.7644180234375Q3.50644,8.7804630234375,3.430459,8.8119350234375Q3.354479,8.8434070234375,3.286099,8.8890970234375Q3.2177189999999998,8.9347870234375,3.159566,8.9929400234375Q3.101413,9.0510930234375,3.055723,9.1194730234375Q3.010033,9.1878530234375,2.978561,9.2638330234375Q2.947089,9.3398140234375,2.931044,9.4204740234375Q2.915,9.5011337234375,2.915,9.5833740234375Q2.915,9.6112012234375,2.916853,9.6389666234375Q2.9363479999999997,12.5370740234375,4.99132,14.5920540234375Q7.06598,16.6667040234375,10,16.6667040234375Q11.1917,16.6667040234375,12.30806,16.2819440234375Q12.37346,16.2636640234375,12.43505,16.235064023437502Q12.49663,16.2064640234375,12.55279,16.1682840234375Q12.60894,16.1301040234375,12.65819,16.0833540234375Q12.70744,16.036604023437498,12.74849,15.9825140234375Q12.78954,15.9284240234375,12.82131,15.868404023437499Q12.85308,15.8083940234375,12.87473,15.7440340234375Q12.89639,15.6796740234375,12.90736,15.6126640234375Q12.91833,15.5456540234375,12.91833,15.4777440234375Q12.91833,15.3955040234375,12.90229,15.3148440234375Q12.88624,15.2341840234375,12.85477,15.1582040234375Q12.8233,15.082224023437501,12.77761,15.0138440234375Q12.73192,14.9454640234375,12.67377,14.8873140234375Q12.61561,14.8291640234375,12.54723,14.783474023437499Q12.47885,14.7377840234375,12.40287,14.7063140234375Q12.32689,14.6748340234375,12.24623,14.658794023437501Q12.16557,14.6427540234375,12.08333,14.642744023437501Q11.91469,14.642744023437501,11.75926,14.7082040234375Q10.9093,15.0000440234375,10,15.0000440234375Q7.75633,15.0000440234375,6.16983,13.413544023437499Q4.6008890000000005,11.8445940234375,4.583527,9.6329521234375Z"
|
||||
fill-rule="evenodd"
|
||||
fill="#FFFFFF"
|
||||
fill-opacity="1"
|
||||
style="mix-blend-mode:passthrough"
|
||||
/></g
|
||||
><g
|
||||
><path
|
||||
d="M10.833333,15.8861049234375Q10.835,15.8597658234375,10.835,15.8333740234375Q10.835,15.7511337234375,10.818956,15.6704740234375Q10.802911,15.5898140234375,10.771439,15.5138330234375Q10.739967,15.4378530234375,10.694277,15.3694730234375Q10.648587,15.3010930234375,10.590434,15.2429400234375Q10.532281,15.1847870234375,10.463901,15.1390970234375Q10.395521,15.0934070234375,10.319541,15.0619350234375Q10.24356,15.0304630234375,10.1629,15.0144180234375Q10.0822403,14.9983740234375,10,14.9983740234375Q9.9177597,14.9983740234375,9.8371,15.0144180234375Q9.75644,15.0304630234375,9.680459,15.0619350234375Q9.604479,15.0934070234375,9.536099,15.1390970234375Q9.467719,15.1847870234375,9.409566,15.2429400234375Q9.351413,15.3010930234375,9.305723,15.3694730234375Q9.260033,15.4378530234375,9.228561,15.5138330234375Q9.197089,15.5898140234375,9.181044,15.6704740234375Q9.165,15.7511337234375,9.165,15.8333740234375Q9.165,15.8597658234375,9.166667,15.8861049234375L9.166667,18.2806440234375Q9.165,18.3069840234375,9.165,18.3333740234375Q9.165,18.4156140234375,9.181044,18.4962740234375Q9.197089,18.5769340234375,9.228561,18.6529140234375Q9.260033,18.7288940234375,9.305723,18.7972740234375Q9.351413,18.8656540234375,9.409566,18.9238040234375Q9.467719,18.9819640234375,9.536099,19.0276540234375Q9.604479,19.0733440234375,9.680459,19.1048140234375Q9.75644,19.1362840234375,9.8371,19.1523340234375Q9.9177597,19.1683740234375,10,19.1683740234375Q10.0822403,19.1683740234375,10.1629,19.1523340234375Q10.24356,19.1362840234375,10.319541,19.1048140234375Q10.395521,19.0733440234375,10.463901,19.0276540234375Q10.532281,18.9819640234375,10.590434,18.9238040234375Q10.648587,18.8656540234375,10.694277,18.7972740234375Q10.739967,18.7288940234375,10.771439,18.6529140234375Q10.802911,18.5769340234375,10.818956,18.4962740234375Q10.835,18.4156140234375,10.835,18.3333740234375Q10.835,18.3069840234375,10.833333,18.2806440234375L10.833333,15.8861049234375Z"
|
||||
fill-rule="evenodd"
|
||||
fill="#FFFFFF"
|
||||
fill-opacity="1"
|
||||
style="mix-blend-mode:passthrough"
|
||||
/></g
|
||||
><g
|
||||
><path
|
||||
d="M1.9480309999999998,3.126542Q1.813081,3.007654,1.7390400000000001,2.843752Q1.665,2.67985,1.665,2.5Q1.665,2.4177597,1.681044,2.3371Q1.697089,2.25644,1.728561,2.180459Q1.760033,2.104479,1.805723,2.036099Q1.851413,1.967719,1.9095659999999999,1.9095659999999999Q1.967719,1.851413,2.036099,1.805723Q2.104479,1.760033,2.180459,1.728561Q2.25644,1.697089,2.3371,1.681044Q2.4177597,1.665,2.5,1.665Q2.67985,1.665,2.843752,1.7390400000000001Q3.007654,1.813081,3.126542,1.9480309999999998L18.052,16.8735Q18.1869,16.9923,18.261,17.1562Q18.335,17.3202,18.335,17.5Q18.335,17.5822,18.319000000000003,17.6629Q18.3029,17.7436,18.2714,17.819499999999998Q18.240000000000002,17.8955,18.1943,17.963900000000002Q18.148600000000002,18.0323,18.090400000000002,18.090400000000002Q18.0323,18.148600000000002,17.963900000000002,18.1943Q17.8955,18.240000000000002,17.819499999999998,18.2714Q17.7436,18.3029,17.6629,18.319000000000003Q17.5822,18.335,17.5,18.335Q17.3202,18.335,17.1562,18.261Q16.9923,18.1869,16.8735,18.052L1.9480309999999998,3.126542Z"
|
||||
fill-rule="evenodd"
|
||||
fill="#FFFFFF"
|
||||
fill-opacity="1"
|
||||
style="mix-blend-mode:passthrough"
|
||||
/></g
|
||||
></g
|
||||
></g
|
||||
></svg
|
||||
>
|
||||
|
After Width: | Height: | Size: 9.8 KiB |
54
frontend/shared/VideoChat/icons/MicOn.svelte
Normal file
54
frontend/shared/VideoChat/icons/MicOn.svelte
Normal file
@@ -0,0 +1,54 @@
|
||||
<svg
|
||||
xmlns="http://www.w3.org/2000/svg"
|
||||
xmlns:xlink="http://www.w3.org/1999/xlink"
|
||||
fill="none"
|
||||
version="1.1"
|
||||
width="20"
|
||||
height="20"
|
||||
viewBox="0 0 20 20"
|
||||
><defs
|
||||
><clipPath id="master_svg0_13_278"
|
||||
><rect x="0" y="0" width="20" height="20" rx="0" /></clipPath
|
||||
><clipPath id="master_svg1_13_278/13_029"
|
||||
><rect x="0" y="0" width="20" height="20" rx="0" /></clipPath
|
||||
></defs
|
||||
><g clip-path="url(#master_svg0_13_278)"
|
||||
><g clip-path="url(#master_svg1_13_278/13_029)"
|
||||
><g
|
||||
><rect
|
||||
x="0"
|
||||
y="0"
|
||||
width="20"
|
||||
height="20"
|
||||
rx="0"
|
||||
fill="#FFFFFF"
|
||||
fill-opacity="0.009999999776482582"
|
||||
style="mix-blend-mode:passthrough"
|
||||
/></g
|
||||
><g
|
||||
><path
|
||||
d="M6.249918953125,9.9999559765625L6.249918953125,4.5832959765625Q6.249918953125,3.0299959765624997,7.348267953125,1.9316419765625Q8.446621953125,0.8332929765625,9.999921953125,0.8332929765625Q11.553221953125,0.8332929765625,12.651571953125,1.9316419765625Q13.749921953125,3.0299959765624997,13.749921953125,4.5832959765625L13.749921953125,9.9999559765625Q13.749921953125,11.5532559765625,12.651571953125,12.6516259765625Q11.553221953125,13.7499259765625,9.999921953125,13.7499259765625Q8.446621953125,13.7499259765625,7.348267953125,12.6516259765625Q6.249918953125,11.5532559765625,6.249918953125,9.9999559765625ZM7.916584953125,9.9999559765625Q7.916584953125,10.8629059765625,8.526781953124999,11.4730959765625Q9.136971953125,12.0833259765625,9.999921953125,12.0833259765625Q10.862861953125,12.0833259765625,11.473061953125,11.4730959765625Q12.083251953125,10.8629059765625,12.083251953125,9.9999559765625L12.083251953125,4.5832959765625Q12.083251953125,3.7203459765625,11.473061953125,3.1101559765625Q10.862861953125,2.4999589765625,9.999921953125,2.4999589765625Q9.136971953125,2.4999589765625,8.526781953124999,3.1101559765625Q7.916584953125,3.7203459765625,7.916584953125,4.5832959765625L7.916584953125,9.9999559765625Z"
|
||||
fill="#FFFFFF"
|
||||
fill-opacity="1"
|
||||
style="mix-blend-mode:passthrough"
|
||||
/></g
|
||||
><g
|
||||
><path
|
||||
d="M4.583527,9.6329521234375Q4.585,9.6081849234375,4.585,9.5833740234375Q4.585,9.5011337234375,4.568956,9.4204740234375Q4.552911,9.3398140234375,4.521439,9.2638330234375Q4.489967,9.1878530234375,4.444277,9.1194730234375Q4.398587,9.0510930234375,4.340434,8.9929400234375Q4.282281,8.9347870234375,4.213901,8.8890970234375Q4.1455210000000005,8.8434070234375,4.069541,8.8119350234375Q3.99356,8.7804630234375,3.9129,8.7644180234375Q3.8322403,8.7483740234375,3.75,8.7483740234375Q3.6677597,8.7483740234375,3.5871,8.7644180234375Q3.50644,8.7804630234375,3.430459,8.8119350234375Q3.354479,8.8434070234375,3.286099,8.8890970234375Q3.2177189999999998,8.9347870234375,3.159566,8.9929400234375Q3.101413,9.0510930234375,3.055723,9.1194730234375Q3.010033,9.1878530234375,2.978561,9.2638330234375Q2.947089,9.3398140234375,2.931044,9.4204740234375Q2.915,9.5011337234375,2.915,9.5833740234375Q2.915,9.6112012234375,2.916853,9.6389666234375Q2.9363479999999997,12.5370740234375,4.99132,14.5920540234375Q7.06598,16.6667040234375,10,16.6667040234375Q12.93402,16.6667040234375,15.0087,14.5920540234375Q17.0636,12.5370940234375,17.0831,9.6390003234375Q17.085,9.6112181234375,17.085,9.5833740234375Q17.085,9.5011337234375,17.069000000000003,9.4204740234375Q17.0529,9.3398140234375,17.0214,9.2638330234375Q16.990000000000002,9.1878530234375,16.9443,9.1194730234375Q16.898600000000002,9.0510930234375,16.840400000000002,8.9929400234375Q16.7823,8.9347870234375,16.713900000000002,8.8890970234375Q16.6455,8.8434070234375,16.569499999999998,8.8119350234375Q16.4936,8.7804630234375,16.4129,8.7644180234375Q16.3322,8.7483740234375,16.25,8.7483740234375Q16.1678,8.7483740234375,16.0871,8.7644180234375Q16.0064,8.7804630234375,15.9305,8.8119350234375Q15.8545,8.8434070234375,15.7861,8.8890970234375Q15.7177,8.9347870234375,15.6596,8.9929400234375Q15.6014,9.0510930234375,15.5557,9.1194730234375Q15.51,9.1878530234375,15.4786,9.2638330234375Q15.4471,9.3398140234375,15.431,9.4204740234375Q15.415,9.5011337234375,15.415,9.5833740234375Q15.415,9.6081817234375,15.4165,9.6329456234375Q15.3991,11.8445940234375,13.8302,13.413544023437499Q12.24366,15.0000440234375,10,15.0000440234375Q7.75633,15.0000440234375,6.16983,13.413544023437499Q4.6008890000000005,11.8445940234375,4.583527,9.6329521234375Z"
|
||||
fill-rule="evenodd"
|
||||
fill="#FFFFFF"
|
||||
fill-opacity="1"
|
||||
style="mix-blend-mode:passthrough"
|
||||
/></g
|
||||
><g
|
||||
><path
|
||||
d="M10.833333,15.8861049234375Q10.835,15.8597658234375,10.835,15.8333740234375Q10.835,15.7511337234375,10.818956,15.6704740234375Q10.802911,15.5898140234375,10.771439,15.5138330234375Q10.739967,15.4378530234375,10.694277,15.3694730234375Q10.648587,15.3010930234375,10.590434,15.2429400234375Q10.532281,15.1847870234375,10.463901,15.1390970234375Q10.395521,15.0934070234375,10.319541,15.0619350234375Q10.24356,15.0304630234375,10.1629,15.0144180234375Q10.0822403,14.9983740234375,10,14.9983740234375Q9.9177597,14.9983740234375,9.8371,15.0144180234375Q9.75644,15.0304630234375,9.680459,15.0619350234375Q9.604479,15.0934070234375,9.536099,15.1390970234375Q9.467719,15.1847870234375,9.409566,15.2429400234375Q9.351413,15.3010930234375,9.305723,15.3694730234375Q9.260033,15.4378530234375,9.228561,15.5138330234375Q9.197089,15.5898140234375,9.181044,15.6704740234375Q9.165,15.7511337234375,9.165,15.8333740234375Q9.165,15.8597658234375,9.166667,15.8861049234375L9.166667,18.2806440234375Q9.165,18.3069840234375,9.165,18.3333740234375Q9.165,18.4156140234375,9.181044,18.4962740234375Q9.197089,18.5769340234375,9.228561,18.6529140234375Q9.260033,18.7288940234375,9.305723,18.7972740234375Q9.351413,18.8656540234375,9.409566,18.9238040234375Q9.467719,18.9819640234375,9.536099,19.0276540234375Q9.604479,19.0733440234375,9.680459,19.1048140234375Q9.75644,19.1362840234375,9.8371,19.1523340234375Q9.9177597,19.1683740234375,10,19.1683740234375Q10.0822403,19.1683740234375,10.1629,19.1523340234375Q10.24356,19.1362840234375,10.319541,19.1048140234375Q10.395521,19.0733440234375,10.463901,19.0276540234375Q10.532281,18.9819640234375,10.590434,18.9238040234375Q10.648587,18.8656540234375,10.694277,18.7972740234375Q10.739967,18.7288940234375,10.771439,18.6529140234375Q10.802911,18.5769340234375,10.818956,18.4962740234375Q10.835,18.4156140234375,10.835,18.3333740234375Q10.835,18.3069840234375,10.833333,18.2806440234375L10.833333,15.8861049234375Z"
|
||||
fill-rule="evenodd"
|
||||
fill="#FFFFFF"
|
||||
fill-opacity="1"
|
||||
style="mix-blend-mode:passthrough"
|
||||
/></g
|
||||
></g
|
||||
></g
|
||||
></svg
|
||||
>
|
||||
|
After Width: | Height: | Size: 6.5 KiB |
23
frontend/shared/VideoChat/icons/PictureInPicture.svelte
Normal file
23
frontend/shared/VideoChat/icons/PictureInPicture.svelte
Normal file
File diff suppressed because one or more lines are too long
|
After Width: | Height: | Size: 5.7 KiB |
9
frontend/shared/VideoChat/icons/Send.svelte
Normal file
9
frontend/shared/VideoChat/icons/Send.svelte
Normal file
@@ -0,0 +1,9 @@
|
||||
<svg
|
||||
class="icon"
|
||||
viewBox="0 0 1024 1024"
|
||||
version="1.1"
|
||||
xmlns="http://www.w3.org/2000/svg"
|
||||
><path
|
||||
d="M899.925333 172.080762a48.761905 48.761905 0 0 1 0 28.525714l-207.969523 679.448381a48.761905 48.761905 0 0 1-81.115429 20.187429l-150.552381-150.552381-96.304762 96.329143a24.380952 24.380952 0 0 1-41.593905-17.237334v-214.966857l275.821715-243.370667-355.57181 161.596953-103.253333-103.228953a48.761905 48.761905 0 0 1 20.23619-81.091047L838.997333 139.702857a48.761905 48.761905 0 0 1 60.903619 32.353524z"
|
||||
></path></svg
|
||||
>
|
||||
|
After Width: | Height: | Size: 544 B |
37
frontend/shared/VideoChat/icons/SideBySide.svelte
Normal file
37
frontend/shared/VideoChat/icons/SideBySide.svelte
Normal file
@@ -0,0 +1,37 @@
|
||||
<svg
|
||||
xmlns="http://www.w3.org/2000/svg"
|
||||
xmlns:xlink="http://www.w3.org/1999/xlink"
|
||||
fill="none"
|
||||
version="1.1"
|
||||
width="20"
|
||||
height="20"
|
||||
viewBox="0 0 20 20"
|
||||
><defs
|
||||
><clipPath id="master_svg0_13_533/13_323"
|
||||
><rect x="0" y="0" width="20" height="20" rx="0" /></clipPath
|
||||
></defs
|
||||
><g clip-path="url(#master_svg0_13_533/13_323)"
|
||||
><g
|
||||
><path
|
||||
d="M7.5,5.0016259765625Q7.58224,5.0016259765625,7.6629,4.9855819765625Q7.74356,4.9695369765625,7.81954,4.9380649765625Q7.89552,4.9065929765625,7.9639,4.8609029765625Q8.03228,4.8152129765625,8.09043,4.7570599765625Q8.14859,4.6989069765625,8.19428,4.6305269765625Q8.23997,4.5621469765625005,8.27144,4.4861669765625Q8.30291,4.4101859765625,8.318950000000001,4.3295259765625Q8.335,4.2488662765625,8.335,4.1666259765625Q8.335,4.0843856765625,8.31896,4.0037259765625Q8.30291,3.9230659765625,8.27144,3.8470849765625Q8.23997,3.7711049765625,8.19428,3.7027249765625Q8.14859,3.6343449765624998,8.09043,3.5761919765625Q8.03228,3.5180389765625,7.9639,3.4723489765625Q7.89552,3.4266589765625,7.81954,3.3951869765625Q7.74356,3.3637149765625,7.6629,3.3476699765625Q7.58224,3.3316259765625,7.5,3.3316259765625Q7.47361,3.3316259765625,7.44727,3.3332929765625L4.16667,3.3332929765625Q3.131133,3.3332929765625,2.3989,4.0655259765625Q1.666667,4.7977589765625,1.666667,5.8332959765625L1.666667,14.1666259765625Q1.666667,15.2021259765625,2.3989,15.9344259765625Q3.131133,16.6666259765625,4.16667,16.6666259765625L7.44728,16.6666259765625Q7.47361,16.6683259765625,7.5,16.6683259765625Q7.58224,16.6683259765625,7.6629,16.652225976562498Q7.74356,16.6362259765625,7.81954,16.6047259765625Q7.89552,16.573225976562497,7.9639,16.5275259765625Q8.03228,16.4819259765625,8.09043,16.4237259765625Q8.14859,16.365525976562502,8.19428,16.2972259765625Q8.23997,16.2288259765625,8.27144,16.1528259765625Q8.30291,16.0768259765625,8.318950000000001,15.9962259765625Q8.335,15.9155259765625,8.335,15.8333259765625Q8.335,15.7510259765625,8.31896,15.6704259765625Q8.30291,15.5897259765625,8.27144,15.5137259765625Q8.23997,15.4377259765625,8.19428,15.3694259765625Q8.14859,15.3010259765625,8.09043,15.2428259765625Q8.03228,15.1847259765625,7.9639,15.1390259765625Q7.89552,15.0933259765625,7.81954,15.0618259765625Q7.74356,15.0304259765625,7.6629,15.0143259765625Q7.58224,14.9983259765625,7.5,14.9983259765625Q7.47361,14.9983259765625,7.44728,14.9999259765625L4.16667,14.9999259765625Q3.82149,14.9999259765625,3.57741,14.7559259765625Q3.333333,14.5118259765625,3.333333,14.1666259765625L3.333333,5.8332959765625Q3.333333,5.4881159765625,3.57741,5.2440359765625Q3.82149,4.9999589765625,4.16667,4.9999589765625L7.44727,4.9999589765625Q7.47361,5.0016259765625,7.5,5.0016259765625Z"
|
||||
fill-rule="evenodd"
|
||||
fill="#FFFFFF"
|
||||
fill-opacity="1"
|
||||
/></g
|
||||
><g
|
||||
><path
|
||||
d="M12.55273,4.9999589765625Q12.5263913,5.0016259765625,12.5,5.0016259765625Q12.4177597,5.0016259765625,12.3371,4.9855819765625Q12.25644,4.9695369765625,12.180459,4.9380649765625Q12.104479,4.9065929765625,12.036099,4.8609029765625Q11.967719,4.8152129765625,11.909566,4.7570599765625Q11.851413,4.6989069765625,11.805723,4.6305269765625Q11.760033,4.5621469765625005,11.728561,4.4861669765625Q11.697089,4.4101859765625,11.681044,4.3295259765625Q11.665,4.2488662765625,11.665,4.1666259765625Q11.665,4.0843856765625,11.681044,4.0037259765625Q11.697089,3.9230659765625,11.728561,3.8470849765625Q11.760033,3.7711049765625,11.805723,3.7027249765625Q11.851413,3.6343449765624998,11.909566,3.5761919765625Q11.967719,3.5180389765625,12.036099,3.4723489765625Q12.104479,3.4266589765625,12.180459,3.3951869765625Q12.25644,3.3637149765625,12.3371,3.3476699765625Q12.4177597,3.3316259765625,12.5,3.3316259765625Q12.5263913,3.3316259765625,12.55273,3.3332929765625L15.83333,3.3332929765625Q16.86887,3.3332929765625,17.6011,4.0655259765625Q18.33333,4.7977589765625,18.33333,5.8332959765625L18.33333,14.1666259765625Q18.33333,15.2021259765625,17.6011,15.9344259765625Q16.86887,16.6666259765625,15.83333,16.6666259765625L12.5527215,16.6666259765625Q12.5263871,16.6683259765625,12.5,16.6683259765625Q12.4177597,16.6683259765625,12.3371,16.652225976562498Q12.25644,16.6362259765625,12.180459,16.6047259765625Q12.104479,16.573225976562497,12.036099,16.5275259765625Q11.967719,16.4819259765625,11.909566,16.4237259765625Q11.851413,16.365525976562502,11.805723,16.2972259765625Q11.760033,16.2288259765625,11.728561,16.1528259765625Q11.697089,16.0768259765625,11.681044,15.9962259765625Q11.665,15.9155259765625,11.665,15.8333259765625Q11.665,15.7510259765625,11.681044,15.6704259765625Q11.697089,15.5897259765625,11.728561,15.5137259765625Q11.760033,15.4377259765625,11.805723,15.3694259765625Q11.851413,15.3010259765625,11.909566,15.2428259765625Q11.967719,15.1847259765625,12.036099,15.1390259765625Q12.104479,15.0933259765625,12.180459,15.0618259765625Q12.25644,15.0304259765625,12.3371,15.0143259765625Q12.4177597,14.9983259765625,12.5,14.9983259765625Q12.5263871,14.9983259765625,12.5527215,14.9999259765625L15.83333,14.9999259765625Q16.17851,14.9999259765625,16.42259,14.7559259765625Q16.66667,14.5118259765625,16.66667,14.1666259765625L16.66667,5.8332959765625Q16.66667,5.4881159765625,16.42259,5.2440359765625Q16.17851,4.9999589765625,15.83333,4.9999589765625L12.55273,4.9999589765625Z"
|
||||
fill-rule="evenodd"
|
||||
fill="#FFFFFF"
|
||||
fill-opacity="1"
|
||||
/></g
|
||||
><g
|
||||
><path
|
||||
d="M10.833333,2.5527319Q10.835,2.5263923,10.835,2.5Q10.835,2.4177597,10.818956,2.3371Q10.802911,2.25644,10.771439,2.180459Q10.739967,2.104479,10.694277,2.036099Q10.648587,1.967719,10.590434,1.9095659999999999Q10.532281,1.851413,10.463901,1.805723Q10.395521,1.760033,10.319541,1.728561Q10.24356,1.697089,10.1629,1.681044Q10.0822403,1.665,10,1.665Q9.9177597,1.665,9.8371,1.681044Q9.75644,1.697089,9.680459,1.728561Q9.604479,1.760033,9.536099,1.805723Q9.467719,1.851413,9.409566,1.9095659999999999Q9.351413,1.967719,9.305723,2.036099Q9.260033,2.104479,9.228561,2.180459Q9.197089,2.25644,9.181044,2.3371Q9.165,2.4177597,9.165,2.5Q9.165,2.5263923,9.166667,2.5527319L9.166667,17.4473Q9.165,17.473599999999998,9.165,17.5Q9.165,17.5822,9.181044,17.6629Q9.197089,17.7436,9.228561,17.819499999999998Q9.260033,17.8955,9.305723,17.963900000000002Q9.351413,18.0323,9.409566,18.090400000000002Q9.467719,18.148600000000002,9.536099,18.1943Q9.604479,18.240000000000002,9.680459,18.2714Q9.75644,18.3029,9.8371,18.319000000000003Q9.9177597,18.335,10,18.335Q10.0822403,18.335,10.1629,18.319000000000003Q10.24356,18.3029,10.319541,18.2714Q10.395521,18.240000000000002,10.463901,18.1943Q10.532281,18.148600000000002,10.590434,18.090400000000002Q10.648587,18.0323,10.694277,17.963900000000002Q10.739967,17.8955,10.771439,17.819499999999998Q10.802911,17.7436,10.818956,17.6629Q10.835,17.5822,10.835,17.5Q10.835,17.473599999999998,10.833333,17.4473L10.833333,2.5527319Z"
|
||||
fill-rule="evenodd"
|
||||
fill="#FFFFFF"
|
||||
fill-opacity="1"
|
||||
/></g
|
||||
></g
|
||||
></svg
|
||||
>
|
||||
|
After Width: | Height: | Size: 6.8 KiB |
13
frontend/shared/VideoChat/icons/Stop.svelte
Normal file
13
frontend/shared/VideoChat/icons/Stop.svelte
Normal file
@@ -0,0 +1,13 @@
|
||||
<svg
|
||||
t="1742449891206"
|
||||
class="icon"
|
||||
viewBox="0 0 1024 1024"
|
||||
version="1.1"
|
||||
xmlns="http://www.w3.org/2000/svg"
|
||||
p-id="2067"
|
||||
xmlns:xlink="http://www.w3.org/1999/xlink"
|
||||
><path
|
||||
d="M950.857143 109.714286l0 804.571429q0 14.857143-10.857143 25.714286t-25.714286 10.857143l-804.571429 0q-14.857143 0-25.714286-10.857143t-10.857143-25.714286l0-804.571429q0-14.857143 10.857143-25.714286t25.714286-10.857143l804.571429 0q14.857143 0 25.714286 10.857143t10.857143 25.714286z"
|
||||
p-id="2068"
|
||||
></path></svg
|
||||
>
|
||||
|
After Width: | Height: | Size: 517 B |
1
frontend/shared/VideoChat/icons/SubtitleOff.svelte
Normal file
1
frontend/shared/VideoChat/icons/SubtitleOff.svelte
Normal file
@@ -0,0 +1 @@
|
||||
<svg t="1744352112173" class="icon" viewBox="0 0 1024 1024" version="1.1" xmlns="http://www.w3.org/2000/svg" p-id="16533" xmlns:xlink="http://www.w3.org/1999/xlink" width="200" height="200"><path d="M824 466.56V213.12q0-13.6512-5.2928-26.1632-5.104-12.064-14.3904-21.3536-9.2864-9.2864-21.3504-14.3872-12.5152-5.2928-26.1664-5.2928H246.4q-13.6512 0-26.1664 5.2928-12.064 5.1008-21.3504 14.3872-9.2864 9.2864-14.3904 21.3536Q179.2 199.4688 179.2 213.12v607.296q0 12.8448 5.0592 24.608 4.8576 11.2896 13.6704 19.9552 8.7616 8.6176 20.1184 13.344Q229.7792 883.2 242.56 883.2h217.6a28.8 28.8 0 0 0 0-57.6h-217.6q-2.528 0-4.2432-1.6864-1.5168-1.4912-1.5168-3.4976V213.12q0-3.9744 2.8128-6.784 2.8096-2.816 6.7872-2.816h510.4q3.9776 0 6.7872 2.816 2.8128 2.8096 2.8128 6.784v253.44a28.8 28.8 0 0 0 28.8 28.8 28.8 28.8 0 0 0 28.8-28.8zM466.0064 338.08l-130.2016 278.784A32 32 0 0 0 364.8 662.4h0.176a31.9904 31.9904 0 0 0 28.8192-18.4576L418.048 592h165.4976l15.2896 32.736q3.1008-3.4144 6.3904-6.704 20.3584-20.3616 45.4816-33.472l-115.1168-246.4832q-4.9408-10.5792-14.8704-16.5952-9.168-5.5552-19.9232-5.5552-10.7552 0-19.9232 5.5552-9.9296 6.016-14.8704 16.5952z m34.7936 76.7456L553.6576 528h-105.7152l52.8576-113.1776zM896 750.4c0 87.4816-70.9184 158.4-158.4 158.4S579.2 837.8816 579.2 750.4s70.9184-158.4 158.4-158.4 158.4 70.9184 158.4 158.4z m-116.3648-82.7648a28.9152 28.9152 0 0 1 7.1648-5.232q-5.8048-3.232-12.096-5.7248Q756.8256 649.6 737.6 649.6q-19.2256 0-37.104 7.0784-19.4112 7.6832-34.1728 22.448-14.7616 14.7616-22.4448 34.1696Q636.8 731.1744 636.8 750.4q0 19.232 7.0784 37.104 2.4896 6.2944 5.7248 12.096a28.7552 28.7552 0 0 1 5.232-7.1648l124.8-124.8zM838.4 750.4q0-19.2256-7.0784-37.104-2.4896-6.2912-5.7248-12.096a28.6944 28.6944 0 0 1-5.232 7.168l-124.8 124.8a28.7552 28.7552 0 0 1-7.1648 5.2288q5.8048 3.2352 12.096 5.728Q718.3744 851.2 737.6 851.2q19.2256 0 37.104-7.072 19.4112-7.6896 34.1728-22.4512 14.7616-14.7616 22.4448-34.1728Q838.4 769.632 838.4 750.4z" p-id="16534"></path></svg>
|
||||
|
After Width: | Height: | Size: 2.0 KiB |
1
frontend/shared/VideoChat/icons/SubtitleOn.svelte
Normal file
1
frontend/shared/VideoChat/icons/SubtitleOn.svelte
Normal file
@@ -0,0 +1 @@
|
||||
<svg t="1744352097285" class="icon" viewBox="0 0 1024 1024" version="1.1" xmlns="http://www.w3.org/2000/svg" p-id="16380" data-spm-anchor-id="a313x.manage_type_myprojects.0.i0.60b03a81nz0mun" xmlns:xlink="http://www.w3.org/1999/xlink" width="200" height="200"><path d="M833.6 213.12v253.44a28.8 28.8 0 0 1-28.8 28.8 28.8 28.8 0 0 1-28.8-28.8V213.12q0-3.9744-2.8128-6.784-2.8096-2.816-6.7872-2.816H256q-3.9776 0-6.7872 2.816Q246.4 209.1424 246.4 213.12v607.296q0 2.0064 1.5168 3.4976 1.7152 1.6864 4.2432 1.6864h217.6a28.8 28.8 0 0 1 0 57.6h-217.6q-12.7808 0-24.512-4.8768-11.3568-4.7232-20.1184-13.3408-8.8128-8.6656-13.6704-19.9584Q188.8 833.264 188.8 820.416V213.12q0-13.6512 5.2928-26.1632 5.104-12.064 14.3904-21.3536 9.2864-9.2864 21.3504-14.3872Q242.3456 145.92 256 145.92h510.4q13.6512 0 26.1664 5.2928 12.064 5.1008 21.3504 14.3872 9.2864 9.2864 14.3904 21.3536 5.2928 12.512 5.2928 26.1664zM345.408 613.664l130.1984-278.784q4.9408-10.5824 14.8704-16.5984 9.168-5.5552 19.9232-5.5552 10.7552 0 19.9232 5.5552 9.9296 6.016 14.8704 16.5952l130.2016 278.784a32 32 0 0 1-28.672 45.5392l-0.3232 0.0032c-12.4288 0-23.7344-7.2-28.9952-18.4608L593.1488 588.8h-165.4976l-24.256 51.9424a32.0064 32.0064 0 0 1-28.8192 18.4576l-0.176 0.0032a32 32 0 0 1-28.992-45.5424z m164.992-202.0416L457.5424 524.8h105.7152L510.4 411.6224z m120.2784 329.44l61.3216 61.5872 162.1248-162.8256a31.9936 31.9936 0 0 1 22.608-9.424H876.8a32 32 0 0 1 32 31.936v0.064a32 32 0 0 1-9.3216 22.5792l-184.8 185.6-0.0992 0.0992a31.9936 31.9936 0 0 1-45.2544-0.096l-83.984-84.352-0.016-0.016a31.9904 31.9904 0 0 1-9.2896-21.1104l30.496-33.4336c0.4896-0.0224 0.9792-0.032 1.4688-0.032h0.0704a32 32 0 0 1 22.608 9.4208z" p-id="16381" data-spm-anchor-id="a313x.manage_type_myprojects.0.i1.60b03a81nz0mun" class="selected"></path></svg>
|
||||
|
After Width: | Height: | Size: 1.8 KiB |
40
frontend/shared/VideoChat/icons/VolumeOff.svelte
Normal file
40
frontend/shared/VideoChat/icons/VolumeOff.svelte
Normal file
@@ -0,0 +1,40 @@
|
||||
<svg
|
||||
xmlns="http://www.w3.org/2000/svg"
|
||||
xmlns:xlink="http://www.w3.org/1999/xlink"
|
||||
fill="none"
|
||||
version="1.1"
|
||||
width="20"
|
||||
height="20"
|
||||
viewBox="0 0 20 20"
|
||||
><defs
|
||||
><clipPath id="master_svg0_20_113"
|
||||
><rect x="0" y="0" width="20" height="20" rx="0" /></clipPath
|
||||
></defs
|
||||
><g clip-path="url(#master_svg0_20_113)"
|
||||
><g
|
||||
><path
|
||||
d="M17.52452171875,9.078936578124999Q17.659471718749998,8.960048578125,17.73351171875,8.796145578125Q17.80755171875,8.632242578125,17.80755171875,8.452392578125Q17.80755171875,8.370152278125,17.79151171875,8.289492578125Q17.77546171875,8.208832578125,17.74399171875,8.132851578125Q17.71252171875,8.056871578125,17.66683171875,7.988491578125Q17.62114171875,7.920111578125,17.56299171875,7.861958578125Q17.50483171875,7.803805578125,17.43645171875,7.758115578125Q17.36807171875,7.712425578125,17.29209171875,7.680953578125Q17.21611171875,7.649481578125,17.13545171875,7.633436578125Q17.05479171875,7.617392578125,16.97255171875,7.617392578125Q16.79270171875,7.617392578125,16.62880171875,7.691433578125Q16.46490171875,7.765474578125,16.34601171875,7.900425578125L12.88504271875,11.361392578124999Q12.75009271875,11.480282578125,12.67605171875,11.644182578125001Q12.60201171875,11.808082578125,12.60201171875,11.987932578125001Q12.60201171875,12.070172578125,12.61805571875,12.150832578125Q12.63410071875,12.231492578125,12.66557271875,12.307472578125001Q12.69704471875,12.383452578125,12.74273471875,12.451832578125Q12.78842471875,12.520212578125001,12.84657771875,12.578362578124999Q12.90473071875,12.636522578125,12.97311071875,12.682212578125Q13.04149071875,12.727902578125,13.11747071875,12.759372578125Q13.19345171875,12.790842578125,13.27411171875,12.806892578125Q13.35477141875,12.822932578125,13.43701171875,12.822932578125Q13.61685971875,12.822932578125,13.78076071875,12.748892578125Q13.94466171875,12.674852578125,14.06354971875,12.539902578125L17.52452171875,9.078936578124999Z"
|
||||
fill-rule="evenodd"
|
||||
fill="#FFFFFF"
|
||||
fill-opacity="1"
|
||||
style="mix-blend-mode:passthrough"
|
||||
/></g
|
||||
><g
|
||||
><path
|
||||
d="M12.88553,9.078933578125Q12.75058,8.960045578125,12.67654,8.796143578125Q12.6025,8.632241578125,12.6025,8.452392578125Q12.6025,8.370152278125,12.618544,8.289492578125Q12.634589,8.208832578125,12.666061,8.132851578125Q12.697533,8.056871578125,12.743223,7.988491578125Q12.788913,7.920111578125,12.847066,7.861958578125Q12.905219,7.803805578125,12.973599,7.758115578125Q13.041979,7.712425578125,13.117959,7.680953578125Q13.19394,7.649481578125,13.2746,7.633436578125Q13.3552597,7.617392578125,13.4375,7.617392578125Q13.617349,7.617392578125,13.781251,7.691432578125Q13.945153,7.765472578125,14.064041,7.900422578125L17.52501,11.361392578124999Q17.659959999999998,11.480282578125,17.734,11.644182578125001Q17.80804,11.808082578125,17.80804,11.987932578125001Q17.80804,12.070172578125,17.792,12.150832578125Q17.77595,12.231492578125,17.74448,12.307472578125001Q17.71301,12.383452578125,17.66732,12.451832578125Q17.62163,12.520212578125001,17.56347,12.578362578124999Q17.50532,12.636522578125,17.43694,12.682212578125Q17.36856,12.727902578125,17.29258,12.759372578125Q17.2166,12.790842578125,17.13594,12.806892578125Q17.05528,12.822932578125,16.97304,12.822932578125Q16.79319,12.822932578125,16.62929,12.748892578125Q16.46539,12.674852578125,16.3465,12.539902578125L12.88553,9.078933578125Z"
|
||||
fill-rule="evenodd"
|
||||
fill="#FFFFFF"
|
||||
fill-opacity="1"
|
||||
style="mix-blend-mode:passthrough"
|
||||
/></g
|
||||
><g
|
||||
><path
|
||||
d="M4.44364390625,5.42117L2.49983690625,5.42117Q1.80948090625,5.42117,1.32132890625,5.90931Q0.83317090625,6.39747,0.83317090625,7.08783L0.83317090625,12.8496Q0.83317090625,13.54,1.32132990625,14.0281Q1.80948090625,14.5163,2.49983690625,14.5163L4.43961390625,14.5163Q6.77175390625,18.3333,9.99983390625,18.3333Q10.08191390625,18.3333,10.16241390625,18.3173Q10.24291390625,18.301299999999998,10.31874390625,18.2699Q10.39456390625,18.238500000000002,10.46281390625,18.1929Q10.53105390625,18.1473,10.58909390625,18.0893Q10.64713390625,18.0312,10.69272390625,17.963Q10.73832390625,17.8947,10.76973390625,17.8189Q10.80114390625,17.7431,10.81715390625,17.662599999999998Q10.83317390625,17.5821,10.83317390625,17.5L10.83317390625,2.5Q10.83317390625,2.4179238,10.81715390625,2.337425Q10.80114390625,2.256926,10.76973390625,2.181097Q10.73832390625,2.105269,10.69272390625,2.037025Q10.64712390625,1.968781,10.58909390625,1.910744Q10.53105390625,1.852708,10.46281390625,1.807109Q10.39456390625,1.76151,10.31874390625,1.7301009999999999Q10.24291390625,1.698691,10.16241390625,1.682679Q10.08191390625,1.666667,9.99983390625,1.666667Q6.77619390625,1.666667,4.44364390625,5.42117ZM4.91587390625,7.08783Q5.02559390625,7.08783,5.13157390625,7.05943Q5.23755390625,7.03103,5.3325739062499995,6.97617Q5.42758390625,6.92131,5.50516390625,6.84372Q5.58274390625,6.76614,5.63759390625,6.67111Q7.22859390625,3.91495,9.16650390625,3.434681L9.16650390625,16.563299999999998Q7.23188390625,16.074199999999998,5.6405439062500005,13.2715Q5.58600390625,13.1754,5.50830390625,13.0969Q5.4306139062500005,13.0184,5.33514390625,12.9628Q5.23966390625,12.9072,5.1330139062499995,12.8784Q5.02635390625,12.8496,4.91587390625,12.8496L2.49983690625,12.8496L2.49983790625,7.08783L4.91587390625,7.08783Z"
|
||||
fill-rule="evenodd"
|
||||
fill="#FFFFFF"
|
||||
fill-opacity="1"
|
||||
style="mix-blend-mode:passthrough"
|
||||
/></g
|
||||
></g
|
||||
></svg
|
||||
>
|
||||
|
After Width: | Height: | Size: 5.3 KiB |
55
frontend/shared/VideoChat/icons/VolumeOn.svelte
Normal file
55
frontend/shared/VideoChat/icons/VolumeOn.svelte
Normal file
@@ -0,0 +1,55 @@
|
||||
<svg
|
||||
xmlns="http://www.w3.org/2000/svg"
|
||||
xmlns:xlink="http://www.w3.org/1999/xlink"
|
||||
fill="none"
|
||||
version="1.1"
|
||||
width="20"
|
||||
height="20"
|
||||
viewBox="0 0 20 20"
|
||||
><defs
|
||||
><clipPath id="master_svg0_13_280"
|
||||
><rect x="0" y="0" width="20" height="20" rx="0" /></clipPath
|
||||
><clipPath id="master_svg1_13_280/13_053"
|
||||
><rect x="0" y="0" width="20" height="20" rx="0" /></clipPath
|
||||
></defs
|
||||
><g clip-path="url(#master_svg0_13_280)"
|
||||
><g clip-path="url(#master_svg1_13_280/13_053)"
|
||||
><g
|
||||
><rect
|
||||
x="0"
|
||||
y="0"
|
||||
width="20"
|
||||
height="20"
|
||||
rx="0"
|
||||
fill="#FFFFFF"
|
||||
fill-opacity="0.009999999776482582"
|
||||
style="mix-blend-mode:passthrough"
|
||||
/></g
|
||||
><g
|
||||
><path
|
||||
d="M4.443888046875,5.42117L2.500081046875,5.42117Q1.809725046875,5.42117,1.321573046875,5.90931Q0.833415046875,6.39747,0.833415046875,7.08783L0.833415046875,12.8496Q0.833415046875,13.54,1.321574046875,14.0281Q1.809725046875,14.5163,2.500081046875,14.5163L4.439858046875,14.5163Q6.771998046875,18.3333,10.000078046875,18.3333Q10.082158046875,18.3333,10.162658046875,18.3173Q10.243158046875,18.301299999999998,10.318988046875,18.2699Q10.394808046875,18.238500000000002,10.463058046875,18.1929Q10.531298046875,18.1473,10.589338046875,18.0893Q10.647378046875,18.0312,10.692968046875,17.963Q10.738568046875,17.8947,10.769978046875,17.8189Q10.801388046875,17.7431,10.817398046875,17.662599999999998Q10.833418046875,17.5821,10.833418046875,17.5L10.833418046875,2.5Q10.833418046875,2.4179238,10.817398046875,2.337425Q10.801388046875,2.256926,10.769978046875,2.181097Q10.738568046875,2.105269,10.692968046875,2.037025Q10.647368046875,1.968781,10.589338046875,1.910744Q10.531298046875,1.852708,10.463058046875,1.807109Q10.394808046875,1.76151,10.318988046875,1.7301009999999999Q10.243158046875,1.698691,10.162658046875,1.682679Q10.082158046875,1.666667,10.000078046875,1.666667Q6.776438046875,1.666667,4.443888046875,5.42117ZM4.916118046875,7.08783Q5.025838046875,7.08783,5.131818046875,7.05943Q5.237798046875,7.03103,5.3328180468749995,6.97617Q5.427828046875,6.92131,5.505408046875,6.84372Q5.582988046875,6.76614,5.637838046875,6.67111Q7.228838046875,3.91495,9.166748046875,3.434681L9.166748046875,16.563299999999998Q7.232128046875,16.074199999999998,5.6407880468750005,13.2715Q5.586248046875,13.1754,5.508548046875,13.0969Q5.4308580468750005,13.0184,5.335388046875,12.9628Q5.239908046875,12.9072,5.1332580468749995,12.8784Q5.026598046875,12.8496,4.916118046875,12.8496L2.500081046875,12.8496L2.500082046875,7.08783L4.916118046875,7.08783Z"
|
||||
fill-rule="evenodd"
|
||||
fill="#FFFFFF"
|
||||
fill-opacity="1"
|
||||
style="mix-blend-mode:passthrough"
|
||||
/></g
|
||||
><g
|
||||
><path
|
||||
d="M12.813896953124999,6.903831Q12.740067953125,6.845187,12.681175953125,6.771557Q12.622282953125,6.697926,12.581291953125,6.613017Q12.540300953125,6.528109,12.519276953125,6.436197Q12.498251953125,6.3442856,12.498251953125,6.25Q12.498251953125,6.1677597,12.514295953125,6.0871Q12.530340953125,6.00644,12.561812953125,5.930459Q12.593284953125,5.8544789999999995,12.638974953125,5.786099Q12.684664953125,5.717719,12.742817953125,5.659566Q12.800970953125,5.601413,12.869350953125,5.555723Q12.937730953125,5.510033,13.013710953125,5.478561Q13.089691953125,5.447089,13.170351953125,5.431044Q13.251011653125,5.415,13.333251953125,5.415Q13.501904953125,5.415,13.657335953125,5.4804580000000005Q13.812766953125,5.545916,13.930607953125,5.66657Q14.362131953125001,6.059567,14.707911953125,6.532997Q15.248111953125001,7.2726299999999995,15.535961953125,8.14354Q15.833251953125,9.04304,15.833251953125,10Q15.833251953125,10.94869,15.540941953125,11.84127Q15.257921953125,12.70551,14.726221953125,13.4418Q14.373671953125,13.92992,13.930609953125,14.33343Q13.812768953125,14.45408,13.657336953125,14.51954Q13.501904953125,14.585,13.333251953125,14.585Q13.251011653125,14.585,13.170351953125,14.56895Q13.089691953125,14.55291,13.013710953125,14.52144Q12.937730953125,14.48997,12.869350953125,14.44428Q12.800970953125,14.39859,12.742817953125,14.34043Q12.684664953125,14.28228,12.638974953125,14.213899999999999Q12.593284953125,14.145520000000001,12.561812953125,14.06954Q12.530340953125,13.99356,12.514295953125,13.9129Q12.498251953125,13.832239999999999,12.498251953125,13.75Q12.498251953125,13.655719999999999,12.519276953125,13.5638Q12.540300953125,13.47189,12.581291953125,13.386980000000001Q12.622282953125,13.30207,12.681174953125,13.228439999999999Q12.740067953125,13.154810000000001,12.813895953125,13.09617Q13.125969953125,12.8109,13.375114153125,12.46595Q14.166584953125,11.36993,14.166584953125,10Q14.166584953125,8.61762,13.362005753125,7.516Q13.117749953125,7.181583,12.813896953124999,6.903831Z"
|
||||
fill-rule="evenodd"
|
||||
fill="#FFFFFF"
|
||||
fill-opacity="1"
|
||||
style="mix-blend-mode:passthrough"
|
||||
/></g
|
||||
><g
|
||||
><path
|
||||
d="M14.863105578125,2.228405984375Q16.823672578125,3.456592984375,17.969842578125,5.468508984375Q19.166602578125,7.569218984375,19.166602578125,10.000018984375Q19.166602578125,12.469708984375,17.933552578125,14.594658984375Q16.751772578125,16.631258984375002,14.739248578125,17.847858984375Q14.525103578125,17.995758984375,14.264892578125,17.995758984375Q14.182652278125,17.995758984375,14.101992578125,17.979658984375Q14.021332578125,17.963658984375,13.945351578125,17.932158984375Q13.869371578125,17.900658984375,13.800991578125,17.854958984375Q13.732611578125,17.809358984375002,13.674458578125,17.751158984375Q13.616305578125,17.692958984375,13.570615578125,17.624658984375Q13.524925578125,17.556258984375,13.493453578125,17.480258984375Q13.461981578125,17.404258984374998,13.445936578125,17.323658984375Q13.429892578125,17.242958984375,13.429892578125,17.160758984375Q13.429892578125,17.045958984374998,13.460862578124999,16.935458984375Q13.491831578125,16.824858984375,13.551473578125,16.726858984375Q13.611115578125,16.628758984375,13.695005578125,16.550458984375Q13.778896578125,16.472058984375,13.880811578125,16.419258984375Q15.525652578125,15.423558984375,16.492012578125,13.758158984375Q17.499932578125,12.021168984375,17.499932578125,10.000018984375Q17.499932578125,8.010658984374999,16.521692578125,6.293508984375Q15.584492578125,4.648418984375,13.982075578125,3.643182984375Q13.882499578125,3.589574984375,13.800770578125,3.511412984375Q13.719041578125,3.433249984375,13.661047578125,3.336162984375Q13.603053578125,3.239076984375,13.572972578125,3.130062984375Q13.542891578125,3.021047984375,13.542891578125,2.907958984375Q13.542891578125,2.825718684375,13.558936578125,2.745058984375Q13.574980578125,2.664398984375,13.606452578125,2.588417984375Q13.637924578125,2.512437984375,13.683614578125,2.444057984375Q13.729305578125,2.3756779843749998,13.787457578125,2.317524984375Q13.845610578125,2.259371984375,13.913990578125,2.213681984375Q13.982370578125,2.167991984375,14.058351578125,2.136519984375Q14.134331578125,2.105047984375,14.214991478125,2.089002984375Q14.295651478125,2.072958984375,14.377891578125,2.072958984375Q14.508378578125,2.072958984375,14.632644578125,2.112769984375Q14.756910578125,2.152579984375,14.863105578125,2.228405984375Z"
|
||||
fill-rule="evenodd"
|
||||
fill="#FFFFFF"
|
||||
fill-opacity="1"
|
||||
style="mix-blend-mode:passthrough"
|
||||
/></g
|
||||
></g
|
||||
></g
|
||||
></svg
|
||||
>
|
||||
|
After Width: | Height: | Size: 7.2 KiB |
32
frontend/shared/VideoChat/icons/index.ts
Normal file
32
frontend/shared/VideoChat/icons/index.ts
Normal file
@@ -0,0 +1,32 @@
|
||||
import './style.css'
|
||||
import IconFont from "./IconFont.svelte";
|
||||
import Send from "./Send.svelte";
|
||||
import Stop from "./Stop.svelte";
|
||||
import CameraOff from "./CameraOff.svelte";
|
||||
import CameraOn from "./CameraOn.svelte";
|
||||
import VolumeOff from "./VolumeOff.svelte";
|
||||
import VolumeOn from "./VolumeOn.svelte";
|
||||
import SubtitleOn from "./SubtitleOn.svelte";
|
||||
import SubtitleOff from "./SubtitleOff.svelte";
|
||||
import MicOff from "./MicOff.svelte";
|
||||
import MicOn from "./MicOn.svelte";
|
||||
import Check from "./Check.svelte";
|
||||
import PictureInPicture from "./PictureInPicture.svelte";
|
||||
import SideBySide from "./SideBySide.svelte";
|
||||
|
||||
export {
|
||||
IconFont,
|
||||
CameraOff,
|
||||
CameraOn,
|
||||
VolumeOff,
|
||||
VolumeOn,
|
||||
SubtitleOn,
|
||||
SubtitleOff,
|
||||
MicOff,
|
||||
MicOn,
|
||||
Send,
|
||||
Check,
|
||||
PictureInPicture,
|
||||
SideBySide,
|
||||
Stop
|
||||
}
|
||||
9
frontend/shared/VideoChat/icons/style.css
Normal file
9
frontend/shared/VideoChat/icons/style.css
Normal file
@@ -0,0 +1,9 @@
|
||||
.icon {
|
||||
width: 1em;
|
||||
height: 1em;
|
||||
vertical-align: -0.15em;
|
||||
fill: currentColor;
|
||||
overflow: hidden;
|
||||
color: inherit;
|
||||
font-size: inherit;
|
||||
}
|
||||
1108
frontend/shared/VideoChat/index.svelte
Normal file
1108
frontend/shared/VideoChat/index.svelte
Normal file
File diff suppressed because it is too large
Load Diff
29
frontend/shared/VideoChat/interface/eventType.ts
Normal file
29
frontend/shared/VideoChat/interface/eventType.ts
Normal file
@@ -0,0 +1,29 @@
|
||||
export enum EventTypes {
|
||||
'ErrorReceived' = 'ErrorReceived',
|
||||
'MessageReceived' = 'MessageReceived',
|
||||
'StartSpeech' = 'StartSpeech',
|
||||
'EndSpeech' = 'EndSpeech',
|
||||
'StateChanged' = 'StateChanged',
|
||||
}
|
||||
|
||||
export enum WsEventTypes {
|
||||
'WS_CLOSE' = 'WS_CLOSE',
|
||||
'WS_ERROR' = 'WS_ERROR',
|
||||
'WS_MESSAGE' = 'WS_MESSAGE',
|
||||
'WS_OPEN' = 'WS_OPEN'
|
||||
}
|
||||
|
||||
export enum PlayerEventTypes {
|
||||
// Player没断
|
||||
'Player_EndSpeaking' = 'Player_EndSpeaking',
|
||||
'Player_NoLegacy' = 'Player_NoLegacy',
|
||||
// Player相关
|
||||
'Player_StartSpeaking' = 'Player_StartSpeaking',
|
||||
'Player_WaitNextAudioClip' = 'Player_WaitNextAudioClip'
|
||||
}
|
||||
// 端测渲染(端到端)、单独输出数字人处理核心数据Processor相关的事件
|
||||
export enum ProcessorEventTypes {
|
||||
'Change_Status' = 'Change_Status',
|
||||
'Chat_BinsizeError' = 'Chat_BinsizeError'
|
||||
}
|
||||
|
||||
6
frontend/shared/VideoChat/interface/voiceChat.ts
Normal file
6
frontend/shared/VideoChat/interface/voiceChat.ts
Normal file
@@ -0,0 +1,6 @@
|
||||
export enum TYVoiceChatState {
|
||||
Idle = 'Idle',
|
||||
Listening = 'Listening',
|
||||
Responding = 'Responding',
|
||||
Thinking = 'Thinking'
|
||||
}
|
||||
131
frontend/shared/VideoChat/stream_utils.ts
Normal file
131
frontend/shared/VideoChat/stream_utils.ts
Normal file
@@ -0,0 +1,131 @@
|
||||
export function get_devices(): Promise<MediaDeviceInfo[]> {
|
||||
return navigator.mediaDevices.enumerateDevices();
|
||||
}
|
||||
|
||||
export function handle_error(error: string): void {
|
||||
throw new Error(error);
|
||||
}
|
||||
|
||||
export function set_local_stream(
|
||||
local_stream: MediaStream | null,
|
||||
video_source: HTMLVideoElement,
|
||||
): void {
|
||||
video_source.srcObject = local_stream;
|
||||
video_source.muted = true;
|
||||
video_source.play();
|
||||
}
|
||||
|
||||
export async function get_stream(
|
||||
audio: boolean | { deviceId: { exact: string } },
|
||||
video: boolean | { deviceId: { exact: string } },
|
||||
video_source: HTMLVideoElement,
|
||||
track_constraints?:
|
||||
| MediaTrackConstraints
|
||||
| { video: MediaTrackConstraints; audio: MediaTrackConstraints },
|
||||
): Promise<MediaStream> {
|
||||
const video_fallback_constraints = (track_constraints as any)?.video ||
|
||||
track_constraints || {
|
||||
width: { ideal: 500 },
|
||||
height: { ideal: 500 },
|
||||
};
|
||||
const audio_fallback_constraints = (track_constraints as any)?.audio ||
|
||||
track_constraints || {
|
||||
echoCancellation: true,
|
||||
noiseSuppression: true,
|
||||
autoGainControl: true,
|
||||
};
|
||||
const constraints = {
|
||||
video:
|
||||
typeof video === "object"
|
||||
? { ...video, ...video_fallback_constraints }
|
||||
: video,
|
||||
audio:
|
||||
typeof audio === "object"
|
||||
? { ...audio, ...audio_fallback_constraints }
|
||||
: audio,
|
||||
};
|
||||
return navigator.mediaDevices
|
||||
.getUserMedia(constraints)
|
||||
.then((local_stream: MediaStream) => {
|
||||
return local_stream;
|
||||
});
|
||||
}
|
||||
|
||||
export function set_available_devices(
|
||||
devices: MediaDeviceInfo[],
|
||||
kind: "videoinput" | "audioinput" = "videoinput",
|
||||
): MediaDeviceInfo[] {
|
||||
const cameras = devices.filter(
|
||||
(device: MediaDeviceInfo) => device.kind === kind,
|
||||
);
|
||||
|
||||
return cameras;
|
||||
}
|
||||
|
||||
let video_track: MediaStreamTrack | null = null;
|
||||
let audio_track: MediaStreamTrack | null = null;
|
||||
|
||||
export function createSimulatedVideoTrack(width = 1, height = 1) {
|
||||
// if (video_track) return video_track
|
||||
// 创建一个 canvas 元素
|
||||
const canvas = document.createElement("canvas");
|
||||
document.body.appendChild(canvas);
|
||||
canvas.width = width || 500;
|
||||
canvas.height = height || 500;
|
||||
canvas.style.width = "1px";
|
||||
canvas.style.height = "1px";
|
||||
canvas.style.position = "fixed";
|
||||
canvas.style.visibility = "hidden";
|
||||
const ctx = canvas.getContext("2d") as CanvasRenderingContext2D;
|
||||
|
||||
ctx.fillStyle = `hsl(0,0, 0, 1)`; // 动态颜色
|
||||
ctx.fillRect(0, 0, canvas.width, canvas.height);
|
||||
let time = 0;
|
||||
// 在 canvas 上绘制动画内容
|
||||
function drawFrame() {
|
||||
// ctx.fillStyle = `rgb(0, ${(Date.now() / 10) % 360}, 1)`; // 动态颜色
|
||||
ctx.fillStyle = `rgb(255, 255, 255)`; // 动态颜色
|
||||
|
||||
ctx.fillRect(0, 0, canvas.width, canvas.height);
|
||||
// ctx.font = 'bold 50px Arial';
|
||||
// ctx.fillStyle = `rgb(0, 0, 0)`;
|
||||
// ctx.fillText(String(time++), 100, 100)
|
||||
requestAnimationFrame(drawFrame);
|
||||
}
|
||||
drawFrame();
|
||||
|
||||
// 捕获 canvas 的视频流
|
||||
const stream = canvas.captureStream(30); // 30 FPS
|
||||
video_track = stream.getVideoTracks()[0]; // 返回视频轨道
|
||||
video_track.stop = () => {
|
||||
canvas.remove();
|
||||
};
|
||||
video_track.onended = () => {
|
||||
video_track?.stop();
|
||||
};
|
||||
return video_track;
|
||||
}
|
||||
|
||||
export function createSimulatedAudioTrack() {
|
||||
if (audio_track) return audio_track;
|
||||
const audioContext = new (window.AudioContext || window.webkitAudioContext)();
|
||||
const oscillator = audioContext.createOscillator();
|
||||
oscillator.frequency.setValueAtTime(0, audioContext.currentTime);
|
||||
|
||||
const gainNode = audioContext.createGain();
|
||||
gainNode.gain.setValueAtTime(0, audioContext.currentTime);
|
||||
|
||||
const destination = audioContext.createMediaStreamDestination();
|
||||
oscillator.connect(gainNode);
|
||||
gainNode.connect(destination);
|
||||
oscillator.start();
|
||||
|
||||
audio_track = destination.stream.getAudioTracks()[0];
|
||||
audio_track.stop = () => {
|
||||
audioContext.close();
|
||||
};
|
||||
audio_track.onended = () => {
|
||||
audio_track?.stop();
|
||||
};
|
||||
return audio_track;
|
||||
}
|
||||
28
frontend/shared/VideoChat/utils.ts
Normal file
28
frontend/shared/VideoChat/utils.ts
Normal file
@@ -0,0 +1,28 @@
|
||||
export function click_outside(node: Node, cb: any): any {
|
||||
const handle_click = (event: MouseEvent): void => {
|
||||
if (
|
||||
node &&
|
||||
!node.contains(event.target as Node) &&
|
||||
!event.defaultPrevented
|
||||
) {
|
||||
cb(event);
|
||||
}
|
||||
};
|
||||
|
||||
document.addEventListener("click", handle_click, true);
|
||||
|
||||
return {
|
||||
destroy() {
|
||||
document.removeEventListener("click", handle_click, true);
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
export function insertStringAt(rawStr: string, insertString: string, index: number) {
|
||||
if (index < 0 || index > rawStr.length) {
|
||||
console.error("索引超出范围");
|
||||
return rawStr;
|
||||
}
|
||||
|
||||
return rawStr.substring(0, index) + insertString + rawStr.substring(index);
|
||||
}
|
||||
193
frontend/shared/VideoChat/webrtc_utils.ts
Normal file
193
frontend/shared/VideoChat/webrtc_utils.ts
Normal file
@@ -0,0 +1,193 @@
|
||||
export function createPeerConnection(pc, node) {
|
||||
// register some listeners to help debugging
|
||||
pc.addEventListener(
|
||||
"icegatheringstatechange",
|
||||
() => {
|
||||
console.debug(pc.iceGatheringState);
|
||||
},
|
||||
false,
|
||||
);
|
||||
|
||||
pc.addEventListener(
|
||||
"iceconnectionstatechange",
|
||||
() => {
|
||||
console.debug(pc.iceConnectionState);
|
||||
},
|
||||
false,
|
||||
);
|
||||
|
||||
pc.addEventListener(
|
||||
"signalingstatechange",
|
||||
() => {
|
||||
console.debug(pc.signalingState);
|
||||
},
|
||||
false,
|
||||
);
|
||||
|
||||
// connect audio / video from server to local
|
||||
pc.addEventListener("track", (evt) => {
|
||||
console.debug("track event listener");
|
||||
if (node && node.srcObject !== evt.streams[0]) {
|
||||
console.debug("streams", evt.streams);
|
||||
node.srcObject = evt.streams[0];
|
||||
console.debug("node.srcOject", node.srcObject);
|
||||
if (evt.track.kind === "audio") {
|
||||
node.volume = 1.0; // Ensure volume is up
|
||||
node.muted = false;
|
||||
node.autoplay = true;
|
||||
// Attempt to play (needed for some browsers)
|
||||
node.play().catch((e) => console.debug("Autoplay failed:", e));
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
return pc;
|
||||
}
|
||||
|
||||
export async function start(
|
||||
stream,
|
||||
pc: RTCPeerConnection,
|
||||
node,
|
||||
server_fn,
|
||||
webrtc_id,
|
||||
modality: "video" | "audio" = "video",
|
||||
on_change_cb: (msg: "change" | "tick") => void = () => {},
|
||||
rtp_params = {},
|
||||
additional_message_cb: (msg: object) => void = () => {},
|
||||
reject_cb: (msg: object) => void = () => {},
|
||||
) {
|
||||
pc = createPeerConnection(pc, node);
|
||||
const data_channel = pc.createDataChannel("text");
|
||||
|
||||
data_channel.onopen = () => {
|
||||
console.debug("Data channel is open");
|
||||
data_channel.send("handshake");
|
||||
data_channel.send(JSON.stringify({type:'init'}));
|
||||
};
|
||||
|
||||
data_channel.onmessage = (event) => {
|
||||
console.debug("Received message:", event.data);
|
||||
let event_json;
|
||||
try {
|
||||
event_json = JSON.parse(event.data);
|
||||
} catch (e) {
|
||||
console.debug("Error parsing JSON");
|
||||
}
|
||||
if (
|
||||
event.data === "change" ||
|
||||
event.data === "tick" ||
|
||||
event.data === "stopword" ||
|
||||
event_json?.type === "warning" ||
|
||||
event_json?.type === "error" ||
|
||||
event_json?.type === "send_input" ||
|
||||
event_json?.type === "fetch_output" ||
|
||||
event_json?.type === "stopword" ||
|
||||
event_json?.type === "end_stream"
|
||||
) {
|
||||
on_change_cb(event_json ?? event.data);
|
||||
}
|
||||
additional_message_cb(event_json ?? event.data);
|
||||
};
|
||||
|
||||
if (stream) {
|
||||
stream.getTracks().forEach(async (track) => {
|
||||
console.debug("Track stream callback", track);
|
||||
const sender = pc.addTrack(track, stream);
|
||||
const params = sender.getParameters();
|
||||
const updated_params = { ...params, ...rtp_params };
|
||||
await sender.setParameters(updated_params);
|
||||
console.debug("sender params", sender.getParameters());
|
||||
});
|
||||
} else {
|
||||
console.debug("Creating transceiver!");
|
||||
pc.addTransceiver(modality, { direction: "recvonly" });
|
||||
}
|
||||
|
||||
await negotiate(pc, server_fn, webrtc_id, reject_cb);
|
||||
return [pc, data_channel] as const;
|
||||
}
|
||||
|
||||
function make_offer(
|
||||
server_fn: any,
|
||||
body,
|
||||
reject_cb: (msg: object) => void = () => {},
|
||||
): Promise<object> {
|
||||
return new Promise((resolve, reject) => {
|
||||
server_fn(body).then((data) => {
|
||||
console.debug("data", data);
|
||||
if (data?.status === "failed") {
|
||||
reject_cb(data);
|
||||
console.debug("rejecting");
|
||||
reject("error");
|
||||
}
|
||||
resolve(data);
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
async function negotiate(
|
||||
pc: RTCPeerConnection,
|
||||
server_fn: any,
|
||||
webrtc_id: string,
|
||||
reject_cb: (msg: object) => void = () => {},
|
||||
): Promise<void> {
|
||||
pc.onicecandidate = ({ candidate }) => {
|
||||
if (candidate) {
|
||||
console.debug("Sending ICE candidate", candidate);
|
||||
server_fn({
|
||||
candidate: candidate.toJSON(),
|
||||
webrtc_id: webrtc_id,
|
||||
type: "ice-candidate",
|
||||
}).catch((err) => console.error("Error sending ICE candidate:", err));
|
||||
}
|
||||
};
|
||||
|
||||
return pc
|
||||
.createOffer()
|
||||
.then((offer) => {
|
||||
return pc.setLocalDescription(offer);
|
||||
})
|
||||
.then(() => {
|
||||
var offer = pc.localDescription;
|
||||
return make_offer(
|
||||
server_fn,
|
||||
{
|
||||
sdp: offer.sdp,
|
||||
type: offer.type,
|
||||
webrtc_id: webrtc_id,
|
||||
},
|
||||
reject_cb,
|
||||
);
|
||||
})
|
||||
.then((response) => {
|
||||
return response;
|
||||
})
|
||||
.then((answer) => {
|
||||
return pc.setRemoteDescription(answer);
|
||||
});
|
||||
}
|
||||
|
||||
export function stop(pc: RTCPeerConnection) {
|
||||
console.debug("Stopping peer connection");
|
||||
// close transceivers
|
||||
if (pc.getTransceivers) {
|
||||
pc.getTransceivers().forEach((transceiver) => {
|
||||
if (transceiver.stop) {
|
||||
transceiver.stop();
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
// close local audio / video
|
||||
if (pc.getSenders()) {
|
||||
pc.getSenders().forEach((sender) => {
|
||||
console.log("sender", sender);
|
||||
if (sender.track && sender.track.stop) sender.track.stop();
|
||||
});
|
||||
}
|
||||
|
||||
// close peer connection
|
||||
setTimeout(() => {
|
||||
pc.close();
|
||||
}, 500);
|
||||
}
|
||||
@@ -8,7 +8,7 @@ build-backend = "hatchling.build"
|
||||
|
||||
[project]
|
||||
name = "fastrtc"
|
||||
version = "0.0.19"
|
||||
version = "0.0.19.dev"
|
||||
description = "The realtime communication library for Python"
|
||||
readme = "README.md"
|
||||
license = "apache-2.0"
|
||||
@@ -33,7 +33,7 @@ dependencies = [
|
||||
"aiortc",
|
||||
"audioop-lts;python_version>='3.13'",
|
||||
"librosa",
|
||||
"numpy>=2.0.2", # because of librosa
|
||||
"numpy<=1.26.4",
|
||||
"numba>=0.60.0",
|
||||
"standard-aifc;python_version>='3.13'",
|
||||
"standard-sunau;python_version>='3.13'",
|
||||
|
||||
Reference in New Issue
Block a user