mirror of
https://github.com/HumanAIGC-Engineering/gradio-webrtc.git
synced 2026-02-05 18:09:23 +08:00
gs对话接入
本次代码评审新增并完善了gs视频聊天功能，包括前后端接口定义、状态管理及UI组件实现，并引入了新的依赖库以支持更多互动特性。 Link: https://code.alibaba-inc.com/xr-paas/gradio_webrtc/codereview/21273476 * 更新python 部分 * 合并videochat前端部分 * Merge branch 'feature/update-fastrtc-0.0.19' of http://gitlab.alibaba-inc.com/xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19 * 替换audiowave * 导入路径修改 * 合并websocket mode逻辑 * feat: gaussian avatar chat * 增加其他渲染的入参 * feat: ws连接和使用 * Merge branch 'feature/update-fastrtc-0.0.19' of http://gitlab.alibaba-inc.com/xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19 * 右边距离超出容器宽度，则向左移动 * 配置传递 * Merge branch 'feature/update-fastrtc-0.0.19' of gitlab.alibaba-inc.com:xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19 * 高斯包异常 * 同步webrtc_utils * 更新webrtc_utils * 兼容on_chat_datachannel * 修复设备名称列表没有正常显示的问题 * copy 传递 webrtc_id * Merge branch 'feature/update-fastrtc-0.0.19' of gitlab.alibaba-inc.com:xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19 * 保证webrtc 完成后再进行websocket连接 * feat: 音频表情数据接入 * dist 上传 * canvas 隐藏 * feat: 高斯文件下载进度透出 * Merge branch 'feature/update-fastrtc-0.0.19' of http://gitlab.alibaba-inc.com/xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19 * 修改无法获取权限问题 * Merge branch 'feature/update-fastrtc-0.0.19' of gitlab.alibaba-inc.com:xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19 * 先获取权限再获取设备 * fix: gs资源下载完成前不处理ws数据 * fix: merge * 话术调整 * Merge branch 'feature/update-fastrtc-0.0.19' of gitlab.alibaba-inc.com:xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19 * 修复设备切换后重新对话，又切换回默认设备的问题 * Merge branch 'feature/update-fastrtc-0.0.19' of http://gitlab.alibaba-inc.com/xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19 * 更新localvideo 尺寸 * Merge branch 'feature/update-fastrtc-0.0.19' of gitlab.alibaba-inc.com:xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19 * 不能默认default * 修改音频权限问题 * 更新打包结果 * fix: 对话按钮状态跟gs资源挂钩，删除无用代码 * fix: merge * feat: gs渲染模块从npm包引入 * fix * 新增对话记录 * Merge branch 'feature/update-fastrtc-0.0.19' of http://gitlab.alibaba-inc.com/xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19 * 样式修改 * 更新包 * fix: gs数字人初始化位置和静音 * 对话记录滚到底部 * 至少100%高度 * Merge branch 'feature/update-fastrtc-0.0.19' of gitlab.alibaba-inc.com:xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19 * 略微上移文本框 * 开始连接时清空对话记录 * fix: update gs render npm * Merge branch 'feature/update-fastrtc-0.0.19' of http://gitlab.alibaba-inc.com/xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19 * 逻辑保证 * Merge branch 'feature/update-fastrtc-0.0.19' of gitlab.alibaba-inc.com:xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19 * feat: 音频初始化配置是否静音 * actionsbar在有字幕时调整位置 * Merge branch 'feature/update-fastrtc-0.0.19' of http://gitlab.alibaba-inc.com/xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19 * 样式优化 * feat: 增加readme * fix: 资源图片 * fix: docs * fix: update gs render sdk * fix: gs模式下画面位置计算 * fix: update readme * 设备判断，太窄处理 * Merge branch 'feature/update-fastrtc-0.0.19' of gitlab.alibaba-inc.com:xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19 * 是否有权限和是否有设备分开 * feat: gs 下载和加载钩子函数分离 * Merge branch 'feature/update-fastrtc-0.0.19' of http://gitlab.alibaba-inc.com/xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19 * fix: update gs render sdk * 替换 * dist * 上传文件 * del
This commit is contained in:
neil.xh
198
demo/video_chat/app.py
Normal file
198
demo/video_chat/app.py
Normal file
@@ -0,0 +1,198 @@
|
||||
import asyncio
|
||||
import base64
|
||||
from io import BytesIO
|
||||
import json
|
||||
import math
|
||||
import queue
|
||||
import time
|
||||
from typing import TypedDict
|
||||
import uuid
|
||||
import threading
|
||||
|
||||
# from fastrtc.utils import Message
|
||||
from fastapi import FastAPI, WebSocket, WebSocketDisconnect
|
||||
import gradio as gr
|
||||
import numpy as np
|
||||
from fastrtc import (
|
||||
AsyncAudioVideoStreamHandler,
|
||||
WebRTC,
|
||||
VideoEmitType,
|
||||
AudioEmitType,
|
||||
)
|
||||
from PIL import Image
|
||||
import uvicorn
|
||||
|
||||
|
||||
class Message(TypedDict):
|
||||
type: str
|
||||
data: any
|
||||
|
||||
def encode_audio(data: np.ndarray) -> dict:
|
||||
"""Encode Audio data to send to the server"""
|
||||
return {"mime_type": "audio/pcm", "data": base64.b64encode(data.tobytes()).decode("UTF-8")}
|
||||
|
||||
|
||||
def encode_image(data: np.ndarray) -> dict:
|
||||
with BytesIO() as output_bytes:
|
||||
pil_image = Image.fromarray(data)
|
||||
pil_image.save(output_bytes, "JPEG")
|
||||
bytes_data = output_bytes.getvalue()
|
||||
base64_str = str(base64.b64encode(bytes_data), "utf-8")
|
||||
return {"mime_type": "image/jpeg", "data": base64_str}
|
||||
|
||||
frame_queue = queue.Queue(maxsize=100)
|
||||
|
||||
class VideoChatHandler(AsyncAudioVideoStreamHandler):
|
||||
def __init__(
|
||||
self, expected_layout="mono", output_sample_rate=24000, output_frame_size=480
|
||||
) -> None:
|
||||
super().__init__(
|
||||
expected_layout,
|
||||
output_sample_rate,
|
||||
output_frame_size,
|
||||
input_sample_rate=24000,
|
||||
)
|
||||
self.audio_queue = asyncio.Queue()
|
||||
self.video_queue = frame_queue
|
||||
self.quit = asyncio.Event()
|
||||
self.session = None
|
||||
self.last_frame_time = 0
|
||||
|
||||
def copy(self, **kwargs) -> "VideoChatHandler":
|
||||
return VideoChatHandler(
|
||||
expected_layout=self.expected_layout,
|
||||
output_sample_rate=self.output_sample_rate,
|
||||
output_frame_size=self.output_frame_size,
|
||||
)
|
||||
|
||||
def on_pc_connected(self, webrtc_id):
|
||||
print(webrtc_id)
|
||||
|
||||
chat_id = ''
|
||||
async def on_chat_datachannel(self,message: Message,channel):
|
||||
# 返回
|
||||
# {"type":"chat",id:"标识属于同一段话", "message":"Hello, world!"}
|
||||
# {"type":"avatar_end"} 表示本次对话结束
|
||||
if message['type'] == 'stop_chat':
|
||||
self.chat_id = ''
|
||||
channel.send(json.dumps({'type':'avatar_end'}))
|
||||
else:
|
||||
id = uuid.uuid4().hex
|
||||
self.chat_id = id
|
||||
data = message["data"]
|
||||
halfLen = math.floor(data.__len__()/2)
|
||||
channel.send(json.dumps({"type":"chat","id":id,"message":data[:halfLen]}))
|
||||
await asyncio.sleep(5)
|
||||
if self.chat_id == id:
|
||||
channel.send(json.dumps({"type":"chat","id":id,"message":data[halfLen:]}))
|
||||
channel.send(json.dumps({'type':'avatar_end'}))
|
||||
|
||||
async def video_receive(self, frame: np.ndarray):
|
||||
# if self.session:
|
||||
# # send image every 1 second
|
||||
# if time.time() - self.last_frame_time > 1:
|
||||
# self.last_frame_time = time.time()
|
||||
# await self.session.send(encode_image(frame))
|
||||
# if self.latest_args[2] is not None:
|
||||
# await self.session.send(encode_image(self.latest_args[2]))
|
||||
# print(frame.shape)
|
||||
newFrame = np.array(frame)
|
||||
newFrame[0:, :, 0] = 255 - newFrame[0:, :, 0]
|
||||
# self.video_queue.put_nowait(newFrame)
|
||||
|
||||
async def video_emit(self) -> VideoEmitType:
|
||||
# print('123123',frame_queue.qsize())
|
||||
return frame_queue.get()
|
||||
|
||||
async def receive(self, frame: tuple[int, np.ndarray]) -> None:
|
||||
frame_size, array = frame
|
||||
self.audio_queue.put_nowait(array)
|
||||
|
||||
async def emit(self) -> AudioEmitType:
|
||||
if not self.args_set.is_set():
|
||||
await self.wait_for_args()
|
||||
array = await self.audio_queue.get()
|
||||
return (self.output_sample_rate, array)
|
||||
|
||||
def shutdown(self) -> None:
|
||||
self.quit.set()
|
||||
self.connection = None
|
||||
self.args_set.clear()
|
||||
self.quit.clear()
|
||||
|
||||
|
||||
css = """
|
||||
footer {
|
||||
display: none !important;
|
||||
}
|
||||
"""
|
||||
|
||||
with gr.Blocks(css=css) as demo:
|
||||
webrtc = WebRTC(
|
||||
label="Video Chat",
|
||||
modality="audio-video",
|
||||
mode="send-receive",
|
||||
video_chat=True,
|
||||
avatar_type="",
|
||||
avatar_assets_path="",
|
||||
avatar_ws_route='/ws',
|
||||
elem_id="video-source",
|
||||
track_constraints={
|
||||
"video": {
|
||||
"facingMode": "user",
|
||||
"width": {"ideal": 500},
|
||||
"height": {"ideal": 500},
|
||||
"frameRate": {"ideal": 30},
|
||||
},
|
||||
"audio": {
|
||||
"echoCancellation": True,
|
||||
"noiseSuppression": {"exact": True},
|
||||
"autoGainControl": {"exact": False},
|
||||
"sampleRate": {"ideal": 24000},
|
||||
"sampleSize": {"ideal": 16},
|
||||
"channelCount": {"exact": 1},
|
||||
},
|
||||
}
|
||||
)
|
||||
handler = VideoChatHandler()
|
||||
webrtc.stream(
|
||||
handler,
|
||||
inputs=[webrtc],
|
||||
outputs=[webrtc],
|
||||
time_limit=1500,
|
||||
concurrency_limit=2,
|
||||
)
|
||||
# 线程函数：随机生成 numpy 帧
|
||||
def generate_frames(width=480, height=960, channels=3):
|
||||
while True:
|
||||
try:
|
||||
# 随机生成一个 RGB 图像帧
|
||||
frame = np.random.randint(188, 256, (height, width, channels), dtype=np.uint8)
|
||||
|
||||
# 将帧放入队列
|
||||
frame_queue.put(frame)
|
||||
# print("生成一帧数据，形状:", frame.shape, frame_queue.qsize())
|
||||
|
||||
# 模拟实时性：避免过度消耗 CPU
|
||||
time.sleep(0.03) # 每秒约生成 30 帧
|
||||
except Exception as e:
|
||||
print(f"生成帧时出错: {e}")
|
||||
break
|
||||
thread = threading.Thread(target=generate_frames, daemon=True)
|
||||
thread.start()
|
||||
|
||||
@demo.app.websocket("/ws")
|
||||
async def on_websocket(websocket: WebSocket):
|
||||
await websocket.accept()
|
||||
while True:
|
||||
try:
|
||||
data = await websocket.receive_text()
|
||||
await websocket.send_text(f"Message text was: {data}")
|
||||
except WebSocketDisconnect:
|
||||
break
|
||||
|
||||
if __name__ == "__main__":
|
||||
demo.launch()
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user