gs对话接入

本次代码评审新增并完善了gs视频聊天功能，包括前后端接口定义、状态管理及UI组件实现，并引入了新的依赖库以支持更多互动特性。
Link: https://code.alibaba-inc.com/xr-paas/gradio_webrtc/codereview/21273476
* 更新python 部分

* 合并videochat前端部分

* Merge branch 'feature/update-fastrtc-0.0.19' of http://gitlab.alibaba-inc.com/xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19

* 替换audiowave

* 导入路径修改

* 合并websocket mode逻辑

* feat: gaussian avatar chat

* 增加其他渲染的入参

* feat: ws连接和使用

* Merge branch 'feature/update-fastrtc-0.0.19' of http://gitlab.alibaba-inc.com/xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19

* 右边距离超出容器宽度，则向左移动

* 配置传递

* Merge branch 'feature/update-fastrtc-0.0.19' of gitlab.alibaba-inc.com:xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19

* 高斯包异常

* 同步webrtc_utils

* 更新webrtc_utils

* 兼容on_chat_datachannel

* 修复设备名称列表没有正常显示的问题

* copy 传递 webrtc_id

* Merge branch 'feature/update-fastrtc-0.0.19' of gitlab.alibaba-inc.com:xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19

* 保证webrtc 完成后再进行websocket连接

* feat: 音频表情数据接入

* dist 上传

* canvas 隐藏

* feat: 高斯文件下载进度透出

* Merge branch 'feature/update-fastrtc-0.0.19' of http://gitlab.alibaba-inc.com/xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19

* 修改无法获取权限问题

* Merge branch 'feature/update-fastrtc-0.0.19' of gitlab.alibaba-inc.com:xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19

* 先获取权限再获取设备

* fix: gs资源下载完成前不处理ws数据

* fix: merge

* 话术调整

* Merge branch 'feature/update-fastrtc-0.0.19' of gitlab.alibaba-inc.com:xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19

* 修复设备切换后重新对话，又切换回默认设备的问题

* Merge branch 'feature/update-fastrtc-0.0.19' of http://gitlab.alibaba-inc.com/xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19

* 更新localvideo 尺寸

* Merge branch 'feature/update-fastrtc-0.0.19' of gitlab.alibaba-inc.com:xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19

* 不能默认default

* 修改音频权限问题

* 更新打包结果

* fix: 对话按钮状态跟gs资源挂钩，删除无用代码

* fix: merge

* feat: gs渲染模块从npm包引入

* fix

* 新增对话记录

* Merge branch 'feature/update-fastrtc-0.0.19' of http://gitlab.alibaba-inc.com/xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19

* 样式修改

* 更新包

* fix: gs数字人初始化位置和静音

* 对话记录滚到底部

* 至少100%高度

* Merge branch 'feature/update-fastrtc-0.0.19' of gitlab.alibaba-inc.com:xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19

* 略微上移文本框

* 开始连接时清空对话记录

* fix: update gs render npm

* Merge branch 'feature/update-fastrtc-0.0.19' of http://gitlab.alibaba-inc.com/xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19

* 逻辑保证

* Merge branch 'feature/update-fastrtc-0.0.19' of gitlab.alibaba-inc.com:xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19

* feat: 音频初始化配置是否静音

* actionsbar在有字幕时调整位置

* Merge branch 'feature/update-fastrtc-0.0.19' of http://gitlab.alibaba-inc.com/xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19

* 样式优化

* feat: 增加readme

* fix: 资源图片

* fix: docs

* fix: update gs render sdk

* fix: gs模式下画面位置计算

* fix: update readme

* 设备判断，太窄处理

* Merge branch 'feature/update-fastrtc-0.0.19' of gitlab.alibaba-inc.com:xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19

* 是否有权限和是否有设备分开

* feat: gs 下载和加载钩子函数分离

* Merge branch 'feature/update-fastrtc-0.0.19' of http://gitlab.alibaba-inc.com/xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19

* fix: update gs render sdk

* 替换

* dist

* 上传文件

* del

backend/fastrtc/webrtc_connection_mixin.py

@@ -4,6 +4,7 @@ from __future__ import annotations
 
 import asyncio
 import inspect
+import json
 import logging
 from collections import defaultdict
 from collections.abc import Callable
@@ -30,6 +31,7 @@ from fastrtc.tracks import (
     ServerToClientAudio,
     ServerToClientVideo,
     StreamHandlerBase,
+    StreamHandlerFactory,
     StreamHandlerImpl,
     VideoCallback,
     VideoEventHandler,
@@ -246,7 +248,7 @@ class WebRTCConnectionMixin:
         self.pcs[body["webrtc_id"]] = pc
 
         if isinstance(self.event_handler, StreamHandlerBase):
-            handler = self.event_handler.copy()
+            handler = self.event_handler.copy(webrtc_id=body['webrtc_id'])
             handler.emit = webrtc_error_handler(handler.emit)  # type: ignore
             handler.receive = webrtc_error_handler(handler.receive)  # type: ignore
             handler.start_up = webrtc_error_handler(handler.start_up)  # type: ignore
@@ -255,6 +257,9 @@ class WebRTCConnectionMixin:
                 handler.video_receive = webrtc_error_handler(handler.video_receive)  # type: ignore
             if hasattr(handler, "video_emit"):
                 handler.video_emit = webrtc_error_handler(handler.video_emit)  # type: ignore
+            if hasattr(handler, "on_pc_connected"):
+                handler.on_pc_connected(body["webrtc_id"])
+
         elif isinstance(self.event_handler, VideoStreamHandler):
             self.event_handler.callable = cast(
                 VideoEventHandler, webrtc_error_handler(self.event_handler.callable)
@@ -265,6 +270,7 @@ class WebRTCConnectionMixin:
 
         self.handlers[body["webrtc_id"]] = handler
 
+
         @pc.on("iceconnectionstatechange")
         async def on_iceconnectionstatechange():
             logger.debug("ICE connection state change %s", pc.iceConnectionState)
@@ -393,6 +399,23 @@ class WebRTCConnectionMixin:
             def _(message):
                 logger.debug(f"Received message: {message}")
                 if channel.readyState == "open":
+                  def parse_json_safely(str: str):
+                    try:
+                        result = json.loads(str)
+                        return result, None
+                    except json.JSONDecodeError as e:
+                        # print(f"JSON解析错误: {e.msg}")
+                        return None, e
+                  msg_dict,error = parse_json_safely(message)
+                  if(error is None and msg_dict['type'] in ['chat','stop_chat', 'init']):
+                    msg_dict = cast(dict, json.loads(message))
+                    handler = self.handlers[body["webrtc_id"]]
+                    if inspect.iscoroutinefunction(handler.on_chat_datachannel):
+                        asyncio.create_task(
+                            handler.on_chat_datachannel(msg_dict,channel))
+                    else: 
+                        handler.on_chat_datachannel(msg_dict,channel)
+                  else:
                     channel.send(
                         create_message("log", data=f"Server received: {message}")
                     )