gs对话接入

本次代码评审新增并完善了gs视频聊天功能,包括前后端接口定义、状态管理及UI组件实现,并引入了新的依赖库以支持更多互动特性。
Link: https://code.alibaba-inc.com/xr-paas/gradio_webrtc/codereview/21273476
* 更新python 部分

* 合并videochat前端部分

* Merge branch 'feature/update-fastrtc-0.0.19' of http://gitlab.alibaba-inc.com/xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19

* 替换audiowave

* 导入路径修改

* 合并websocket mode逻辑

* feat: gaussian avatar chat

* 增加其他渲染的入参

* feat: ws连接和使用

* Merge branch 'feature/update-fastrtc-0.0.19' of http://gitlab.alibaba-inc.com/xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19

* 右边距离超出容器宽度,则向左移动

* 配置传递

* Merge branch 'feature/update-fastrtc-0.0.19' of gitlab.alibaba-inc.com:xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19

* 高斯包异常

* 同步webrtc_utils

* 更新webrtc_utils

* 兼容on_chat_datachannel

* 修复设备名称列表没有正常显示的问题

* copy 传递 webrtc_id

* Merge branch 'feature/update-fastrtc-0.0.19' of gitlab.alibaba-inc.com:xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19

* 保证webrtc 完成后再进行websocket连接

* feat: 音频表情数据接入

* dist 上传

* canvas 隐藏

* feat: 高斯文件下载进度透出

* Merge branch 'feature/update-fastrtc-0.0.19' of http://gitlab.alibaba-inc.com/xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19

* 修改无法获取权限问题

* Merge branch 'feature/update-fastrtc-0.0.19' of gitlab.alibaba-inc.com:xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19

* 先获取权限再获取设备

* fix: gs资源下载完成前不处理ws数据

* fix: merge

* 话术调整

* Merge branch 'feature/update-fastrtc-0.0.19' of gitlab.alibaba-inc.com:xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19

* 修复设备切换后重新对话,又切换回默认设备的问题

* Merge branch 'feature/update-fastrtc-0.0.19' of http://gitlab.alibaba-inc.com/xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19

* 更新localvideo 尺寸

* Merge branch 'feature/update-fastrtc-0.0.19' of gitlab.alibaba-inc.com:xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19

* 不能默认default

* 修改音频权限问题

* 更新打包结果

* fix: 对话按钮状态跟gs资源挂钩,删除无用代码

* fix: merge

* feat: gs渲染模块从npm包引入

* fix

* 新增对话记录

* Merge branch 'feature/update-fastrtc-0.0.19' of http://gitlab.alibaba-inc.com/xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19

* 样式修改

* 更新包

* fix: gs数字人初始化位置和静音

* 对话记录滚到底部

* 至少100%高度

* Merge branch 'feature/update-fastrtc-0.0.19' of gitlab.alibaba-inc.com:xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19

* 略微上移文本框

* 开始连接时清空对话记录

* fix: update gs render npm

* Merge branch 'feature/update-fastrtc-0.0.19' of http://gitlab.alibaba-inc.com/xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19

* 逻辑保证

* Merge branch 'feature/update-fastrtc-0.0.19' of gitlab.alibaba-inc.com:xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19

* feat: 音频初始化配置是否静音

* actionsbar在有字幕时调整位置

* Merge branch 'feature/update-fastrtc-0.0.19' of http://gitlab.alibaba-inc.com/xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19

* 样式优化

* feat: 增加readme

* fix: 资源图片

* fix: docs

* fix: update gs render sdk

* fix: gs模式下画面位置计算

* fix: update readme

* 设备判断,太窄处理

* Merge branch 'feature/update-fastrtc-0.0.19' of gitlab.alibaba-inc.com:xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19

* 是否有权限和是否有设备分开

* feat: gs 下载和加载钩子函数分离

* Merge branch 'feature/update-fastrtc-0.0.19' of http://gitlab.alibaba-inc.com/xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19

* fix: update gs render sdk

* 替换

* dist

* 上传文件

* del
This commit is contained in:
neil.xh
2025-04-16 19:09:04 +08:00
parent 06885d06c4
commit f476f9cf29
54 changed files with 4980 additions and 23257 deletions

View File

@@ -0,0 +1,225 @@
<script lang="ts">
import { onDestroy } from "svelte";
import type { ComponentType } from "svelte";
import PulsingIcon from "../../PulsingIcon.svelte";
export let numBars = 16;
export let stream_state: "open" | "closed" | "waiting" = "closed";
export let audio_source_callback: () => MediaStream;
export let icon: string | undefined | ComponentType = undefined;
export let icon_button_color: string = "var(--color-accent)";
export let pulse_color: string = "var(--color-accent)";
export let wave_color: string = "var(--color-accent)";
let audioContext: AudioContext;
let analyser: AnalyserNode;
let dataArray: Uint8Array;
let animationId: number;
export let pulseScale = 1;
$: containerWidth = icon
? "128px"
: `calc((var(--boxSize) + var(--gutter)) * ${numBars} + 80px)`;
$: if (stream_state === "open") setupAudioContext();
onDestroy(() => {
if (animationId) {
cancelAnimationFrame(animationId);
}
if (audioContext) {
audioContext.close();
}
});
function setupAudioContext() {
// @ts-ignore
audioContext = new (window.AudioContext || window.webkitAudioContext)();
analyser = audioContext.createAnalyser();
const streamSource = audio_source_callback()
if(!streamSource)return
const source = audioContext.createMediaStreamSource(
streamSource,
);
source.connect(analyser);
analyser.fftSize = 64;
analyser.smoothingTimeConstant = 0.8;
dataArray = new Uint8Array(analyser.frequencyBinCount);
updateVisualization();
}
function updateVisualization() {
analyser.getByteFrequencyData(dataArray);
// Update bars
const bars = document.querySelectorAll('.gradio-webrtc-waveContainer .gradio-webrtc-box');
for (let i = 0; i < bars.length; i++) {
const barHeight = (dataArray[transformIndex(i)] / 255);
bars[i].style.transform = `scaleY(${Math.max(0.1, barHeight)})`;
bars[i].style.background = wave_color;
bars[i].style.opacity = 0.5;
}
animationId = requestAnimationFrame(updateVisualization);
}
// 声波高度从两侧向中间收拢
function transformIndex(index: number): number {
const mapping = [0, 2, 4, 6, 8, 10, 12, 14, 15, 13, 11, 9, 7, 5, 3, 1];
if (index < 0 || index >= mapping.length) {
throw new Error("Index must be between 0 and 15");
}
return mapping[index];
}
</script>
<div class="gradio-webrtc-waveContainer">
{#if icon && !pending}
<div class="gradio-webrtc-icon-container">
<div
class="gradio-webrtc-icon"
style:transform={`scale(${pulseScale})`}
style:background={icon_button_color}
>
<PulsingIcon
{stream_state}
{pulse_color}
{icon}
{icon_button_color}
{icon_radius}
{audio_source_callback}
/>
</div>
</div>
{:else}
<div class="gradio-webrtc-boxContainer" style:width={containerWidth}>
{#each Array(numBars/2) as _}
<div class="gradio-webrtc-box"></div>
{/each}
<div class="split-container"></div>
{#each Array(numBars/2) as _}
<div class="gradio-webrtc-box"></div>
{/each}
</div>
{/if}
</div>
<style>
.gradio-webrtc-waveContainer {
position: relative;
display: flex;
min-height: 100px;
max-height: 128px;
justify-content: center;
align-items: center;
}
.gradio-webrtc-boxContainer {
display: flex;
justify-content: space-between;
height: 64px;
--boxSize: 4px;
--gutter: 4px;
}
.split-container {
width: 80px;
}
.gradio-webrtc-box {
height: 100%;
width: var(--boxSize);
background: var(--color-accent);
border-radius: 8px;
transition: transform 0.05s ease;
}
.gradio-webrtc-icon-container {
position: relative;
width: 128px;
height: 128px;
display: flex;
justify-content: center;
align-items: center;
}
.gradio-webrtc-icon {
position: relative;
width: 48px;
height: 48px;
border-radius: 50%;
transition: transform 0.1s ease;
display: flex;
justify-content: center;
align-items: center;
z-index: 2;
}
.icon-image {
width: 32px;
height: 32px;
object-fit: contain;
filter: brightness(0) invert(1);
}
.pulse-ring {
position: absolute;
top: 50%;
left: 50%;
transform: translate(-50%, -50%);
width: 48px;
height: 48px;
border-radius: 50%;
animation: pulse 2s cubic-bezier(0.4, 0, 0.6, 1) infinite;
opacity: 0.5;
}
@keyframes pulse {
0% {
transform: translate(-50%, -50%) scale(1);
opacity: 0.5;
}
100% {
transform: translate(-50%, -50%) scale(var(--max-scale, 3));
opacity: 0;
}
}
.dots {
display: flex;
gap: 8px;
align-items: center;
height: 64px;
}
.dot {
width: 12px;
height: 12px;
border-radius: 50%;
opacity: 0.5;
animation: pulse 1.5s infinite;
}
.dot:nth-child(2) {
animation-delay: 0.2s;
}
.dot:nth-child(3) {
animation-delay: 0.4s;
}
@keyframes pulse {
0%,
100% {
opacity: 0.4;
transform: scale(1);
}
50% {
opacity: 1;
transform: scale(1.1);
}
}
</style>

View File

@@ -0,0 +1,105 @@
<script lang="ts">
import { Spinner } from "@gradio/icons";
import AudioWave from "./AudioWave.svelte";
import { createEventDispatcher } from 'svelte';
const dispatch = createEventDispatcher();
export let stream_state;
export let onStartChat
export let audio_source_callback
export let wave_color
export let assetLoaded = true
</script>
<div class="player-controls">
<!-- svelte-ignore a11y-click-events-have-key-events -->
<!-- svelte-ignore a11y-no-static-element-interactions -->
<div
class="chat-btn"
class:start-chat={stream_state === "closed"}
class:stop-chat={stream_state === "open" && assetLoaded === true}
on:click={onStartChat}
>
{#if stream_state === "closed"}
<span>点击开始对话</span>
{:else if stream_state === "waiting" || assetLoaded === false}
<div class="waiting-icon-text">
<div class="icon" title="spinner">
<Spinner />
</div>
<span>等待中</span>
</div>
{:else}
<div class="stop-chat-inner"></div>
{/if}
</div>
{#if stream_state === "open" && assetLoaded === true}
<div class="input-audio-wave">
<AudioWave {audio_source_callback} {stream_state} {wave_color} />
</div>
{/if}
</div>
<style lang="less">
.player-controls {
height: 15%;
position: relative;
display: flex;
justify-content: center;
align-items: center;
min-height: 84px;
.chat-btn {
height: 64px;
width: 296px;
display: flex;
justify-content: center;
align-items: center;
border-radius: 999px;
opacity: 1;
background: linear-gradient(180deg, #7873f6 0%, #524de1 100%);
transition: all 0.3s;
z-index: 2;
cursor: pointer;
}
.start-chat {
font-size: 16px;
font-weight: 500;
text-align: center;
color: #ffffff;
}
.waiting-icon-text {
width: 80px;
align-items: center;
font-size: 16px;
font-weight: 500;
color: #ffffff;
margin: 0 var(--spacing-sm);
display: flex;
justify-content: space-evenly;
gap: var(--size-1);
.icon {
width: 25px;
height: 25px;
fill: #ffffff;
stroke: #ffffff;
color: #ffffff;
}
}
.stop-chat {
width: 64px;
.stop-chat-inner {
width: 25px;
height: 25px;
border-radius: 6.25px;
background: #fafafa;
}
}
.input-audio-wave {
position: absolute;
}
}
</style>

View File

@@ -0,0 +1,181 @@
<script lang="ts">
import { IconFont, Send, Stop } from "../icons";
import { insertStringAt } from "../utils";
export let replying;
export let onSend;
export let onStop;
export let onInterrupt;
let inputHeight = 24;
let rowsDivRef: HTMLDivElement;
let chatInputRef: HTMLTextAreaElement;
let inputValue = "";
function on_chat_input_keydown(event: KeyboardEvent) {
if (event.key === "Enter") {
if (event.altKey) {
chatInputRef.value = insertStringAt(
chatInputRef.value,
"\n",
chatInputRef.selectionStart,
);
chatInputRef.dispatchEvent(new InputEvent("input"));
} else {
event.preventDefault();
on_send();
}
}
}
async function on_send() {
await onSend(chatInputRef.value);
chatInputRef.value = "";
}
function on_chat_input(event: InputEvent) {
if (rowsDivRef) {
rowsDivRef.textContent = (event.target as any).value.replace(
/\n$/,
"\n\n",
);
inputHeight = rowsDivRef.offsetHeight;
}
}
</script>
<div class="chat-input-container">
<!-- svelte-ignore a11y-click-events-have-key-events -->
<!-- svelte-ignore a11y-no-static-element-interactions -->
<div class="stop-chat-btn" on:click={onStop}></div>
<div class="chat-input-inner">
<div class="chat-input-wrapper">
<textarea
class="chat-input"
bind:this={chatInputRef}
on:keydown={on_chat_input_keydown}
on:input={on_chat_input}
style={`height:${inputHeight}px`}
/>
<div class="rowsDiv" bind:this={rowsDivRef}>{inputValue}</div>
</div>
{#if replying}
<button class="interrupt-btn" on:click={onInterrupt}></button>
{:else}
<button class="send-btn" on:click={on_send}>
<IconFont icon={Send} color={"#fff"} ></IconFont>
</button>
{/if}
<div class="chat-tip">Texts are ignored during responding.</div>
</div>
</div>
<style lang="less">
.chat-input-container {
height: 15%;
position: relative;
display: flex;
justify-content: center;
align-items: center;
min-height: 84px;
// padding: 0 12px;
.chat-input-inner {
position: relative;
padding: 0 12px;
background-color: #fff;
height: 64px;
flex: 1;
display: flex;
align-items: center;
border: 1px solid #e8eaf2;
border-radius: 12px;
border-radius: 20px;
box-shadow:
0 12px 24px -16px rgba(54, 54, 73, 0.04),
0 12px 40px 0 rgba(51, 51, 71, 0.08),
0 0 1px 0 rgba(44, 44, 54, 0.02);
.chat-tip {
position: absolute;
top: 100%;
color: #cecece;
}
.chat-input::placeholder {
font-size: 12px;
}
.chat-input-wrapper {
flex: 1;
position: relative;
display: flex;
align-items: center;
.chat-input {
width: 100%;
border: none;
outline: none;
color: #26244c;
font-size: 16px;
font-weight: 400;
resize: none;
padding: 0;
margin: 8px 0;
line-height: 24px;
max-height: 48px;
min-height: 24px;
}
.rowsDiv {
position: absolute;
left: 0;
right: 0;
z-index: -1;
visibility: hidden;
font-size: 16px;
font-weight: 400;
line-height: 24px;
white-space: pre-wrap;
word-wrap: break-word;
}
}
.send-btn,.interrupt-btn {
flex: 0 0 auto;
background: #615ced;
border-radius: 20px;
height: 28px;
width: 28px;
display: flex;
align-items: center;
justify-content: center;
margin-left: 16px;
cursor: pointer;
}
.interrupt-btn{
&::after {
content: " ";
width: 12px;
height: 12px;
border-radius: 2px;
background: #fafafa;
}
}
}
.stop-chat-btn {
cursor: pointer;
margin-right: 12px;
height: 28px;
width: 28px;
display: flex;
justify-content: center;
align-items: center;
border-radius: 999px;
opacity: 1;
background: linear-gradient(180deg, #7873f6 0%, #524de1 100%);
&::after {
content: " ";
width: 12px;
height: 12px;
border-radius: 2px;
background: #fafafa;
}
}
}
</style>

View File

@@ -0,0 +1,34 @@
<script lang="ts">
export let message;
export let style = "";
export let role = "";
$: classnames = `answer-message-container ${role}`
</script>
<div class={classnames} {style}>
<div class="answer-message-text">
{message}
</div>
</div>
<style lang="less">
.answer-message-container {
padding: 6px 12px;
background: rgba(255, 255, 255, 0.8);
border-radius: 12px;
color: #26244c;
&.human {
background: #dddddd99;
// margin-left: 20px;
margin-right: 0;
}
&.avatar {
background: #9189fa;
color: #ffffff;
// margin-right: 20px;
}
}
</style>

View File

@@ -0,0 +1,65 @@
<script lang="ts">
import { tick } from "svelte";
import ChatMessage from "./ChatMessage.svelte";
export let chatRecords;
let containerRef: HTMLElement
$: if(chatRecords){
tick().then(() => {
scrollToBottom()
})
}
function scrollToBottom() {
// console.log("🚀 ~ scrollToBottom ~ scrollToBottom:")
if(containerRef){
containerRef.scrollTop = containerRef.scrollHeight;
}
}
export const expose = { scrollToBottom };
</script>
<div class="chat-records" bind:this={containerRef}>
<div class="chat-records-inner">
{#each chatRecords as item, i (item.id)}
<div class={`chat-message ${item.role}`}>
<ChatMessage message={item.message} role={item.role}></ChatMessage>
</div>
{/each}
</div>
</div>
<style lang="less">
.chat-records{
width: 100%;
height: 100%;
overflow-y: auto;
&::-webkit-scrollbar {
display: none;
}
}
.chat-records-inner {
display: flex;
flex-direction: column;
align-items: flex-end;
justify-content: end;
width: 100%;
// height: 100%;
height: auto;
min-height: 100%;
.chat-message {
margin-bottom: 12px;
max-width: 80%;
&.human {
align-self: flex-end;
}
&.avatar {
align-self: flex-start;
}
&:last-child {
margin-bottom: 0;
}
}
}
</style>