mirror of
https://github.com/HumanAIGC-Engineering/gradio-webrtc.git
synced 2026-02-05 18:09:23 +08:00
gs对话接入
本次代码评审新增并完善了gs视频聊天功能，包括前后端接口定义、状态管理及UI组件实现，并引入了新的依赖库以支持更多互动特性。 Link: https://code.alibaba-inc.com/xr-paas/gradio_webrtc/codereview/21273476 * 更新python 部分 * 合并videochat前端部分 * Merge branch 'feature/update-fastrtc-0.0.19' of http://gitlab.alibaba-inc.com/xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19 * 替换audiowave * 导入路径修改 * 合并websocket mode逻辑 * feat: gaussian avatar chat * 增加其他渲染的入参 * feat: ws连接和使用 * Merge branch 'feature/update-fastrtc-0.0.19' of http://gitlab.alibaba-inc.com/xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19 * 右边距离超出容器宽度，则向左移动 * 配置传递 * Merge branch 'feature/update-fastrtc-0.0.19' of gitlab.alibaba-inc.com:xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19 * 高斯包异常 * 同步webrtc_utils * 更新webrtc_utils * 兼容on_chat_datachannel * 修复设备名称列表没有正常显示的问题 * copy 传递 webrtc_id * Merge branch 'feature/update-fastrtc-0.0.19' of gitlab.alibaba-inc.com:xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19 * 保证webrtc 完成后再进行websocket连接 * feat: 音频表情数据接入 * dist 上传 * canvas 隐藏 * feat: 高斯文件下载进度透出 * Merge branch 'feature/update-fastrtc-0.0.19' of http://gitlab.alibaba-inc.com/xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19 * 修改无法获取权限问题 * Merge branch 'feature/update-fastrtc-0.0.19' of gitlab.alibaba-inc.com:xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19 * 先获取权限再获取设备 * fix: gs资源下载完成前不处理ws数据 * fix: merge * 话术调整 * Merge branch 'feature/update-fastrtc-0.0.19' of gitlab.alibaba-inc.com:xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19 * 修复设备切换后重新对话，又切换回默认设备的问题 * Merge branch 'feature/update-fastrtc-0.0.19' of http://gitlab.alibaba-inc.com/xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19 * 更新localvideo 尺寸 * Merge branch 'feature/update-fastrtc-0.0.19' of gitlab.alibaba-inc.com:xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19 * 不能默认default * 修改音频权限问题 * 更新打包结果 * fix: 对话按钮状态跟gs资源挂钩，删除无用代码 * fix: merge * feat: gs渲染模块从npm包引入 * fix * 新增对话记录 * Merge branch 'feature/update-fastrtc-0.0.19' of http://gitlab.alibaba-inc.com/xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19 * 样式修改 * 更新包 * fix: gs数字人初始化位置和静音 * 对话记录滚到底部 * 至少100%高度 * Merge branch 'feature/update-fastrtc-0.0.19' of gitlab.alibaba-inc.com:xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19 * 略微上移文本框 * 开始连接时清空对话记录 * fix: update gs render npm * Merge branch 'feature/update-fastrtc-0.0.19' of http://gitlab.alibaba-inc.com/xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19 * 逻辑保证 * Merge branch 'feature/update-fastrtc-0.0.19' of gitlab.alibaba-inc.com:xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19 * feat: 音频初始化配置是否静音 * actionsbar在有字幕时调整位置 * Merge branch 'feature/update-fastrtc-0.0.19' of http://gitlab.alibaba-inc.com/xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19 * 样式优化 * feat: 增加readme * fix: 资源图片 * fix: docs * fix: update gs render sdk * fix: gs模式下画面位置计算 * fix: update readme * 设备判断，太窄处理 * Merge branch 'feature/update-fastrtc-0.0.19' of gitlab.alibaba-inc.com:xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19 * 是否有权限和是否有设备分开 * feat: gs 下载和加载钩子函数分离 * Merge branch 'feature/update-fastrtc-0.0.19' of http://gitlab.alibaba-inc.com/xr-paas/gradio_webrtc into feature/update-fastrtc-0.0.19 * fix: update gs render sdk * 替换 * dist * 上传文件 * del
This commit is contained in:
neil.xh
225
frontend/shared/VideoChat/components/AudioWave.svelte
Normal file
225
frontend/shared/VideoChat/components/AudioWave.svelte
Normal file
@@ -0,0 +1,225 @@
|
||||
<script lang="ts">
|
||||
import { onDestroy } from "svelte";
|
||||
import type { ComponentType } from "svelte";
|
||||
|
||||
import PulsingIcon from "../../PulsingIcon.svelte";
|
||||
|
||||
export let numBars = 16;
|
||||
export let stream_state: "open" | "closed" | "waiting" = "closed";
|
||||
export let audio_source_callback: () => MediaStream;
|
||||
export let icon: string | undefined | ComponentType = undefined;
|
||||
export let icon_button_color: string = "var(--color-accent)";
|
||||
export let pulse_color: string = "var(--color-accent)";
|
||||
export let wave_color: string = "var(--color-accent)";
|
||||
|
||||
let audioContext: AudioContext;
|
||||
let analyser: AnalyserNode;
|
||||
let dataArray: Uint8Array;
|
||||
let animationId: number;
|
||||
export let pulseScale = 1;
|
||||
|
||||
$: containerWidth = icon
|
||||
? "128px"
|
||||
: `calc((var(--boxSize) + var(--gutter)) * ${numBars} + 80px)`;
|
||||
|
||||
$: if (stream_state === "open") setupAudioContext();
|
||||
|
||||
onDestroy(() => {
|
||||
if (animationId) {
|
||||
cancelAnimationFrame(animationId);
|
||||
}
|
||||
if (audioContext) {
|
||||
audioContext.close();
|
||||
}
|
||||
});
|
||||
|
||||
function setupAudioContext() {
|
||||
// @ts-ignore
|
||||
audioContext = new (window.AudioContext || window.webkitAudioContext)();
|
||||
analyser = audioContext.createAnalyser();
|
||||
const streamSource = audio_source_callback()
|
||||
if(!streamSource)return
|
||||
const source = audioContext.createMediaStreamSource(
|
||||
streamSource,
|
||||
);
|
||||
|
||||
source.connect(analyser);
|
||||
|
||||
analyser.fftSize = 64;
|
||||
analyser.smoothingTimeConstant = 0.8;
|
||||
dataArray = new Uint8Array(analyser.frequencyBinCount);
|
||||
|
||||
updateVisualization();
|
||||
}
|
||||
|
||||
function updateVisualization() {
|
||||
analyser.getByteFrequencyData(dataArray);
|
||||
|
||||
// Update bars
|
||||
const bars = document.querySelectorAll('.gradio-webrtc-waveContainer .gradio-webrtc-box');
|
||||
for (let i = 0; i < bars.length; i++) {
|
||||
const barHeight = (dataArray[transformIndex(i)] / 255);
|
||||
bars[i].style.transform = `scaleY(${Math.max(0.1, barHeight)})`;
|
||||
bars[i].style.background = wave_color;
|
||||
bars[i].style.opacity = 0.5;
|
||||
}
|
||||
|
||||
animationId = requestAnimationFrame(updateVisualization);
|
||||
}
|
||||
|
||||
// 声波高度从两侧向中间收拢
|
||||
function transformIndex(index: number): number {
|
||||
const mapping = [0, 2, 4, 6, 8, 10, 12, 14, 15, 13, 11, 9, 7, 5, 3, 1];
|
||||
if (index < 0 || index >= mapping.length) {
|
||||
throw new Error("Index must be between 0 and 15");
|
||||
}
|
||||
return mapping[index];
|
||||
}
|
||||
</script>
|
||||
|
||||
<div class="gradio-webrtc-waveContainer">
|
||||
{#if icon && !pending}
|
||||
<div class="gradio-webrtc-icon-container">
|
||||
<div
|
||||
class="gradio-webrtc-icon"
|
||||
style:transform={`scale(${pulseScale})`}
|
||||
style:background={icon_button_color}
|
||||
>
|
||||
<PulsingIcon
|
||||
{stream_state}
|
||||
{pulse_color}
|
||||
{icon}
|
||||
{icon_button_color}
|
||||
{icon_radius}
|
||||
{audio_source_callback}
|
||||
/>
|
||||
</div>
|
||||
</div>
|
||||
{:else}
|
||||
<div class="gradio-webrtc-boxContainer" style:width={containerWidth}>
|
||||
{#each Array(numBars/2) as _}
|
||||
<div class="gradio-webrtc-box"></div>
|
||||
{/each}
|
||||
<div class="split-container"></div>
|
||||
{#each Array(numBars/2) as _}
|
||||
<div class="gradio-webrtc-box"></div>
|
||||
{/each}
|
||||
</div>
|
||||
{/if}
|
||||
</div>
|
||||
|
||||
<style>
|
||||
.gradio-webrtc-waveContainer {
|
||||
position: relative;
|
||||
display: flex;
|
||||
min-height: 100px;
|
||||
max-height: 128px;
|
||||
justify-content: center;
|
||||
align-items: center;
|
||||
}
|
||||
|
||||
.gradio-webrtc-boxContainer {
|
||||
display: flex;
|
||||
justify-content: space-between;
|
||||
height: 64px;
|
||||
--boxSize: 4px;
|
||||
--gutter: 4px;
|
||||
|
||||
}
|
||||
.split-container {
|
||||
width: 80px;
|
||||
}
|
||||
.gradio-webrtc-box {
|
||||
height: 100%;
|
||||
width: var(--boxSize);
|
||||
background: var(--color-accent);
|
||||
border-radius: 8px;
|
||||
transition: transform 0.05s ease;
|
||||
}
|
||||
|
||||
.gradio-webrtc-icon-container {
|
||||
position: relative;
|
||||
width: 128px;
|
||||
height: 128px;
|
||||
display: flex;
|
||||
justify-content: center;
|
||||
align-items: center;
|
||||
}
|
||||
|
||||
.gradio-webrtc-icon {
|
||||
position: relative;
|
||||
width: 48px;
|
||||
height: 48px;
|
||||
border-radius: 50%;
|
||||
transition: transform 0.1s ease;
|
||||
display: flex;
|
||||
justify-content: center;
|
||||
align-items: center;
|
||||
z-index: 2;
|
||||
}
|
||||
|
||||
.icon-image {
|
||||
width: 32px;
|
||||
height: 32px;
|
||||
object-fit: contain;
|
||||
filter: brightness(0) invert(1);
|
||||
}
|
||||
|
||||
.pulse-ring {
|
||||
position: absolute;
|
||||
top: 50%;
|
||||
left: 50%;
|
||||
transform: translate(-50%, -50%);
|
||||
width: 48px;
|
||||
height: 48px;
|
||||
border-radius: 50%;
|
||||
animation: pulse 2s cubic-bezier(0.4, 0, 0.6, 1) infinite;
|
||||
opacity: 0.5;
|
||||
}
|
||||
|
||||
@keyframes pulse {
|
||||
0% {
|
||||
transform: translate(-50%, -50%) scale(1);
|
||||
opacity: 0.5;
|
||||
}
|
||||
100% {
|
||||
transform: translate(-50%, -50%) scale(var(--max-scale, 3));
|
||||
opacity: 0;
|
||||
}
|
||||
}
|
||||
|
||||
.dots {
|
||||
display: flex;
|
||||
gap: 8px;
|
||||
align-items: center;
|
||||
height: 64px;
|
||||
}
|
||||
|
||||
.dot {
|
||||
width: 12px;
|
||||
height: 12px;
|
||||
border-radius: 50%;
|
||||
opacity: 0.5;
|
||||
animation: pulse 1.5s infinite;
|
||||
}
|
||||
|
||||
.dot:nth-child(2) {
|
||||
animation-delay: 0.2s;
|
||||
}
|
||||
|
||||
.dot:nth-child(3) {
|
||||
animation-delay: 0.4s;
|
||||
}
|
||||
|
||||
@keyframes pulse {
|
||||
0%,
|
||||
100% {
|
||||
opacity: 0.4;
|
||||
transform: scale(1);
|
||||
}
|
||||
50% {
|
||||
opacity: 1;
|
||||
transform: scale(1.1);
|
||||
}
|
||||
}
|
||||
</style>
|
||||
Reference in New Issue
Block a user