mirror of
https://github.com/OpenBMB/MiniCPM-V.git
synced 2026-02-04 09:49:20 +08:00
Update README
This commit is contained in:
@@ -136,12 +136,12 @@ MiniCPM-o 2.6 can be easily used in various ways: (1) [llama.cpp](https://github
|
||||
|
||||
**Model Architecture.**
|
||||
|
||||
- **End-to-end Omni-modal Architecture.** Different modality encoder/decoders are connected and trained in an **end-to-end** fashion to fully exploit rich multimodal knowledge.
|
||||
- **End-to-end Omni-modal Architecture.** Different modality encoder/decoders are connected and trained in an **end-to-end** fashion to fully exploit rich multimodal knowledge. The model is trained in a fully end-to-end manner with only CE loss.
|
||||
- **Omni-modal Live Streaming Mechanism.** (1) We change the offline modality encoder/decoders into online ones for **streaminig inputs/outputs.** (2) We devise a **time-division multiplexing (TDM) mechanism** for omni-modality streaminig processing in the LLM backbone. It divides parallel omni-modality streams into sequential info within small periodic time slices.
|
||||
- **Configurable Speech Modeling Design.** We devise a multimodal system prompt, including traditional text system prompt, and **a new audio system prompt to determine the assistant voice**. This enables flexible voice configurations in inference time, and also facilitates end-to-end voice cloning and description-based voice creation.
|
||||
|
||||
<div align="center">
|
||||
<img src="./assets/minicpm-o-26-framework.png" , width=80%>
|
||||
<img src="./assets/minicpm-o-26-framework-v2.png" , width=80%>
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
@@ -126,12 +126,12 @@ MiniCPM-o 2.6 可以通过多种方式轻松使用:(1) [llama.cpp](https://git
|
||||
|
||||
**模型架构。**
|
||||
|
||||
- **端到端全模态架构。** 通过**端到端**的方式连接和训练不同模态的编/解码模块以充分利用丰富的多模态知识。
|
||||
- **端到端全模态架构。** 通过**端到端**的方式连接和训练不同模态的编/解码模块以充分利用丰富的多模态知识。模型完全使用 CE 损失端到端训练。
|
||||
- **全模态流式机制。** (1) 我们将不同模态的离线编/解码器改造为适用于**流式输入/输出**的在线模块。 (2) 我们针对大语言模型基座设计了**时分复用的全模态流式信息处理机制**,将平行的不同模态的信息流拆分重组为周期性时间片序列。
|
||||
- **可配置的声音方案。** 我们设计了新的多模态系统提示,包含传统文本系统提示词,和**用于指定模型声音的语音系统提示词**。模型可在推理时灵活地通过文字或语音样例控制声音风格,并支持端到端声音克隆和音色创建等高级能力。
|
||||
|
||||
<div align="center">
|
||||
<img src="./assets/minicpm-o-26-framework.png" , width=80%>
|
||||
<img src="./assets/minicpm-o-26-framework-v2.png" , width=80%>
|
||||
</div>
|
||||
|
||||
<br>
|
||||
|
||||
BIN
assets/minicpm-o-26-framework-v2.png
Normal file
BIN
assets/minicpm-o-26-framework-v2.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 1.1 MiB |
Reference in New Issue
Block a user