diff --git a/README.md b/README.md index e8e6c29..a498122 100644 --- a/README.md +++ b/README.md @@ -1,43 +1,54 @@
- + -**A GPT-4V Level MLLM for Single Image, Multi Image and Video on Your Phone** +**A GPT-4o Level MLLM for Vision, Speech and Multimodal Live Streaming on Your Phone** [中文](./README_zh.md) | English -Join our 💬 WeChat | View MiniCPM-V 📖 best practices + + + + WeChat + WeChat  | + +  + +Discord + Discord + +

- MiniCPM-V 2.6 🤗 🤖 | MiniCPM-Llama3-V 2.5 🤗 🤖 | - MiniCPM-Llama3-V 2.5 Technical Report + MiniCPM-o 2.6 🤗 CN🤖 US🤖 | MiniCPM-V 2.6 🤗 🤖 | + Technical Blog Coming Soon

+**MiniCPM-o** is the latest series of end-side multimodal LLMs (MLLMs) ungraded from MiniCPM-V. The models can now take image, video, text, and audio as inputs and provide high-quality text and speech outputs in an end-to-end fashion. Since February 2024, we have released 6 versions of the model, aiming to achieve **strong performance and efficient deployment**. The most notable models in the series currently include: -**MiniCPM-V** is a series of end-side multimodal LLMs (MLLMs) designed for vision-language understanding. The models take image, video and text as inputs and provide high-quality text outputs. Since February 2024, we have released 5 versions of the model, aiming to achieve **strong performance and efficient deployment**. The most notable models in this series currently include: +- **MiniCPM-o 2.6**: 🔥🔥🔥 The latest and most capable model in the MiniCPM-o series. With a total of 8B parameters, this end-to-end model **achieves comparable performance to GPT-4o-202405 in vision, speech, and multimodal live streaming**, making it one of the most versatile and performant models in the open-source community. For the new voice mode, MiniCPM-o 2.6 **supports bilingual real-time speech conversation with configurable voices**, and also allows for fun capabilities such as emotion/speed/style control, end-to-end voice cloning, role play, etc. It also advances MiniCPM-V 2.6's visual capabilities such **strong OCR capability, trustworthy behavior, multilingual support, and video understanding**. Due to its superior token density, MiniCPM-o 2.6 can for the first time **support multimodal live streaming on end-side devices** such as iPad. -- **MiniCPM-V 2.6**: 🔥🔥🔥 The latest and most capable model in the MiniCPM-V series. With a total of 8B parameters, the model **surpasses GPT-4V in single image, multi-image and video understanding**. It outperforms **GPT-4o mini, Gemini 1.5 Pro and Claude 3.5 Sonnet** in single image understanding, and advances MiniCPM-Llama3-V 2.5's features such as strong OCR capability, trustworthy behavior, multilingual support, and end-side deployment. Due to its superior token density, MiniCPM-V 2.6 can for the first time support real-time video understanding on end-side devices such as iPad. +- **MiniCPM-V 2.6**: The most capable model in the MiniCPM-V series. With a total of 8B parameters, the model **surpasses GPT-4V in single image, multi-image and video understanding**. It outperforms **GPT-4o mini, Gemini 1.5 Pro and Claude 3.5 Sonnet** in single image understanding, and can for the first time support real-time video understanding on iPad. -- **MiniCPM-V 2.0**: The lightest model in the MiniCPM-V series. With 2B parameters, it surpasses larger models such as Yi-VL 34B, CogVLM-Chat 17B, and Qwen-VL-Chat 10B in overall performance. It can accept image inputs of any aspect ratio and up to 1.8 million pixels (e.g., 1344x1344), achieving comparable performance with Gemini Pro in understanding scene-text and matches GPT-4V in low hallucination rates. ## News #### 📌 Pinned +* [2025.01.13] 🔥🔥🔥 We open-source MiniCPM-o 2.6, which matches GPT-4o-202405 on vision, speech and multimodal live streaming. It advances popular capabitlies of MiniCPM-V 2.6, and supports various new fun features. Try it now! + + * [2024.08.17] 🚀🚀🚀 MiniCPM-V 2.6 is now fully supported by [official](https://github.com/ggerganov/llama.cpp) llama.cpp! GGUF models of various sizes are available [here](https://huggingface.co/openbmb/MiniCPM-V-2_6-gguf). -* [2024.08.15] We now also support multi-image SFT. For more details, please refer to the [document](https://github.com/OpenBMB/MiniCPM-V/tree/main/finetune). -* [2024.08.14] MiniCPM-V 2.6 now also supports [fine-tuning](https://github.com/modelscope/ms-swift/issues/1613) with the SWIFT framework! -* [2024.08.10] 🚀🚀🚀 MiniCPM-Llama3-V 2.5 is now fully supported by [official](https://github.com/ggerganov/llama.cpp) llama.cpp! GGUF models of various sizes are available [here](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5-gguf). + * [2024.08.06] 🔥🔥🔥 We open-source MiniCPM-V 2.6, which outperforms GPT-4V on single image, multi-image and video understanding. It advances popular features of MiniCPM-Llama3-V 2.5, and can support real-time video understanding on iPad. Try it now! + * [2024.08.03] MiniCPM-Llama3-V 2.5 technical report is released! See [here](https://arxiv.org/abs/2408.01800). -* [2024.07.19] MiniCPM-Llama3-V 2.5 supports vLLM now! See [here](#inference-with-vllm). -* [2024.05.28] 💫 We now support LoRA fine-tuning for MiniCPM-Llama3-V 2.5, using only 2 V100 GPUs! See more statistics [here](https://github.com/OpenBMB/MiniCPM-V/tree/main/finetune#model-fine-tuning-memory-usage-statistics). -* [2024.05.23] 🔍 We've released a comprehensive comparison between Phi-3-vision-128k-instruct and MiniCPM-Llama3-V 2.5, including benchmarks evaluations, multilingual capabilities, and inference efficiency 🌟📊🌍🚀. Click [here](./docs/compare_with_phi-3_vision.md) to view more details. + * [2024.05.23] 🔥🔥🔥 MiniCPM-V tops GitHub Trending and Hugging Face Trending! Our demo, recommended by Hugging Face Gradio’s official account, is available [here](https://huggingface.co/spaces/openbmb/MiniCPM-Llama3-V-2_5). Come and try it out!
@@ -45,10 +56,22 @@ Join our 💬 WeChat | View MiniC
Click to view more news. +* [2024.08.15] We now also support multi-image SFT. For more details, please refer to the [document](https://github.com/OpenBMB/MiniCPM-V/tree/main/finetune). +* [2024.08.14] MiniCPM-V 2.6 now also supports [fine-tuning](https://github.com/modelscope/ms-swift/issues/1613) with the SWIFT framework! +* [2024.08.10] 🚀🚀🚀 MiniCPM-Llama3-V 2.5 is now fully supported by [official](https://github.com/ggerganov/llama.cpp) llama.cpp! GGUF models of various sizes are available [here](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5-gguf). + +* [2024.07.19] MiniCPM-Llama3-V 2.5 supports vLLM now! See [here](#inference-with-vllm). + * [2024.06.03] Now, you can run MiniCPM-Llama3-V 2.5 on multiple low VRAM GPUs(12 GB or 16 GB) by distributing the model's layers across multiple GPUs. For more details, Check this [link](https://github.com/OpenBMB/MiniCPM-V/blob/main/docs/inference_on_multiple_gpus.md). * [2024.05.28] 🚀🚀🚀 MiniCPM-Llama3-V 2.5 now fully supports its feature in llama.cpp and ollama! Please pull the latest code **of our provided forks** ([llama.cpp](https://github.com/OpenBMB/llama.cpp/blob/minicpm-v2.5/examples/minicpmv/README.md), [ollama](https://github.com/OpenBMB/ollama/tree/minicpm-v2.5/examples/minicpm-v2.5)). GGUF models in various sizes are available [here](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5-gguf/tree/main). MiniCPM-Llama3-V 2.5 series is **not supported by the official repositories yet**, and we are working hard to merge PRs. Please stay tuned! + +* [2024.05.28] 💫 We now support LoRA fine-tuning for MiniCPM-Llama3-V 2.5, using only 2 V100 GPUs! See more statistics [here](https://github.com/OpenBMB/MiniCPM-V/tree/main/finetune#model-fine-tuning-memory-usage-statistics). + * [2024.05.25] MiniCPM-Llama3-V 2.5 now supports streaming outputs and customized system prompts. Try it [here](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5#usage)! * [2024.05.24] We release the MiniCPM-Llama3-V 2.5 [gguf](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5-gguf), which supports [llama.cpp](#inference-with-llamacpp) inference and provides a 6~8 token/s smooth decoding on mobile phones. Try it now! + +* [2024.05.23] 🔍 We've released a comprehensive comparison between Phi-3-vision-128k-instruct and MiniCPM-Llama3-V 2.5, including benchmarks evaluations, multilingual capabilities, and inference efficiency 🌟📊🌍🚀. Click [here](./docs/compare_with_phi-3_vision.md) to view more details. + * [2024.05.20] We open-soure MiniCPM-Llama3-V 2.5, it has improved OCR capability and supports 30+ languages, representing the first end-side MLLM achieving GPT-4V level performance! We provide [efficient inference](#deployment-on-mobile-phone) and [simple fine-tuning](./finetune/readme.md). Try it now! * [2024.04.23] MiniCPM-V-2.0 supports vLLM now! Click [here](#inference-with-vllm) to view more details. * [2024.04.18] We create a HuggingFace Space to host the demo of MiniCPM-V 2.0 at [here](https://huggingface.co/spaces/openbmb/MiniCPM-V-2)! @@ -64,29 +87,943 @@ Join our 💬 WeChat | View MiniC ## Contents +- [MiniCPM-o 2.6](#minicpm-o-26) - [MiniCPM-V 2.6](#minicpm-v-26) -- [MiniCPM-Llama3-V 2.5](#minicpm-llama3-v-25) -- [MiniCPM-V 2.0](#minicpm-v-20) - [Chat with Our Demo on Gradio 🤗](#chat-with-our-demo-on-gradio-) - [Install](#install) - [Inference](#inference) - [Model Zoo](#model-zoo) - [Multi-turn Conversation](#multi-turn-conversation) - - [Chat with multiple images](#chat-with-multiple-images) - - [In-context few-shot learning](#in-context-few-shot-learning) - - [Chat with video](#chat-with-video) + - [Chat with Multiple Images](#chat-with-multiple-images) + - [In-context Few-shot Learning](#in-context-few-shot-learning) + - [Chat with Video](#chat-with-video) + - [Speech Conversation](#speech-conversation) + - [Mimick](#mimick) + - [General Speech Conversation with Configurable Voices](#general-speech-conversation-with-configurable-voices) + - [Addressing Various Audio Tasks](#addressing-various-audio-tasks) + - [Multimodal Live Streaming](#multimodal-live-streaming) - [Inference on Multiple GPUs](#inference-on-multiple-gpus) - [Inference on Mac](#inference-on-mac) - [Deployment on Mobile Phone](#deployment-on-mobile-phone) - - [Inference with llama.cpp](#inference-with-llamacpp) - - [Inference with ollama](#inference-with-ollama) - - [Inference with vLLM](#inference-with-vllm) + - [Efficient Inference with llama.cpp, ollama, vLLM](#efficient-inference-with-llamacpp-ollama-vllm) - [Fine-tuning](#fine-tuning) - [FAQs](#faqs) +- [Limitations](#limitations) + + +## MiniCPM-o 2.6 + +**MiniCPM-o 2.6** is the latest and most capable model in the MiniCPM-o series. The model is built in an end-to-end fashion based on SigLip-400M, Whisper-medium-300M, ChatTTS-200M, and Qwen2.5-7B with a total of 8B parameters. It exhibits a significant performance improvement over MiniCPM-V 2.6, and introduces new features for real-time speech conversation and multimodal live streaming. Notable features of MiniCPM-o 2.6 include: + +- 🔥 **Leading Visual Capability.** + MiniCPM-o 2.6 achieves an average score of 70.2 on OpenCompass, a comprehensive evaluation over 8 popular benchmarks. **With only 8B parameters, it surpasses widely used proprietary models like GPT-4o-202405, Gemini 1.5 Pro, and Claude 3.5 Sonnet** for single image understanding. It also **outperforms GPT-4V and Claude 3.5 Sonnet** in mutli-image and video understanding, and shows promising in-context learning capability. + +- 🎙 **State-of-the-art Speech Capability.** MiniCPM-o 2.6 supports **bilingual real-time speech conversation with configurable voices** in English and Chinese. It **outperforms GPT-4o-realtime on audio understanding tasks** such as ASR and STT translation, and shows **state-of-the-art performance on speech conversation in both semantic and acoustic evaluations in the open-source community**. It also allows for fun features such as emotion/speed/style control, end-to-end voice cloning, role play, etc. + +- 🎬 **Strong Multimodal Live Streaming Capability.** As a new feature, MiniCPM-o 2.6 can **accept continous video and audio streams independent of user queries, and support real-time speech interaction**. It **outperforms GPT-4o-202408 and Claude 3.5 Sonnet and shows state-of-art performance in open-source community on StreamingBench**, a comprehensive benchmark for real-time video understanding, omni-source (video & audio) understanding, and multimodal contextual understanding. + +- 💪 **Strong OCR Capability and Others.** +Advancing popular visual capabilites from MiniCPM-V series, MiniCPM-o 2.6 can process images with any aspect ratio and up to 1.8 million pixels (e.g., 1344x1344). It achieves **state-of-the-art performance on OCRBench for models under 25B, surpassing proprietary models such as GPT-4o-202405**. + Based on the the latest [RLAIF-V](https://github.com/RLHF-V/RLAIF-V/) and [VisCPM](https://github.com/OpenBMB/VisCPM) techniques, it features **trustworthy behaviors**, outperforming GPT-4o and Claude 3.5 Sonnet on MMHal-Bench, and supports **multilingual capabilities** on more than 30 languages. + + +- 🚀 **Superior Efficiency.** + In addition to its friendly size, MiniCPM-o 2.6 also shows **state-of-the-art token density** (i.e., number of pixels encoded into each visual token). **It produces only 640 tokens when processing a 1.8M pixel image, which is 75% fewer than most models**. This directly improves the inference speed, first-token latency, memory usage, and power consumption. As a result, MiniCPM-o 2.6 can efficiently support **multimodal live streaming** on end-side devices such as iPad. + +- 💫 **Easy Usage.** +MiniCPM-o 2.6 can be easily used in various ways: (1) [llama.cpp](https://github.com/OpenBMB/llama.cpp/blob/minicpm-omni/examples/llava/README-minicpmo2.6.md) support for efficient CPU inference on local devices, (2) [int4](https://huggingface.co/openbmb/MiniCPM-o-2_6-int4) and [GGUF](https://huggingface.co/openbmb/MiniCPM-o-2_6-gguf) format quantized models in 16 sizes, (3) [vLLM](#efficient-inference-with-llamacpp-ollama-vllm) support for high-throughput and memory-efficient inference, (4) fine-tuning on new domains and tasks with [LLaMA-Factory](./docs/llamafactory_train.md), (5) quick local WebUI demo setup with [Gradio](#chat-with-our-demo-on-gradio), and (6) online web demo on [CN](https://minicpm-omni-webdemo.modelbest.cn/ +) server and [US](https://minicpm-omni-webdemo-us.modelbest.cn/) server. + + +**Model Architecture.** + +- **End-to-end Omni-modal Architecture.** Different modality encoder/decoders are connected and trained in an **end-to-end** fashion to fully exploit rich multimodal knowledge. +- **Omni-modal Live Streaming Mechanism.** (1) We change the offline modality encoder/decoders into online ones for **streaminig inputs/outputs.** (2) We devise a **time-division multiplexing (TDM) mechanism** for omni-modality streaminig processing in the LLM backbone. It divides parallel omni-modality streams into sequential info within small periodic time slices. +- **Configurable Speech Modeling Design.** We devise a multimodal system prompt, including traditional text system prompt, and **a new audio system prompt to determine the assistant voice**. This enables flexible voice configurations in inference time, and also facilitates end-to-end voice cloning and description-based voice creation. + +
+ +
+ + +### Evaluation + +
+ +
+ +
+Click to view visual understanding results. + +**Image Understanding** + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ModelSizeToken Density+OpenCompassOCRBenchMathVista miniChartQAMMVetMMStarMMEMMB1.1 testAI2DMMMU valHallusionBenchTextVQA valDocVQA testMathVerse miniMathVisionMMHal Score
Proprietary
GPT-4o-20240513-108869.973661.385.769.163.92328.782.284.669.255.0-92.850.230.43.6
Claude3.5-Sonnet-75067.978861.690.866.062.21920.078.580.265.949.9-95.2--3.4
Gemini-1.5-Pro--64.475457.781.364.059.12110.673.979.160.645.673.586.5-19.2-
GPT-4o-mini-20240718-108864.178552.4-66.954.82003.476.077.860.046.1----3.3
Open Source
Cambrian-34B34B182058.359150.375.653.254.22049.977.879.550.441.676.775.5---
GLM-4V-9B13B78459.177651.1-58.054.82018.867.971.246.945.0-----
Pixtral-12B12B25661.068556.981.858.554.5-72.779.051.147.075.790.7---
DeepSeek-VL2-27B (4B)27B67266.480963.986.060.061.92253.081.283.854.045.384.293.3--3.0
Qwen2-VL-7B8B78467.186658.283.062.060.72326.081.883.054.150.684.394.531.916.33.2
LLaVA-OneVision-72B72B18268.174167.583.760.665.82261.085.085.656.849.080.591.339.1-3.5
InternVL-2.5-8B8B70668.382264.484.862.862.82344.083.684.556.050.179.193.039.519.73.4
MiniCPM-V 2.68B282265.2852*60.679.460.057.52348.4*78.082.149.8*48.1*80.190.825.718.33.6
MiniCPM-o 2.68B282270.2897*71.9*86.9*67.564.02372.0*80.585.850.4*51.982.093.541.4*23.1*3.8
+
+* We evaluate this benchmark using chain-of-thought prompting. Specifically, for MME, we used this technique only for the Cognition set. + + ++ Token Density: number of pixels encoded into each visual token at maximum resolution, i.e., # pixels at maximum resolution / # visual tokens. + +Note: For proprietary models, we calculate token density based on the image encoding charging strategy defined in the official API documentation, which provides an upper-bound estimation. + + +**Multi-image and Video Understanding** + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ModelSizeBLINK-valMantis-EvalMIRBVideo-MME (wo / w subs)
Proprietary
GPT-4o-20240513-68.0--71.9/77.2
GPT4V-54.662.753.159.9/63.3
Open-source
LLaVA-NeXT-Interleave 14B14B52.666.430.2-
LLaVA-One-Vision-72B72B55.477.6-66.2/69.5
MANTIS 8B8B49.159.534.8-
Qwen2-VL-7B8B53.269.6*67.6*63.3/69.0
InternVL-2.5-8B8B54.867.752.564.2/66.9
MiniCPM-V 2.68B53.069.153.860.9/63.6
MiniCPM-o 2.68B56.771.958.663.9/67.9
+ +
+* We evaluate officially released checkpoints by ourselves. + +
+ + +
+Click to view audio understanding and speech conversation results. + +**Audio Understanding** + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
TaskSizeASR (zh)ASR (en)ASTEmotion
MetricCER↓WER↓BLEU↑ACC↑
DatasetAISHELL-1Fleurs zhWenetSpeech test-netLibriSpeech test-cleanGigaSpeechTED-LIUMCoVoST en2zhCoVoST zh2enMELD emotion
Proprietary
GPT-4o-Realtime-7.3*5.4*28.9*2.6*12.9*4.8*37.1*15.7*33.2*
Gemini-1.5-Pro-4.5*5.9*14.3*2.9*10.6*3.0*47.3*22.6*48.4*
Open-Source
Qwen2-Audio-Base8B-7.5-1.6--45.224.455.3
Qwen2-Audio-Instruction8B2.6*6.9*10.3*3.1*9.7*5.9*39.5*22.9*17.4*
GLM-4-Voice-Base9B2.5--2.8----
MiniCPM-o 2.68B1.64.46.91.78.73.048.227.252.4
+
+* We evaluate officially released checkpoints by ourselves.

+ +**Speech Generation** + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
TaskSizeSpeechQA
MetricACC↑G-Eval (10 point)↑Semantic ELO score↑Acoustic ELO score↑Overall ELO score↑UTMOS↑ASR-WER↓
DatasetSpeech Llama Q.Speech Web Q.Speech Trivia QASpeech AlpacaEvalAudioArena
Proprietary
GPT-4o-Realtime71.751.669.77.41157120312004.22.3
Open-Source
GLM-4-Voice9B50.032.036.45.1999114710354.111.7
Llama-Omni8B45.322.910.73.99608788973.224.3
Moshi7B43.723.816.72.48718088752.88.2
Mini-Omni1B22.012.86.92.59268038653.410.0
MiniCPM-o 2.68B61.040.040.25.11088116311314.29.8
+
+All results are from AudioEvals, and the evaluation methods along with further details can be found in AudioEvals.

+ +**End-to-end Voice Cloning** + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
TaskVoice cloning
MetricSIMO↑SIMO↑
DatasetSeed-TTS test-zhSeed-TTS test-en
F5-TTS7667
CosyVoice7564
FireRedTTS6346
MiniCPM-o 2.65747
+
+ +
+ +
+Click to view multimodal live streaming results. + +**Multimodal Live Streaming**: results on StreamingBench + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ModelSizeReal-Time Video UnderstandingOmni-Source UnderstandingContextual UnderstandingOverall
Proprietary
Gemini 1.5 Pro-77.467.851.170.3
GPT-4o-202408-74.551.048.064.1
Claude-3.5-Sonnet-74.041.437.859.7
Open-source
VILA-1.58B61.537.526.749.5
LongVA7B63.135.930.250.7
LLaVA-Next-Video-34B34B69.841.734.356.7
Qwen2-VL-7B8B71.240.733.157.0
InternVL2-8B8B70.142.734.157.0
VITA-1.58B70.940.835.857.4
LLaVA-OneVision-7B8B74.340.831.058.4
InternLM-XC2.5-OL-7B8B75.446.233.660.8
MiniCPM-V 2.68B72.440.233.457.7
MiniCPM-o 2.68B79.953.438.566.0
+ +
+ + +### Examples + +We deploy MiniCPM-o 2.6 on end devices. The demo video is the raw-speed recording on an iPad Pro and a Web demo. + +
+ +
+ +
+ math + diagram + bike +
## MiniCPM-V 2.6 +
+Click to view more details of MiniCPM-V 2.6 + **MiniCPM-V 2.6** is the latest and most capable model in the MiniCPM-V series. The model is built on SigLip-400M and Qwen2-7B with a total of 8B parameters. It exhibits a significant performance improvement over MiniCPM-Llama3-V 2.5, and introduces new features for multi-image and video understanding. Notable features of MiniCPM-V 2.6 include: - 🔥 **Leading Performance.** @@ -364,7 +1301,7 @@ MiniCPM-V 2.6 can be easily used in various ways: (1) [llama.cpp](https://github 42.4 10.3 - + MiniCPM-V 2.6 8B 2822 @@ -489,7 +1426,7 @@ Note: For proprietary models, we calculate token density based on the image enco 34.4* 56.9* - + MiniCPM-V 2.6 8B 69.1 @@ -634,7 +1571,7 @@ Note: For proprietary models, we calculate token density based on the image enco 2.64 3.28 - + MiniCPM-V 2.6 8B 60.9 @@ -775,7 +1712,7 @@ Note: For proprietary models, we calculate token density based on the image enco 70.9 54.1 - + MiniCPM-V 2.6+ 8B 0 @@ -784,14 +1721,14 @@ Note: For proprietary models, we calculate token density based on the image enco 45.4 23.9 - + 4 63.6 60.5 65.5 50.1 - + 8 64.6 63.4 @@ -850,408 +1787,46 @@ We deploy MiniCPM-V 2.6 on end devices. The demo video is the raw screen recordi

-## MiniCPM-Llama3-V 2.5 - -
-Click to view more details of MiniCPM-Llama3-V 2.5 - -**MiniCPM-Llama3-V 2.5** is the latest model in the MiniCPM-V series. The model is built on SigLip-400M and Llama3-8B-Instruct with a total of 8B parameters. It exhibits a significant performance improvement over MiniCPM-V 2.0. Notable features of MiniCPM-Llama3-V 2.5 include: - -- 🔥 **Leading Performance.** - MiniCPM-Llama3-V 2.5 has achieved an average score of 65.1 on OpenCompass, a comprehensive evaluation over 11 popular benchmarks. **With only 8B parameters, it surpasses widely used proprietary models like GPT-4V-1106, Gemini Pro, Claude 3 and Qwen-VL-Max** and greatly outperforms other Llama 3-based MLLMs. - -- 💪 **Strong OCR Capabilities.** - MiniCPM-Llama3-V 2.5 can process images with any aspect ratio and up to 1.8 million pixels (e.g., 1344x1344), achieving a **700+ score on OCRBench, surpassing proprietary models such as GPT-4o, GPT-4V-0409, Qwen-VL-Max and Gemini Pro**. Based on recent user feedback, MiniCPM-Llama3-V 2.5 has now enhanced full-text OCR extraction, table-to-markdown conversion, and other high-utility capabilities, and has further strengthened its instruction-following and complex reasoning abilities, enhancing multimodal interaction experiences. - -- 🏆 **Trustworthy Behavior.** - Leveraging the latest [RLAIF-V](https://github.com/RLHF-V/RLAIF-V/) method (the newest technique in the [RLHF-V](https://github.com/RLHF-V) [CVPR'24] series), MiniCPM-Llama3-V 2.5 exhibits more trustworthy behavior. It achieves a **10.3%** hallucination rate on Object HalBench, lower than GPT-4V-1106 (13.6%), achieving the best-level performance within the open-source community. [Data released](https://huggingface.co/datasets/openbmb/RLAIF-V-Dataset). - -- 🌏 **Multilingual Support.** - Thanks to the strong multilingual capabilities of Llama 3 and the cross-lingual generalization technique from [VisCPM](https://github.com/OpenBMB/VisCPM), MiniCPM-Llama3-V 2.5 extends its bilingual (Chinese-English) multimodal capabilities to **over 30 languages including German, French, Spanish, Italian, Korean etc.** [All Supported Languages](./assets/minicpm-llama-v-2-5_languages.md). - -- 🚀 **Efficient Deployment.** - MiniCPM-Llama3-V 2.5 systematically employs **model quantization, CPU optimizations, NPU optimizations and compilation optimizations**, achieving high-efficiency deployment on end-side devices. For mobile phones with Qualcomm chips, we have integrated the NPU acceleration framework QNN into llama.cpp for the first time. After systematic optimization, MiniCPM-Llama3-V 2.5 has realized a **150x acceleration in end-side MLLM image encoding** and a **3x speedup in language decoding**. - -- 💫 **Easy Usage.** -MiniCPM-Llama3-V 2.5 can be easily used in various ways: (1) [llama.cpp](https://github.com/OpenBMB/llama.cpp/blob/minicpm-v2.5/examples/minicpmv/README.md) and [ollama](https://github.com/OpenBMB/ollama/tree/minicpm-v2.5/examples/minicpm-v2.5) support for efficient CPU inference on local devices, (2) [GGUF](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5-gguf) format quantized models in 16 sizes, (3) efficient [LoRA](https://github.com/OpenBMB/MiniCPM-V/tree/main/finetune#lora-finetuning) fine-tuning with only 2 V100 GPUs, (4) [streaming output](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5#usage), (5) quick local WebUI demo setup with [Gradio](https://github.com/OpenBMB/MiniCPM-V/blob/main/web_demo_2.5.py) and [Streamlit](https://github.com/OpenBMB/MiniCPM-V/blob/main/web_demo_streamlit-2_5.py), and (6) interactive demos on [HuggingFace Spaces](https://huggingface.co/spaces/openbmb/MiniCPM-Llama3-V-2_5). - -### Evaluation - -
- -
-
-Click to view results on TextVQA, DocVQA, OCRBench, OpenCompass, MME, MMBench, MMMU, MathVista, LLaVA Bench, RealWorld QA, Object HalBench. -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
ModelSizeOCRBenchTextVQA valDocVQA testOpen-CompassMMEMMB test (en)MMB test (cn)MMMU valMath-VistaLLaVA BenchRealWorld QAObject HalBench
Proprietary
Gemini Pro-68074.688.162.92148.973.674.348.945.879.960.4-
GPT-4V (2023.11.06)-64578.088.463.51771.577.074.453.847.893.163.086.4
Open-source
Mini-Gemini2.2B-56.234.2*-1653.0--31.7----
Qwen-VL-Chat9.6B48861.562.651.61860.061.856.337.033.867.749.356.2
DeepSeek-VL-7B7.3B43564.7*47.0*54.61765.473.871.438.336.877.854.2-
Yi-VL-34B34B29043.4*16.9*52.22050.272.470.745.130.762.354.879.3
CogVLM-Chat17.4B59070.433.3*54.21736.665.855.937.334.773.960.373.6
TextMonkey9.7B55864.366.7---------
Idefics28.0B-73.074.057.21847.675.768.645.252.249.160.7-
Bunny-LLama-3-8B8.4B---54.31920.377.073.941.331.561.258.8-
LLaVA-NeXT Llama-3-8B8.4B--78.2-1971.5--41.737.580.160.0-
Phi-3-vision-128k-instruct4.2B639*70.9--1537.5*--40.444.564.2*58.8*-
MiniCPM-V 1.02.8B36660.638.247.51650.264.162.638.328.951.351.278.4
MiniCPM-V 2.02.8B60574.171.954.51808.669.166.538.238.769.255.885.5
MiniCPM-Llama3-V 2.58.5B72576.684.865.12024.677.274.245.854.386.763.589.7
- - -
-* We evaluate the officially released checkpoint by ourselves. - -
- -
- -
- Evaluation results of multilingual LLaVA Bench -
- -### Examples - - -

- -

-
- -
- - -## MiniCPM-V 2.0 - -
-Click to view more details of MiniCPM-V 2.0 - - -**MiniCPM-V 2.0** is an efficient version with promising performance for deployment. The model is built based on SigLip-400M and [MiniCPM-2.4B](https://github.com/OpenBMB/MiniCPM/), connected by a perceiver resampler. Our latest version, MiniCPM-V 2.0 has several notable features. - -- 🔥 **State-of-the-art Performance.** - - MiniCPM-V 2.0 achieves **state-of-the-art performance** on multiple benchmarks (including OCRBench, TextVQA, MME, MMB, MathVista, etc) among models under 7B parameters. It even **outperforms strong Qwen-VL-Chat 9.6B, CogVLM-Chat 17.4B, and Yi-VL 34B on OpenCompass, a comprehensive evaluation over 11 popular benchmarks**. Notably, MiniCPM-V 2.0 shows **strong OCR capability**, achieving **comparable performance to Gemini Pro in scene-text understanding**, and **state-of-the-art performance on OCRBench** among open-source models. - -- 🏆 **Trustworthy Behavior.** - - LMMs are known for suffering from hallucination, often generating text not factually grounded in images. MiniCPM-V 2.0 is **the first end-side LMM aligned via multimodal RLHF for trustworthy behavior** (using the recent [RLHF-V](https://rlhf-v.github.io/) [CVPR'24] series technique). This allows the model to **match GPT-4V in preventing hallucinations** on Object HalBench. - -- 🌟 **High-Resolution Images at Any Aspect Raito.** - - MiniCPM-V 2.0 can accept **1.8 million pixels (e.g., 1344x1344) images at any aspect ratio**. This enables better perception of fine-grained visual information such as small objects and optical characters, which is achieved via a recent technique from [LLaVA-UHD](https://arxiv.org/pdf/2403.11703.pdf). - -- ⚡️ **High Efficiency.** - - MiniCPM-V 2.0 can be **efficiently deployed on most GPU cards and personal computers**, and **even on end devices such as mobile phones**. For visual encoding, we compress the image representations into much fewer tokens via a perceiver resampler. This allows MiniCPM-V 2.0 to operate with **favorable memory cost and speed during inference even when dealing with high-resolution images**. - -- 🙌 **Bilingual Support.** - - MiniCPM-V 2.0 **supports strong bilingual multimodal capabilities in both English and Chinese**. This is enabled by generalizing multimodal capabilities across languages, a technique from [VisCPM](https://arxiv.org/abs/2308.12038) [ICLR'24]. - -### Examples - - -

- -

-
- -We deploy MiniCPM-V 2.0 on end devices. The demo video is the raw screen recording on a Xiaomi 14 Pro without edition. - - -

- - -

-
-
## Legacy Models | Model | Introduction and Guidance | |:----------------------|:-------------------:| -| MiniCPM-V 1.0 | [Document](./minicpm_v1.md) | -| OmniLMM-12B | [Document](./omnilmm_en.md) | +| MiniCPM-Llama3-V 2.5 | [Document](./docs/minicpm_llama3_v2dot5.md) | +| MiniCPM-V 2.0 | [Document](./docs/minicpm_v2.md) | +| MiniCPM-V 1.0 | [Document](./docs/minicpm_v1.md) | +| OmniLMM-12B | [Document](././docs/omnilmm_en.md) | ## Chat with Our Demo on Gradio 🤗 -We provide online and local demos powered by Hugging Face Gradio , the most popular model deployment framework nowadays. It supports streaming outputs, progress bars, queuing, alerts, and other useful features. +We provide online and local demos powered by Hugging Face Gradio , the most popular model deployment framework nowadays. It supports streaming outputs, progress bars, queuing, alerts, and other useful features. ### Online Demo -Click here to try out the online demo of [MiniCPM-V 2.6](http://120.92.209.146:8887/) | [MiniCPM-Llama3-V 2.5](https://huggingface.co/spaces/openbmb/MiniCPM-Llama3-V-2_5) | [MiniCPM-V 2.0](https://huggingface.co/spaces/openbmb/MiniCPM-V-2). +Click here to try out the online demo of MiniCPM-o 2.6 ([CN](https://minicpm-omni-webdemo.modelbest.cn) | [US](https://minicpm-omni-webdemo-us.modelbest.cn/)) | [MiniCPM-V 2.6](http://120.92.209.146:8887/) | [MiniCPM-Llama3-V 2.5](https://huggingface.co/spaces/openbmb/MiniCPM-Llama3-V-2_5) | [MiniCPM-V 2.0](https://huggingface.co/spaces/openbmb/MiniCPM-V-2). ### Local WebUI Demo -You can easily build your own local WebUI demo with Gradio using the following commands. - -```shell -pip install -r requirements.txt -``` - -```shell -# For NVIDIA GPUs, run: -python web_demo_2.6.py --device cuda +You can easily build your own local WebUI demo using the following commands. +1. launch model server: +```shell +pip install -r requirements_o2.6.txt + +python web_demos/minicpm-o_2.6/model_server.py +``` + +2. launch web server: + +```shell +# Make sure Node and PNPM is installed. +cd web_demos/minicpm-o_2.6/web_server +pnpm install # install requirements + +pnpm run dev # start server ``` @@ -1284,21 +1859,19 @@ pip install -r requirements.txt | Model | Device | Memory |          Description | Download | |:-----------|:--:|:-----------:|:-------------------|:---------------:| -| MiniCPM-V 2.6| GPU | 17 GB | The latest version, achieving state-of-the-art end-side performance for single image, multi-image and video understanding. | [🤗](https://huggingface.co/openbmb/MiniCPM-V-2_6)    [](https://modelscope.cn/models/OpenBMB/MiniCPM-V-2_6) | +| MiniCPM-o 2.6| GPU | 18 GB | The latest version, achieving GPT-4o level performance for vision, speech and multimodal live streaming on end-side devices. | [🤗](https://huggingface.co/openbmb/MiniCPM-o-2_6)    [](https://modelscope.cn/models/OpenBMB/MiniCPM-o-2_6) | +| MiniCPM-o 2.6 gguf | CPU | 8 GB | The gguf version, lower memory usage and faster inference. | [🤗](https://huggingface.co/openbmb/MiniCPM-o-2_6-gguf)    [](https://modelscope.cn/models/OpenBMB/MiniCPM-o-2_6-gguf) | +| MiniCPM-o 2.6 int4 | GPU | 9 GB | The int4 quantized version, lower GPU memory usage. | [🤗](https://huggingface.co/openbmb/MiniCPM-o-2_6-int4)    [](https://modelscope.cn/models/OpenBMB/MiniCPM-o-2_6-int4) | +| MiniCPM-V 2.6| GPU | 17 GB | Strong end-side multimodal performance for single image, multi-image and video understanding. | [🤗](https://huggingface.co/openbmb/MiniCPM-V-2_6)    [](https://modelscope.cn/models/OpenBMB/MiniCPM-V-2_6) | | MiniCPM-V 2.6 gguf | CPU | 6 GB | The gguf version, lower memory usage and faster inference. | [🤗](https://huggingface.co/openbmb/MiniCPM-V-2_6-gguf)    [](https://modelscope.cn/models/OpenBMB/MiniCPM-V-2_6-gguf) | | MiniCPM-V 2.6 int4 | GPU | 7 GB | The int4 quantized version, lower GPU memory usage. | [🤗](https://huggingface.co/openbmb/MiniCPM-V-2_6-int4)    [](https://modelscope.cn/models/OpenBMB/MiniCPM-V-2_6-int4) | -| MiniCPM-Llama3-V 2.5 | GPU | 19 GB | Strong end-side multimodal performance. | [🤗](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5/)    [](https://modelscope.cn/models/OpenBMB/MiniCPM-Llama3-V-2_5) | -| MiniCPM-Llama3-V 2.5 gguf | CPU | 6 GB | The gguf version, lower memory usage and faster inference. | [🤗](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5-gguf)   [](https://modelscope.cn/models/OpenBMB/MiniCPM-Llama3-V-2_5-gguf) | -| MiniCPM-Llama3-V 2.5 int4 | GPU | 8 GB | The int4 quantized version, lower GPU memory usage. | [🤗](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5-int4/)    [](https://modelscope.cn/models/OpenBMB/MiniCPM-Llama3-V-2_5-int4) | -| MiniCPM-V 2.0 | GPU | 8 GB | Light version, balance the performance the computation cost. | [🤗](https://huggingface.co/openbmb/MiniCPM-V-2)    [](https://modelscope.cn/models/OpenBMB/MiniCPM-V-2) | -| MiniCPM-V 1.0 | GPU | 7 GB | Lightest version, achieving the fastest inference. | [🤗](https://huggingface.co/openbmb/MiniCPM-V)    [](https://modelscope.cn/models/OpenBMB/MiniCPM-V) | ### Multi-turn Conversation Please refer to the following codes to run.
- +
@@ -1307,33 +1880,30 @@ import torch from PIL import Image from transformers import AutoModel, AutoTokenizer -torch.manual_seed(0) +torch.manual_seed(100) -model = AutoModel.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True, +model = AutoModel.from_pretrained('openbmb/MiniCPM-o-2_6', trust_remote_code=True, attn_implementation='sdpa', torch_dtype=torch.bfloat16) # sdpa or flash_attention_2, no eager model = model.eval().cuda() -tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True) +tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-o-2_6', trust_remote_code=True) -image = Image.open('./assets/airplane.jpeg').convert('RGB') +image = Image.open('./assets/minicpmo2_6/show_demo.jpg').convert('RGB') # First round chat -question = "Tell me the model of this aircraft." +question = "What is the landform in the picture?" msgs = [{'role': 'user', 'content': [image, question]}] answer = model.chat( - image=None, msgs=msgs, tokenizer=tokenizer ) print(answer) -# Second round chat -# pass history context of multi-turn conversation +# Second round chat, pass history context of multi-turn conversation msgs.append({"role": "assistant", "content": [answer]}) -msgs.append({"role": "user", "content": ["Introduce something about Airbus A380."]}) +msgs.append({"role": "user", "content": ["What should I pay attention to when traveling here?"]}) answer = model.chat( - image=None, msgs=msgs, tokenizer=tokenizer ) @@ -1343,24 +1913,24 @@ print(answer) You will get the following output: ``` -"The aircraft in the image is an Airbus A380, which can be identified by its large size, double-deck structure, and the distinctive shape of its wings and engines. The A380 is a wide-body aircraft known for being the world's largest passenger airliner, designed for long-haul flights. It has four engines, which are characteristic of large commercial aircraft. The registration number on the aircraft can also provide specific information about the model if looked up in an aviation database." +"The landform in the picture is a mountain range. The mountains appear to be karst formations, characterized by their steep, rugged peaks and smooth, rounded shapes. These types of mountains are often found in regions with limestone bedrock and are shaped by processes such as erosion and weathering. The reflection of the mountains in the water adds to the scenic beauty of the landscape." -"The Airbus A380 is a double-deck, wide-body, four-engine jet airliner made by Airbus. It is the world's largest passenger airliner and is known for its long-haul capabilities. The aircraft was developed to improve efficiency and comfort for passengers traveling over long distances. It has two full-length passenger decks, which can accommodate more passengers than a typical single-aisle airplane. The A380 has been operated by airlines such as Lufthansa, Singapore Airlines, and Emirates, among others. It is widely recognized for its unique design and significant impact on the aviation industry." +"When traveling to this scenic location, it's important to pay attention to the weather conditions, as the area appears to be prone to fog and mist, especially during sunrise or sunset. Additionally, ensure you have proper footwear for navigating the potentially slippery terrain around the water. Lastly, respect the natural environment by not disturbing the local flora and fauna." ``` -#### Chat with multiple images +#### Chat with Multiple Images
- Click to view Python code running MiniCPM-V 2.6 with multiple images input. + Click to view Python code running MiniCPM-o 2.6 with multiple images input. ```python import torch from PIL import Image from transformers import AutoModel, AutoTokenizer -model = AutoModel.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True, +model = AutoModel.from_pretrained('openbmb/MiniCPM-o-2_6', trust_remote_code=True, attn_implementation='sdpa', torch_dtype=torch.bfloat16) # sdpa or flash_attention_2, no eager model = model.eval().cuda() -tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True) +tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-o-2_6', trust_remote_code=True) image1 = Image.open('image1.jpg').convert('RGB') image2 = Image.open('image2.jpg').convert('RGB') @@ -1369,7 +1939,6 @@ question = 'Compare image 1 and image 2, tell me about the differences between i msgs = [{'role': 'user', 'content': [image1, image2, question]}] answer = model.chat( - image=None, msgs=msgs, tokenizer=tokenizer ) @@ -1377,19 +1946,19 @@ print(answer) ```
-#### In-context few-shot learning +#### In-context Few-shot Learning
- Click to view Python code running MiniCPM-V 2.6 with few-shot input. + Click to view Python code running MiniCPM-o 2.6 with few-shot input. ```python import torch from PIL import Image from transformers import AutoModel, AutoTokenizer -model = AutoModel.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True, +model = AutoModel.from_pretrained('openbmb/MiniCPM-o-2_6', trust_remote_code=True, attn_implementation='sdpa', torch_dtype=torch.bfloat16) # sdpa or flash_attention_2, no eager model = model.eval().cuda() -tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True) +tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-o-2_6', trust_remote_code=True) question = "production date" image1 = Image.open('example1.jpg').convert('RGB') @@ -1405,7 +1974,6 @@ msgs = [ ] answer = model.chat( - image=None, msgs=msgs, tokenizer=tokenizer ) @@ -1413,9 +1981,9 @@ print(answer) ```
-#### Chat with video +#### Chat with Video
- Click to view Python code running MiniCPM-V 2.6 with video input. + Click to view Python code running MiniCPM-o 2.6 with video input. ```python import torch @@ -1423,10 +1991,10 @@ from PIL import Image from transformers import AutoModel, AutoTokenizer from decord import VideoReader, cpu # pip install decord -model = AutoModel.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True, +model = AutoModel.from_pretrained('openbmb/MiniCPM-o-2_6', trust_remote_code=True, attn_implementation='sdpa', torch_dtype=torch.bfloat16) # sdpa or flash_attention_2, no eager model = model.eval().cuda() -tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True) +tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-o-2_6', trust_remote_code=True) MAX_NUM_FRAMES=64 # if cuda OOM set a smaller number @@ -1459,7 +2027,6 @@ params["use_image_id"] = False params["max_slice_nums"] = 2 # use 1 if cuda OOM and video resolution > 448*448 answer = model.chat( - image=None, msgs=msgs, tokenizer=tokenizer, **params @@ -1469,6 +2036,295 @@ print(answer)
+#### Speech Conversation +
Model initialization + +```python +import torch +import librosa +from transformers import AutoModel, AutoTokenizer + +model = AutoModel.from_pretrained('openbmb/MiniCPM-o-2_6', trust_remote_code=True, + attn_implementation='sdpa', torch_dtype=torch.bfloat16) # sdpa or flash_attention_2, no eager +model = model.eval().cuda() +tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-o-2_6', trust_remote_code=True) + +model.init_tts() +model.tts.float() +``` + +
+ +##### Mimick + +
Click here to experience the capability of end-to-end audio understanding and generation. + +`Mimick` task reflects a model's end-to-end speech modeling capability. The model takes audio input, and outputs an ASR transcription and subsequently reconstructs the original audio with high similarity. The higher the similarity between the reconstructed audio and the original audio, the stronger the model's foundational capability in end-to-end speech modeling. + +```python +mimick_prompt = "Please repeat each user's speech, including voice style and speech content." +audio_input, _ = librosa.load('xxx.wav', sr=16000, mono=True) +msgs = [{'role': 'user', 'content': [mimick_prompt,audio_input]}] +res = model.chat( + msgs=msgs, + tokenizer=tokenizer, + sampling=True, + max_new_tokens=128, + use_tts_template=True, + temperature=0.3, + generate_audio=True, + output_audio_path='output.wav', # save the tts result to output_audio_path +) +``` + +
+ +##### General Speech Conversation with Configurable Voices +
Click to view the Python code for enabling MiniCPM-o 2.6 to interact with you in a specified voice. + +```python +ref_audio, _ = librosa.load('./assets/voice_01.wav', sr=16000, mono=True) # load the reference audio + +# Audio RolePlay: # With this mode, model will role-play the character based on the audio prompt. +sys_prompt = model.get_sys_prompt(ref_audio=ref_audio, mode='audio_roleplay', language='en') +user_question = {'role': 'user', 'content': [librosa.load('xxx.wav', sr=16000, mono=True)[0]]} + +# Audio Assistant: # With this mode, model will speak with the voice in ref_audio as a AI assistant. +# sys_prompt = model.get_sys_prompt(ref_audio=ref_audio, mode='audio_assistant', language='en') +# user_question = {'role': 'user', 'content': [librosa.load('xxx.wav', sr=16000, mono=True)[0]]} # Try to ask something! +``` +```python +msgs = [sys_prompt, user_question] +res = model.chat( + msgs=msgs, + tokenizer=tokenizer, + sampling=True, + max_new_tokens=128, + use_tts_template=True, + generate_audio=True, + temperature=0.3, + output_audio_path='result.wav', +) + +# round two +history = msgs.append({'role': 'assistant', 'content': res}) +user_question = {'role': 'user', 'content': [librosa.load('xxx.wav', sr=16000, mono=True)[0]]} +msgs = history.append(user_question) +res = model.chat( + msgs=msgs, + tokenizer=tokenizer, + sampling=True, + max_new_tokens=128, + use_tts_template=True, + generate_audio=True, + temperature=0.3, + output_audio_path='result_round_2.wav', +) +print(res) +``` + +
+ +##### Addressing Various Audio Tasks +
+ Click to show Python code running MiniCPM-o 2.6 with specific audioQA task. + +```python +''' +Audio Understanding Task Prompt: +Speech: + ASR with ZH(same as AST en2zh): 请仔细听这段音频片段，并将其内容逐字记录。 + ASR with EN(same as AST zh2en): Please listen to the audio snippet carefully and transcribe the content. + Speaker Analysis: Based on the speaker's content, speculate on their gender, condition, age range, and health status. +General Audio: + Audio Caption: Summarize the main content of the audio. + Sound Scene Tagging: Utilize one keyword to convey the audio's content or the associated scene. +''' +task_prompt = "\n" +audio_input, _ = librosa.load('xxx.wav', sr=16000, mono=True) + +msgs = [{'role': 'user', 'content': [task_prompt,audio_input]}] + +res = model.chat( + msgs=msgs, + tokenizer=tokenizer, + sampling=True, + max_new_tokens=128, + use_tts_template=True, + generate_audio=True, + temperature=0.3, + output_audio_path='result.wav', +) +print(res) +``` +```python +''' +Speech Generation Task Prompt: + Human Instruction-to-Speech: see https://voxinstruct.github.io/VoxInstruct/ + Example: + # 在新闻中，一个年轻男性兴致勃勃地说：“祝福亲爱的祖国母亲美丽富强！”他用低音调和低音量，慢慢地说出了这句话。 + # Delighting in a surprised tone, an adult male with low pitch and low volume comments:"One even gave my little dog a biscuit" This dialogue takes place at a leisurely pace, delivering a sense of excitement and surprise in the context. + + Voice Cloning or Voice Creation: With this mode, model will act like a TTS model. +''' +# Human Instruction-to-Speech: +task_prompt = '' #Try to make some Human Instruction-to-Speech prompt +msgs = [{'role': 'user', 'content': [task_prompt]}] # you can try to use the same audio question. (Voice Creation) + +# Voice Cloning mode: With this mode, model will act like a TTS model. +# sys_prompt = model.get_sys_prompt(ref_audio=ref_audio, mode='voice_cloning', language='en') +# text_prompt = f"Please read the text below." +# user_question = {'role': 'user', 'content': [text_prompt, "content that you want to read"]} # using same voice in sys_prompt to read the text. (Voice Cloning) +# user_question = {'role': 'user', 'content': [text_prompt, librosa.load('xxx.wav', sr=16000, mono=True)[0]]} # using same voice in sys_prompt to read 'xxx.wav'. (Voice Conversion) + +msgs = [sys_prompt, user_question] +res = model.chat( + msgs=msgs, + tokenizer=tokenizer, + sampling=True, + max_new_tokens=128, + use_tts_template=True, + generate_audio=True, + temperature=0.3, + output_audio_path='result.wav', +) + + +``` + +
+ +#### Multimodal Live Streaming +
+ Click to view Python code running MiniCPM-o 2.6 with chat inference. + +```python +import math +import numpy as np +from PIL import Image +from moviepy.editor import VideoFileClip +import tempfile +import librosa +import soundfile as sf + +## make sure The model has been initialized and `model.init_tts()` has been executed + +def get_video_chunk_content(video_path, flatten=True): + video = VideoFileClip(video_path) + print('video_duration:', video.duration) + + with tempfile.NamedTemporaryFile(suffix=".wav", delete=True) as temp_audio_file: + temp_audio_file_path = temp_audio_file.name + video.audio.write_audiofile(temp_audio_file_path, codec="pcm_s16le", fps=16000) + audio_np, sr = librosa.load(temp_audio_file_path, sr=16000, mono=True) + num_units = math.ceil(video.duration) + + # 1 frame + 1s audio chunk + contents= [] + for i in range(num_units): + frame = video.get_frame(i+1) + image = Image.fromarray((frame).astype(np.uint8)) + audio = audio_np[sr*i:sr*(i+1)] + if flatten: + contents.extend(["", image, audio]) + else: + contents.append(["", image, audio]) + + return contents + +video_path="/path/to/video" +sys_msg = model.get_sys_prompt(mode='omni', language='en') +# if use voice clone prompt, please set ref_audio +# ref_audio_path = '/path/to/ref_audio' +# ref_audio, _ = librosa.load(ref_audio_path, sr=16000, mono=True) +# sys_msg = model.get_sys_prompt(ref_audio=ref_audio, mode='omni', language='en') + +contents = get_video_chunk_content(video_path) +msg = {"role":"user", "content": contents} +msgs = [sys_msg, msg] + +# please set generate_audio=True and output_audio_path to save the tts result +generate_audio = True +output_audio_path = 'output.wav' + +res = model.chat( + msgs=msgs, + tokenizer=tokenizer, + sampling=True, + temperature=0.5, + max_new_tokens=4096, + omni_input=True, # please set omni_input=True when omni inference + use_tts_template=True, + generate_audio=generate_audio, + output_audio_path=output_audio_path, + max_slice_nums=1, + use_image_id=False, + return_dict=True +) +print(res) +``` +
+ +
+ Click to view Python code running MiniCPM-o 2.6 with streaming inference. + +Note: The streaming inference has a slight performance degradation because the audio encoding is not global. +```python +# a new conversation need reset session first, it will reset the kv-cache +model.reset_session() + +contents = get_video_chunk_content(video_path, flatten=False) +session_id = '123' +generate_audio = True + +# 1. prefill system prompt +res = model.streaming_prefill( + session_id=session_id, + msgs=[sys_msg], + tokenizer=tokenizer +) + +# 2. prefill video/audio chunks +for content in contents: + msgs = [{"role":"user", "content": content}] + res = model.streaming_prefill( + session_id=session_id, + msgs=msgs, + tokenizer=tokenizer + ) + +# 3. generate +res = model.streaming_generate( + session_id=session_id, + tokenizer=tokenizer, + temperature=0.5, + generate_audio=generate_audio +) + +audios = [] +text = "" + +if generate_audio: + for r in res: + audio_wav = r.audio_wav + sampling_rate = r.sampling_rate + txt = r.text + + audios.append(audio_wav) + text += txt + + res = np.concatenate(audios) + sf.write("output.wav", res, samplerate=sampling_rate) + print("text:", text) + print("audio saved to output.wav") +else: + for r in res: + text += r['text'] + print("text:", text) +``` + +
+ ### Inference on Multiple GPUs You can run MiniCPM-Llama3-V 2.5 on multiple low VRAM GPUs (12 GB or 16 GB) by distributing the model's layers across multiple GPUs. Please refer to this [tutorial](https://github.com/OpenBMB/MiniCPM-V/blob/main/docs/inference_on_multiple_gpus.md) for detailed instructions on how to load the model and inference using multiple low VRAM GPUs. @@ -1511,108 +2367,129 @@ PYTORCH_ENABLE_MPS_FALLBACK=1 python test.py ### Deployment on Mobile Phone MiniCPM-V 2.0 can be deployed on mobile phones with Android operating systems. 🚀 Click [MiniCPM-V 2.0](https://github.com/OpenBMB/mlc-MiniCPM) to install apk. -### Inference with llama.cpp -MiniCPM-V 2.6 can run with llama.cpp now! See [our fork of llama.cpp](https://github.com/OpenBMB/llama.cpp/tree/minicpmv-main/examples/llava/README-minicpmv2.6.md) for more detail. This implementation supports smooth inference of 16~18 token/s on iPad (test environment：iPad Pro + M4). +### Efficient Inference with llama.cpp, ollama, vLLM -### Inference with ollama -MiniCPM-V 2.6 can run with ollama now! See [our fork of ollama](https://github.com/OpenBMB/ollama/blob/minicpm-v2.6/examples/minicpm-v2.6/README.md) for more detail. This implementation supports smooth inference of 16~18 token/s on iPad (test environment：iPad Pro + M4). +See [our fork of llama.cpp](https://github.com/OpenBMB/llama.cpp/tree/minicpmv-main/examples/llava/README-minicpmv2.6.md) for more detail. This implementation supports smooth inference of 16~18 token/s on iPad (test environment：iPad Pro + M4). + +See [our fork of ollama](https://github.com/OpenBMB/ollama/blob/minicpm-v2.6/examples/minicpm-v2.6/README.md) for more detail. This implementation supports smooth inference of 16~18 token/s on iPad (test environment：iPad Pro + M4). -### Inference with vLLM
- vLLM now officially supports MiniCPM-V 2.6, MiniCPM-Llama3-V 2.5 and MiniCPM-V 2.0, Click to see. + vLLM now officially supports MiniCPM-V 2.6, MiniCPM-Llama3-V 2.5 and MiniCPM-V 2.0. And you can use our fork to run MiniCPM-o 2.6 for now. Click to see. -1. Install vLLM(>=0.5.4): -```shell -pip install vllm -``` -2. Install timm: (optional, MiniCPM-V 2.0 need timm) -```shell -pip install timm==0.9.10 -``` -3. Run the example(for image): -```python -from transformers import AutoTokenizer -from PIL import Image -from vllm import LLM, SamplingParams +1. For MiniCPM-o 2.6 + 1. Clone our fork of vLLM: + ```shell + git clone https://github.com/OpenBMB/vllm.git + cd vllm + git checkout minicpmo + ``` + 2. Install vLLM from source: + ```shell + VLLM_USE_PRECOMPILED=1 pip install --editable . + ``` + 3. Run MiniCPM-o 2.6 in the same way as the previous models (shown in the following example). -MODEL_NAME = "openbmb/MiniCPM-V-2_6" -# Also available for previous models -# MODEL_NAME = "openbmb/MiniCPM-Llama3-V-2_5" -# MODEL_NAME = "HwwwH/MiniCPM-V-2" +2. For previous MiniCPM-V models + 1. Install vLLM(>=0.5.4): + ```shell + pip install vllm + ``` + 2. Install timm: (optional, MiniCPM-V 2.0 need timm) + ```shell + pip install timm==0.9.10 + ``` + 3. Run the example(for image): + ```python + from transformers import AutoTokenizer + from PIL import Image + from vllm import LLM, SamplingParams -image = Image.open("xxx.png").convert("RGB") -tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME, trust_remote_code=True) -llm = LLM( - model=MODEL_NAME, - trust_remote_code=True, - gpu_memory_utilization=1, - max_model_len=2048 -) + MODEL_NAME = "openbmb/MiniCPM-V-2_6" + # MODEL_NAME = "openbmb/MiniCPM-O-2_6" + # Also available for previous models + # MODEL_NAME = "openbmb/MiniCPM-Llama3-V-2_5" + # MODEL_NAME = "HwwwH/MiniCPM-V-2" -messages = [{ - "role": - "user", - "content": - # Number of images - "(./)" + \ - "\nWhat is the content of this image?" -}] -prompt = tokenizer.apply_chat_template( - messages, - tokenize=False, - add_generation_prompt=True -) + image = Image.open("xxx.png").convert("RGB") + tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME, trust_remote_code=True) + llm = LLM( + model=MODEL_NAME, + trust_remote_code=True, + gpu_memory_utilization=1, + max_model_len=2048 + ) -# Single Inference -inputs = { - "prompt": prompt, - "multi_modal_data": { - "image": image - # Multi images, the number of images should be equal to that of `(./)` - # "image": [image, image] - }, -} -# Batch Inference -# inputs = [{ -# "prompt": prompt, -# "multi_modal_data": { -# "image": image -# }, -# } for _ in 2] + messages = [{ + "role": + "user", + "content": + # Number of images + "(./)" + \ + "\nWhat is the content of this image?" + }] + prompt = tokenizer.apply_chat_template( + messages, + tokenize=False, + add_generation_prompt=True + ) + + # Single Inference + inputs = { + "prompt": prompt, + "multi_modal_data": { + "image": image + # Multi images, the number of images should be equal to that of `(./)` + # "image": [image, image] + }, + } + # Batch Inference + # inputs = [{ + # "prompt": prompt, + # "multi_modal_data": { + # "image": image + # }, + # } for _ in 2] -# 2.6 -stop_tokens = ['<|im_end|>', '<|endoftext|>'] -stop_token_ids = [tokenizer.convert_tokens_to_ids(i) for i in stop_tokens] -# 2.0 -# stop_token_ids = [tokenizer.eos_id] -# 2.5 -# stop_token_ids = [tokenizer.eos_id, tokenizer.eot_id] + # 2.6 + stop_tokens = ['<|im_end|>', '<|endoftext|>'] + stop_token_ids = [tokenizer.convert_tokens_to_ids(i) for i in stop_tokens] + # 2.0 + # stop_token_ids = [tokenizer.eos_id] + # 2.5 + # stop_token_ids = [tokenizer.eos_id, tokenizer.eot_id] -sampling_params = SamplingParams( - stop_token_ids=stop_token_ids, - use_beam_search=True, - temperature=0, - best_of=3, - max_tokens=1024 -) + sampling_params = SamplingParams( + stop_token_ids=stop_token_ids, + use_beam_search=True, + temperature=0, + best_of=3, + max_tokens=1024 + ) -outputs = llm.generate(inputs, sampling_params=sampling_params) + outputs = llm.generate(inputs, sampling_params=sampling_params) -print(outputs[0].outputs[0].text) -``` -4. click [here](https://modelbest.feishu.cn/wiki/C2BWw4ZP0iCDy7kkCPCcX2BHnOf?from=from_copylink) if you want to use it with *video*, or get more details about `vLLM`. -
+ print(outputs[0].outputs[0].text) + ``` + 4. click [here](https://modelbest.feishu.cn/wiki/C2BWw4ZP0iCDy7kkCPCcX2BHnOf?from=from_copylink) if you want to use it with *video*, or get more details about `vLLM`. +
## Fine-tuning ### Simple Fine-tuning -We support simple fine-tuning with Hugging Face for MiniCPM-V 2.0 and MiniCPM-Llama3-V 2.5. +We support simple fine-tuning with Hugging Face for MiniCPM-o 2.6, MiniCPM-V 2.6, MiniCPM-Llama3-V 2.5 and MiniCPM-V 2.0. [Reference Document](./finetune/readme.md) +### With LLaMA-Factory + +We support fine-tuning MiniCPM-o-2.6 and MiniCPM-V 2.6 with the LLaMA-Factory framework. LLaMA-Factory provides a solution for flexibly customizing the fine-tuning (Lora/Full/Qlora) of 200+ LLMs without the need for coding through the built-in web UI LLaMABoard. It supports various training methods like sft/ppo/dpo/kto and advanced algorithms like Galore/BAdam/LLaMA-Pro/Pissa/LongLoRA. + +Best Practices: [MiniCPM-V-2.6 | MiniCPM-o-2.6](./docs/llamafactory_train.md). + + ### With the SWIFT Framework We now support MiniCPM-V series fine-tuning with the SWIFT framework. SWIFT supports training, inference, evaluation and deployment of nearly 200 LLMs and MLLMs . It supports the lightweight training solutions provided by PEFT and a complete Adapters Library including techniques such as NEFTune, LoRA+ and LLaMA-PRO. @@ -1622,20 +2499,25 @@ Best Practices：[MiniCPM-V 1.0](https://github.com/modelscope/swift/blob/main/d ## FAQs Click here to view the [FAQs](./docs/faqs.md) +## Limitations +As an experimental trial, we find MiniCPM-o 2.6 has notable limitations worth further investigation and improvement. +- **Unstable speech output.** The speech generation can be flawed with noisy background and unmeaningful sound. +- **Repeated response.** The model tends to repeat its response when encounting similar consecutive user queries. +- **High-latency on Web Demo.** Users may experience unusual high-latency when using web demo hosted on overseas servers. We recommend deploying the demo locally or with good network connections. + ## Model License * This repository is released under the [Apache-2.0](https://github.com/OpenBMB/MiniCPM/blob/main/LICENSE) License. -* The usage of MiniCPM-V model weights must strictly follow [MiniCPM Model License.md](https://github.com/OpenBMB/MiniCPM/blob/main/MiniCPM%20Model%20License.md). +* The usage of MiniCPM-o/V model weights must strictly follow [MiniCPM Model License.md](https://github.com/OpenBMB/MiniCPM/blob/main/MiniCPM%20Model%20License.md). -* The models and weights of MiniCPM are completely free for academic research. after filling out a ["questionnaire"](https://modelbest.feishu.cn/share/base/form/shrcnpV5ZT9EJ6xYjh3Kx0J6v8g) for registration, are also available for free commercial use. - +* The models and weights of MiniCPM are completely free for academic research. after filling out a ["questionnaire"](https://modelbest.feishu.cn/share/base/form/shrcnpV5ZT9EJ6xYjh3Kx0J6v8g) for registration, are also available for free commercial use. ## Statement -As LMMs, MiniCPM-V models (including OmniLMM) generate contents by learning a large amount of multimodal corpora, but they cannot comprehend, express personal opinions or make value judgement. Anything generated by MiniCPM-V models does not represent the views and positions of the model developers +As MLLMs, MiniCPM-o/V models generate contents by learning a large amount of multimodal corpora, but they cannot comprehend, express personal opinions or make value judgement. Anything generated by MiniCPM-o/V models does not represent the views and positions of the model developers -We will not be liable for any problems arising from the use of MiniCPM-V models, including but not limited to data security issues, risk of public opinion, or any risks and problems arising from the misdirection, misuse, dissemination or misuse of the model. +We will not be liable for any problems arising from the use of MiniCPM-o/V models, including but not limited to data security issues, risk of public opinion, or any risks and problems arising from the misdirection, misuse, dissemination or misuse of the model. ## Institutions @@ -1644,7 +2526,6 @@ This project is developed by the following institutions: - [THUNLP](https://nlp.csai.tsinghua.edu.cn/) - [ModelBest](https://modelbest.cn/) -- [Zhihu](https://www.zhihu.com/ ) ## 🌟 Star History @@ -1676,14 +2557,14 @@ This project is developed by the following institutions: ## Key Techniques and Other Multimodal Projects -👏 Welcome to explore key techniques of MiniCPM-V and other multimodal projects of our team: +👏 Welcome to explore key techniques of MiniCPM-o/V and other multimodal projects of our team: [VisCPM](https://github.com/OpenBMB/VisCPM/tree/main) | [RLHF-V](https://github.com/RLHF-V/RLHF-V) | [LLaVA-UHD](https://github.com/thunlp/LLaVA-UHD) | [RLAIF-V](https://github.com/RLHF-V/RLAIF-V) ## Citation -If you find our model/code/paper helpful, please consider cite our papers 📝 and star us ⭐️！ +If you find our model/code/paper helpful, please consider citing our papers 📝 and staring us ⭐️！ ```bib @article{yao2024minicpm, diff --git a/README_en.md b/README_en.md deleted file mode 100644 index e8e6c29..0000000 --- a/README_en.md +++ /dev/null @@ -1,1695 +0,0 @@ -
- - - -**A GPT-4V Level MLLM for Single Image, Multi Image and Video on Your Phone** - - [中文](./README_zh.md) | - English - -Join our 💬 WeChat | View MiniCPM-V 📖 best practices - - -

- MiniCPM-V 2.6 🤗 🤖 | MiniCPM-Llama3-V 2.5 🤗 🤖 | - MiniCPM-Llama3-V 2.5 Technical Report -

- -
- - -**MiniCPM-V** is a series of end-side multimodal LLMs (MLLMs) designed for vision-language understanding. The models take image, video and text as inputs and provide high-quality text outputs. Since February 2024, we have released 5 versions of the model, aiming to achieve **strong performance and efficient deployment**. The most notable models in this series currently include: - -- **MiniCPM-V 2.6**: 🔥🔥🔥 The latest and most capable model in the MiniCPM-V series. With a total of 8B parameters, the model **surpasses GPT-4V in single image, multi-image and video understanding**. It outperforms **GPT-4o mini, Gemini 1.5 Pro and Claude 3.5 Sonnet** in single image understanding, and advances MiniCPM-Llama3-V 2.5's features such as strong OCR capability, trustworthy behavior, multilingual support, and end-side deployment. Due to its superior token density, MiniCPM-V 2.6 can for the first time support real-time video understanding on end-side devices such as iPad. - -- **MiniCPM-V 2.0**: The lightest model in the MiniCPM-V series. With 2B parameters, it surpasses larger models such as Yi-VL 34B, CogVLM-Chat 17B, and Qwen-VL-Chat 10B in overall performance. It can accept image inputs of any aspect ratio and up to 1.8 million pixels (e.g., 1344x1344), achieving comparable performance with Gemini Pro in understanding scene-text and matches GPT-4V in low hallucination rates. - - -## News - -#### 📌 Pinned - -* [2024.08.17] 🚀🚀🚀 MiniCPM-V 2.6 is now fully supported by [official](https://github.com/ggerganov/llama.cpp) llama.cpp! GGUF models of various sizes are available [here](https://huggingface.co/openbmb/MiniCPM-V-2_6-gguf). -* [2024.08.15] We now also support multi-image SFT. For more details, please refer to the [document](https://github.com/OpenBMB/MiniCPM-V/tree/main/finetune). -* [2024.08.14] MiniCPM-V 2.6 now also supports [fine-tuning](https://github.com/modelscope/ms-swift/issues/1613) with the SWIFT framework! -* [2024.08.10] 🚀🚀🚀 MiniCPM-Llama3-V 2.5 is now fully supported by [official](https://github.com/ggerganov/llama.cpp) llama.cpp! GGUF models of various sizes are available [here](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5-gguf). -* [2024.08.06] 🔥🔥🔥 We open-source MiniCPM-V 2.6, which outperforms GPT-4V on single image, multi-image and video understanding. It advances popular features of MiniCPM-Llama3-V 2.5, and can support real-time video understanding on iPad. Try it now! -* [2024.08.03] MiniCPM-Llama3-V 2.5 technical report is released! See [here](https://arxiv.org/abs/2408.01800). -* [2024.07.19] MiniCPM-Llama3-V 2.5 supports vLLM now! See [here](#inference-with-vllm). -* [2024.05.28] 💫 We now support LoRA fine-tuning for MiniCPM-Llama3-V 2.5, using only 2 V100 GPUs! See more statistics [here](https://github.com/OpenBMB/MiniCPM-V/tree/main/finetune#model-fine-tuning-memory-usage-statistics). -* [2024.05.23] 🔍 We've released a comprehensive comparison between Phi-3-vision-128k-instruct and MiniCPM-Llama3-V 2.5, including benchmarks evaluations, multilingual capabilities, and inference efficiency 🌟📊🌍🚀. Click [here](./docs/compare_with_phi-3_vision.md) to view more details. -* [2024.05.23] 🔥🔥🔥 MiniCPM-V tops GitHub Trending and Hugging Face Trending! Our demo, recommended by Hugging Face Gradio’s official account, is available [here](https://huggingface.co/spaces/openbmb/MiniCPM-Llama3-V-2_5). Come and try it out! - -
- -
-Click to view more news. - -* [2024.06.03] Now, you can run MiniCPM-Llama3-V 2.5 on multiple low VRAM GPUs(12 GB or 16 GB) by distributing the model's layers across multiple GPUs. For more details, Check this [link](https://github.com/OpenBMB/MiniCPM-V/blob/main/docs/inference_on_multiple_gpus.md). -* [2024.05.28] 🚀🚀🚀 MiniCPM-Llama3-V 2.5 now fully supports its feature in llama.cpp and ollama! Please pull the latest code **of our provided forks** ([llama.cpp](https://github.com/OpenBMB/llama.cpp/blob/minicpm-v2.5/examples/minicpmv/README.md), [ollama](https://github.com/OpenBMB/ollama/tree/minicpm-v2.5/examples/minicpm-v2.5)). GGUF models in various sizes are available [here](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5-gguf/tree/main). MiniCPM-Llama3-V 2.5 series is **not supported by the official repositories yet**, and we are working hard to merge PRs. Please stay tuned! -* [2024.05.25] MiniCPM-Llama3-V 2.5 now supports streaming outputs and customized system prompts. Try it [here](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5#usage)! -* [2024.05.24] We release the MiniCPM-Llama3-V 2.5 [gguf](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5-gguf), which supports [llama.cpp](#inference-with-llamacpp) inference and provides a 6~8 token/s smooth decoding on mobile phones. Try it now! -* [2024.05.20] We open-soure MiniCPM-Llama3-V 2.5, it has improved OCR capability and supports 30+ languages, representing the first end-side MLLM achieving GPT-4V level performance! We provide [efficient inference](#deployment-on-mobile-phone) and [simple fine-tuning](./finetune/readme.md). Try it now! -* [2024.04.23] MiniCPM-V-2.0 supports vLLM now! Click [here](#inference-with-vllm) to view more details. -* [2024.04.18] We create a HuggingFace Space to host the demo of MiniCPM-V 2.0 at [here](https://huggingface.co/spaces/openbmb/MiniCPM-V-2)! -* [2024.04.17] MiniCPM-V-2.0 supports deploying [WebUI Demo](#webui-demo) now! -* [2024.04.15] MiniCPM-V-2.0 now also supports [fine-tuning](https://github.com/modelscope/swift/blob/main/docs/source/Multi-Modal/minicpm-v-2最佳实践.md) with the SWIFT framework! -* [2024.04.12] We open-source MiniCPM-V 2.0, which achieves comparable performance with Gemini Pro in understanding scene text and outperforms strong Qwen-VL-Chat 9.6B and Yi-VL 34B on OpenCompass, a comprehensive evaluation over 11 popular benchmarks. Click here to view the MiniCPM-V 2.0 technical blog. -* [2024.03.14] MiniCPM-V now supports [fine-tuning](https://github.com/modelscope/swift/blob/main/docs/source/Multi-Modal/minicpm-v最佳实践.md) with the SWIFT framework. Thanks to [Jintao](https://github.com/Jintao-Huang) for the contribution！ -* [2024.03.01] MiniCPM-V now can be deployed on Mac! -* [2024.02.01] We open-source MiniCPM-V and OmniLMM-12B, which support efficient end-side deployment and powerful multimodal capabilities correspondingly. -
- - -## Contents - - -- [MiniCPM-V 2.6](#minicpm-v-26) -- [MiniCPM-Llama3-V 2.5](#minicpm-llama3-v-25) -- [MiniCPM-V 2.0](#minicpm-v-20) -- [Chat with Our Demo on Gradio 🤗](#chat-with-our-demo-on-gradio-) -- [Install](#install) -- [Inference](#inference) - - [Model Zoo](#model-zoo) - - [Multi-turn Conversation](#multi-turn-conversation) - - [Chat with multiple images](#chat-with-multiple-images) - - [In-context few-shot learning](#in-context-few-shot-learning) - - [Chat with video](#chat-with-video) - - [Inference on Multiple GPUs](#inference-on-multiple-gpus) - - [Inference on Mac](#inference-on-mac) - - [Deployment on Mobile Phone](#deployment-on-mobile-phone) - - [Inference with llama.cpp](#inference-with-llamacpp) - - [Inference with ollama](#inference-with-ollama) - - [Inference with vLLM](#inference-with-vllm) -- [Fine-tuning](#fine-tuning) -- [FAQs](#faqs) - - -## MiniCPM-V 2.6 - -**MiniCPM-V 2.6** is the latest and most capable model in the MiniCPM-V series. The model is built on SigLip-400M and Qwen2-7B with a total of 8B parameters. It exhibits a significant performance improvement over MiniCPM-Llama3-V 2.5, and introduces new features for multi-image and video understanding. Notable features of MiniCPM-V 2.6 include: - -- 🔥 **Leading Performance.** - MiniCPM-V 2.6 achieves an average score of 65.2 on the latest version of OpenCompass, a comprehensive evaluation over 8 popular benchmarks. **With only 8B parameters, it surpasses widely used proprietary models like GPT-4o mini, GPT-4V, Gemini 1.5 Pro, and Claude 3.5 Sonnet** for single image understanding. - -- 🖼️ **Multi Image Understanding and In-context Learning.** MiniCPM-V 2.6 can also perform **conversation and reasoning over multiple images**. It achieves **state-of-the-art performance** on popular multi-image benchmarks such as Mantis-Eval, BLINK, Mathverse mv and Sciverse mv, and also shows promising in-context learning capability. - -- 🎬 **Video Understanding.** MiniCPM-V 2.6 can also **accept video inputs**, performing conversation and providing dense captions for spatial-temporal information. It outperforms **GPT-4V, Claude 3.5 Sonnet and LLaVA-NeXT-Video-34B** on Video-MME with/without subtitles. - -- 💪 **Strong OCR Capability and Others.** - MiniCPM-V 2.6 can process images with any aspect ratio and up to 1.8 million pixels (e.g., 1344x1344). It achieves **state-of-the-art performance on OCRBench, surpassing proprietary models such as GPT-4o, GPT-4V, and Gemini 1.5 Pro**. - Based on the the latest [RLAIF-V](https://github.com/RLHF-V/RLAIF-V/) and [VisCPM](https://github.com/OpenBMB/VisCPM) techniques, it features **trustworthy behaviors**, with significantly lower hallucination rates than GPT-4o and GPT-4V on Object HalBench, and supports **multilingual capabilities** on English, Chinese, German, French, Italian, Korean, etc. - - -- 🚀 **Superior Efficiency.** - In addition to its friendly size, MiniCPM-V 2.6 also shows **state-of-the-art token density** (i.e., number of pixels encoded into each visual token). **It produces only 640 tokens when processing a 1.8M pixel image, which is 75% fewer than most models**. This directly improves the inference speed, first-token latency, memory usage, and power consumption. As a result, MiniCPM-V 2.6 can efficiently support **real-time video understanding** on end-side devices such as iPad. - -- 💫 **Easy Usage.** -MiniCPM-V 2.6 can be easily used in various ways: (1) [llama.cpp](https://github.com/OpenBMB/llama.cpp/blob/minicpmv-main/examples/llava/README-minicpmv2.6.md) and [ollama](https://github.com/OpenBMB/ollama/blob/minicpm-v2.6/examples/minicpm-v2.6/README.md) support for efficient CPU inference on local devices, (2) [int4](https://huggingface.co/openbmb/MiniCPM-V-2_6-int4) and [GGUF](https://huggingface.co/openbmb/MiniCPM-V-2_6-gguf) format quantized models in 16 sizes, (3) [vLLM](#inference-with-vllm) support for high-throughput and memory-efficient inference, (4) fine-tuning on new domains and tasks, (5) quick local WebUI demo setup with [Gradio](#chat-with-our-demo-on-gradio), and (6) online web [demo](http://120.92.209.146:8887/). - -### Evaluation -
- -
- -
-Click to view single image results on OpenCompass, MME, MMVet, OCRBench, MMMU, MathVista, MMB, AI2D, TextVQA, DocVQA, HallusionBench, Object HalBench. -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
ModelSizeToken Density+OpenCompassMMEMMVetOCRBenchMMMU valMathVista miniMMB1.1 testAI2DTextVQA valDocVQA testHallusionBenchObject HalBench
Proprietary
GPT-4o-108869.92328.769.173669.261.382.284.6-92.855.017.6
Claude 3.5 Sonnet-75067.91920.066.078865.961.678.580.2-95.249.913.8
Gemini 1.5 Pro--64.42110.664.075460.657.773.979.173.586.545.6-
GPT-4o mini-108864.12003.466.978560.052.476.077.8--46.112.4
GPT-4V-108863.52070.267.565661.754.779.878.678.087.243.914.2
Step-1V--59.52206.463.362549.944.878.079.271.6-48.4-
Qwen-VL-Max-78458.32281.761.868452.043.474.675.779.593.141.213.4
Open-source
LLaVA-NeXT-Yi-34B34B15755.02006.550.757448.840.477.878.969.3-34.812.6
Mini-Gemini-HD-34B34B157-2141.059.351848.043.3-80.574.178.9--
Cambrian-34B34B182058.32049.953.259150.450.377.879.576.775.541.614.7
GLM-4V-9B13B78459.12018.858.077646.951.167.971.2--45.0-
InternVL2-8B8B70664.12215.154.379451.258.379.483.677.491.645.021.3
MiniCPM-Llama-V 2.58B188258.82024.652.872545.854.372.078.476.684.842.410.3
MiniCPM-V 2.68B282265.22348.4*60.0852*49.8*60.678.082.180.190.848.1*8.2
- -
-* We evaluate this benchmark using chain-of-thought prompting. Specifically, for MME, we used this technique only for the Cognition set. - -+ Token Density: number of pixels encoded into each visual token at maximum resolution, i.e., # pixels at maximum resolution / # visual tokens. - -Note: For proprietary models, we calculate token density based on the image encoding charging strategy defined in the official API documentation, which provides an upper-bound estimation. - -
- - -
-Click to view multi-image results on Mantis Eval, BLINK, Mathverse mv, Sciverse mv, MIRB. -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
ModelSizeMantis EvalBLINK valMathverse mvSciverse mvMIRB
Proprietary
GPT-4V-62.754.660.366.953.1
LLaVA-NeXT-Interleave-14B14B66.452.632.730.2-
Open-source
Emu2-Chat37B37.836.2-27.2-
CogVLM17B45.241.1---
VPG-C7B52.443.124.323.1-
VILA 8B8B51.239.3-36.5-
InternLM-XComposer-2.58B53.1*48.932.1*-42.5
InternVL2-8B8B59.0*50.930.5*34.4*56.9*
MiniCPM-V 2.68B69.153.084.974.953.8
- -
-* We evaluate the officially released checkpoint by ourselves. -
- -
-Click to view video results on Video-MME and Video-ChatGPT. -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
ModelSizeVideo-MMEVideo-ChatGPT
w/o subsw subsCorrectnessDetailContextTemporalConsistency
Proprietary
Claude 3.5 Sonnet-60.062.9-----
GPT-4V-59.963.3-----
Open-source
LLaVA-NeXT-7B7B--3.393.293.922.603.12
LLaVA-NeXT-34B34B--3.293.233.832.513.47
CogVLM2-Video12B--3.493.463.232.983.64
LongVA7B52.454.33.053.093.772.443.64
InternVL2-8B8B54.056.9-----
InternLM-XComposer-2.58B55.8------
LLaVA-NeXT-Video32B60.263.03.483.373.952.643.28
MiniCPM-V 2.68B60.963.63.593.283.932.733.62
-
-
- - -
-Click to view few-shot results on TextVQA, VizWiz, VQAv2, OK-VQA. -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
ModelSizeShotTextVQA valVizWiz test-devVQAv2 test-devOK-VQA val
Flamingo80B0*35.031.656.340.6
436.539.663.157.4
837.344.865.657.5
IDEFICS80B0*30.936.060.045.2
434.340.463.652.4
835.746.164.855.1
OmniCorpus7B0*43.049.863.245.5
445.451.364.546.5
845.652.264.746.6
Emu237B026.440.433.526.7
448.254.667.053.2
849.354.767.854.1
MM130B026.240.448.926.7
849.354.770.954.1
MiniCPM-V 2.6+8B043.933.845.423.9
463.660.565.550.1
864.663.468.251.4
- - -
-* denotes zero image shot and two additional text shots following Flamingo. - -+ We evaluate the pretraining ckpt without SFT. -
- -### Examples - -
- Bike - Menu - Code - Mem - medal -
-
- Click to view more cases. -
- elec - Menu -
-
- -We deploy MiniCPM-V 2.6 on end devices. The demo video is the raw screen recording on a iPad Pro without edition. - - -

- -      - -

-
- - -

- -      - -

-
- - -

- - -

-
- -## MiniCPM-Llama3-V 2.5 - -
-Click to view more details of MiniCPM-Llama3-V 2.5 - -**MiniCPM-Llama3-V 2.5** is the latest model in the MiniCPM-V series. The model is built on SigLip-400M and Llama3-8B-Instruct with a total of 8B parameters. It exhibits a significant performance improvement over MiniCPM-V 2.0. Notable features of MiniCPM-Llama3-V 2.5 include: - -- 🔥 **Leading Performance.** - MiniCPM-Llama3-V 2.5 has achieved an average score of 65.1 on OpenCompass, a comprehensive evaluation over 11 popular benchmarks. **With only 8B parameters, it surpasses widely used proprietary models like GPT-4V-1106, Gemini Pro, Claude 3 and Qwen-VL-Max** and greatly outperforms other Llama 3-based MLLMs. - -- 💪 **Strong OCR Capabilities.** - MiniCPM-Llama3-V 2.5 can process images with any aspect ratio and up to 1.8 million pixels (e.g., 1344x1344), achieving a **700+ score on OCRBench, surpassing proprietary models such as GPT-4o, GPT-4V-0409, Qwen-VL-Max and Gemini Pro**. Based on recent user feedback, MiniCPM-Llama3-V 2.5 has now enhanced full-text OCR extraction, table-to-markdown conversion, and other high-utility capabilities, and has further strengthened its instruction-following and complex reasoning abilities, enhancing multimodal interaction experiences. - -- 🏆 **Trustworthy Behavior.** - Leveraging the latest [RLAIF-V](https://github.com/RLHF-V/RLAIF-V/) method (the newest technique in the [RLHF-V](https://github.com/RLHF-V) [CVPR'24] series), MiniCPM-Llama3-V 2.5 exhibits more trustworthy behavior. It achieves a **10.3%** hallucination rate on Object HalBench, lower than GPT-4V-1106 (13.6%), achieving the best-level performance within the open-source community. [Data released](https://huggingface.co/datasets/openbmb/RLAIF-V-Dataset). - -- 🌏 **Multilingual Support.** - Thanks to the strong multilingual capabilities of Llama 3 and the cross-lingual generalization technique from [VisCPM](https://github.com/OpenBMB/VisCPM), MiniCPM-Llama3-V 2.5 extends its bilingual (Chinese-English) multimodal capabilities to **over 30 languages including German, French, Spanish, Italian, Korean etc.** [All Supported Languages](./assets/minicpm-llama-v-2-5_languages.md). - -- 🚀 **Efficient Deployment.** - MiniCPM-Llama3-V 2.5 systematically employs **model quantization, CPU optimizations, NPU optimizations and compilation optimizations**, achieving high-efficiency deployment on end-side devices. For mobile phones with Qualcomm chips, we have integrated the NPU acceleration framework QNN into llama.cpp for the first time. After systematic optimization, MiniCPM-Llama3-V 2.5 has realized a **150x acceleration in end-side MLLM image encoding** and a **3x speedup in language decoding**. - -- 💫 **Easy Usage.** -MiniCPM-Llama3-V 2.5 can be easily used in various ways: (1) [llama.cpp](https://github.com/OpenBMB/llama.cpp/blob/minicpm-v2.5/examples/minicpmv/README.md) and [ollama](https://github.com/OpenBMB/ollama/tree/minicpm-v2.5/examples/minicpm-v2.5) support for efficient CPU inference on local devices, (2) [GGUF](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5-gguf) format quantized models in 16 sizes, (3) efficient [LoRA](https://github.com/OpenBMB/MiniCPM-V/tree/main/finetune#lora-finetuning) fine-tuning with only 2 V100 GPUs, (4) [streaming output](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5#usage), (5) quick local WebUI demo setup with [Gradio](https://github.com/OpenBMB/MiniCPM-V/blob/main/web_demo_2.5.py) and [Streamlit](https://github.com/OpenBMB/MiniCPM-V/blob/main/web_demo_streamlit-2_5.py), and (6) interactive demos on [HuggingFace Spaces](https://huggingface.co/spaces/openbmb/MiniCPM-Llama3-V-2_5). - -### Evaluation - -
- -
-
-Click to view results on TextVQA, DocVQA, OCRBench, OpenCompass, MME, MMBench, MMMU, MathVista, LLaVA Bench, RealWorld QA, Object HalBench. -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
ModelSizeOCRBenchTextVQA valDocVQA testOpen-CompassMMEMMB test (en)MMB test (cn)MMMU valMath-VistaLLaVA BenchRealWorld QAObject HalBench
Proprietary
Gemini Pro-68074.688.162.92148.973.674.348.945.879.960.4-
GPT-4V (2023.11.06)-64578.088.463.51771.577.074.453.847.893.163.086.4
Open-source
Mini-Gemini2.2B-56.234.2*-1653.0--31.7----
Qwen-VL-Chat9.6B48861.562.651.61860.061.856.337.033.867.749.356.2
DeepSeek-VL-7B7.3B43564.7*47.0*54.61765.473.871.438.336.877.854.2-
Yi-VL-34B34B29043.4*16.9*52.22050.272.470.745.130.762.354.879.3
CogVLM-Chat17.4B59070.433.3*54.21736.665.855.937.334.773.960.373.6
TextMonkey9.7B55864.366.7---------
Idefics28.0B-73.074.057.21847.675.768.645.252.249.160.7-
Bunny-LLama-3-8B8.4B---54.31920.377.073.941.331.561.258.8-
LLaVA-NeXT Llama-3-8B8.4B--78.2-1971.5--41.737.580.160.0-
Phi-3-vision-128k-instruct4.2B639*70.9--1537.5*--40.444.564.2*58.8*-
MiniCPM-V 1.02.8B36660.638.247.51650.264.162.638.328.951.351.278.4
MiniCPM-V 2.02.8B60574.171.954.51808.669.166.538.238.769.255.885.5
MiniCPM-Llama3-V 2.58.5B72576.684.865.12024.677.274.245.854.386.763.589.7
- - -
-* We evaluate the officially released checkpoint by ourselves. - -
- -
- -
- Evaluation results of multilingual LLaVA Bench -
- -### Examples - - -

- -

-
- -
- - -## MiniCPM-V 2.0 - -
-Click to view more details of MiniCPM-V 2.0 - - -**MiniCPM-V 2.0** is an efficient version with promising performance for deployment. The model is built based on SigLip-400M and [MiniCPM-2.4B](https://github.com/OpenBMB/MiniCPM/), connected by a perceiver resampler. Our latest version, MiniCPM-V 2.0 has several notable features. - -- 🔥 **State-of-the-art Performance.** - - MiniCPM-V 2.0 achieves **state-of-the-art performance** on multiple benchmarks (including OCRBench, TextVQA, MME, MMB, MathVista, etc) among models under 7B parameters. It even **outperforms strong Qwen-VL-Chat 9.6B, CogVLM-Chat 17.4B, and Yi-VL 34B on OpenCompass, a comprehensive evaluation over 11 popular benchmarks**. Notably, MiniCPM-V 2.0 shows **strong OCR capability**, achieving **comparable performance to Gemini Pro in scene-text understanding**, and **state-of-the-art performance on OCRBench** among open-source models. - -- 🏆 **Trustworthy Behavior.** - - LMMs are known for suffering from hallucination, often generating text not factually grounded in images. MiniCPM-V 2.0 is **the first end-side LMM aligned via multimodal RLHF for trustworthy behavior** (using the recent [RLHF-V](https://rlhf-v.github.io/) [CVPR'24] series technique). This allows the model to **match GPT-4V in preventing hallucinations** on Object HalBench. - -- 🌟 **High-Resolution Images at Any Aspect Raito.** - - MiniCPM-V 2.0 can accept **1.8 million pixels (e.g., 1344x1344) images at any aspect ratio**. This enables better perception of fine-grained visual information such as small objects and optical characters, which is achieved via a recent technique from [LLaVA-UHD](https://arxiv.org/pdf/2403.11703.pdf). - -- ⚡️ **High Efficiency.** - - MiniCPM-V 2.0 can be **efficiently deployed on most GPU cards and personal computers**, and **even on end devices such as mobile phones**. For visual encoding, we compress the image representations into much fewer tokens via a perceiver resampler. This allows MiniCPM-V 2.0 to operate with **favorable memory cost and speed during inference even when dealing with high-resolution images**. - -- 🙌 **Bilingual Support.** - - MiniCPM-V 2.0 **supports strong bilingual multimodal capabilities in both English and Chinese**. This is enabled by generalizing multimodal capabilities across languages, a technique from [VisCPM](https://arxiv.org/abs/2308.12038) [ICLR'24]. - -### Examples - - -

- -

-
- -We deploy MiniCPM-V 2.0 on end devices. The demo video is the raw screen recording on a Xiaomi 14 Pro without edition. - - -

- - -

-
- -
- -## Legacy Models - -| Model | Introduction and Guidance | -|:----------------------|:-------------------:| -| MiniCPM-V 1.0 | [Document](./minicpm_v1.md) | -| OmniLMM-12B | [Document](./omnilmm_en.md) | - - -## Chat with Our Demo on Gradio 🤗 - -We provide online and local demos powered by Hugging Face Gradio , the most popular model deployment framework nowadays. It supports streaming outputs, progress bars, queuing, alerts, and other useful features. - - -### Online Demo - -Click here to try out the online demo of [MiniCPM-V 2.6](http://120.92.209.146:8887/) | [MiniCPM-Llama3-V 2.5](https://huggingface.co/spaces/openbmb/MiniCPM-Llama3-V-2_5) | [MiniCPM-V 2.0](https://huggingface.co/spaces/openbmb/MiniCPM-V-2). - -### Local WebUI Demo - -You can easily build your own local WebUI demo with Gradio using the following commands. - -```shell -pip install -r requirements.txt -``` - -```shell -# For NVIDIA GPUs, run: -python web_demo_2.6.py --device cuda - -``` - - -## Install - -1. Clone this repository and navigate to the source folder - -```bash -git clone https://github.com/OpenBMB/MiniCPM-V.git -cd MiniCPM-V -``` - -2. Create conda environment - -```Shell -conda create -n MiniCPM-V python=3.10 -y -conda activate MiniCPM-V -``` - -3. Install dependencies - -```shell -pip install -r requirements.txt -``` - -## Inference - - -### Model Zoo - -| Model | Device | Memory |          Description | Download | -|:-----------|:--:|:-----------:|:-------------------|:---------------:| -| MiniCPM-V 2.6| GPU | 17 GB | The latest version, achieving state-of-the-art end-side performance for single image, multi-image and video understanding. | [🤗](https://huggingface.co/openbmb/MiniCPM-V-2_6)    [](https://modelscope.cn/models/OpenBMB/MiniCPM-V-2_6) | -| MiniCPM-V 2.6 gguf | CPU | 6 GB | The gguf version, lower memory usage and faster inference. | [🤗](https://huggingface.co/openbmb/MiniCPM-V-2_6-gguf)    [](https://modelscope.cn/models/OpenBMB/MiniCPM-V-2_6-gguf) | -| MiniCPM-V 2.6 int4 | GPU | 7 GB | The int4 quantized version, lower GPU memory usage. | [🤗](https://huggingface.co/openbmb/MiniCPM-V-2_6-int4)    [](https://modelscope.cn/models/OpenBMB/MiniCPM-V-2_6-int4) | -| MiniCPM-Llama3-V 2.5 | GPU | 19 GB | Strong end-side multimodal performance. | [🤗](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5/)    [](https://modelscope.cn/models/OpenBMB/MiniCPM-Llama3-V-2_5) | -| MiniCPM-Llama3-V 2.5 gguf | CPU | 6 GB | The gguf version, lower memory usage and faster inference. | [🤗](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5-gguf)   [](https://modelscope.cn/models/OpenBMB/MiniCPM-Llama3-V-2_5-gguf) | -| MiniCPM-Llama3-V 2.5 int4 | GPU | 8 GB | The int4 quantized version, lower GPU memory usage. | [🤗](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5-int4/)    [](https://modelscope.cn/models/OpenBMB/MiniCPM-Llama3-V-2_5-int4) | -| MiniCPM-V 2.0 | GPU | 8 GB | Light version, balance the performance the computation cost. | [🤗](https://huggingface.co/openbmb/MiniCPM-V-2)    [](https://modelscope.cn/models/OpenBMB/MiniCPM-V-2) | -| MiniCPM-V 1.0 | GPU | 7 GB | Lightest version, achieving the fastest inference. | [🤗](https://huggingface.co/openbmb/MiniCPM-V)    [](https://modelscope.cn/models/OpenBMB/MiniCPM-V) | - -### Multi-turn Conversation - -Please refer to the following codes to run. - -
- -
- - -```python -import torch -from PIL import Image -from transformers import AutoModel, AutoTokenizer - -torch.manual_seed(0) - -model = AutoModel.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True, - attn_implementation='sdpa', torch_dtype=torch.bfloat16) # sdpa or flash_attention_2, no eager -model = model.eval().cuda() -tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True) - -image = Image.open('./assets/airplane.jpeg').convert('RGB') - -# First round chat -question = "Tell me the model of this aircraft." -msgs = [{'role': 'user', 'content': [image, question]}] - -answer = model.chat( - image=None, - msgs=msgs, - tokenizer=tokenizer -) -print(answer) - -# Second round chat -# pass history context of multi-turn conversation -msgs.append({"role": "assistant", "content": [answer]}) -msgs.append({"role": "user", "content": ["Introduce something about Airbus A380."]}) - -answer = model.chat( - image=None, - msgs=msgs, - tokenizer=tokenizer -) -print(answer) -``` - -You will get the following output: - -``` -"The aircraft in the image is an Airbus A380, which can be identified by its large size, double-deck structure, and the distinctive shape of its wings and engines. The A380 is a wide-body aircraft known for being the world's largest passenger airliner, designed for long-haul flights. It has four engines, which are characteristic of large commercial aircraft. The registration number on the aircraft can also provide specific information about the model if looked up in an aviation database." - -"The Airbus A380 is a double-deck, wide-body, four-engine jet airliner made by Airbus. It is the world's largest passenger airliner and is known for its long-haul capabilities. The aircraft was developed to improve efficiency and comfort for passengers traveling over long distances. It has two full-length passenger decks, which can accommodate more passengers than a typical single-aisle airplane. The A380 has been operated by airlines such as Lufthansa, Singapore Airlines, and Emirates, among others. It is widely recognized for its unique design and significant impact on the aviation industry." -``` - -#### Chat with multiple images -
- Click to view Python code running MiniCPM-V 2.6 with multiple images input. - -```python -import torch -from PIL import Image -from transformers import AutoModel, AutoTokenizer - -model = AutoModel.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True, - attn_implementation='sdpa', torch_dtype=torch.bfloat16) # sdpa or flash_attention_2, no eager -model = model.eval().cuda() -tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True) - -image1 = Image.open('image1.jpg').convert('RGB') -image2 = Image.open('image2.jpg').convert('RGB') -question = 'Compare image 1 and image 2, tell me about the differences between image 1 and image 2.' - -msgs = [{'role': 'user', 'content': [image1, image2, question]}] - -answer = model.chat( - image=None, - msgs=msgs, - tokenizer=tokenizer -) -print(answer) -``` -
- -#### In-context few-shot learning -
- Click to view Python code running MiniCPM-V 2.6 with few-shot input. - -```python -import torch -from PIL import Image -from transformers import AutoModel, AutoTokenizer - -model = AutoModel.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True, - attn_implementation='sdpa', torch_dtype=torch.bfloat16) # sdpa or flash_attention_2, no eager -model = model.eval().cuda() -tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True) - -question = "production date" -image1 = Image.open('example1.jpg').convert('RGB') -answer1 = "2023.08.04" -image2 = Image.open('example2.jpg').convert('RGB') -answer2 = "2007.04.24" -image_test = Image.open('test.jpg').convert('RGB') - -msgs = [ - {'role': 'user', 'content': [image1, question]}, {'role': 'assistant', 'content': [answer1]}, - {'role': 'user', 'content': [image2, question]}, {'role': 'assistant', 'content': [answer2]}, - {'role': 'user', 'content': [image_test, question]} -] - -answer = model.chat( - image=None, - msgs=msgs, - tokenizer=tokenizer -) -print(answer) -``` -
- -#### Chat with video -
- Click to view Python code running MiniCPM-V 2.6 with video input. - -```python -import torch -from PIL import Image -from transformers import AutoModel, AutoTokenizer -from decord import VideoReader, cpu # pip install decord - -model = AutoModel.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True, - attn_implementation='sdpa', torch_dtype=torch.bfloat16) # sdpa or flash_attention_2, no eager -model = model.eval().cuda() -tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True) - -MAX_NUM_FRAMES=64 # if cuda OOM set a smaller number - -def encode_video(video_path): - def uniform_sample(l, n): - gap = len(l) / n - idxs = [int(i * gap + gap / 2) for i in range(n)] - return [l[i] for i in idxs] - - vr = VideoReader(video_path, ctx=cpu(0)) - sample_fps = round(vr.get_avg_fps() / 1) # FPS - frame_idx = [i for i in range(0, len(vr), sample_fps)] - if len(frame_idx) > MAX_NUM_FRAMES: - frame_idx = uniform_sample(frame_idx, MAX_NUM_FRAMES) - frames = vr.get_batch(frame_idx).asnumpy() - frames = [Image.fromarray(v.astype('uint8')) for v in frames] - print('num frames:', len(frames)) - return frames - -video_path="video_test.mp4" -frames = encode_video(video_path) -question = "Describe the video" -msgs = [ - {'role': 'user', 'content': frames + [question]}, -] - -# Set decode params for video -params = {} -params["use_image_id"] = False -params["max_slice_nums"] = 2 # use 1 if cuda OOM and video resolution > 448*448 - -answer = model.chat( - image=None, - msgs=msgs, - tokenizer=tokenizer, - **params -) -print(answer) -``` -
- - -### Inference on Multiple GPUs -You can run MiniCPM-Llama3-V 2.5 on multiple low VRAM GPUs (12 GB or 16 GB) by distributing the model's layers across multiple GPUs. Please refer to this [tutorial](https://github.com/OpenBMB/MiniCPM-V/blob/main/docs/inference_on_multiple_gpus.md) for detailed instructions on how to load the model and inference using multiple low VRAM GPUs. - - -### Inference on Mac -
-Click to view an example, to run MiniCPM-Llama3-V 2.5 on 💻 Mac with MPS (Apple silicon or AMD GPUs). - -```python -# test.py Need more than 16GB memory. -import torch -from PIL import Image -from transformers import AutoModel, AutoTokenizer - -model = AutoModel.from_pretrained('openbmb/MiniCPM-Llama3-V-2_5', trust_remote_code=True, low_cpu_mem_usage=True) -model = model.to(device='mps') - -tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-Llama3-V-2_5', trust_remote_code=True) -model.eval() - -image = Image.open('./assets/hk_OCR.jpg').convert('RGB') -question = 'Where is this photo taken?' -msgs = [{'role': 'user', 'content': question}] - -answer, context, _ = model.chat( - image=image, - msgs=msgs, - context=None, - tokenizer=tokenizer, - sampling=True -) -print(answer) -``` -Run with command: -```shell -PYTORCH_ENABLE_MPS_FALLBACK=1 python test.py -``` -
- -### Deployment on Mobile Phone -MiniCPM-V 2.0 can be deployed on mobile phones with Android operating systems. 🚀 Click [MiniCPM-V 2.0](https://github.com/OpenBMB/mlc-MiniCPM) to install apk. - -### Inference with llama.cpp -MiniCPM-V 2.6 can run with llama.cpp now! See [our fork of llama.cpp](https://github.com/OpenBMB/llama.cpp/tree/minicpmv-main/examples/llava/README-minicpmv2.6.md) for more detail. This implementation supports smooth inference of 16~18 token/s on iPad (test environment：iPad Pro + M4). - -### Inference with ollama -MiniCPM-V 2.6 can run with ollama now! See [our fork of ollama](https://github.com/OpenBMB/ollama/blob/minicpm-v2.6/examples/minicpm-v2.6/README.md) for more detail. This implementation supports smooth inference of 16~18 token/s on iPad (test environment：iPad Pro + M4). - -### Inference with vLLM - -
- vLLM now officially supports MiniCPM-V 2.6, MiniCPM-Llama3-V 2.5 and MiniCPM-V 2.0, Click to see. - -1. Install vLLM(>=0.5.4): -```shell -pip install vllm -``` -2. Install timm: (optional, MiniCPM-V 2.0 need timm) -```shell -pip install timm==0.9.10 -``` -3. Run the example(for image): -```python -from transformers import AutoTokenizer -from PIL import Image -from vllm import LLM, SamplingParams - -MODEL_NAME = "openbmb/MiniCPM-V-2_6" -# Also available for previous models -# MODEL_NAME = "openbmb/MiniCPM-Llama3-V-2_5" -# MODEL_NAME = "HwwwH/MiniCPM-V-2" - -image = Image.open("xxx.png").convert("RGB") -tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME, trust_remote_code=True) -llm = LLM( - model=MODEL_NAME, - trust_remote_code=True, - gpu_memory_utilization=1, - max_model_len=2048 -) - -messages = [{ - "role": - "user", - "content": - # Number of images - "(./)" + \ - "\nWhat is the content of this image?" -}] -prompt = tokenizer.apply_chat_template( - messages, - tokenize=False, - add_generation_prompt=True -) - -# Single Inference -inputs = { - "prompt": prompt, - "multi_modal_data": { - "image": image - # Multi images, the number of images should be equal to that of `(./)` - # "image": [image, image] - }, -} -# Batch Inference -# inputs = [{ -# "prompt": prompt, -# "multi_modal_data": { -# "image": image -# }, -# } for _ in 2] - - -# 2.6 -stop_tokens = ['<|im_end|>', '<|endoftext|>'] -stop_token_ids = [tokenizer.convert_tokens_to_ids(i) for i in stop_tokens] -# 2.0 -# stop_token_ids = [tokenizer.eos_id] -# 2.5 -# stop_token_ids = [tokenizer.eos_id, tokenizer.eot_id] - -sampling_params = SamplingParams( - stop_token_ids=stop_token_ids, - use_beam_search=True, - temperature=0, - best_of=3, - max_tokens=1024 -) - -outputs = llm.generate(inputs, sampling_params=sampling_params) - -print(outputs[0].outputs[0].text) -``` -4. click [here](https://modelbest.feishu.cn/wiki/C2BWw4ZP0iCDy7kkCPCcX2BHnOf?from=from_copylink) if you want to use it with *video*, or get more details about `vLLM`. -
- -## Fine-tuning - -### Simple Fine-tuning - -We support simple fine-tuning with Hugging Face for MiniCPM-V 2.0 and MiniCPM-Llama3-V 2.5. - -[Reference Document](./finetune/readme.md) - -### With the SWIFT Framework - -We now support MiniCPM-V series fine-tuning with the SWIFT framework. SWIFT supports training, inference, evaluation and deployment of nearly 200 LLMs and MLLMs . It supports the lightweight training solutions provided by PEFT and a complete Adapters Library including techniques such as NEFTune, LoRA+ and LLaMA-PRO. - -Best Practices：[MiniCPM-V 1.0](https://github.com/modelscope/swift/blob/main/docs/source/Multi-Modal/minicpm-v最佳实践.md), [MiniCPM-V 2.0](https://github.com/modelscope/swift/blob/main/docs/source/Multi-Modal/minicpm-v-2最佳实践.md), [MiniCPM-V 2.6](https://github.com/modelscope/ms-swift/issues/1613). - -## FAQs -Click here to view the [FAQs](./docs/faqs.md) - -## Model License - -* This repository is released under the [Apache-2.0](https://github.com/OpenBMB/MiniCPM/blob/main/LICENSE) License. - -* The usage of MiniCPM-V model weights must strictly follow [MiniCPM Model License.md](https://github.com/OpenBMB/MiniCPM/blob/main/MiniCPM%20Model%20License.md). - -* The models and weights of MiniCPM are completely free for academic research. after filling out a ["questionnaire"](https://modelbest.feishu.cn/share/base/form/shrcnpV5ZT9EJ6xYjh3Kx0J6v8g) for registration, are also available for free commercial use. - - -## Statement - -As LMMs, MiniCPM-V models (including OmniLMM) generate contents by learning a large amount of multimodal corpora, but they cannot comprehend, express personal opinions or make value judgement. Anything generated by MiniCPM-V models does not represent the views and positions of the model developers - -We will not be liable for any problems arising from the use of MiniCPM-V models, including but not limited to data security issues, risk of public opinion, or any risks and problems arising from the misdirection, misuse, dissemination or misuse of the model. - - -## Institutions - -This project is developed by the following institutions: - -- [THUNLP](https://nlp.csai.tsinghua.edu.cn/) -- [ModelBest](https://modelbest.cn/) -- [Zhihu](https://www.zhihu.com/ ) - -## 🌟 Star History - - - -

- -

-
- - - -## Key Techniques and Other Multimodal Projects - -👏 Welcome to explore key techniques of MiniCPM-V and other multimodal projects of our team: - -[VisCPM](https://github.com/OpenBMB/VisCPM/tree/main) | [RLHF-V](https://github.com/RLHF-V/RLHF-V) | [LLaVA-UHD](https://github.com/thunlp/LLaVA-UHD) | [RLAIF-V](https://github.com/RLHF-V/RLAIF-V) - - -## Citation - -If you find our model/code/paper helpful, please consider cite our papers 📝 and star us ⭐️！ - -```bib -@article{yao2024minicpm, - title={MiniCPM-V: A GPT-4V Level MLLM on Your Phone}, - author={Yao, Yuan and Yu, Tianyu and Zhang, Ao and Wang, Chongyi and Cui, Junbo and Zhu, Hongji and Cai, Tianchi and Li, Haoyu and Zhao, Weilin and He, Zhihui and others}, - journal={arXiv preprint arXiv:2408.01800}, - year={2024} -} -``` diff --git a/README_zh.md b/README_zh.md index 0399a37..35d8040 100644 --- a/README_zh.md +++ b/README_zh.md @@ -1,34 +1,36 @@
- + - - -**端侧可用的 GPT-4V 级单图、多图、视频多模态大模型** +**端侧可用的 GPT-4o 级视觉、语音、多模态实时流式大模型** 中文 | - [English](./README_en.md) + [English](./README.md) + + + + + 微信社区  | + + + MiniCPM-V   📖 最佳实践 + - 加入我们的 💬 微信社区 -｜ 了解 MiniCPM-V 📖 最佳实践 - -

- MiniCPM-V 2.6 🤗 🤖 | MiniCPM-Llama3-V 2.5 🤗 🤖 | - MiniCPM-Llama3-V 2.5 技术报告 +

+ MiniCPM-o 2.6 🤗 国内🤖 国外🤖 | MiniCPM-V 2.6 🤗 🤖 | + 技术报告近期将发布

-
-**MiniCPM-V**是面向图文理解的端侧多模态大模型系列。该系列模型接受图像和文本输入，并提供高质量的文本输出。自2024年2月以来，我们共发布了5个版本模型，旨在实现**领先的性能和高效的部署**，目前该系列最值得关注的模型包括： +**MiniCPM-o** 是从 MiniCPM-V 升级的最新端侧多模态大模型系列。该系列模型可以以端到端方式，接受图像、视频、文本、音频作为输入，并生成高质量文本和语音输出。自2024年2月以来，我们以实现高性能和高效部署为目标，发布了6个版本的模型。目前系列中最值得关注的模型包括： -- **MiniCPM-V 2.6**: 🔥🔥🔥 MiniCPM-V系列的最新、性能最佳模型。总参数量 8B，单图、多图和视频理解性能**超越了 GPT-4V**。在单图理解上，它取得了优于 **GPT-4o mini、Gemini 1.5 Pro 和 Claude 3.5 Sonnet**等商用闭源模型的表现，并进一步优化了 MiniCPM-Llama3-V 2.5 的 OCR、可信行为、多语言支持以及端侧部署等诸多特性。基于其领先的视觉 token 密度，MiniCPM-V 2.6 成为了首个支持在 iPad 等端侧设备上进行实时视频理解的多模态大模型。 - -- **MiniCPM-V 2.0**：MiniCPM-V系列的最轻量级模型。总参数量2B，多模态综合性能超越 Yi-VL 34B、CogVLM-Chat 17B、Qwen-VL-Chat 10B 等更大参数规模的模型，可接受 180 万像素的任意长宽比图像输入，实现了和 Gemini Pro 相近的场景文字识别能力以及和 GPT-4V 相匹的低幻觉率。 +- **MiniCPM-o 2.6**: 🔥🔥🔥 MiniCPM-o 系列的最新、性能最佳模型。总参数量 8B，**视觉、语音和多模态流式能力达到了 GPT-4o-202405 级别**，是开源社区中模态支持最丰富、性能最佳的模型之一。在新的语音模式中，MiniCPM-o 2.6 **支持可配置声音的中英双语语音对话，还具备情感/语速/风格控制、端到端声音克隆、角色扮演等进阶能力**。模型也进一步提升了 MiniCPM-V 2.6 的 **OCR、可信行为、多语言支持和视频理解等视觉能力**。基于其领先的视觉 token 密度，MiniCPM-V 2.6 成为了**首个支持在 iPad 等端侧设备上进行多模态实时流式交互**的多模态大模型。 +- **MiniCPM-V 2.6**: MiniCPM-V 系列中性能最佳的模型。总参数量 8B，单图、多图和视频理解性能**超越了 GPT-4V**。它取得了优于 **GPT-4o mini、Gemini 1.5 Pro 和 Claude 3.5 Sonnet**等的单图理解表现，并成为了首个支持在 iPad 等端侧设备上进行实时视频理解的多模态大模型。 ## 更新日志 @@ -36,15 +38,10 @@ #### 📌 置顶 +* [2025.01.13] 🔥🔥🔥 我们开源了 MiniCPM-o 2.6，该模型视觉、语音和多模态流式能力达到了 GPT-4o-202405 级别，进一步优化了 MiniCPM-V 2.6 的众多亮点能力，还支持了很多有趣的新功能。欢迎试用！ * [2024.08.17] 🚀🚀🚀 llama.cpp [官方仓库](https://github.com/ggerganov/llama.cpp)正式支持 MiniCPM-V 2.6 啦！点击[这里](https://huggingface.co/openbmb/MiniCPM-V-2_6-gguf)查看各种大小的 GGUF 版本。 -* [2024.08.15] MiniCPM-V 2.6 现在支持多图像 SFT。有关更多详细信息，请参阅[微调文档](https://github.com/OpenBMB/MiniCPM-V/tree/main/finetune) -* [2024.08.14] MiniCPM-V 2.6 现在可以通过 SWIFT 框架 [微调](https://github.com/modelscope/ms-swift/issues/1613) 了！ -* [2024.08.10] 🚀🚀🚀 llama.cpp [官方仓库](https://github.com/ggerganov/llama.cpp)正式支持 MiniCPM-Llama3-V 2.5 啦！点击[这里](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5-gguf/tree/main)查看各种大小的 GGUF 版本。 * [2024.08.06] 🔥🔥🔥 我们开源了 MiniCPM-V 2.6，该模型在单图、多图和视频理解方面取得了优于 GPT-4V 的表现。我们还进一步提升了 MiniCPM-Llama3-V 2.5 的多项亮点能力，并首次支持了 iPad 上的实时视频理解。欢迎试用！ * [2024.08.03] MiniCPM-Llama3-V 2.5 技术报告已发布！欢迎点击[这里](https://arxiv.org/abs/2408.01800)查看。 -* [2024.07.19] MiniCPM-Llama3-V 2.5 现已支持[vLLM](#vllm-部署-) ！ -* [2024.05.28] 💫 我们现在支持 MiniCPM-Llama3-V 2.5 的 LoRA 微调，更多内存使用统计信息可以在[这里](https://github.com/OpenBMB/MiniCPM-V/tree/main/finetune#model-fine-tuning-memory-usage-statistics)找到。 -* [2024.05.23] 🔍 我们添加了Phi-3-vision-128k-instruct 与 MiniCPM-Llama3-V 2.5的全面对比，包括基准测试评估、多语言能力和推理效率 🌟📊🌍🚀。点击[这里](./docs/compare_with_phi-3_vision.md)查看详细信息。 * [2024.05.23] 🔥🔥🔥 MiniCPM-V 在 GitHub Trending 和 Hugging Face Trending 上登顶！MiniCPM-Llama3-V 2.5 Demo 被 Hugging Face 的 Gradio 官方账户推荐，欢迎点击[这里](https://huggingface.co/spaces/openbmb/MiniCPM-Llama3-V-2_5)体验！ @@ -53,10 +50,16 @@
点击查看完整更新日志。 +* [2024.08.15] MiniCPM-V 2.6 现在支持多图像 SFT。有关更多详细信息，请参阅[微调文档](https://github.com/OpenBMB/MiniCPM-V/tree/main/finetune) +* [2024.08.14] MiniCPM-V 2.6 现在可以通过 SWIFT 框架 [微调](https://github.com/modelscope/ms-swift/issues/1613) 了！ +* [2024.08.10] 🚀🚀🚀 llama.cpp [官方仓库](https://github.com/ggerganov/llama.cpp)正式支持 MiniCPM-Llama3-V 2.5 啦！点击[这里](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5-gguf/tree/main)查看各种大小的 GGUF 版本。 +* [2024.07.19] MiniCPM-Llama3-V 2.5 现已支持[vLLM](#vllm-部署-) ！ * [2024.06.03] 现在，你可以利用多张低显存显卡（12G/16G）进行GPU串行推理。详情请参见该[文档](https://github.com/OpenBMB/MiniCPM-V/blob/main/docs/inference_on_multiple_gpus.md)配置。 +* [2024.05.28] 💫 我们现在支持 MiniCPM-Llama3-V 2.5 的 LoRA 微调，更多内存使用统计信息可以在[这里](https://github.com/OpenBMB/MiniCPM-V/tree/main/finetune#model-fine-tuning-memory-usage-statistics)找到。 * [2024.05.28] 💥 MiniCPM-Llama3-V 2.5 现在在 llama.cpp 和 ollama 中完全支持其功能！**请拉取我们最新的 fork 来使用**：[llama.cpp](https://github.com/OpenBMB/llama.cpp/blob/minicpm-v2.5/examples/minicpmv/README.md) & [ollama](https://github.com/OpenBMB/ollama/tree/minicpm-v2.5/examples/minicpm-v2.5)。我们还发布了各种大小的 GGUF 版本，请点击[这里](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5-gguf/tree/main)查看。请注意，**目前官方仓库尚未支持 MiniCPM-Llama3-V 2.5**，我们也正积极推进将这些功能合并到 llama.cpp & ollama 官方仓库，敬请关注！ * [2024.05.25] MiniCPM-Llama3-V 2.5 [支持流式输出和自定义系统提示词](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5#usage)了，欢迎试用! * [2024.05.24] 我们开源了 MiniCPM-Llama3-V 2.5 [gguf](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5-gguf)，支持 [llama.cpp](#llamacpp-部署) 推理！实现端侧 6-8 tokens/s 的流畅解码，欢迎试用！ +* [2024.05.23] 🔍 我们添加了Phi-3-vision-128k-instruct 与 MiniCPM-Llama3-V 2.5的全面对比，包括基准测试评估、多语言能力和推理效率 🌟📊🌍🚀。点击[这里](./docs/compare_with_phi-3_vision.md)查看详细信息。 * [2024.05.20] 我们开源了 MiniCPM-Llama3-V 2.5，增强了 OCR 能力，支持 30 多种语言，并首次在端侧实现了 GPT-4V 级的多模态能力！我们提供了[高效推理](#手机端部署)和[简易微调](./finetune/readme.md)的支持，欢迎试用！ * [2024.04.23] 我们增加了MiniCPM-V 2.0对 [vLLM](#vllm-部署-) 的支持，欢迎体验！ * [2024.04.18] 我们在 HuggingFace Space 新增了 MiniCPM-V 2.0 的 [demo](https://huggingface.co/spaces/openbmb/MiniCPM-V-2)，欢迎体验！ @@ -71,26 +74,945 @@ ## 目录 +- [MiniCPM-o 2.6](#minicpm-o-26) - [MiniCPM-V 2.6](#minicpm-v-26) -- [MiniCPM-Llama3-V 2.5](#minicpm-llama3-v-25) -- [MiniCPM-V 2.0](#minicpm-v-20) - [Gradio Demo 🤗](#gradio-demo-) - [安装](#安装) - [推理](#推理) - [模型库](#模型库) - [多轮对话](#多轮对话) - - [多图理解](#多图理解) - - [少样本上下文学习](#少样本上下文学习) - - [视频理解](#视频理解) + - [多图对话](#多图对话) + - [少样本上下文对话](#少样本上下文对话) + - [视频对话](#视频对话) + - [语音对话](#语音对话) + - [Mimick](#mimick) + - [可配置声音的语音对话](#可配置声音的语音对话) + - [更多语音任务](#更多语音任务) + - [多模态流式交互](#多模态流式交互) - [多卡推理](#多卡推理) - [Mac 推理](#mac-推理) - [手机端部署](#手机端部署) - [本地WebUI Demo部署](#本地webui-demo部署) - - [llama.cpp 部署](#llamacpp-部署) - - [ollama 部署](#ollama-部署) - - [vLLM 部署 ](#vllm-部署-) + - [基于 llama.cpp、ollama、vLLM 的高效推理](#基于-llamacppollamavllm-的高效推理) - [微调](#微调) - [FAQs](#faqs) +- [模型局限性](#模型局限性) + +## MiniCPM-o 2.6 + + +MiniCPM-o 2.6 是 MiniCPM-o 系列的最新、性能最佳模型。该模型基于 SigLip-400M、Whisper-medium-300M、ChatTTS-200M 和 Qwen2.5-7B 构建，共 8B 参数，通过端到端方式训练和推理。相比 MiniCPM-V 2.6，该模型在性能上有了显著提升，并支持了实时语音对话和多模态流式交互的新功能。MiniCPM-o 2.6 的主要特性包括： + + +- 🔥 **领先的视觉能力。** +MiniCPM-o 2.6 在 OpenCompass 榜单上（综合 8 个主流多模态评测基准）平均得分 70.2，**以 8B 量级的大小在单图理解方面超越了 GPT-4o-202405、Gemini 1.5 Pro 和 Claude 3.5 Sonnet 等主流商用闭源多模态大模型**。此外，它的多图和视频理解表现也**优于 GPT-4V 和 Claude 3.5 Sonnet**，并展现出了优秀的上下文学习能力。 + +- 🎙 **出色的语音能力。** +MiniCPM-o 2.6 **支持可配置声音的中英双语实时对话**。MiniCPM-o 2.6 在语音理解任务（如 ASR 和 STT 等）**优于 GPT-4o-realtime**，并在语音对话的语义和声学评估中展现了**开源模型中最高的语音生成性能**。它还支持情绪/语速/风格控制、语音克隆、角色扮演等进阶能力。 + +- 🎬 **强大的多模态流式交互能力。** +作为一项新功能，MiniCPM-o 2.6 能够**接受连续的视频和音频流，并和用户进行实时语音交互**。在针对实时视频理解、全模态视音频理解、多模态上下文理解的综合评测基准 StreamingBench 中，MiniCPM-o 2.6 取得开源社区最佳水平，并**超过了 GPT-4o-202408 和 Claude 3.5 Sonnet**。 + +- 💪 **强大的 OCR 能力及其他功能。** +MiniCPM-o 2.6 进一步优化了 MiniCPM-V 2.6 的众多视觉理解能力，其可以处理任意长宽比的图像，像素数可达 180 万（如 1344x1344）。在 OCRBench 上取得**25B 以下最佳水平，超过 GPT-4o-202405 等商用闭源模型**。基于最新的 [RLHF-V](https://rlhf-v.github.io/)、[RLAIF-V](https://github.com/RLHF-V/RLAIF-V/) 和 [VisCPM](https://github.com/OpenBMB/VisCPM) 技术，其具备了**可信的多模态行为**，在 MMHal-Bench 上超过了 GPT-4o 和 Claude 3.5，并支持英语、中文、德语、法语、意大利语、韩语等**30多种语言**。 + +- 🚀 **卓越的效率。** +除了对个人用户友好的模型大小，MiniCPM-o 2.6 还表现出**最先进的视觉 token 密度**（即每个视觉 token 编码的像素数量）。它**仅需 640 个 token 即可处理 180 万像素图像，比大多数模型少 75%**。这一特性优化了模型的推理速度、首 token 延迟、内存占用和功耗。因此，MiniCPM-o 2.6 可以支持 iPad 等终端设备上的高效**多模态实时流式交互**。 + + +- 💫 **易于使用。** +MiniCPM-o 2.6 可以通过多种方式轻松使用：(1) [llama.cpp](https://github.com/OpenBMB/llama.cpp/blob/minicpm-omni/examples/llava/README-minicpmo2.6.md) 支持在本地设备上进行高效的 CPU 推理，(2) [int4](https://huggingface.co/openbmb/MiniCPM-V-2_6-int4) 和 [GGUF](https://huggingface.co/openbmb/MiniCPM-V-2_6-gguf) 格式的量化模型，有 16 种尺寸，(3) [vLLM](#基于-llamacppollamavllm-的高效推理) 支持高吞吐量和内存高效的推理，(4) 通过[LLaMA-Factory](./docs/llamafactory_train.md)框架针对新领域和任务进行微调，(5) 使用 [Gradio](#本地-webui-demo-) 快速设置本地 WebUI 演示，(6) 部署于[国内](https://minicpm-omni-webdemo.modelbest.cn/ +) 或 [国外](https://minicpm-omni-webdemo-us.modelbest.cn/)服务器的在线 demo。 + + +**模型架构。** + +- **端到端全模态架构。** 通过**端到端**的方式连接和训练不同模态的编/解码模块以充分利用丰富的多模态知识。 +- **全模态流式机制。** (1) 我们将不同模态的离线编/解码器改造为适用于**流式输入/输出**的在线模块。 (2) 我们针对大语言模型基座设计了**时分复用的全模态流式信息处理机制**，将平行的不同模态的信息流拆分重组为周期性时间片序列。 +- **可配置的声音方案。** 我们设计了新的多模态系统提示，包含传统文本系统提示词，和**用于指定模型声音的语音系统提示词**。模型可在推理时灵活地通过文字或语音样例控制声音风格，并支持端到端声音克隆和音色创建等高级能力。 + +
+ +
+ +
+ + + +### 性能评估 + +
+ +
+ +
+点击查看视觉理解能力详细评测结果。 + +**图像理解能力** + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ModelSizeToken Density+OpenCompassOCRBenchMathVista miniChartQAMMVetMMStarMMEMMB1.1 testAI2DMMMU valHallusionBenchTextVQA valDocVQA testMathVerse miniMathVisionMMHal Score
Proprietary
GPT-4o-20240513-108869.973661.385.769.163.92328.782.284.669.255.0-92.850.230.43.6
Claude3.5-Sonnet-75067.978861.690.866.062.21920.078.580.265.949.9-95.2--3.4
Gemini-1.5-Pro--64.475457.781.364.059.12110.673.979.160.645.673.586.5-19.2-
GPT-4o-mini-20240718-108864.178552.4-66.954.82003.476.077.860.046.1----3.3
Open Source
Cambrian-34B34B182058.359150.375.653.254.22049.977.879.550.441.676.775.5---
GLM-4V-9B13B78459.177651.1-58.054.82018.867.971.246.945.0-----
Pixtral-12B12B25661.068556.981.858.554.5-72.779.051.147.075.790.7---
DeepSeek-VL2-27B (4B)27B67266.480963.986.060.061.92253.081.283.854.045.384.293.3--3.0
Qwen2-VL-7B8B78467.186658.283.062.060.72326.081.883.054.150.684.394.531.916.33.2
LLaVA-OneVision-72B72B18268.174167.583.760.665.82261.085.085.656.849.080.591.339.1-3.5
InternVL-2.5-8B8B70668.382264.484.862.862.82344.083.684.556.050.179.193.039.519.73.4
MiniCPM-V 2.68B282265.2852*60.679.460.057.52348.4*78.082.149.8*48.1*80.190.825.718.33.6
MiniCPM-o 2.68B282270.2897*71.9*86.9*67.564.02372.0*80.585.850.4*51.982.093.541.4*23.1*3.8
+
+* 我们使用思维链提示词来评估这些基准，对于 MME 我们只在 Cognition 任务上使用了思维链。 ++ Token Density：每个视觉 token 在最大分辨率下编码的像素数，即最大分辨率下的像素数 / 视觉 token 数。 + +注意：闭源模型的 Token Density 由 API 收费方式估算得到。 + +**多图和视频理解能力** + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ModelSizeBLINK-valMantis-EvalMIRBVideo-MME (wo / w subs)
Proprietary
GPT-4o-20240513-68--71.9/77.2
GPT4V-54.662.753.159.9/63.3
Open-source
LLaVA-NeXT-Interleave 14B14B52.666.430.2-
LLaVA-One-Vision-72B72B55.477.6-66.2/69.5
MANTIS 8B8B49.159.534.8-
Qwen2-VL-7B8B53.269.6*67.6*63.3/69.0
InternVL-2.5-8B8B54.867.752.564.2/66.9
MiniCPM-V 2.68B5369.153.860.9/63.6
MiniCPM-o 2.68B56.771.958.663.9/67.9
+ +
+* 正式开源模型权重的评测结果。 + +
+ + +
+点击查看语音理解和生成能力的详细评测结果。 + +**语音理解能力** + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
TaskSizeASR (zh)ASR (en)ASTEmotion
MetricCER↓WER↓BLEU↑ACC↑
DatasetAISHELL-1Fleurs zhWenetSpeech test-netLibriSpeech test-cleanGigaSpeechTED-LIUMCoVoST en2zhCoVoST zh2enMELD emotion
Proprietary
GPT-4o-Realtime-7.3*5.4*28.9*2.6*12.9*4.8*37.1*15.7*33.2*
Gemini-1.5-Pro-4.5*5.9*14.3*2.9*10.6*3.0*47.3*22.6*48.4*
Open-Source
Qwen2-Audio-Base8B-7.5-1.6--45.224.455.3
Qwen2-Audio-Instruction8B2.6*6.9*10.3*3.1*9.7*5.9*39.5*22.9*17.4*
GLM-4-Voice-Base9B2.5--2.8----
MiniCPM-o 2.68B1.64.46.91.78.73.048.227.252.4
+
+* 正式开源模型权重的评测结果。

+ +**语音生成能力。** + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
TaskSizeSpeechQA
MetricACC↑G-Eval (10 point)↑Semantic ELO score↑Acoustic ELO score↑Overall ELO score↑UTMOS↑ASR-WER↓
DatasetSpeech Llama Q.Speech Web Q.Speech Trivia QASpeech AlpacaEvalAudioArena
Proprietary
GPT-4o-Realtime71.751.669.77.41157120312004.22.3
Open-Source
GLM-4-Voice9B50.032.036.45.1999114710354.111.7
Llama-Omni8B45.322.910.73.99608788973.224.3
Moshi7B43.723.816.72.48718088752.88.2
Mini-Omni1B22.012.86.92.59268038653.410.0
MiniCPM-o 2.68B61.040.040.25.11088116311314.29.8
+
+所有的结果都基于 AudioEvals。

+ +**端到端声音克隆能力。** + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
TaskTTS
MetricSIMO↑SIMO↑
DatasetSeed-TTS test-zhSeed-TTS test-en
F5-TTS7667
CosyVoice7564
FireRedTTS6346
MiniCPM-o 2.65747
+
+ +
+ +
+点击查看多模态流式交互能力评测详细结果。 + +**多模态流式交互能力**: StreamingBench 分数 + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ModelSizeReal-Time Video UnderstandingOmni-Source UnderstandingContextual UnderstandingOverall
Proprietary
Gemini 1.5 Pro-77.467.851.170.3
GPT-4o-202408-74.551.048.064.1
Claude-3.5-Sonnet-74.041.437.859.7
Open-source
VILA-1.58B61.537.526.749.5
LongVA7B63.135.930.250.7
LLaVA-Next-Video-34B34B69.841.734.356.7
Qwen2-VL-7B8B71.240.733.157.0
InternVL2-8B8B70.142.734.157.0
VITA-1.58B70.940.835.857.4
LLaVA-OneVision-7B8B74.340.831.058.4
InternLM-XC2.5-OL-7B8B75.446.233.660.8
MiniCPM-V 2.68B72.440.233.457.7
MiniCPM-o 2.68B79.953.438.566.0
+ +
+ + +### 典型示例 + +以下为 MiniCPM-o 2.6 的 iPad Pro 实机演示和 web demo 演示样例： + + +
+ +
+ +
+ math + diagram + bike +
+ + +
+Click to view more details of MiniCPM-V 2.6 + ## MiniCPM-V 2.6 @@ -372,7 +1294,7 @@ 42.4 10.3 - + MiniCPM-V 2.6 8B 2822 @@ -496,7 +1418,7 @@ 34.4* 56.9* - + MiniCPM-V 2.6 8B 69.1 @@ -643,7 +1565,7 @@ 2.64 3.28 - + MiniCPM-V 2.6 8B 60.9 @@ -785,7 +1707,7 @@ 70.9 54.1 - + MiniCPM-V 2.6+ 8B 0 @@ -794,14 +1716,14 @@ 45.4 23.9 - + 4 63.6 60.5 65.5 50.1 - + 8 64.6 63.4 @@ -852,390 +1774,16 @@

-## MiniCPM-Llama3-V 2.5 - -
-查看 MiniCPM-Llama3-V 2.5 的详细信息 - -**MiniCPM-Llama3-V 2.5** 是 MiniCPM-V 系列的最新版本模型，基于 SigLip-400M 和 Llama3-8B-Instruct 构建，共 8B 参数量，相较于 MiniCPM-V 2.0 性能取得较大幅度提升。MiniCPM-Llama3-V 2.5 值得关注的特点包括： - -- 🔥 **领先的性能。** - MiniCPM-Llama3-V 2.5 在综合了 11 个主流多模态大模型评测基准的 OpenCompass 榜单上平均得分 65.1，**以 8B 量级的大小超过了 GPT-4V-1106、Gemini Pro、Claude 3、Qwen-VL-Max 等主流商用闭源多模态大模型**，大幅超越基于Llama 3构建的其他多模态大模型。 - -- 💪 **优秀的 OCR 能力。** - MiniCPM-Llama3-V 2.5 可接受 180 万像素的任意宽高比图像输入，**OCRBench 得分达到 725，超越 GPT-4o、GPT-4V、Gemini Pro、Qwen-VL-Max 等商用闭源模型**，达到最佳水平。基于近期用户反馈建议，MiniCPM-Llama3-V 2.5 增强了全文 OCR 信息提取、表格图像转 markdown 等高频实用能力，并且进一步加强了指令跟随、复杂推理能力，带来更好的多模态交互体感。 - -- 🏆 **可信行为。** - 借助最新的 [RLAIF-V](https://github.com/RLHF-V/RLAIF-V/) 对齐技术（[RLHF-V](https://github.com/RLHF-V/) [CVPR'24]系列的最新技术），MiniCPM-Llama3-V 2.5 具有更加可信的多模态行为，在 Object HalBench 的幻觉率降低到了 **10.3%**，显著低于 GPT-4V-1106 (13.6%)，达到开源社区最佳水平。[数据集已发布](https://huggingface.co/datasets/openbmb/RLAIF-V-Dataset)。 - -- 🌏 **多语言支持。** - 得益于 Llama 3 强大的多语言能力和 VisCPM 的跨语言泛化技术，MiniCPM-Llama3-V 2.5 在中英双语多模态能力的基础上，仅通过少量翻译的多模态数据的指令微调，高效泛化支持了**德语、法语、西班牙语、意大利语、韩语等 30+ 种语言**的多模态能力，并表现出了良好的多语言多模态对话性能。[查看所有支持语言](./assets/minicpm-llama-v-2-5_languages.md) - -- 🚀 **高效部署。** - MiniCPM-Llama3-V 2.5 较为系统地通过**模型量化、CPU、NPU、编译优化**等高效加速技术，实现高效的终端设备部署。对于高通芯片的移动手机，我们首次将 NPU 加速框架 QNN 整合进了 llama.cpp。经过系统优化后，MiniCPM-Llama3-V 2.5 实现了多模态大模型端侧**语言解码速度 3 倍加速**、**图像编码 150 倍加速**的巨大提升。 - -- 💫 **易于使用。** - MiniCPM-Llama3-V 2.5 可以通过多种方式轻松使用：（1）[llama.cpp](https://github.com/OpenBMB/llama.cpp/blob/minicpm-v2.5/examples/minicpmv/README.md) 和 [ollama](https://github.com/OpenBMB/ollama/tree/minicpm-v2.5/examples/minicpm-v2.5) 支持在本地设备上进行高效的 CPU 推理；（2）提供 16 种尺寸的 [GGUF](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5-gguf) 格式量化模型；（3）仅需 2 张 V100 GPU 即可进行高效的 [LoRA](https://github.com/OpenBMB/MiniCPM-V/tree/main/finetune#lora-finetuning) 微调；（ 4）支持[流式输出](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5#usage)；（5）快速搭建 [Gradio](https://github.com/OpenBMB/MiniCPM-V/blob/main/web_demo_2.5.py) 和 [Streamlit](https://github.com/OpenBMB/MiniCPM-V/blob/main/web_demo_streamlit-2_5.py) 本地 WebUI demo；（ 6.）[HuggingFace Spaces](https://huggingface.co/spaces/openbmb/MiniCPM-Llama3-V-2_5) 交互式 demo。 - -### 性能评估 - -
- -
-
-TextVQA, DocVQA, OCRBench, OpenCompass MultiModal Avg Score, MME, MMBench, MMMU, MathVista, LLaVA Bench, RealWorld QA, Object HalBench上的详细评测结果。 -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
ModelSizeOCRBenchTextVQA valDocVQA testOpen-CompassMMEMMB test (en)MMB test (cn)MMMU valMath-VistaLLaVA BenchRealWorld QAObject HalBench
Proprietary
Gemini Pro-68074.688.162.92148.973.674.348.945.879.960.4-
GPT-4V (2023.11.06)-64578.088.463.51771.577.074.453.847.893.163.086.4
Open-source
Mini-Gemini2.2B-56.234.2*-1653.0--31.7----
Qwen-VL-Chat9.6B48861.562.651.61860.061.856.337.033.867.749.356.2
DeepSeek-VL-7B7.3B43564.7*47.0*54.61765.473.871.438.336.877.854.2-
Yi-VL-34B34B29043.4*16.9*52.22050.272.470.745.130.762.354.879.3
CogVLM-Chat17.4B59070.433.3*54.21736.665.855.937.334.773.960.373.6
TextMonkey9.7B55864.366.7---------
Idefics28.0B-73.074.057.21847.675.768.645.252.249.160.7-
Bunny-LLama-3-8B8.4B---54.31920.377.073.941.331.561.258.8-
LLaVA-NeXT Llama-3-8B8.4B----1971.5--41.7-80.160.0-
Phi-3-vision-128k-instruct4.2B639*70.9--1537.5*--40.444.564.2*58.8*-
MiniCPM-V 1.02.8B36660.638.247.51650.264.162.638.328.951.351.278.4
MiniCPM-V 2.02.8B60574.171.954.51808.669.166.538.238.769.255.885.5
MiniCPM-Llama3-V 2.58.5B72576.684.865.12024.677.274.245.854.386.763.589.7
- -
-* 正式开源模型权重的评测结果。
-
- -
- 多语言LLaVA Bench评测结果 -
- - -### 典型示例 - -

- -

-
- - -
- -## MiniCPM-V 2.0 - -
-查看 MiniCPM-V 2.0 的详细信息 - -**MiniCPM-V 2.0**可以高效部署到终端设备。该模型基于 SigLip-400M 和 [MiniCPM-2.4B](https://github.com/OpenBMB/MiniCPM/)构建，通过perceiver resampler连接。其特点包括： - -- 🔥 **优秀的性能。** - - MiniCPM-V 2.0 在多个测试基准（如 OCRBench, TextVQA, MME, MMB, MathVista 等）中实现了 7B 以下模型的**最佳性能**。**在综合了 11 个主流多模态大模型评测基准的 OpenCompass 榜单上超过了 Qwen-VL-Chat 9.6B、CogVLM-Chat 17.4B 和 Yi-VL 34B 等更大参数规模的模型**。MiniCPM-V 2.0 还展现出**领先的 OCR 能力**，在场景文字识别能力上**接近 Gemini Pro**，OCRBench 得分达到**开源模型第一**。 - - -- 🏆 **可信行为。** - - 多模态大模型深受幻觉问题困扰，模型经常生成和图像中的事实不符的文本。MiniCPM-V 2.0 是 **第一个通过多模态 RLHF 对齐的端侧多模态大模型**（借助 [RLHF-V](https://rlhf-v.github.io/) [CVPR'24] 系列技术）。该模型在 [Object HalBench](https://arxiv.org/abs/2312.00849) 达到**和 GPT-4V 相仿**的性能。 - - -- 🌟 **高清图像高效编码。** - - MiniCPM-V 2.0 可以接受 **180 万像素的任意长宽比图像输入**（基于最新的[LLaVA-UHD](https://arxiv.org/pdf/2403.11703.pdf) 技术），这使得模型可以感知到小物体、密集文字等更加细粒度的视觉信息。 - - -- ⚡️ **高效部署。** - - MiniCPM-V 2.0 可以**高效部署在大多数消费级显卡和个人电脑上**，包括**移动手机等终端设备**。在视觉编码方面，我们通过perceiver resampler将图像表示压缩为更少的 token。这使得 MiniCPM-V 2.0 即便是**面对高分辨率图像，也能占用较低的存储并展现优秀的推理速度**。 - -- 🙌 **双语支持。** - - MiniCPM-V 2.0 **提供领先的中英双语多模态能力支持**。 - 该能力通过 [VisCPM](https://arxiv.org/abs/2308.12038) [ICLR'24] 论文中提出的多模态能力的跨语言泛化技术实现。 - -### 典型示例 - - - -

- -

-
- -我们将 MiniCPM-V 2.0 部署在小米 14 Pro 上，并录制了以下演示视频，未经任何视频剪辑。 - - -

- - -

-
- -
- - - - ## 历史版本模型 | 模型 | 介绍信息和使用教程 | |:----------------------|:-------------------:| -| MiniCPM-V 1.0 | [文档](./minicpm_v1.md) | +| MiniCPM-Llama3-V 2.5 | [文档](./docs/minicpm_llama3_v2dot5.md) | +| MiniCPM-V 2.0 | [文档](./docs/minicpm_v2.md) | +| MiniCPM-V 1.0 | [文档](./docs/minicpm_v1.md) | | OmniLMM-12B | [文档](./omnilmm.md) | @@ -1291,23 +1839,21 @@ pip install -r requirements.txt | 模型 | 设备 | 资源 |          简介 | 下载链接 | |:--------------|:-:|:----------:|:-------------------|:---------------:| -| MiniCPM-V 2.6| GPU | 17 GB | 最新版本，提供最佳的端侧单图、多图、视频理解能力。 | [🤗](https://huggingface.co/openbmb/MiniCPM-V-2_6)    [](https://modelscope.cn/models/OpenBMB/MiniCPM-V-2_6) | +| MiniCPM-o 2.6| GPU | 18 GB | 最新版本，提供端侧 GPT-4o 级的视觉、语音、多模态流式交互能力。 | [🤗](https://huggingface.co/openbmb/MiniCPM-o-2_6)    [](https://modelscope.cn/models/OpenBMB/MiniCPM-o-2_6) | +| MiniCPM-o 2.6 gguf | CPU | 8 GB | gguf 版本，更低的内存占用和更高的推理效率。 | [🤗](https://huggingface.co/openbmb/MiniCPM-o-2_6-gguf)    [](https://modelscope.cn/models/OpenBMB/MiniCPM-o-2_6-gguf) | +| MiniCPM-o 2.6 int4 | GPU | 9 GB | int4量化版，更低显存占用。 | [🤗](https://huggingface.co/openbmb/MiniCPM-o-2_6-int4)    [](https://modelscope.cn/models/OpenBMB/MiniCPM-o-2_6-int4) | +| MiniCPM-V 2.6| GPU | 17 GB | 提供出色的端侧单图、多图、视频理解能力。 | [🤗](https://huggingface.co/openbmb/MiniCPM-V-2_6)    [](https://modelscope.cn/models/OpenBMB/MiniCPM-V-2_6) | | MiniCPM-V 2.6 gguf | CPU | 6 GB | gguf 版本，更低的内存占用和更高的推理效率。 | [🤗](https://huggingface.co/openbmb/MiniCPM-V-2_6-gguf)    [](https://modelscope.cn/models/OpenBMB/MiniCPM-V-2_6-gguf) | | MiniCPM-V 2.6 int4 | GPU | 7 GB | int4量化版，更低显存占用。 | [🤗](https://huggingface.co/openbmb/MiniCPM-V-2_6-int4)    [](https://modelscope.cn/models/OpenBMB/MiniCPM-V-2_6-int4) | -| MiniCPM-Llama3-V 2.5| GPU | 19 GB | 提供出色的端侧多模态理解能力。 | [🤗](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5/)    [](https://modelscope.cn/models/OpenBMB/MiniCPM-Llama3-V-2_5) | -| MiniCPM-Llama3-V 2.5 gguf | CPU | 6 GB | gguf 版本，更低的内存占用和更高的推理效率。 | [🤗](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5-gguf)    [](https://modelscope.cn/models/OpenBMB/MiniCPM-Llama3-V-2_5-gguf) | -| MiniCPM-Llama3-V 2.5 int4 | GPU | 8 GB | int4量化版，更低显存占用。 | [🤗](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5-int4/)    [](https://modelscope.cn/models/OpenBMB/MiniCPM-Llama3-V-2_5-int4) | -| MiniCPM-V 2.0 | GPU | 8 GB | 轻量级版本，平衡计算开销和多模态理解能力。 | [🤗](https://huggingface.co/openbmb/MiniCPM-V-2)    [](https://modelscope.cn/models/OpenBMB/MiniCPM-V-2) | -| MiniCPM-V 1.0 | GPU | 7 GB | 最轻量版本， 提供最快的推理速度。 | [🤗](https://huggingface.co/openbmb/MiniCPM-V)    [](https://modelscope.cn/models/OpenBMB/MiniCPM-V) | 更多[历史版本模型](#legacy-models) + ### 多轮对话 -请参考以下代码进行推理。
- +
@@ -1316,60 +1862,57 @@ import torch from PIL import Image from transformers import AutoModel, AutoTokenizer -torch.manual_seed(0) +torch.manual_seed(100) -model = AutoModel.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True, +model = AutoModel.from_pretrained('openbmb/MiniCPM-o-2_6', trust_remote_code=True, attn_implementation='sdpa', torch_dtype=torch.bfloat16) # sdpa or flash_attention_2, no eager model = model.eval().cuda() -tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True) +tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-o-2_6', trust_remote_code=True) -image = Image.open('./assets/airplane.jpeg').convert('RGB') +image = Image.open('./assets/minicpmo2_6/show_demo.jpg').convert('RGB') # First round chat -question = "Tell me the model of this aircraft." +question = "What is the landform in the picture?" msgs = [{'role': 'user', 'content': [image, question]}] answer = model.chat( - image=None, msgs=msgs, tokenizer=tokenizer ) print(answer) -# Second round chat -# pass history context of multi-turn conversation +# Second round chat, pass history context of multi-turn conversation msgs.append({"role": "assistant", "content": [answer]}) -msgs.append({"role": "user", "content": ["Introduce something about Airbus A380."]}) +msgs.append({"role": "user", "content": ["What should I pay attention to when traveling here?"]}) answer = model.chat( - image=None, msgs=msgs, tokenizer=tokenizer ) print(answer) ``` -可以得到以下输出: +你可以得到如下推理结果： ``` -"The aircraft in the image is an Airbus A380, which can be identified by its large size, double-deck structure, and the distinctive shape of its wings and engines. The A380 is a wide-body aircraft known for being the world's largest passenger airliner, designed for long-haul flights. It has four engines, which are characteristic of large commercial aircraft. The registration number on the aircraft can also provide specific information about the model if looked up in an aviation database." +"The landform in the picture is a mountain range. The mountains appear to be karst formations, characterized by their steep, rugged peaks and smooth, rounded shapes. These types of mountains are often found in regions with limestone bedrock and are shaped by processes such as erosion and weathering. The reflection of the mountains in the water adds to the scenic beauty of the landscape." -"The Airbus A380 is a double-deck, wide-body, four-engine jet airliner made by Airbus. It is the world's largest passenger airliner and is known for its long-haul capabilities. The aircraft was developed to improve efficiency and comfort for passengers traveling over long distances. It has two full-length passenger decks, which can accommodate more passengers than a typical single-aisle airplane. The A380 has been operated by airlines such as Lufthansa, Singapore Airlines, and Emirates, among others. It is widely recognized for its unique design and significant impact on the aviation industry." +"When traveling to this scenic location, it's important to pay attention to the weather conditions, as the area appears to be prone to fog and mist, especially during sunrise or sunset. Additionally, ensure you have proper footwear for navigating the potentially slippery terrain around the water. Lastly, respect the natural environment by not disturbing the local flora and fauna." ``` -#### 多图理解 +#### 多图对话
- 点击查看使用 MiniCPM-V 2.6 进行多图理解的Python示例 + 点击查看 MiniCPM-o 2.6 多图输入的 Python 代码。 ```python import torch from PIL import Image from transformers import AutoModel, AutoTokenizer -model = AutoModel.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True, +model = AutoModel.from_pretrained('openbmb/MiniCPM-o-2_6', trust_remote_code=True, attn_implementation='sdpa', torch_dtype=torch.bfloat16) # sdpa or flash_attention_2, no eager model = model.eval().cuda() -tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True) +tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-o-2_6', trust_remote_code=True) image1 = Image.open('image1.jpg').convert('RGB') image2 = Image.open('image2.jpg').convert('RGB') @@ -1378,7 +1921,6 @@ question = 'Compare image 1 and image 2, tell me about the differences between i msgs = [{'role': 'user', 'content': [image1, image2, question]}] answer = model.chat( - image=None, msgs=msgs, tokenizer=tokenizer ) @@ -1386,20 +1928,19 @@ print(answer) ```
-#### 少样本上下文学习 - +#### 少样本上下文对话
- 点击查看使用 MiniCPM-V 2.6 进行few-shot推理的Python示例 + 点击查看 MiniCPM-o 2.6 少样本上下文对话的 Python 代码。 ```python import torch from PIL import Image from transformers import AutoModel, AutoTokenizer -model = AutoModel.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True, +model = AutoModel.from_pretrained('openbmb/MiniCPM-o-2_6', trust_remote_code=True, attn_implementation='sdpa', torch_dtype=torch.bfloat16) # sdpa or flash_attention_2, no eager model = model.eval().cuda() -tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True) +tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-o-2_6', trust_remote_code=True) question = "production date" image1 = Image.open('example1.jpg').convert('RGB') @@ -1415,7 +1956,6 @@ msgs = [ ] answer = model.chat( - image=None, msgs=msgs, tokenizer=tokenizer ) @@ -1423,9 +1963,9 @@ print(answer) ```
-#### 视频理解 +#### 视频对话
- 点击查看使用 MiniCPM-V 2.6 进行视频理解的Python示例 + 点击查看 MiniCPM-o 2.6 视频输入的 Python 代码。 ```python import torch @@ -1433,10 +1973,10 @@ from PIL import Image from transformers import AutoModel, AutoTokenizer from decord import VideoReader, cpu # pip install decord -model = AutoModel.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True, +model = AutoModel.from_pretrained('openbmb/MiniCPM-o-2_6', trust_remote_code=True, attn_implementation='sdpa', torch_dtype=torch.bfloat16) # sdpa or flash_attention_2, no eager model = model.eval().cuda() -tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True) +tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-o-2_6', trust_remote_code=True) MAX_NUM_FRAMES=64 # if cuda OOM set a smaller number @@ -1466,10 +2006,9 @@ msgs = [ # Set decode params for video params = {} params["use_image_id"] = False -params["max_slice_nums"] = 2 # 如果cuda OOM且视频分辨率大于448*448可设为1 +params["max_slice_nums"] = 2 # use 1 if cuda OOM and video resolution > 448*448 answer = model.chat( - image=None, msgs=msgs, tokenizer=tokenizer, **params @@ -1479,6 +2018,295 @@ print(answer)
+#### 语音对话 +
初始化模型 + +```python +import torch +import librosa +from transformers import AutoModel, AutoTokenizer + +model = AutoModel.from_pretrained('openbmb/MiniCPM-o-2_6', trust_remote_code=True, + attn_implementation='sdpa', torch_dtype=torch.bfloat16) # sdpa or flash_attention_2, no eager +model = model.eval().cuda() +tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-o-2_6', trust_remote_code=True) + +model.init_tts() +model.tts.float() +``` + +
+ +##### Mimick + +
点击查看 MiniCPM-o 2.6 端到端语音理解生成的 Python 代码。 + +- `Mimick` 任务反映了模型的端到端语音建模能力。模型接受音频输入，输出语音识别（ASR）转录结果，并随后以高相似度重建原始音频。重建的音频相似度和原始音频越高，表明模型有越高的语音端到端建模基础能力。 +```python +mimick_prompt = "Please repeat each user's speech, including voice style and speech content." +audio_input, _ = librosa.load('xxx.wav', sr=16000, mono=True) +msgs = [{'role': 'user', 'content': [mimick_prompt,audio_input]}] +res = model.chat( + msgs=msgs, + tokenizer=tokenizer, + sampling=True, + max_new_tokens=128, + use_tts_template=True, + temperature=0.3, + generate_audio=True, + output_audio_path='output.wav', # save the tts result to output_audio_path +) +``` + +
+ +##### 可配置声音的语音对话 +
点击查看个性化配置 MiniCPM-o 2.6 对话声音的 Python 代码。 + +```python +ref_audio, _ = librosa.load('./assets/voice_01.wav', sr=16000, mono=True) # load the reference audio + +# Audio RolePlay: # With this mode, model will role-play the character based on the audio prompt. +sys_prompt = model.get_sys_prompt(ref_audio=ref_audio, mode='audio_roleplay', language='en') +user_question = {'role': 'user', 'content': [librosa.load('xxx.wav', sr=16000, mono=True)[0]]} + +# Audio Assistant: # With this mode, model will speak with the voice in ref_audio as a AI assistant. +# sys_prompt = model.get_sys_prompt(ref_audio=ref_audio, mode='audio_assistant', language='en') +# user_question = {'role': 'user', 'content': [librosa.load('xxx.wav', sr=16000, mono=True)[0]]} # Try to ask something! +``` +```python +msgs = [sys_prompt, user_question] +res = model.chat( + msgs=msgs, + tokenizer=tokenizer, + sampling=True, + max_new_tokens=128, + use_tts_template=True, + generate_audio=True, + temperature=0.3, + output_audio_path='result.wav', +) + +# round two +history = msgs.append({'role': 'assistant', 'content': res}) +user_question = {'role': 'user', 'content': [librosa.load('xxx.wav', sr=16000, mono=True)[0]]} +msgs = history.append(user_question) +res = model.chat( + msgs=msgs, + tokenizer=tokenizer, + sampling=True, + max_new_tokens=128, + use_tts_template=True, + generate_audio=True, + temperature=0.3, + output_audio_path='result_round_2.wav', +) +print(res) +``` + +
+ +##### 更多语音任务 +
+ 点击查看 MiniCPM-o 2.6 完成更多语音任务的 Python 代码。 + +```python +''' +Audio Understanding Task Prompt: +Speech: + ASR with ZH(same as AST en2zh): 请仔细听这段音频片段，并将其内容逐字记录。 + ASR with EN(same as AST zh2en): Please listen to the audio snippet carefully and transcribe the content. + Speaker Analysis: Based on the speaker's content, speculate on their gender, condition, age range, and health status. +General Audio: + Audio Caption: Summarize the main content of the audio. + Sound Scene Tagging: Utilize one keyword to convey the audio's content or the associated scene. +''' +task_prompt = "\n" +audio_input, _ = librosa.load('xxx.wav', sr=16000, mono=True) + +msgs = [{'role': 'user', 'content': [task_prompt,audio_input]}] + +res = model.chat( + msgs=msgs, + tokenizer=tokenizer, + sampling=True, + max_new_tokens=128, + use_tts_template=True, + generate_audio=True, + temperature=0.3, + output_audio_path='result.wav', +) +print(res) +``` +```python +''' +Speech Generation Task Prompt: + Human Instruction-to-Speech: see https://voxinstruct.github.io/VoxInstruct/ + Example: + # 在新闻中，一个年轻男性兴致勃勃地说：“祝福亲爱的祖国母亲美丽富强！”他用低音调和低音量，慢慢地说出了这句话。 + # Delighting in a surprised tone, an adult male with low pitch and low volume comments:"One even gave my little dog a biscuit" This dialogue takes place at a leisurely pace, delivering a sense of excitement and surprise in the context. + + Voice Cloning or Voice Creation: With this mode, model will act like a TTS model. +''' +# Human Instruction-to-Speech: +task_prompt = '' #Try to make some Human Instruction-to-Speech prompt +msgs = [{'role': 'user', 'content': [task_prompt]}] # you can try to use the same audio question + +# Voice Cloning mode: With this mode, model will act like a TTS model. +# sys_prompt = model.get_sys_prompt(ref_audio=ref_audio, mode='voice_cloning', language='en') +# text_prompt = f"Please read the text below." +# user_question = {'role': 'user', 'content': [text_prompt, "content that you want to read"]} # using same voice in sys_prompt to read the text. (Voice Cloning) +# user_question = {'role': 'user', 'content': [text_prompt, librosa.load('xxx.wav', sr=16000, mono=True)[0]]} # using same voice in sys_prompt to read 'xxx.wav'. (Voice Creation) + +msgs = [sys_prompt, user_question] +res = model.chat( + msgs=msgs, + tokenizer=tokenizer, + sampling=True, + max_new_tokens=128, + use_tts_template=True, + generate_audio=True, + temperature=0.3, + output_audio_path='result.wav', +) + + +``` + +
+ +#### 多模态流式交互 +
+ 点击查看 MiniCPM-o 2.6 多模态流式交互的 Python 代码。 + +```python +import math +import numpy as np +from PIL import Image +from moviepy.editor import VideoFileClip +import tempfile +import librosa +import soundfile as sf + +## make sure The model has been initialized and `model.init_tts()` has been executed + +def get_video_chunk_content(video_path, flatten=True): + video = VideoFileClip(video_path) + print('video_duration:', video.duration) + + with tempfile.NamedTemporaryFile(suffix=".wav", delete=True) as temp_audio_file: + temp_audio_file_path = temp_audio_file.name + video.audio.write_audiofile(temp_audio_file_path, codec="pcm_s16le", fps=16000) + audio_np, sr = librosa.load(temp_audio_file_path, sr=16000, mono=True) + num_units = math.ceil(video.duration) + + # 1 frame + 1s audio chunk + contents= [] + for i in range(num_units): + frame = video.get_frame(i+1) + image = Image.fromarray((frame).astype(np.uint8)) + audio = audio_np[sr*i:sr*(i+1)] + if flatten: + contents.extend(["", image, audio]) + else: + contents.append(["", image, audio]) + + return contents + +video_path="/path/to/video" +sys_msg = model.get_sys_prompt(mode='omni', language='en') +# if use voice clone prompt, please set ref_audio +# ref_audio_path = '/path/to/ref_audio' +# ref_audio, _ = librosa.load(ref_audio_path, sr=16000, mono=True) +# sys_msg = model.get_sys_prompt(ref_audio=ref_audio, mode='omni', language='en') + +contents = get_video_chunk_content(video_path) +msg = {"role":"user", "content": contents} +msgs = [sys_msg, msg] + +# please set generate_audio=True and output_audio_path to save the tts result +generate_audio = True +output_audio_path = 'output.wav' + +res = model.chat( + msgs=msgs, + tokenizer=tokenizer, + sampling=True, + temperature=0.5, + max_new_tokens=4096, + omni_input=True, # please set omni_input=True when omni inference + use_tts_template=True, + generate_audio=generate_audio, + output_audio_path=output_audio_path, + max_slice_nums=1, + use_image_id=False, + return_dict=True +) +print(res) +``` +
+ +
+ 点击查看多模态流式推理设置。 + +注意：流式推理存在轻微的性能下降，因为音频编码并非全局的。 +```python +# a new conversation need reset session first, it will reset the kv-cache +model.reset_session() + +contents = get_video_chunk_content(video_path, flatten=False) +session_id = '123' +generate_audio = True + +# 1. prefill system prompt +res = model.streaming_prefill( + session_id=session_id, + msgs=[sys_msg], + tokenizer=tokenizer +) + +# 2. prefill video/audio chunks +for content in contents: + msgs = [{"role":"user", "content": content}] + res = model.streaming_prefill( + session_id=session_id, + msgs=msgs, + tokenizer=tokenizer + ) + +# 3. generate +res = model.streaming_generate( + session_id=session_id, + tokenizer=tokenizer, + temperature=0.5, + generate_audio=generate_audio +) + +audios = [] +text = "" + +if generate_audio: + for r in res: + audio_wav = r.audio_wav + sampling_rate = r.sampling_rate + txt = r.text + + audios.append(audio_wav) + text += txt + + res = np.concatenate(audios) + sf.write("output.wav", res, samplerate=sampling_rate) + print("text:", text) + print("audio saved to output.wav") +else: + for r in res: + text += r['text'] + print("text:", text) +``` + +
+ + ### 多卡推理 您可以通过将模型的层分布在多个低显存显卡（12 GB 或 16 GB）上，运行 MiniCPM-Llama3-V 2.5。请查看该[教程](https://github.com/OpenBMB/MiniCPM-V/blob/main/docs/inference_on_multiple_gpus.md)，详细了解如何使用多张低显存显卡载入模型并进行推理。 @@ -1536,97 +2364,110 @@ python web_demo_2.6.py --device cuda ```
-### llama.cpp 部署 -MiniCPM-V 2.6 现在支持llama.cpp啦! 用法请参考[我们的fork llama.cpp](https://github.com/OpenBMB/llama.cpp/tree/minicpmv-main/examples/llava/README-minicpmv2.6.md)， 在iPad上可以支持 16~18 token/s 的流畅推理（测试环境：iPad Pro + M4）。 +### 基于 llama.cpp、ollama、vLLM 的高效推理 -### ollama 部署 -MiniCPM-V 2.6 现在支持ollama啦! 用法请参考[我们的fork ollama](https://github.com/OpenBMB/ollama/blob/minicpm-v2.6/examples/minicpm-v2.6/README.md)， 在iPad上可以支持 16~18 token/s 的流畅推理（测试环境：iPad Pro + M4）。 +llama.cpp 用法请参考[我们的fork llama.cpp](https://github.com/OpenBMB/llama.cpp/tree/minicpmv-main/examples/llava/README-minicpmv2.6.md)， 在iPad上可以支持 16~18 token/s 的流畅推理（测试环境：iPad Pro + M4）。 + +ollama 用法请参考[我们的fork ollama](https://github.com/OpenBMB/ollama/blob/minicpm-v2.6/examples/minicpm-v2.6/README.md)， 在iPad上可以支持 16~18 token/s 的流畅推理（测试环境：iPad Pro + M4）。 -### vLLM 部署
-点击查看, vLLM 现已官方支持MiniCPM-V 2.6、MiniCPM-Llama3-V 2.5 和 MiniCPM-V 2.0 +点击查看, vLLM 现已官方支持MiniCPM-V 2.6、MiniCPM-Llama3-V 2.5 和 MiniCPM-V 2.0，MiniCPM-o 2.6 模型也可以临时用我们的 fork 仓库运行。 +1. MiniCPM-o 2.6 + 1. 克隆我们的 vLLM fork 仓库: + ```shell + git clone https://github.com/OpenBMB/vllm.git + cd vllm + git checkout minicpmo + ``` + 2. 从源码进行安装: + ```shell + VLLM_USE_PRECOMPILED=1 pip install --editable . + ``` + 3. 用和之前同样的方式运行（下有样例）. -1. 安装 vLLM(>=0.5.4): -```shell -pip install vllm -``` -3. 安装 timm 库: （可选，MiniCPM-V 2.0需安装） -```shell -pip install timm=0.9.10 -``` -4. 运行示例代码:（注意：如果使用本地路径的模型，请确保模型代码已更新到Hugging Face上的最新版) -```python -from transformers import AutoTokenizer -from PIL import Image -from vllm import LLM, SamplingParams +2. 之前版本的 MiniCPM-V + 1. 安装 vLLM(>=0.5.4): + ```shell + pip install vllm + ``` + 3. 安装 timm 库: （可选，MiniCPM-V 2.0需安装） + ```shell + pip install timm=0.9.10 + ``` + 4. 运行示例代码:（注意：如果使用本地路径的模型，请确保模型代码已更新到Hugging Face上的最新版) + ```python + from transformers import AutoTokenizer + from PIL import Image + from vllm import LLM, SamplingParams -MODEL_NAME = "openbmb/MiniCPM-V-2_6" -# Also available for previous models -# MODEL_NAME = "openbmb/MiniCPM-Llama3-V-2_5" -# MODEL_NAME = "HwwwH/MiniCPM-V-2" + MODEL_NAME = "openbmb/MiniCPM-V-2_6" + # MODEL_NAME = "openbmb/MiniCPM-O-2_6" + # Also available for previous models + # MODEL_NAME = "openbmb/MiniCPM-Llama3-V-2_5" + # MODEL_NAME = "HwwwH/MiniCPM-V-2" -image = Image.open("xxx.png").convert("RGB") -tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME, trust_remote_code=True) -llm = LLM( - model=MODEL_NAME, - trust_remote_code=True, - gpu_memory_utilization=1, - max_model_len=2048 -) + image = Image.open("xxx.png").convert("RGB") + tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME, trust_remote_code=True) + llm = LLM( + model=MODEL_NAME, + trust_remote_code=True, + gpu_memory_utilization=1, + max_model_len=2048 + ) -messages = [{ - "role": - "user", - "content": - # Number of images - "(./)" + \ - "\nWhat is the content of this image?" -}] -prompt = tokenizer.apply_chat_template( - messages, - tokenize=False, - add_generation_prompt=True -) + messages = [{ + "role": + "user", + "content": + # Number of images + "(./)" + \ + "\nWhat is the content of this image?" + }] + prompt = tokenizer.apply_chat_template( + messages, + tokenize=False, + add_generation_prompt=True + ) -# Single Inference -inputs = { - "prompt": prompt, - "multi_modal_data": { - "image": image - # Multi images, the number of images should be equal to that of `(./)` - # "image": [image, image] - }, -} -# Batch Inference -# inputs = [{ -# "prompt": prompt, -# "multi_modal_data": { -# "image": image -# }, -# } for _ in 2] + # Single Inference + inputs = { + "prompt": prompt, + "multi_modal_data": { + "image": image + # Multi images, the number of images should be equal to that of `(./)` + # "image": [image, image] + }, + } + # Batch Inference + # inputs = [{ + # "prompt": prompt, + # "multi_modal_data": { + # "image": image + # }, + # } for _ in 2] -# 2.6 -stop_tokens = ['<|im_end|>', '<|endoftext|>'] -stop_token_ids = [tokenizer.convert_tokens_to_ids(i) for i in stop_tokens] -# 2.0 -# stop_token_ids = [tokenizer.eos_id] -# 2.5 -# stop_token_ids = [tokenizer.eos_id, tokenizer.eot_id] + # 2.6 + stop_tokens = ['<|im_end|>', '<|endoftext|>'] + stop_token_ids = [tokenizer.convert_tokens_to_ids(i) for i in stop_tokens] + # 2.0 + # stop_token_ids = [tokenizer.eos_id] + # 2.5 + # stop_token_ids = [tokenizer.eos_id, tokenizer.eot_id] -sampling_params = SamplingParams( - stop_token_ids=stop_token_ids, - use_beam_search=True, - temperature=0, - best_of=3, - max_tokens=1024 -) + sampling_params = SamplingParams( + stop_token_ids=stop_token_ids, + use_beam_search=True, + temperature=0, + best_of=3, + max_tokens=1024 + ) -outputs = llm.generate(inputs, sampling_params=sampling_params) + outputs = llm.generate(inputs, sampling_params=sampling_params) -print(outputs[0].outputs[0].text) -``` -4. [点击此处](https://modelbest.feishu.cn/wiki/C2BWw4ZP0iCDy7kkCPCcX2BHnOf?from=from_copylink)查看带视频推理和其他有关 `vLLM` 的信息。 + print(outputs[0].outputs[0].text) + ``` + 4. [点击此处](https://modelbest.feishu.cn/wiki/C2BWw4ZP0iCDy7kkCPCcX2BHnOf?from=from_copylink)查看带视频推理和其他有关 `vLLM` 的信息。
@@ -1635,10 +2476,17 @@ print(outputs[0].outputs[0].text) ### 简易微调 -我们支持使用 Huggingface Transformers 库简易地微调 MiniCPM-V 2.0 和 MiniCPM-Llama3-V 2.5 模型。 +我们支持使用 Huggingface Transformers 库简易地微调 MiniCPM-o 2.6、MiniCPM-V 2.6、MiniCPM-Llama3-V 2.5 和 MiniCPM-V 2.0 模型。 [参考文档](./finetune/readme.md) +### 使用 LLaMA-Factory + +我们支持使用 LLaMA-Factory 微调 MiniCPM-o-2.6 和 MiniCPM-V 2.6。LLaMA-Factory 提供了一种灵活定制 200 多个大型语言模型（LLM）微调（Lora/Full/Qlora）解决方案，无需编写代码，通过内置的 Web 用户界面 LLaMABoard 即可实现训练/推理/评估。它支持多种训练方法，如 sft/ppo/dpo/kto，并且还支持如 Galore/BAdam/LLaMA-Pro/Pissa/LongLoRA 等高级算法。 + +最佳实践: [MiniCPM-V-2.6 | MiniCPM-o-2.6](https://github.com/openbmb/MiniCPM-V/blob/main/docs/llamafactory_train.md). + + ### 使用 SWIFT 框架 我们支持使用 SWIFT 框架微调 MiniCPM-V 系列模型。SWIFT 支持近 200 种大语言模型和多模态大模型的训练、推理、评测和部署。支持 PEFT 提供的轻量训练方案和完整的 Adapters 库支持的最新训练技术如 NEFTune、LoRA+、LLaMA-PRO 等。 @@ -1649,15 +2497,23 @@ print(outputs[0].outputs[0].text) 点击查看 [FAQs](./docs/faqs.md) +## 模型局限性 + +我们实验发现 MiniCPM-o 2.6 存在一些显著的局限性，需要进一步研究和改进： +- **不稳定的语音输出。** 语音生成可能会受到背景噪音和无意义声音的影响，表现不稳定。 +- **重复响应。** 当遇到连续相似的用户请求时，模型往往会重复相同的回答。 +- **Web Demo 延迟较高。** 用户在使用远程服务器上部署的 web demo 时可能会产生较高延迟。我们推荐用户在本地部署来获得更低延迟的体验。 + + ## 模型协议 * 本仓库中代码依照 [Apache-2.0](https://github.com/OpenBMB/MiniCPM/blob/main/LICENSE) 协议开源 -* MiniCPM-V 模型权重的使用则需要遵循 [“MiniCPM模型商用许可协议.md”](https://github.com/OpenBMB/MiniCPM/blob/main/MiniCPM%E6%A8%A1%E5%9E%8B%E5%95%86%E7%94%A8%E8%AE%B8%E5%8F%AF%E5%8D%8F%E8%AE%AE.md)。 +* MiniCPM-o/V 模型权重的使用则需要遵循 [“MiniCPM模型商用许可协议.md”](https://github.com/OpenBMB/MiniCPM/blob/main/MiniCPM%E6%A8%A1%E5%9E%8B%E5%95%86%E7%94%A8%E8%AE%B8%E5%8F%AF%E5%8D%8F%E8%AE%AE.md)。 * MiniCPM 模型权重对学术研究完全开放，在填写[“问卷”](https://modelbest.feishu.cn/share/base/form/shrcnpV5ZT9EJ6xYjh3Kx0J6v8g)进行登记后亦允许免费商业使用。 ## 声明 -作为多模态大模型，MiniCPM-V 系列模型（包括 OmniLMM）通过学习大量的多模态数据来生成内容，但它无法理解、表达个人观点或价值判断，它所输出的任何内容都不代表模型开发者的观点和立场。 +作为多模态大模型，MiniCPM-o/V 系列模型（包括 OmniLMM）通过学习大量的多模态数据来生成内容，但它无法理解、表达个人观点或价值判断，它所输出的任何内容都不代表模型开发者的观点和立场。 因此用户在使用本项目的系列模型生成的内容时，应自行负责对其进行评估和验证。如果由于使用本项目的系列开源模型而导致的任何问题，包括但不限于数据安全问题、公共舆论风险，或模型被误导、滥用、传播或不当利用所带来的任何风险和问题，我们将不承担任何责任。 @@ -1668,7 +2524,6 @@ print(outputs[0].outputs[0].text) - [清华大学自然语言处理实验室](https://nlp.csai.tsinghua.edu.cn/) - [面壁智能](https://modelbest.cn/) -- [知乎](https://www.zhihu.com/ ) ## 🌟 Star History @@ -1700,7 +2555,7 @@ print(outputs[0].outputs[0].text) ## 支持技术和其他多模态项目 -👏 欢迎了解 MiniCPM-V 背后的支持技术和更多我们的多模态项目！ +👏 欢迎了解 MiniCPM-o/V 背后的支持技术和更多我们的多模态项目！ [VisCPM](https://github.com/OpenBMB/VisCPM/tree/main) | [RLHF-V](https://github.com/RLHF-V/RLHF-V) | [LLaVA-UHD](https://github.com/thunlp/LLaVA-UHD) | [RLAIF-V](https://github.com/RLHF-V/RLAIF-V) diff --git a/assets/MiniCPM-o.png b/assets/MiniCPM-o.png new file mode 100644 index 0000000..20fa91e Binary files /dev/null and b/assets/MiniCPM-o.png differ diff --git a/assets/discord.png b/assets/discord.png new file mode 100644 index 0000000..c3067a4 Binary files /dev/null and b/assets/discord.png differ diff --git a/assets/logo.html b/assets/logo.html new file mode 100644 index 0000000..71257de --- /dev/null +++ b/assets/logo.html @@ -0,0 +1,3 @@ + + MiniCPM-o + \ No newline at end of file diff --git a/assets/minicpm-o-26-framework.png b/assets/minicpm-o-26-framework.png new file mode 100644 index 0000000..459887e Binary files /dev/null and b/assets/minicpm-o-26-framework.png differ diff --git a/assets/minicpmo2_6/minicpmo2_6_diagram_train_NN.png b/assets/minicpmo2_6/minicpmo2_6_diagram_train_NN.png new file mode 100644 index 0000000..eeef5f2 Binary files /dev/null and b/assets/minicpmo2_6/minicpmo2_6_diagram_train_NN.png differ diff --git a/assets/minicpmo2_6/minicpmo2_6_math_intersect.png b/assets/minicpmo2_6/minicpmo2_6_math_intersect.png new file mode 100644 index 0000000..f526b1c Binary files /dev/null and b/assets/minicpmo2_6/minicpmo2_6_math_intersect.png differ diff --git a/assets/minicpmo2_6/minicpmo2_6_multi-image_bike.png b/assets/minicpmo2_6/minicpmo2_6_multi-image_bike.png new file mode 100644 index 0000000..090337b Binary files /dev/null and b/assets/minicpmo2_6/minicpmo2_6_multi-image_bike.png differ diff --git a/assets/minicpmo2_6/show_demo.jpg b/assets/minicpmo2_6/show_demo.jpg new file mode 100644 index 0000000..40ec4fb Binary files /dev/null and b/assets/minicpmo2_6/show_demo.jpg differ diff --git a/assets/o-2dot6-demo-video-preview.png b/assets/o-2dot6-demo-video-preview.png new file mode 100644 index 0000000..8e34ab4 Binary files /dev/null and b/assets/o-2dot6-demo-video-preview.png differ diff --git a/assets/radar.jpg b/assets/radar.jpg new file mode 100644 index 0000000..51f75bc Binary files /dev/null and b/assets/radar.jpg differ diff --git a/assets/ref_audios/default.wav b/assets/ref_audios/default.wav new file mode 100644 index 0000000..8171eee Binary files /dev/null and b/assets/ref_audios/default.wav differ diff --git a/assets/ref_audios/female_example.wav b/assets/ref_audios/female_example.wav new file mode 100644 index 0000000..4f795b2 Binary files /dev/null and b/assets/ref_audios/female_example.wav differ diff --git a/assets/ref_audios/male_example.wav b/assets/ref_audios/male_example.wav new file mode 100644 index 0000000..09e725b Binary files /dev/null and b/assets/ref_audios/male_example.wav differ diff --git a/assets/ref_audios/video_default.wav b/assets/ref_audios/video_default.wav new file mode 100644 index 0000000..2e6061b Binary files /dev/null and b/assets/ref_audios/video_default.wav differ diff --git a/assets/wechat.png b/assets/wechat.png new file mode 100644 index 0000000..8a109ef Binary files /dev/null and b/assets/wechat.png differ diff --git a/docs/minicpm_llama3_v2dot5.md b/docs/minicpm_llama3_v2dot5.md new file mode 100644 index 0000000..7ab8700 --- /dev/null +++ b/docs/minicpm_llama3_v2dot5.md @@ -0,0 +1,333 @@ +## MiniCPM-Llama3-V 2.5 + +> Archieve at: 2025-01-13 + + +**MiniCPM-Llama3-V 2.5** is the latest model in the MiniCPM-V series. The model is built on SigLip-400M and Llama3-8B-Instruct with a total of 8B parameters. It exhibits a significant performance improvement over MiniCPM-V 2.0. Notable features of MiniCPM-Llama3-V 2.5 include: + +- 🔥 **Leading Performance.** + MiniCPM-Llama3-V 2.5 has achieved an average score of 65.1 on OpenCompass, a comprehensive evaluation over 11 popular benchmarks. **With only 8B parameters, it surpasses widely used proprietary models like GPT-4V-1106, Gemini Pro, Claude 3 and Qwen-VL-Max** and greatly outperforms other Llama 3-based MLLMs. + +- 💪 **Strong OCR Capabilities.** + MiniCPM-Llama3-V 2.5 can process images with any aspect ratio and up to 1.8 million pixels (e.g., 1344x1344), achieving a **700+ score on OCRBench, surpassing proprietary models such as GPT-4o, GPT-4V-0409, Qwen-VL-Max and Gemini Pro**. Based on recent user feedback, MiniCPM-Llama3-V 2.5 has now enhanced full-text OCR extraction, table-to-markdown conversion, and other high-utility capabilities, and has further strengthened its instruction-following and complex reasoning abilities, enhancing multimodal interaction experiences. + +- 🏆 **Trustworthy Behavior.** + Leveraging the latest [RLAIF-V](https://github.com/RLHF-V/RLAIF-V/) method (the newest technique in the [RLHF-V](https://github.com/RLHF-V) [CVPR'24] series), MiniCPM-Llama3-V 2.5 exhibits more trustworthy behavior. It achieves a **10.3%** hallucination rate on Object HalBench, lower than GPT-4V-1106 (13.6%), achieving the best-level performance within the open-source community. [Data released](https://huggingface.co/datasets/openbmb/RLAIF-V-Dataset). + +- 🌏 **Multilingual Support.** + Thanks to the strong multilingual capabilities of Llama 3 and the cross-lingual generalization technique from [VisCPM](https://github.com/OpenBMB/VisCPM), MiniCPM-Llama3-V 2.5 extends its bilingual (Chinese-English) multimodal capabilities to **over 30 languages including German, French, Spanish, Italian, Korean etc.** [All Supported Languages](./assets/minicpm-llama-v-2-5_languages.md). + +- 🚀 **Efficient Deployment.** + MiniCPM-Llama3-V 2.5 systematically employs **model quantization, CPU optimizations, NPU optimizations and compilation optimizations**, achieving high-efficiency deployment on end-side devices. For mobile phones with Qualcomm chips, we have integrated the NPU acceleration framework QNN into llama.cpp for the first time. After systematic optimization, MiniCPM-Llama3-V 2.5 has realized a **150x acceleration in end-side MLLM image encoding** and a **3x speedup in language decoding**. + +- 💫 **Easy Usage.** +MiniCPM-Llama3-V 2.5 can be easily used in various ways: (1) [llama.cpp](https://github.com/OpenBMB/llama.cpp/blob/minicpm-v2.5/examples/minicpmv/README.md) and [ollama](https://github.com/OpenBMB/ollama/tree/minicpm-v2.5/examples/minicpm-v2.5) support for efficient CPU inference on local devices, (2) [GGUF](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5-gguf) format quantized models in 16 sizes, (3) efficient [LoRA](https://github.com/OpenBMB/MiniCPM-V/tree/main/finetune#lora-finetuning) fine-tuning with only 2 V100 GPUs, (4) [streaming output](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5#usage), (5) quick local WebUI demo setup with [Gradio](https://github.com/OpenBMB/MiniCPM-V/blob/main/web_demo_2.5.py) and [Streamlit](https://github.com/OpenBMB/MiniCPM-V/blob/main/web_demo_streamlit-2_5.py), and (6) interactive demos on [HuggingFace Spaces](https://huggingface.co/spaces/openbmb/MiniCPM-Llama3-V-2_5). + +### Evaluation + +
+ +
+
+Click to view results on TextVQA, DocVQA, OCRBench, OpenCompass, MME, MMBench, MMMU, MathVista, LLaVA Bench, RealWorld QA, Object HalBench. +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ModelSizeOCRBenchTextVQA valDocVQA testOpen-CompassMMEMMB test (en)MMB test (cn)MMMU valMath-VistaLLaVA BenchRealWorld QAObject HalBench
Proprietary
Gemini Pro-68074.688.162.92148.973.674.348.945.879.960.4-
GPT-4V (2023.11.06)-64578.088.463.51771.577.074.453.847.893.163.086.4
Open-source
Mini-Gemini2.2B-56.234.2*-1653.0--31.7----
Qwen-VL-Chat9.6B48861.562.651.61860.061.856.337.033.867.749.356.2
DeepSeek-VL-7B7.3B43564.7*47.0*54.61765.473.871.438.336.877.854.2-
Yi-VL-34B34B29043.4*16.9*52.22050.272.470.745.130.762.354.879.3
CogVLM-Chat17.4B59070.433.3*54.21736.665.855.937.334.773.960.373.6
TextMonkey9.7B55864.366.7---------
Idefics28.0B-73.074.057.21847.675.768.645.252.249.160.7-
Bunny-LLama-3-8B8.4B---54.31920.377.073.941.331.561.258.8-
LLaVA-NeXT Llama-3-8B8.4B--78.2-1971.5--41.737.580.160.0-
Phi-3-vision-128k-instruct4.2B639*70.9--1537.5*--40.444.564.2*58.8*-
MiniCPM-V 1.02.8B36660.638.247.51650.264.162.638.328.951.351.278.4
MiniCPM-V 2.02.8B60574.171.954.51808.669.166.538.238.769.255.885.5
MiniCPM-Llama3-V 2.58.5B72576.684.865.12024.677.274.245.854.386.763.589.7
+ + +
+* We evaluate the officially released checkpoint by ourselves. + +
+ +
+ +
+ Evaluation results of multilingual LLaVA Bench +
+ +### Examples + + +

+ +

+
+ +
+ + +### Model Zoo + +| Model | Device | Memory |          Description | Download | +|:-----------|:--:|:-----------:|:-------------------|:---------------:| +| MiniCPM-Llama3-V 2.5 | GPU | 19 GB | Strong end-side multimodal performance. | [🤗](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5/)    [](https://modelscope.cn/models/OpenBMB/MiniCPM-Llama3-V-2_5) | +| MiniCPM-Llama3-V 2.5 gguf | CPU | 6 GB | The gguf version, lower memory usage and faster inference. | [🤗](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5-gguf)   [](https://modelscope.cn/models/OpenBMB/MiniCPM-Llama3-V-2_5-gguf) | +| MiniCPM-Llama3-V 2.5 int4 | GPU | 8 GB | The int4 quantized version, lower GPU memory usage. | [🤗](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5-int4/)    [](https://modelscope.cn/models/OpenBMB/MiniCPM-Llama3-V-2_5-int4) | diff --git a/minicpm_v1.md b/docs/minicpm_v1.md similarity index 100% rename from minicpm_v1.md rename to docs/minicpm_v1.md diff --git a/docs/minicpm_v2.md b/docs/minicpm_v2.md new file mode 100644 index 0000000..9dcb5a0 --- /dev/null +++ b/docs/minicpm_v2.md @@ -0,0 +1,294 @@ +## MiniCPM-V 2.0 + + +> Archive at：2025-01-13 + + + +**MiniCPM-V 2.0** is an efficient version with promising performance for deployment. The model is built based on SigLip-400M and [MiniCPM-2.4B](https://github.com/OpenBMB/MiniCPM/), connected by a perceiver resampler. Our latest version, MiniCPM-V 2.0 has several notable features. + +- 🔥 **State-of-the-art Performance.** + + MiniCPM-V 2.0 achieves **state-of-the-art performance** on multiple benchmarks (including OCRBench, TextVQA, MME, MMB, MathVista, etc) among models under 7B parameters. It even **outperforms strong Qwen-VL-Chat 9.6B, CogVLM-Chat 17.4B, and Yi-VL 34B on OpenCompass, a comprehensive evaluation over 11 popular benchmarks**. Notably, MiniCPM-V 2.0 shows **strong OCR capability**, achieving **comparable performance to Gemini Pro in scene-text understanding**, and **state-of-the-art performance on OCRBench** among open-source models. + +- 🏆 **Trustworthy Behavior.** + + LMMs are known for suffering from hallucination, often generating text not factually grounded in images. MiniCPM-V 2.0 is **the first end-side LMM aligned via multimodal RLHF for trustworthy behavior** (using the recent [RLHF-V](https://rlhf-v.github.io/) [CVPR'24] series technique). This allows the model to **match GPT-4V in preventing hallucinations** on Object HalBench. + +- 🌟 **High-Resolution Images at Any Aspect Raito.** + + MiniCPM-V 2.0 can accept **1.8 million pixels (e.g., 1344x1344) images at any aspect ratio**. This enables better perception of fine-grained visual information such as small objects and optical characters, which is achieved via a recent technique from [LLaVA-UHD](https://arxiv.org/pdf/2403.11703.pdf). + +- ⚡️ **High Efficiency.** + + MiniCPM-V 2.0 can be **efficiently deployed on most GPU cards and personal computers**, and **even on end devices such as mobile phones**. For visual encoding, we compress the image representations into much fewer tokens via a perceiver resampler. This allows MiniCPM-V 2.0 to operate with **favorable memory cost and speed during inference even when dealing with high-resolution images**. + +- 🙌 **Bilingual Support.** + + MiniCPM-V 2.0 **supports strong bilingual multimodal capabilities in both English and Chinese**. This is enabled by generalizing multimodal capabilities across languages, a technique from [VisCPM](https://arxiv.org/abs/2308.12038) [ICLR'24]. + + +### Evaluation + +
+ +
+
+Click to view results on TextVQA, DocVQA, OCRBench, OpenCompass, MME, MMBench, MMMU, MathVista, LLaVA Bench, Object HalBench. +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ModelSizeTextVQA valDocVQA testOCRBenchOpenCompassMMEMMB dev(en)MMB dev(zh)MMMU valMathVistaLLaVA BenchObject HalBench
Proprietary models
Gemini Pro Vision- 74.688.168063.82148.975.274.048.945.879.9-
GPT-4V- 78.088.464563.21771.575.175.053.847.893.186.4 / 92.7
Open-source models 6B~34B
Yi-VL-6B6.7B45.5*17.1*29049.31915.1 68.6 68.3 40.3 28.8 51.9 -
Qwen-VL-Chat9.6B61.562.6488 52.1 1860.0 60.6 56.7 37.0 33.8 67.7 56.2 / 80.0
Yi-VL-34B34B43.4*16.9*29052.6 2050.271.171.445.130.762.3-
DeepSeek-VL-7B7.3B64.7*47.0* 43555.6 1765.4 74.1 72.8 38.3 36.877.8 -
TextMonkey9.7B64.366.7 558- - - - - -- -
CogVLM-Chat17.4B70.433.3*590 52.5 1736.6 63.7 53.8 37.3 34.7 73.9 73.6 / 87.4
Open-source models 1B~3B
DeepSeek-VL-1.3B1.7B58.4*37.9*41346.0 1531.6 64.0 61.2 33.8 29.4 51.1 -
MobileVLM V23.1B57.519.4*--1440.5(P) 63.2 -----
Mini-Gemini2.2B56.234.2*--1653.0 59.8 - 31.7 -- -
MiniCPM-V2.8B 60.638.2 36647.61650.2 67.9 65.3 38.328.951.3 78.4 / 88.5
MiniCPM-V 2.02.8B 74.171.9 60555.01808.6 69.6 68.1 38.2 38.769.2 85.5 / 92.2
+ +
+* We evaluate the officially released checkpoint by ourselves. +
+ +### Examples + + +

+ +

+
+ +We deploy MiniCPM-V 2.0 on end devices. The demo video is the raw screen recording on a Xiaomi 14 Pro without edition. + + +

+ + +

+
+ + + +### Model Zoo + +| Model | Device | Memory |          Description | Download | +|:-----------|:--:|:-----------:|:-------------------|:---------------:| +| MiniCPM-V 2.0 | GPU | 8 GB | Light version, balance the performance the computation cost. | [🤗](https://huggingface.co/openbmb/MiniCPM-V-2)    [](https://modelscope.cn/models/OpenBMB/MiniCPM-V-2) | +| MiniCPM-V 1.0 | GPU | 7 GB | Lightest version, achieving the fastest inference. | [🤗](https://huggingface.co/openbmb/MiniCPM-V)    [](https://modelscope.cn/models/OpenBMB/MiniCPM-V) | diff --git a/docs/minicpm_v2dot6.md b/docs/minicpm_v2dot6.md new file mode 100644 index 0000000..9ef6dac --- /dev/null +++ b/docs/minicpm_v2dot6.md @@ -0,0 +1,945 @@ +## MiniCPM-V 2.6 + +> Archieve at: 2025-01-13 + +**MiniCPM-V 2.6** is the latest and most capable model in the MiniCPM-V series. The model is built on SigLip-400M and Qwen2-7B with a total of 8B parameters. It exhibits a significant performance improvement over MiniCPM-Llama3-V 2.5, and introduces new features for multi-image and video understanding. Notable features of MiniCPM-V 2.6 include: + +- 🔥 **Leading Performance.** + MiniCPM-V 2.6 achieves an average score of 65.2 on the latest version of OpenCompass, a comprehensive evaluation over 8 popular benchmarks. **With only 8B parameters, it surpasses widely used proprietary models like GPT-4o mini, GPT-4V, Gemini 1.5 Pro, and Claude 3.5 Sonnet** for single image understanding. + +- 🖼️ **Multi Image Understanding and In-context Learning.** MiniCPM-V 2.6 can also perform **conversation and reasoning over multiple images**. It achieves **state-of-the-art performance** on popular multi-image benchmarks such as Mantis-Eval, BLINK, Mathverse mv and Sciverse mv, and also shows promising in-context learning capability. + +- 🎬 **Video Understanding.** MiniCPM-V 2.6 can also **accept video inputs**, performing conversation and providing dense captions for spatial-temporal information. It outperforms **GPT-4V, Claude 3.5 Sonnet and LLaVA-NeXT-Video-34B** on Video-MME with/without subtitles. + +- 💪 **Strong OCR Capability and Others.** + MiniCPM-V 2.6 can process images with any aspect ratio and up to 1.8 million pixels (e.g., 1344x1344). It achieves **state-of-the-art performance on OCRBench, surpassing proprietary models such as GPT-4o, GPT-4V, and Gemini 1.5 Pro**. + Based on the the latest [RLAIF-V](https://github.com/RLHF-V/RLAIF-V/) and [VisCPM](https://github.com/OpenBMB/VisCPM) techniques, it features **trustworthy behaviors**, with significantly lower hallucination rates than GPT-4o and GPT-4V on Object HalBench, and supports **multilingual capabilities** on English, Chinese, German, French, Italian, Korean, etc. + + +- 🚀 **Superior Efficiency.** + In addition to its friendly size, MiniCPM-V 2.6 also shows **state-of-the-art token density** (i.e., number of pixels encoded into each visual token). **It produces only 640 tokens when processing a 1.8M pixel image, which is 75% fewer than most models**. This directly improves the inference speed, first-token latency, memory usage, and power consumption. As a result, MiniCPM-V 2.6 can efficiently support **real-time video understanding** on end-side devices such as iPad. + +- 💫 **Easy Usage.** +MiniCPM-V 2.6 can be easily used in various ways: (1) [llama.cpp](https://github.com/OpenBMB/llama.cpp/blob/minicpmv-main/examples/llava/README-minicpmv2.6.md) and [ollama](https://github.com/OpenBMB/ollama/blob/minicpm-v2.6/examples/minicpm-v2.6/README.md) support for efficient CPU inference on local devices, (2) [int4](https://huggingface.co/openbmb/MiniCPM-V-2_6-int4) and [GGUF](https://huggingface.co/openbmb/MiniCPM-V-2_6-gguf) format quantized models in 16 sizes, (3) [vLLM](#inference-with-vllm) support for high-throughput and memory-efficient inference, (4) fine-tuning on new domains and tasks, (5) quick local WebUI demo setup with [Gradio](#chat-with-our-demo-on-gradio), and (6) online web [demo](http://120.92.209.146:8887/). + +### Evaluation +
+ +
+ +
+Click to view single image results on OpenCompass, MME, MMVet, OCRBench, MMMU, MathVista, MMB, AI2D, TextVQA, DocVQA, HallusionBench, Object HalBench. +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ModelSizeToken Density+OpenCompassMMEMMVetOCRBenchMMMU valMathVista miniMMB1.1 testAI2DTextVQA valDocVQA testHallusionBenchObject HalBench
Proprietary
GPT-4o-108869.92328.769.173669.261.382.284.6-92.855.017.6
Claude 3.5 Sonnet-75067.91920.066.078865.961.678.580.2-95.249.913.8
Gemini 1.5 Pro--64.42110.664.075460.657.773.979.173.586.545.6-
GPT-4o mini-108864.12003.466.978560.052.476.077.8--46.112.4
GPT-4V-108863.52070.267.565661.754.779.878.678.087.243.914.2
Step-1V--59.52206.463.362549.944.878.079.271.6-48.4-
Qwen-VL-Max-78458.32281.761.868452.043.474.675.779.593.141.213.4
Open-source
LLaVA-NeXT-Yi-34B34B15755.02006.550.757448.840.477.878.969.3-34.812.6
Mini-Gemini-HD-34B34B157-2141.059.351848.043.3-80.574.178.9--
Cambrian-34B34B182058.32049.953.259150.450.377.879.576.775.541.614.7
GLM-4V-9B13B78459.12018.858.077646.951.167.971.2--45.0-
InternVL2-8B8B70664.12215.154.379451.258.379.483.677.491.645.021.3
MiniCPM-Llama-V 2.58B188258.82024.652.872545.854.372.078.476.684.842.410.3
MiniCPM-V 2.68B282265.22348.4*60.0852*49.8*60.678.082.180.190.848.1*8.2
+ +
+* We evaluate this benchmark using chain-of-thought prompting. Specifically, for MME, we used this technique only for the Cognition set. + ++ Token Density: number of pixels encoded into each visual token at maximum resolution, i.e., # pixels at maximum resolution / # visual tokens. + +Note: For proprietary models, we calculate token density based on the image encoding charging strategy defined in the official API documentation, which provides an upper-bound estimation. + +
+ + +
+Click to view multi-image results on Mantis Eval, BLINK, Mathverse mv, Sciverse mv, MIRB. +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ModelSizeMantis EvalBLINK valMathverse mvSciverse mvMIRB
Proprietary
GPT-4V-62.754.660.366.953.1
LLaVA-NeXT-Interleave-14B14B66.452.632.730.2-
Open-source
Emu2-Chat37B37.836.2-27.2-
CogVLM17B45.241.1---
VPG-C7B52.443.124.323.1-
VILA 8B8B51.239.3-36.5-
InternLM-XComposer-2.58B53.1*48.932.1*-42.5
InternVL2-8B8B59.0*50.930.5*34.4*56.9*
MiniCPM-V 2.68B69.153.084.974.953.8
+ +
+* We evaluate the officially released checkpoint by ourselves. +
+ +
+Click to view video results on Video-MME and Video-ChatGPT. +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ModelSizeVideo-MMEVideo-ChatGPT
w/o subsw subsCorrectnessDetailContextTemporalConsistency
Proprietary
Claude 3.5 Sonnet-60.062.9-----
GPT-4V-59.963.3-----
Open-source
LLaVA-NeXT-7B7B--3.393.293.922.603.12
LLaVA-NeXT-34B34B--3.293.233.832.513.47
CogVLM2-Video12B--3.493.463.232.983.64
LongVA7B52.454.33.053.093.772.443.64
InternVL2-8B8B54.056.9-----
InternLM-XComposer-2.58B55.8------
LLaVA-NeXT-Video32B60.263.03.483.373.952.643.28
MiniCPM-V 2.68B60.963.63.593.283.932.733.62
+
+
+ + +
+Click to view few-shot results on TextVQA, VizWiz, VQAv2, OK-VQA. +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ModelSizeShotTextVQA valVizWiz test-devVQAv2 test-devOK-VQA val
Flamingo80B0*35.031.656.340.6
436.539.663.157.4
837.344.865.657.5
IDEFICS80B0*30.936.060.045.2
434.340.463.652.4
835.746.164.855.1
OmniCorpus7B0*43.049.863.245.5
445.451.364.546.5
845.652.264.746.6
Emu237B026.440.433.526.7
448.254.667.053.2
849.354.767.854.1
MM130B026.240.448.926.7
849.354.770.954.1
MiniCPM-V 2.6+8B043.933.845.423.9
463.660.565.550.1
864.663.468.251.4
+ + +
+* denotes zero image shot and two additional text shots following Flamingo. + ++ We evaluate the pretraining ckpt without SFT. +
+ +### Examples + +
+ Bike + Menu + Code + Mem + medal +
+
+ Click to view more cases. +
+ elec + Menu +
+
+ +We deploy MiniCPM-V 2.6 on end devices. The demo video is the raw screen recording on a iPad Pro without edition. + + +

+ +      + +

+
+ + +

+ +      + +

+
+ + +

+ + +

+
+ +
+ + + +### Multi-turn Conversation + + +
+ +
+ + +```python +import torch +from PIL import Image +from transformers import AutoModel, AutoTokenizer + +torch.manual_seed(0) + +model = AutoModel.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True, + attn_implementation='sdpa', torch_dtype=torch.bfloat16) # sdpa or flash_attention_2, no eager +model = model.eval().cuda() +tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True) + +image = Image.open('./assets/airplane.jpeg').convert('RGB') + +# First round chat +question = "Tell me the model of this aircraft." +msgs = [{'role': 'user', 'content': [image, question]}] + +answer = model.chat( + image=None, + msgs=msgs, + tokenizer=tokenizer +) +print(answer) + +# Second round chat +# pass history context of multi-turn conversation +msgs.append({"role": "assistant", "content": [answer]}) +msgs.append({"role": "user", "content": ["Introduce something about Airbus A380."]}) + +answer = model.chat( + image=None, + msgs=msgs, + tokenizer=tokenizer +) +print(answer) +``` + +You could get the following output: + +``` +"The aircraft in the image is an Airbus A380, which can be identified by its large size, double-deck structure, and the distinctive shape of its wings and engines. The A380 is a wide-body aircraft known for being the world's largest passenger airliner, designed for long-haul flights. It has four engines, which are characteristic of large commercial aircraft. The registration number on the aircraft can also provide specific information about the model if looked up in an aviation database." + +"The Airbus A380 is a double-deck, wide-body, four-engine jet airliner made by Airbus. It is the world's largest passenger airliner and is known for its long-haul capabilities. The aircraft was developed to improve efficiency and comfort for passengers traveling over long distances. It has two full-length passenger decks, which can accommodate more passengers than a typical single-aisle airplane. The A380 has been operated by airlines such as Lufthansa, Singapore Airlines, and Emirates, among others. It is widely recognized for its unique design and significant impact on the aviation industry." +``` + +#### Multi-image Understanding +
+ Click to view Python example of MiniCPM-V 2.6 multi-image understanding + +```python +import torch +from PIL import Image +from transformers import AutoModel, AutoTokenizer + +model = AutoModel.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True, + attn_implementation='sdpa', torch_dtype=torch.bfloat16) # sdpa or flash_attention_2, no eager +model = model.eval().cuda() +tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True) + +image1 = Image.open('image1.jpg').convert('RGB') +image2 = Image.open('image2.jpg').convert('RGB') +question = 'Compare image 1 and image 2, tell me about the differences between image 1 and image 2.' + +msgs = [{'role': 'user', 'content': [image1, image2, question]}] + +answer = model.chat( + image=None, + msgs=msgs, + tokenizer=tokenizer +) +print(answer) +``` +
+ +#### Few-shot In-Context-Learning + +
+ Click to view Python example of MiniCPM-V 2.6 few-shot in-context-learning example + +```python +import torch +from PIL import Image +from transformers import AutoModel, AutoTokenizer + +model = AutoModel.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True, + attn_implementation='sdpa', torch_dtype=torch.bfloat16) # sdpa or flash_attention_2, no eager +model = model.eval().cuda() +tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True) + +question = "production date" +image1 = Image.open('example1.jpg').convert('RGB') +answer1 = "2023.08.04" +image2 = Image.open('example2.jpg').convert('RGB') +answer2 = "2007.04.24" +image_test = Image.open('test.jpg').convert('RGB') + +msgs = [ + {'role': 'user', 'content': [image1, question]}, {'role': 'assistant', 'content': [answer1]}, + {'role': 'user', 'content': [image2, question]}, {'role': 'assistant', 'content': [answer2]}, + {'role': 'user', 'content': [image_test, question]} +] + +answer = model.chat( + image=None, + msgs=msgs, + tokenizer=tokenizer +) +print(answer) +``` +
+ +#### Video understanding +
+ Click to view Python example of MiniCPM-V 2.6 video understanding + +```python +import torch +from PIL import Image +from transformers import AutoModel, AutoTokenizer +from decord import VideoReader, cpu # pip install decord + +model = AutoModel.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True, + attn_implementation='sdpa', torch_dtype=torch.bfloat16) # sdpa or flash_attention_2, no eager +model = model.eval().cuda() +tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True) + +MAX_NUM_FRAMES=64 # if cuda OOM set a smaller number + +def encode_video(video_path): + def uniform_sample(l, n): + gap = len(l) / n + idxs = [int(i * gap + gap / 2) for i in range(n)] + return [l[i] for i in idxs] + + vr = VideoReader(video_path, ctx=cpu(0)) + sample_fps = round(vr.get_avg_fps() / 1) # FPS + frame_idx = [i for i in range(0, len(vr), sample_fps)] + if len(frame_idx) > MAX_NUM_FRAMES: + frame_idx = uniform_sample(frame_idx, MAX_NUM_FRAMES) + frames = vr.get_batch(frame_idx).asnumpy() + frames = [Image.fromarray(v.astype('uint8')) for v in frames] + print('num frames:', len(frames)) + return frames + +video_path="video_test.mp4" +frames = encode_video(video_path) +question = "Describe the video" +msgs = [ + {'role': 'user', 'content': frames + [question]}, +] + +# Set decode params for video +params = {} +params["use_image_id"] = False +params["max_slice_nums"] = 2 # 如果cuda OOM且视频分辨率大于448*448可设为1 + +answer = model.chat( + image=None, + msgs=msgs, + tokenizer=tokenizer, + **params +) +print(answer) +``` +
diff --git a/omnilmm.md b/docs/omnilmm.md similarity index 100% rename from omnilmm.md rename to docs/omnilmm.md diff --git a/omnilmm_en.md b/docs/omnilmm_en.md similarity index 98% rename from omnilmm_en.md rename to docs/omnilmm_en.md index 62f7992..6782d44 100644 --- a/omnilmm_en.md +++ b/docs/omnilmm_en.md @@ -1,6 +1,6 @@ ## OmniLMM-12B -> OmniLMM-12B is released at early time of this project. We recommond you to use our [recently released models](./README_en.md), for better performance and efficiency. +> OmniLMM-12B is released at early time of this project. We recommond you to use our [recently released models](./README.md), for better performance and efficiency. > Archieve at: 2024-05-19 diff --git a/finetune/dataset.py b/finetune/dataset.py index 885ae14..5012ec6 100644 --- a/finetune/dataset.py +++ b/finetune/dataset.py @@ -7,7 +7,6 @@ import re import random from dataclasses import dataclass, field from typing import Dict, List, Optional -from decord import VideoReader, cpu # pip install decord import numpy as np import torch @@ -21,26 +20,6 @@ logger = logging.getLogger(__name__) llama3_chat_template = "{% set loop_messages = messages %}{% for message in loop_messages %}{% set content = '<|start_header_id|>' + message['role'] + '<|end_header_id|>\n\n'+ message['content'] | trim + '<|eot_id|>' %}{% if loop.index0 == 0 %}{% set content = bos_token + content %}{% endif %}{{ content }}{% endfor %}" -MAX_NUM_FRAMES=64 -def encode_video(video_path, max_num_frames=64): - max_num_frames = min(max_num_frames, MAX_NUM_FRAMES) - def uniform_sample(l, n): - gap = len(l) / n - idxs = [int(i * gap + gap / 2) for i in range(n)] - return [l[i] for i in idxs] - - vr = VideoReader(video_path, ctx=cpu(0)) - sample_fps = round(vr.get_avg_fps() / 1) # FPS - frame_idx = [i for i in range(0, len(vr), sample_fps)] - if len(frame_idx) > max_num_frames: - if max_num_frames==1: - frame_idx = [frame_idx[len(frame_idx)//2]] - else: - frame_idx = uniform_sample(frame_idx, max_num_frames) - frames = vr.get_batch(frame_idx).asnumpy() - frames = [Image.fromarray(v.astype('uint8')) for v in frames] - return frames - class SupervisedDataset(Dataset): """Dataset for supervised fine-tuning.""" @@ -55,8 +34,6 @@ class SupervisedDataset(Dataset): query_nums=64, batch_vision=False, max_length=2048, - video_max_slice_nums=2, - max_num_frames=1, ): super(SupervisedDataset, self).__init__() self.raw_data = raw_data @@ -68,58 +45,17 @@ class SupervisedDataset(Dataset): self.query_nums=query_nums self.batch_vision = batch_vision self.max_length = max_length - # video config - self.video_slice_config = copy.deepcopy(slice_config) - self.video_slice_config['max_slice_nums'] = video_max_slice_nums - self.max_num_frames = max_num_frames def __len__(self): return len(self.raw_data) def __getitem__(self, i) -> Dict[str, torch.Tensor]: try: - # default: sft image - use_image_id = True - slice_config = self.slice_config - if "image" in self.raw_data[i]: - if isinstance(self.raw_data[i]["image"], str): - images_dict = { "" : Image.open(self.raw_data[i]["image"]).convert("RGB") } - elif isinstance(self.raw_data[i]["image"], Dict): - ### for multi-images input, the template for every image is , such as , - images_dict = {img_name : Image.open(img_path).convert("RGB") for img_name, img_path in self.raw_data[i]["image"].items()} - elif "video" in self.raw_data[i]: - if isinstance(self.raw_data[i]["video"], str): - frames = encode_video(self.raw_data[i]["video"], max_num_frames=self.max_num_frames) - image_names = [] - images_dict = {} - for j, frame in enumerate(frames): - image_name = "".format(j) - images_dict[image_name] = frame - image_names.append(image_name) - for j in range(len(self.raw_data[i]["conversations"])): - content = self.raw_data[i]["conversations"][j]['content'] - self.raw_data[i]["conversations"][j]['content'] = content.replace("