@@ -1170,8 +385,20 @@ MiniCPM-Llama3-V 2.5 can be easily used in various ways: (1) [llama.cpp](https:/
-
+We deploy MiniCPM-Llama3-V 2.5 on end devices. The demo video is the raw screen recording on a Xiaomi 14 Pro without edition.
+
+
+ 
+ 
+
+
+
+
+
+ 
+
+
## MiniCPM-V 2.0
@@ -1228,29 +455,9 @@ We deploy MiniCPM-V 2.0 on end devices. The demo video is the raw screen recordi
| OmniLMM-12B | [Document](./omnilmm_en.md) |
-## Chat with Our Demo on Gradio 🤗
-
-We provide online and local demos powered by Hugging Face Gradio
, the most popular model deployment framework nowadays. It supports streaming outputs, progress bars, queuing, alerts, and other useful features.
-
-
-### Online Demo
-
-Click here to try out the online demo of [MiniCPM-V 2.6](https://huggingface.co/spaces/openbmb/MiniCPM-V-2_6) | [MiniCPM-Llama3-V 2.5](https://huggingface.co/spaces/openbmb/MiniCPM-Llama3-V-2_5) | [MiniCPM-V 2.0](https://huggingface.co/spaces/openbmb/MiniCPM-V-2).
-
-### Local WebUI Demo
-
-You can easily build your own local WebUI demo with Gradio using the following commands.
-
-```shell
-pip install -r requirements.txt
-```
-
-```shell
-# For NVIDIA GPUs, run:
-python web_demo_2.6.py --device cuda
-
-```
+## Online Demo
+Click here to try out the Demo of [MiniCPM-Llama3-V 2.5](https://huggingface.co/spaces/openbmb/MiniCPM-Llama3-V-2_5) ｜ [MiniCPM-V 2.0](https://huggingface.co/spaces/openbmb/MiniCPM-V-2).
## Install
@@ -1281,12 +488,9 @@ pip install -r requirements.txt
| Model | Device | Memory | Description | Download |
|:-----------|:--:|:-----------:|:-------------------|:---------------:|
-| MiniCPM-V 2.6| GPU | 17 GB | The latest version, achieving state-of-the-art end-side performance for single image, multi-image and video understanding. | [🤗](https://huggingface.co/openbmb/MiniCPM-V-2_6) [
](https://modelscope.cn/models/OpenBMB/MiniCPM-V-2_6) |
-| MiniCPM-V 2.6 gguf | CPU | 6 GB | The gguf version, lower memory usage and faster inference. | [🤗](https://huggingface.co/openbmb/MiniCPM-V-2_6-gguf) [
](https://modelscope.cn/models/OpenBMB/MiniCPM-V-2_6-gguf) |
-| MiniCPM-V 2.6 int4 | GPU | 7 GB | The int4 quantized version, lower GPU memory usage. | [🤗](https://huggingface.co/openbmb/MiniCPM-V-2_6-int4) [
](https://modelscope.cn/models/OpenBMB/MiniCPM-V-2_6-int4) |
-| MiniCPM-Llama3-V 2.5 | GPU | 19 GB | Strong end-side multimodal performance. | [🤗](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5/) [
](https://modelscope.cn/models/OpenBMB/MiniCPM-Llama3-V-2_5) |
-| MiniCPM-Llama3-V 2.5 gguf | CPU | 6 GB | The gguf version, lower memory usage and faster inference. | [🤗](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5-gguf) [
](https://modelscope.cn/models/OpenBMB/MiniCPM-Llama3-V-2_5-gguf) |
-| MiniCPM-Llama3-V 2.5 int4 | GPU | 8 GB | The int4 quantized version, lower GPU memory usage. | [🤗](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5-int4/) [
](https://modelscope.cn/models/OpenBMB/MiniCPM-Llama3-V-2_5-int4) |
+| MiniCPM-Llama3-V 2.5 | GPU | 19 GB | The lastest version, achieving state-of-the end-side multimodal performance. | [🤗](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5/) [
](https://modelscope.cn/models/OpenBMB/MiniCPM-Llama3-V-2_5) |
+| MiniCPM-Llama3-V 2.5 gguf | CPU | 5 GB | The gguf version, lower GPU memory and faster inference. | [🤗](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5-gguf) [
](https://modelscope.cn/models/OpenBMB/MiniCPM-Llama3-V-2_5-gguf) |
+| MiniCPM-Llama3-V 2.5 int4 | GPU | 8 GB | The int4 quantized version，lower GPU memory usage. | [🤗](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5-int4/) [
](https://modelscope.cn/models/OpenBMB/MiniCPM-Llama3-V-2_5-int4) |
| MiniCPM-V 2.0 | GPU | 8 GB | Light version, balance the performance the computation cost. | [🤗](https://huggingface.co/openbmb/MiniCPM-V-2) [
](https://modelscope.cn/models/OpenBMB/MiniCPM-V-2) |
| MiniCPM-V 1.0 | GPU | 7 GB | Lightest version, achieving the fastest inference. | [🤗](https://huggingface.co/openbmb/MiniCPM-V) [
](https://modelscope.cn/models/OpenBMB/MiniCPM-V) |
@@ -1300,40 +504,30 @@ Please refer to the following codes to run.
```python
+from chat import MiniCPMVChat, img2base64
import torch
-from PIL import Image
-from transformers import AutoModel, AutoTokenizer
+import json
torch.manual_seed(0)
-model = AutoModel.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True,
- attn_implementation='sdpa', torch_dtype=torch.bfloat16) # sdpa or flash_attention_2, no eager
-model = model.eval().cuda()
-tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True)
+chat_model = MiniCPMVChat('openbmb/MiniCPM-Llama3-V-2_5')
-image = Image.open('./assets/airplane.jpeg').convert('RGB')
+im_64 = img2base64('./assets/airplane.jpeg')
# First round chat
-question = "Tell me the model of this aircraft."
-msgs = [{'role': 'user', 'content': [image, question]}]
+msgs = [{"role": "user", "content": "Tell me the model of this aircraft."}]
-answer = model.chat(
- image=None,
- msgs=msgs,
- tokenizer=tokenizer
-)
+inputs = {"image": im_64, "question": json.dumps(msgs)}
+answer = chat_model.chat(inputs)
print(answer)
# Second round chat
# pass history context of multi-turn conversation
-msgs.append({"role": "assistant", "content": [answer]})
-msgs.append({"role": "user", "content": ["Introduce something about Airbus A380."]})
+msgs.append({"role": "assistant", "content": answer})
+msgs.append({"role": "user", "content": "Introduce something about Airbus A380."})
-answer = model.chat(
- image=None,
- msgs=msgs,
- tokenizer=tokenizer
-)
+inputs = {"image": im_64, "question": json.dumps(msgs)}
+answer = chat_model.chat(inputs)
print(answer)
```
@@ -1345,129 +539,6 @@ You will get the following output:
"The Airbus A380 is a double-deck, wide-body, four-engine jet airliner made by Airbus. It is the world's largest passenger airliner and is known for its long-haul capabilities. The aircraft was developed to improve efficiency and comfort for passengers traveling over long distances. It has two full-length passenger decks, which can accommodate more passengers than a typical single-aisle airplane. The A380 has been operated by airlines such as Lufthansa, Singapore Airlines, and Emirates, among others. It is widely recognized for its unique design and significant impact on the aviation industry."
```
-#### Chat with multiple images
-
- Click to view Python code running MiniCPM-V 2.6 with multiple images input.
-
-```python
-import torch
-from PIL import Image
-from transformers import AutoModel, AutoTokenizer
-
-model = AutoModel.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True,
- attn_implementation='sdpa', torch_dtype=torch.bfloat16) # sdpa or flash_attention_2, no eager
-model = model.eval().cuda()
-tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True)
-
-image1 = Image.open('image1.jpg').convert('RGB')
-image2 = Image.open('image2.jpg').convert('RGB')
-question = 'Compare image 1 and image 2, tell me about the differences between image 1 and image 2.'
-
-msgs = [{'role': 'user', 'content': [image1, image2, question]}]
-
-answer = model.chat(
- image=None,
- msgs=msgs,
- tokenizer=tokenizer
-)
-print(answer)
-```
-
-
-#### In-context few-shot learning
-
- Click to view Python code running MiniCPM-V 2.6 with few-shot input.
-
-```python
-import torch
-from PIL import Image
-from transformers import AutoModel, AutoTokenizer
-
-model = AutoModel.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True,
- attn_implementation='sdpa', torch_dtype=torch.bfloat16) # sdpa or flash_attention_2, no eager
-model = model.eval().cuda()
-tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True)
-
-question = "production date"
-image1 = Image.open('example1.jpg').convert('RGB')
-answer1 = "2023.08.04"
-image2 = Image.open('example2.jpg').convert('RGB')
-answer2 = "2007.04.24"
-image_test = Image.open('test.jpg').convert('RGB')
-
-msgs = [
- {'role': 'user', 'content': [image1, question]}, {'role': 'assistant', 'content': [answer1]},
- {'role': 'user', 'content': [image2, question]}, {'role': 'assistant', 'content': [answer2]},
- {'role': 'user', 'content': [image_test, question]}
-]
-
-answer = model.chat(
- image=None,
- msgs=msgs,
- tokenizer=tokenizer
-)
-print(answer)
-```
-
-
-#### Chat with video
-
- Click to view Python code running MiniCPM-V 2.6 with video input.
-
-```python
-import torch
-from PIL import Image
-from transformers import AutoModel, AutoTokenizer
-from decord import VideoReader, cpu # pip install decord
-
-model = AutoModel.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True,
- attn_implementation='sdpa', torch_dtype=torch.bfloat16) # sdpa or flash_attention_2, no eager
-model = model.eval().cuda()
-tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True)
-
-MAX_NUM_FRAMES=64 # if cuda OOM set a smaller number
-
-def encode_video(video_path):
- def uniform_sample(l, n):
- gap = len(l) / n
- idxs = [int(i * gap + gap / 2) for i in range(n)]
- return [l[i] for i in idxs]
-
- vr = VideoReader(video_path, ctx=cpu(0))
- sample_fps = round(vr.get_avg_fps() / 1) # FPS
- frame_idx = [i for i in range(0, len(vr), sample_fps)]
- if len(frame_idx) > MAX_NUM_FRAMES:
- frame_idx = uniform_sample(frame_idx, MAX_NUM_FRAMES)
- frames = vr.get_batch(frame_idx).asnumpy()
- frames = [Image.fromarray(v.astype('uint8')) for v in frames]
- print('num frames:', len(frames))
- return frames
-
-video_path="video_test.mp4"
-frames = encode_video(video_path)
-question = "Describe the video"
-msgs = [
- {'role': 'user', 'content': frames + [question]},
-]
-
-# Set decode params for video
-params = {}
-params["use_image_id"] = False
-params["max_slice_nums"] = 2 # use 1 if cuda OOM and video resolution > 448*448
-
-answer = model.chat(
- image=None,
- msgs=msgs,
- tokenizer=tokenizer,
- **params
-)
-print(answer)
-```
-
-
-
-### Inference on Multiple GPUs
-You can run MiniCPM-Llama3-V 2.5 on multiple low VRAM GPUs (12 GB or 16 GB) by distributing the model's layers across multiple GPUs. Please refer to this [tutorial](https://github.com/OpenBMB/MiniCPM-V/blob/main/docs/inference_on_multiple_gpus.md) for detailed instructions on how to load the model and inference using multiple low VRAM GPUs.
### Inference on Mac
@@ -1506,100 +577,52 @@ PYTORCH_ENABLE_MPS_FALLBACK=1 python test.py
### Deployment on Mobile Phone
-MiniCPM-V 2.0 can be deployed on mobile phones with Android operating systems. 🚀 Click [MiniCPM-V 2.0](https://github.com/OpenBMB/mlc-MiniCPM) to install apk.
+MiniCPM-V 2.0 can be deployed on mobile phones with Android operating systems. 🚀 Click [here](https://github.com/OpenBMB/mlc-MiniCPM) to install apk. MiniCPM-Llama3-V 2.5 coming soon.
-### Inference with llama.cpp
-MiniCPM-V 2.6 can run with llama.cpp now! See [our fork of llama.cpp](https://github.com/OpenBMB/llama.cpp/tree/minicpmv-main/examples/llava/README-minicpmv2.6.md) for more detail. This implementation supports smooth inference of 16~18 token/s on iPad (test environment：iPad Pro + M4).
-
-### Inference with ollama
-MiniCPM-V 2.6 can run with ollama now! See [our fork of ollama](https://github.com/OpenBMB/ollama/blob/minicpm-v2.6/examples/minicpm-v2.6/README.md) for more detail. This implementation supports smooth inference of 16~18 token/s on iPad (test environment：iPad Pro + M4).
-
-### Inference with vLLM
+### WebUI Demo
- vLLM now officially supports MiniCPM-V 2.6, MiniCPM-Llama3-V 2.5 and MiniCPM-V 2.0, Click to see.
-
-1. Install vLLM(>=0.5.4):
+Click to see how to deploy WebUI demo on different devices
+
```shell
-pip install vllm
+pip install -r requirements.txt
```
-2. Install timm: (optional, MiniCPM-V 2.0 need timm)
+
```shell
-pip install timm==0.9.10
+# For NVIDIA GPUs, run:
+python web_demo_2.5.py --device cuda
+
+# For Mac with MPS (Apple silicon or AMD GPUs), run:
+PYTORCH_ENABLE_MPS_FALLBACK=1 python web_demo_2.5.py --device mps
```
-3. Run the example(for image):
-```python
-from transformers import AutoTokenizer
-from PIL import Image
-from vllm import LLM, SamplingParams
+
-MODEL_NAME = "openbmb/MiniCPM-V-2_6"
-# Also available for previous models
-# MODEL_NAME = "openbmb/MiniCPM-Llama3-V-2_5"
-# MODEL_NAME = "HwwwH/MiniCPM-V-2"
+### Inference with llama.cpp
+MiniCPM-Llama3-V 2.5 can run with llama.cpp now! See our fork of [llama.cpp](https://github.com/OpenBMB/llama.cpp/tree/minicpm-v2.5/examples/minicpmv) for more detail. This implementation supports smooth inference of 6~8 token/s on mobile phones (test environment：Xiaomi 14 pro + Snapdragon 8 Gen 3).
-image = Image.open("xxx.png").convert("RGB")
-tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME, trust_remote_code=True)
-llm = LLM(
- model=MODEL_NAME,
- trust_remote_code=True,
- gpu_memory_utilization=1,
- max_model_len=2048
-)
+### Inference with vLLM
-messages = [{
- "role":
- "user",
- "content":
- # Number of images
- "(
./)" + \
- "\nWhat is the content of this image?"
-}]
-prompt = tokenizer.apply_chat_template(
- messages,
- tokenize=False,
- add_generation_prompt=True
-)
+
+Click to see how to inference with vLLM
+Because our pull request to vLLM is still waiting for reviewing, we fork this repository to build and test our vLLM demo. Here are the steps:
-# Single Inference
-inputs = {
- "prompt": prompt,
- "multi_modal_data": {
- "image": image
- # Multi images, the number of images should be equal to that of `(./)`
- # "image": [image, image]
- },
-}
-# Batch Inference
-# inputs = [{
-# "prompt": prompt,
-# "multi_modal_data": {
-# "image": image
-# },
-# } for _ in 2]
-
-
-# 2.6
-stop_tokens = ['<|im_end|>', '<|endoftext|>']
-stop_token_ids = [tokenizer.convert_tokens_to_ids(i) for i in stop_tokens]
-# 2.0
-# stop_token_ids = [tokenizer.eos_id]
-# 2.5
-# stop_token_ids = [tokenizer.eos_id, tokenizer.eot_id]
-
-sampling_params = SamplingParams(
- stop_token_ids=stop_token_ids,
- use_beam_search=True,
- temperature=0,
- best_of=3,
- max_tokens=1024
-)
-
-outputs = llm.generate(inputs, sampling_params=sampling_params)
-
-print(outputs[0].outputs[0].text)
+1. Clone our version of vLLM:
+```shell
+git clone https://github.com/OpenBMB/vllm.git
+```
+2. Install vLLM:
+```shell
+cd vllm
+pip install -e .
+```
+3. Install timm:
+```shell
+pip install timm=0.9.10
+```
+4. Run our demo:
+```shell
+python examples/minicpmv_example.py
```
-4. click [here](https://modelbest.feishu.cn/wiki/C2BWw4ZP0iCDy7kkCPCcX2BHnOf?from=from_copylink) if you want to use it with *video*, or get more details about `vLLM`.
## Fine-tuning
@@ -1616,23 +639,28 @@ We now support MiniCPM-V series fine-tuning with the SWIFT framework. SWIFT supp
Best Practices：[MiniCPM-V 1.0](https://github.com/modelscope/swift/blob/main/docs/source/Multi-Modal/minicpm-v最佳实践.md), [MiniCPM-V 2.0](https://github.com/modelscope/swift/blob/main/docs/source/Multi-Modal/minicpm-v-2最佳实践.md)
-## FAQs
-Click here to view the [FAQs](./docs/faqs.md)
+
+
+## TODO
+
+- [x] MiniCPM-V fine-tuning support
+- [ ] Code release for real-time interactive assistant
## Model License
-* This repository is released under the [Apache-2.0](https://github.com/OpenBMB/MiniCPM/blob/main/LICENSE) License.
+The code in this repo is released according to [Apache-2.0](https://github.com/OpenBMB/MiniCPM/blob/main/LICENSE)
-* The usage of MiniCPM-V model weights must strictly follow [MiniCPM Model License.md](https://github.com/OpenBMB/MiniCPM/blob/main/MiniCPM%20Model%20License.md).
+The usage of MiniCPM-V's and OmniLMM's parameters is subject to "[General Model License Agreement - Source Notes - Publicity Restrictions - Commercial License](https://github.com/OpenBMB/General-Model-License/blob/main/通用模型许可协议-来源说明-宣传限制-商业授权.md)"
-* The models and weights of MiniCPM are completely free for academic research. after filling out a ["questionnaire"](https://modelbest.feishu.cn/share/base/form/shrcnpV5ZT9EJ6xYjh3Kx0J6v8g) for registration, are also available for free commercial use.
-
+The parameters are fully open to academic research
+
+Please contact cpm@modelbest.cn to obtain written authorization for commercial uses. Free commercial use is also allowed after registration.
## Statement
As LMMs, MiniCPM-V models (including OmniLMM) generate contents by learning a large amount of multimodal corpora, but they cannot comprehend, express personal opinions or make value judgement. Anything generated by MiniCPM-V models does not represent the views and positions of the model developers
-We will not be liable for any problems arising from the use of MiniCPM-V models, including but not limited to data security issues, risk of public opinion, or any risks and problems arising from the misdirection, misuse, dissemination or misuse of the model.
+We will not be liable for any problems arising from the use of MiniCPMV-V models, including but not limited to data security issues, risk of public opinion, or any risks and problems arising from the misdirection, misuse, dissemination or misuse of the model.
## Institutions
@@ -1643,16 +671,15 @@ This project is developed by the following institutions:
-
[ModelBest](https://modelbest.cn/)
-
[Zhihu](https://www.zhihu.com/ )
-## 🌟 Star History
+## Other Multimodal Projects from Our Team
+👏 Welcome to explore other multimodal projects of our team:
-
-
- 
-
-
+[VisCPM](https://github.com/OpenBMB/VisCPM/tree/main) | [RLHF-V](https://github.com/RLHF-V/RLHF-V) | [LLaVA-UHD](https://github.com/thunlp/LLaVA-UHD) | [RLAIF-V](https://github.com/RLHF-V/RLAIF-V)
-
+
-## Key Techniques and Other Multimodal Projects
-
-👏 Welcome to explore key techniques of MiniCPM-V and other multimodal projects of our team:
-
-[VisCPM](https://github.com/OpenBMB/VisCPM/tree/main) | [RLHF-V](https://github.com/RLHF-V/RLHF-V) | [LLaVA-UHD](https://github.com/thunlp/LLaVA-UHD) | [RLAIF-V](https://github.com/RLHF-V/RLAIF-V)
-
-
-## Citation
+## Citation
If you find our model/code/paper helpful, please consider cite our papers 📝 and star us ⭐️！
```bib
-@article{yao2024minicpm,
- title={MiniCPM-V: A GPT-4V Level MLLM on Your Phone},
- author={Yao, Yuan and Yu, Tianyu and Zhang, Ao and Wang, Chongyi and Cui, Junbo and Zhu, Hongji and Cai, Tianchi and Li, Haoyu and Zhao, Weilin and He, Zhihui and others},
- journal={arXiv preprint arXiv:2408.01800},
+@article{yu2023rlhf,
+ title={Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback},
+ author={Yu, Tianyu and Yao, Yuan and Zhang, Haoye and He, Taiwen and Han, Yifeng and Cui, Ganqu and Hu, Jinyi and Liu, Zhiyuan and Zheng, Hai-Tao and Sun, Maosong and others},
+ journal={arXiv preprint arXiv:2312.00849},
+ year={2023}
+}
+@article{viscpm,
+ title={Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languages},
+ author={Jinyi Hu and Yuan Yao and Chongyi Wang and Shan Wang and Yinxu Pan and Qianyu Chen and Tianyu Yu and Hanghao Wu and Yue Zhao and Haoye Zhang and Xu Han and Yankai Lin and Jiao Xue and Dahai Li and Zhiyuan Liu and Maosong Sun},
+ journal={arXiv preprint arXiv:2308.12038},
+ year={2023}
+}
+@article{xu2024llava-uhd,
+ title={{LLaVA-UHD}: an LMM Perceiving Any Aspect Ratio and High-Resolution Images},
+ author={Xu, Ruyi and Yao, Yuan and Guo, Zonghao and Cui, Junbo and Ni, Zanlin and Ge, Chunjiang and Chua, Tat-Seng and Liu, Zhiyuan and Huang, Gao},
+ journal={arXiv preprint arXiv:2403.11703},
+ year={2024}
+}
+@article{yu2024rlaifv,
+ title={RLAIF-V: Aligning MLLMs through Open-Source AI Feedback for Super GPT-4V Trustworthiness},
+ author={Yu, Tianyu and Zhang, Haoye and Yao, Yuan and Dang, Yunkai and Chen, Da and Lu, Xiaoman and Cui, Ganqu and He, Taiwen and Liu, Zhiyuan and Chua, Tat-Seng and Sun, Maosong},
+ journal={arXiv preprint arXiv:2405.17220},
year={2024}
}
```
diff --git a/README_en.md b/README_en.md
index 1e48533..ce1a180 100644
--- a/README_en.md
+++ b/README_en.md
@@ -2,25 +2,23 @@
-**A GPT-4V Level MLLM for Single Image, Multi Image and Video on Your Phone**
+**A GPT-4V Level Multimodal LLM on Your Phone**
[中文](./README_zh.md) |
English
-Join our
💬 WeChat
-
-
- MiniCPM-V 2.6 🤗 🤖 | MiniCPM-Llama3-V 2.5 🤗 🤖 |
- MiniCPM-Llama3-V 2.5 Technical Report
+ MiniCPM-Llama3-V 2.5 🤗 🤖 |
+ MiniCPM-V 2.0 🤗 🤖 |
+ Technical Blog
-Click to view more details of MiniCPM-Llama3-V 2.5
-
**MiniCPM-Llama3-V 2.5** is the latest model in the MiniCPM-V series. The model is built on SigLip-400M and Llama3-8B-Instruct with a total of 8B parameters. It exhibits a significant performance improvement over MiniCPM-V 2.0. Notable features of MiniCPM-Llama3-V 2.5 include:
- 🔥 **Leading Performance.**
@@ -869,9 +87,6 @@ We deploy MiniCPM-V 2.6 on end devices. The demo video is the raw screen recordi
- 🚀 **Efficient Deployment.**
MiniCPM-Llama3-V 2.5 systematically employs **model quantization, CPU optimizations, NPU optimizations and compilation optimizations**, achieving high-efficiency deployment on end-side devices. For mobile phones with Qualcomm chips, we have integrated the NPU acceleration framework QNN into llama.cpp for the first time. After systematic optimization, MiniCPM-Llama3-V 2.5 has realized a **150x acceleration in end-side MLLM image encoding** and a **3x speedup in language decoding**.
-- 💫 **Easy Usage.**
-MiniCPM-Llama3-V 2.5 can be easily used in various ways: (1) [llama.cpp](https://github.com/OpenBMB/llama.cpp/blob/minicpm-v2.5/examples/minicpmv/README.md) and [ollama](https://github.com/OpenBMB/ollama/tree/minicpm-v2.5/examples/minicpm-v2.5) support for efficient CPU inference on local devices, (2) [GGUF](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5-gguf) format quantized models in 16 sizes, (3) efficient [LoRA](https://github.com/OpenBMB/MiniCPM-V/tree/main/finetune#lora-finetuning) fine-tuning with only 2 V100 GPUs, (4) [streaming output](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5#usage), (5) quick local WebUI demo setup with [Gradio](https://github.com/OpenBMB/MiniCPM-V/blob/main/web_demo_2.5.py) and [Streamlit](https://github.com/OpenBMB/MiniCPM-V/blob/main/web_demo_streamlit-2_5.py), and (6) interactive demos on [HuggingFace Spaces](https://huggingface.co/spaces/openbmb/MiniCPM-Llama3-V-2_5).
-
### Evaluation
@@ -1170,8 +385,20 @@ MiniCPM-Llama3-V 2.5 can be easily used in various ways: (1) [llama.cpp](https:/
-
+We deploy MiniCPM-Llama3-V 2.5 on end devices. The demo video is the raw screen recording on a Xiaomi 14 Pro without edition.
+
+
+
+
+
+
+
+
+
+
+
+
## MiniCPM-V 2.0
@@ -1228,29 +455,9 @@ We deploy MiniCPM-V 2.0 on end devices. The demo video is the raw screen recordi
| OmniLMM-12B | [Document](./omnilmm_en.md) |
-## Chat with Our Demo on Gradio 🤗
-
-We provide online and local demos powered by Hugging Face Gradio
, the most popular model deployment framework nowadays. It supports streaming outputs, progress bars, queuing, alerts, and other useful features.
-
-
-### Online Demo
-
-Click here to try out the online demo of [MiniCPM-V 2.6](https://huggingface.co/spaces/openbmb/MiniCPM-V-2_6) | [MiniCPM-Llama3-V 2.5](https://huggingface.co/spaces/openbmb/MiniCPM-Llama3-V-2_5) | [MiniCPM-V 2.0](https://huggingface.co/spaces/openbmb/MiniCPM-V-2).
-
-### Local WebUI Demo
-
-You can easily build your own local WebUI demo with Gradio using the following commands.
-
-```shell
-pip install -r requirements.txt
-```
-
-```shell
-# For NVIDIA GPUs, run:
-python web_demo_2.6.py --device cuda
-
-```
+## Online Demo
+Click here to try out the Demo of [MiniCPM-Llama3-V 2.5](https://huggingface.co/spaces/openbmb/MiniCPM-Llama3-V-2_5) ｜ [MiniCPM-V 2.0](https://huggingface.co/spaces/openbmb/MiniCPM-V-2).
## Install
@@ -1281,12 +488,9 @@ pip install -r requirements.txt
| Model | Device | Memory | Description | Download |
|:-----------|:--:|:-----------:|:-------------------|:---------------:|
-| MiniCPM-V 2.6| GPU | 17 GB | The latest version, achieving state-of-the-art end-side performance for single image, multi-image and video understanding. | [🤗](https://huggingface.co/openbmb/MiniCPM-V-2_6) [
](https://modelscope.cn/models/OpenBMB/MiniCPM-V-2_6) |
-| MiniCPM-V 2.6 gguf | CPU | 6 GB | The gguf version, lower memory usage and faster inference. | [🤗](https://huggingface.co/openbmb/MiniCPM-V-2_6-gguf) [
](https://modelscope.cn/models/OpenBMB/MiniCPM-V-2_6-gguf) |
-| MiniCPM-V 2.6 int4 | GPU | 7 GB | The int4 quantized version, lower GPU memory usage. | [🤗](https://huggingface.co/openbmb/MiniCPM-V-2_6-int4) [
](https://modelscope.cn/models/OpenBMB/MiniCPM-V-2_6-int4) |
-| MiniCPM-Llama3-V 2.5 | GPU | 19 GB | Strong end-side multimodal performance. | [🤗](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5/) [
](https://modelscope.cn/models/OpenBMB/MiniCPM-Llama3-V-2_5) |
-| MiniCPM-Llama3-V 2.5 gguf | CPU | 6 GB | The gguf version, lower memory usage and faster inference. | [🤗](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5-gguf) [
](https://modelscope.cn/models/OpenBMB/MiniCPM-Llama3-V-2_5-gguf) |
-| MiniCPM-Llama3-V 2.5 int4 | GPU | 8 GB | The int4 quantized version, lower GPU memory usage. | [🤗](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5-int4/) [
](https://modelscope.cn/models/OpenBMB/MiniCPM-Llama3-V-2_5-int4) |
+| MiniCPM-Llama3-V 2.5 | GPU | 19 GB | The lastest version, achieving state-of-the end-side multimodal performance. | [🤗](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5/) [
](https://modelscope.cn/models/OpenBMB/MiniCPM-Llama3-V-2_5) |
+| MiniCPM-Llama3-V 2.5 gguf | CPU | 5 GB | The gguf version, lower GPU memory and faster inference. | [🤗](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5-gguf) [
](https://modelscope.cn/models/OpenBMB/MiniCPM-Llama3-V-2_5-gguf) |
+| MiniCPM-Llama3-V 2.5 int4 | GPU | 8 GB | The int4 quantized version，lower GPU memory usage. | [🤗](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5-int4/) [
](https://modelscope.cn/models/OpenBMB/MiniCPM-Llama3-V-2_5-int4) |
| MiniCPM-V 2.0 | GPU | 8 GB | Light version, balance the performance the computation cost. | [🤗](https://huggingface.co/openbmb/MiniCPM-V-2) [
](https://modelscope.cn/models/OpenBMB/MiniCPM-V-2) |
| MiniCPM-V 1.0 | GPU | 7 GB | Lightest version, achieving the fastest inference. | [🤗](https://huggingface.co/openbmb/MiniCPM-V) [
](https://modelscope.cn/models/OpenBMB/MiniCPM-V) |
@@ -1300,40 +504,30 @@ Please refer to the following codes to run.
```python
+from chat import MiniCPMVChat, img2base64
import torch
-from PIL import Image
-from transformers import AutoModel, AutoTokenizer
+import json
torch.manual_seed(0)
-model = AutoModel.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True,
- attn_implementation='sdpa', torch_dtype=torch.bfloat16) # sdpa or flash_attention_2, no eager
-model = model.eval().cuda()
-tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True)
+chat_model = MiniCPMVChat('openbmb/MiniCPM-Llama3-V-2_5')
-image = Image.open('./assets/airplane.jpeg').convert('RGB')
+im_64 = img2base64('./assets/airplane.jpeg')
# First round chat
-question = "Tell me the model of this aircraft."
-msgs = [{'role': 'user', 'content': [image, question]}]
+msgs = [{"role": "user", "content": "Tell me the model of this aircraft."}]
-answer = model.chat(
- image=None,
- msgs=msgs,
- tokenizer=tokenizer
-)
+inputs = {"image": im_64, "question": json.dumps(msgs)}
+answer = chat_model.chat(inputs)
print(answer)
# Second round chat
# pass history context of multi-turn conversation
-msgs.append({"role": "assistant", "content": [answer]})
-msgs.append({"role": "user", "content": ["Introduce something about Airbus A380."]})
+msgs.append({"role": "assistant", "content": answer})
+msgs.append({"role": "user", "content": "Introduce something about Airbus A380."})
-answer = model.chat(
- image=None,
- msgs=msgs,
- tokenizer=tokenizer
-)
+inputs = {"image": im_64, "question": json.dumps(msgs)}
+answer = chat_model.chat(inputs)
print(answer)
```
@@ -1345,129 +539,6 @@ You will get the following output:
"The Airbus A380 is a double-deck, wide-body, four-engine jet airliner made by Airbus. It is the world's largest passenger airliner and is known for its long-haul capabilities. The aircraft was developed to improve efficiency and comfort for passengers traveling over long distances. It has two full-length passenger decks, which can accommodate more passengers than a typical single-aisle airplane. The A380 has been operated by airlines such as Lufthansa, Singapore Airlines, and Emirates, among others. It is widely recognized for its unique design and significant impact on the aviation industry."
```
-#### Chat with multiple images
-
- Click to view Python code running MiniCPM-V 2.6 with multiple images input.
-
-```python
-import torch
-from PIL import Image
-from transformers import AutoModel, AutoTokenizer
-
-model = AutoModel.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True,
- attn_implementation='sdpa', torch_dtype=torch.bfloat16) # sdpa or flash_attention_2, no eager
-model = model.eval().cuda()
-tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True)
-
-image1 = Image.open('image1.jpg').convert('RGB')
-image2 = Image.open('image2.jpg').convert('RGB')
-question = 'Compare image 1 and image 2, tell me about the differences between image 1 and image 2.'
-
-msgs = [{'role': 'user', 'content': [image1, image2, question]}]
-
-answer = model.chat(
- image=None,
- msgs=msgs,
- tokenizer=tokenizer
-)
-print(answer)
-```
-
-
-#### In-context few-shot learning
-
- Click to view Python code running MiniCPM-V 2.6 with few-shot input.
-
-```python
-import torch
-from PIL import Image
-from transformers import AutoModel, AutoTokenizer
-
-model = AutoModel.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True,
- attn_implementation='sdpa', torch_dtype=torch.bfloat16) # sdpa or flash_attention_2, no eager
-model = model.eval().cuda()
-tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True)
-
-question = "production date"
-image1 = Image.open('example1.jpg').convert('RGB')
-answer1 = "2023.08.04"
-image2 = Image.open('example2.jpg').convert('RGB')
-answer2 = "2007.04.24"
-image_test = Image.open('test.jpg').convert('RGB')
-
-msgs = [
- {'role': 'user', 'content': [image1, question]}, {'role': 'assistant', 'content': [answer1]},
- {'role': 'user', 'content': [image2, question]}, {'role': 'assistant', 'content': [answer2]},
- {'role': 'user', 'content': [image_test, question]}
-]
-
-answer = model.chat(
- image=None,
- msgs=msgs,
- tokenizer=tokenizer
-)
-print(answer)
-```
-
-
-#### Chat with video
-
- Click to view Python code running MiniCPM-V 2.6 with video input.
-
-```python
-import torch
-from PIL import Image
-from transformers import AutoModel, AutoTokenizer
-from decord import VideoReader, cpu # pip install decord
-
-model = AutoModel.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True,
- attn_implementation='sdpa', torch_dtype=torch.bfloat16) # sdpa or flash_attention_2, no eager
-model = model.eval().cuda()
-tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True)
-
-MAX_NUM_FRAMES=64 # if cuda OOM set a smaller number
-
-def encode_video(video_path):
- def uniform_sample(l, n):
- gap = len(l) / n
- idxs = [int(i * gap + gap / 2) for i in range(n)]
- return [l[i] for i in idxs]
-
- vr = VideoReader(video_path, ctx=cpu(0))
- sample_fps = round(vr.get_avg_fps() / 1) # FPS
- frame_idx = [i for i in range(0, len(vr), sample_fps)]
- if len(frame_idx) > MAX_NUM_FRAMES:
- frame_idx = uniform_sample(frame_idx, MAX_NUM_FRAMES)
- frames = vr.get_batch(frame_idx).asnumpy()
- frames = [Image.fromarray(v.astype('uint8')) for v in frames]
- print('num frames:', len(frames))
- return frames
-
-video_path="video_test.mp4"
-frames = encode_video(video_path)
-question = "Describe the video"
-msgs = [
- {'role': 'user', 'content': frames + [question]},
-]
-
-# Set decode params for video
-params = {}
-params["use_image_id"] = False
-params["max_slice_nums"] = 2 # use 1 if cuda OOM and video resolution > 448*448
-
-answer = model.chat(
- image=None,
- msgs=msgs,
- tokenizer=tokenizer,
- **params
-)
-print(answer)
-```
-
-
-
-### Inference on Multiple GPUs
-You can run MiniCPM-Llama3-V 2.5 on multiple low VRAM GPUs (12 GB or 16 GB) by distributing the model's layers across multiple GPUs. Please refer to this [tutorial](https://github.com/OpenBMB/MiniCPM-V/blob/main/docs/inference_on_multiple_gpus.md) for detailed instructions on how to load the model and inference using multiple low VRAM GPUs.
### Inference on Mac
@@ -1506,100 +577,52 @@ PYTORCH_ENABLE_MPS_FALLBACK=1 python test.py
### Deployment on Mobile Phone
-MiniCPM-V 2.0 can be deployed on mobile phones with Android operating systems. 🚀 Click [MiniCPM-V 2.0](https://github.com/OpenBMB/mlc-MiniCPM) to install apk.
+MiniCPM-V 2.0 can be deployed on mobile phones with Android operating systems. 🚀 Click [here](https://github.com/OpenBMB/mlc-MiniCPM) to install apk. MiniCPM-Llama3-V 2.5 coming soon.
-### Inference with llama.cpp
-MiniCPM-V 2.6 can run with llama.cpp now! See [our fork of llama.cpp](https://github.com/OpenBMB/llama.cpp/tree/minicpmv-main/examples/llava/README-minicpmv2.6.md) for more detail. This implementation supports smooth inference of 16~18 token/s on iPad (test environment：iPad Pro + M4).
-
-### Inference with ollama
-MiniCPM-V 2.6 can run with ollama now! See [our fork of ollama](https://github.com/OpenBMB/ollama/blob/minicpm-v2.6/examples/minicpm-v2.6/README.md) for more detail. This implementation supports smooth inference of 16~18 token/s on iPad (test environment：iPad Pro + M4).
-
-### Inference with vLLM
+### WebUI Demo
- vLLM now officially supports MiniCPM-V 2.6, MiniCPM-Llama3-V 2.5 and MiniCPM-V 2.0, Click to see.
-
-1. Install vLLM(>=0.5.4):
+Click to see how to deploy WebUI demo on different devices
+
```shell
-pip install vllm
+pip install -r requirements.txt
```
-2. Install timm: (optional, MiniCPM-V 2.0 need timm)
+
```shell
-pip install timm==0.9.10
+# For NVIDIA GPUs, run:
+python web_demo_2.5.py --device cuda
+
+# For Mac with MPS (Apple silicon or AMD GPUs), run:
+PYTORCH_ENABLE_MPS_FALLBACK=1 python web_demo_2.5.py --device mps
```
-3. Run the example(for image):
-```python
-from transformers import AutoTokenizer
-from PIL import Image
-from vllm import LLM, SamplingParams
+
-MODEL_NAME = "openbmb/MiniCPM-V-2_6"
-# Also available for previous models
-# MODEL_NAME = "openbmb/MiniCPM-Llama3-V-2_5"
-# MODEL_NAME = "HwwwH/MiniCPM-V-2"
+### Inference with llama.cpp
+MiniCPM-Llama3-V 2.5 can run with llama.cpp now! See our fork of [llama.cpp](https://github.com/OpenBMB/llama.cpp/tree/minicpm-v2.5/examples/minicpmv) for more detail. This implementation supports smooth inference of 6~8 token/s on mobile phones (test environment：Xiaomi 14 pro + Snapdragon 8 Gen 3).
-image = Image.open("xxx.png").convert("RGB")
-tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME, trust_remote_code=True)
-llm = LLM(
- model=MODEL_NAME,
- trust_remote_code=True,
- gpu_memory_utilization=1,
- max_model_len=2048
-)
+### Inference with vLLM
-messages = [{
- "role":
- "user",
- "content":
- # Number of images
- "(
./)" + \
- "\nWhat is the content of this image?"
-}]
-prompt = tokenizer.apply_chat_template(
- messages,
- tokenize=False,
- add_generation_prompt=True
-)
+
+Click to see how to inference with vLLM
+Because our pull request to vLLM is still waiting for reviewing, we fork this repository to build and test our vLLM demo. Here are the steps:
-# Single Inference
-inputs = {
- "prompt": prompt,
- "multi_modal_data": {
- "image": image
- # Multi images, the number of images should be equal to that of `(./)`
- # "image": [image, image]
- },
-}
-# Batch Inference
-# inputs = [{
-# "prompt": prompt,
-# "multi_modal_data": {
-# "image": image
-# },
-# } for _ in 2]
-
-
-# 2.6
-stop_tokens = ['<|im_end|>', '<|endoftext|>']
-stop_token_ids = [tokenizer.convert_tokens_to_ids(i) for i in stop_tokens]
-# 2.0
-# stop_token_ids = [tokenizer.eos_id]
-# 2.5
-# stop_token_ids = [tokenizer.eos_id, tokenizer.eot_id]
-
-sampling_params = SamplingParams(
- stop_token_ids=stop_token_ids,
- use_beam_search=True,
- temperature=0,
- best_of=3,
- max_tokens=1024
-)
-
-outputs = llm.generate(inputs, sampling_params=sampling_params)
-
-print(outputs[0].outputs[0].text)
+1. Clone our version of vLLM:
+```shell
+git clone https://github.com/OpenBMB/vllm.git
+```
+2. Install vLLM:
+```shell
+cd vllm
+pip install -e .
+```
+3. Install timm:
+```shell
+pip install timm=0.9.10
+```
+4. Run our demo:
+```shell
+python examples/minicpmv_example.py
```
-4. click [here](https://modelbest.feishu.cn/wiki/C2BWw4ZP0iCDy7kkCPCcX2BHnOf?from=from_copylink) if you want to use it with *video*, or get more details about `vLLM`.
## Fine-tuning
@@ -1616,23 +639,28 @@ We now support MiniCPM-V series fine-tuning with the SWIFT framework. SWIFT supp
Best Practices：[MiniCPM-V 1.0](https://github.com/modelscope/swift/blob/main/docs/source/Multi-Modal/minicpm-v最佳实践.md), [MiniCPM-V 2.0](https://github.com/modelscope/swift/blob/main/docs/source/Multi-Modal/minicpm-v-2最佳实践.md)
-## FAQs
-Click here to view the [FAQs](./docs/faqs.md)
+
+
+## TODO
+
+- [x] MiniCPM-V fine-tuning support
+- [ ] Code release for real-time interactive assistant
## Model License
-* This repository is released under the [Apache-2.0](https://github.com/OpenBMB/MiniCPM/blob/main/LICENSE) License.
+The code in this repo is released according to [Apache-2.0](https://github.com/OpenBMB/MiniCPM/blob/main/LICENSE)
-* The usage of MiniCPM-V model weights must strictly follow [MiniCPM Model License.md](https://github.com/OpenBMB/MiniCPM/blob/main/MiniCPM%20Model%20License.md).
+The usage of MiniCPM-V's and OmniLMM's parameters is subject to "[General Model License Agreement - Source Notes - Publicity Restrictions - Commercial License](https://github.com/OpenBMB/General-Model-License/blob/main/通用模型许可协议-来源说明-宣传限制-商业授权.md)"
-* The models and weights of MiniCPM are completely free for academic research. after filling out a ["questionnaire"](https://modelbest.feishu.cn/share/base/form/shrcnpV5ZT9EJ6xYjh3Kx0J6v8g) for registration, are also available for free commercial use.
-
+The parameters are fully open to academic research
+
+Please contact cpm@modelbest.cn to obtain written authorization for commercial uses. Free commercial use is also allowed after registration.
## Statement
As LMMs, MiniCPM-V models (including OmniLMM) generate contents by learning a large amount of multimodal corpora, but they cannot comprehend, express personal opinions or make value judgement. Anything generated by MiniCPM-V models does not represent the views and positions of the model developers
-We will not be liable for any problems arising from the use of MiniCPM-V models, including but not limited to data security issues, risk of public opinion, or any risks and problems arising from the misdirection, misuse, dissemination or misuse of the model.
+We will not be liable for any problems arising from the use of MiniCPMV-V models, including but not limited to data security issues, risk of public opinion, or any risks and problems arising from the misdirection, misuse, dissemination or misuse of the model.
## Institutions
@@ -1643,16 +671,15 @@ This project is developed by the following institutions:
-
[ModelBest](https://modelbest.cn/)
-
[Zhihu](https://www.zhihu.com/ )
-## 🌟 Star History
+## Other Multimodal Projects from Our Team
+👏 Welcome to explore other multimodal projects of our team:
-
-
-
-
-
+[VisCPM](https://github.com/OpenBMB/VisCPM/tree/main) | [RLHF-V](https://github.com/RLHF-V/RLHF-V) | [LLaVA-UHD](https://github.com/thunlp/LLaVA-UHD) | [RLAIF-V](https://github.com/RLHF-V/RLAIF-V)
-
+
-## Key Techniques and Other Multimodal Projects
-
-👏 Welcome to explore key techniques of MiniCPM-V and other multimodal projects of our team:
-
-[VisCPM](https://github.com/OpenBMB/VisCPM/tree/main) | [RLHF-V](https://github.com/RLHF-V/RLHF-V) | [LLaVA-UHD](https://github.com/thunlp/LLaVA-UHD) | [RLAIF-V](https://github.com/RLHF-V/RLAIF-V)
-
-
-## Citation
+## Citation
If you find our model/code/paper helpful, please consider cite our papers 📝 and star us ⭐️！
```bib
-@article{yao2024minicpm,
- title={MiniCPM-V: A GPT-4V Level MLLM on Your Phone},
- author={Yao, Yuan and Yu, Tianyu and Zhang, Ao and Wang, Chongyi and Cui, Junbo and Zhu, Hongji and Cai, Tianchi and Li, Haoyu and Zhao, Weilin and He, Zhihui and others},
- journal={arXiv preprint arXiv:2408.01800},
+@article{yu2023rlhf,
+ title={Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback},
+ author={Yu, Tianyu and Yao, Yuan and Zhang, Haoye and He, Taiwen and Han, Yifeng and Cui, Ganqu and Hu, Jinyi and Liu, Zhiyuan and Zheng, Hai-Tao and Sun, Maosong and others},
+ journal={arXiv preprint arXiv:2312.00849},
+ year={2023}
+}
+@article{viscpm,
+ title={Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languages},
+ author={Jinyi Hu and Yuan Yao and Chongyi Wang and Shan Wang and Yinxu Pan and Qianyu Chen and Tianyu Yu and Hanghao Wu and Yue Zhao and Haoye Zhang and Xu Han and Yankai Lin and Jiao Xue and Dahai Li and Zhiyuan Liu and Maosong Sun},
+ journal={arXiv preprint arXiv:2308.12038},
+ year={2023}
+}
+@article{xu2024llava-uhd,
+ title={{LLaVA-UHD}: an LMM Perceiving Any Aspect Ratio and High-Resolution Images},
+ author={Xu, Ruyi and Yao, Yuan and Guo, Zonghao and Cui, Junbo and Ni, Zanlin and Ge, Chunjiang and Chua, Tat-Seng and Liu, Zhiyuan and Huang, Gao},
+ journal={arXiv preprint arXiv:2403.11703},
+ year={2024}
+}
+@article{yu2024rlaifv,
+ title={RLAIF-V: Aligning MLLMs through Open-Source AI Feedback for Super GPT-4V Trustworthiness},
+ author={Yu, Tianyu and Zhang, Haoye and Yao, Yuan and Dang, Yunkai and Chen, Da and Lu, Xiaoman and Cui, Ganqu and He, Taiwen and Liu, Zhiyuan and Chua, Tat-Seng and Sun, Maosong},
+ journal={arXiv preprint arXiv:2405.17220},
year={2024}
}
```
diff --git a/README_zh.md b/README_zh.md
index 23e830b..574d6da 100644
--- a/README_zh.md
+++ b/README_zh.md
@@ -4,26 +4,23 @@
-**端侧可用的 GPT-4V 级单图、多图、视频多模态大模型**
+**端侧可用的 GPT-4V 级多模态大模型**
中文 |
[English](./README_en.md)
- 加入我们的
💬 微信社区
-
- MiniCPM-V 2.6 🤗 🤖 | MiniCPM-Llama3-V 2.5 🤗 🤖 |
- MiniCPM-Llama3-V 2.5 技术报告
+ MiniCPM-Llama3-V 2.5 🤗 🤖 |
+ MiniCPM-V 2.0 🤗 🤖 |
+ MiniCPM-V 2.0 技术博客
-
-
-点击查看完整更新日志。
-
-* [2024.06.03] 现在，你可以利用多张低显存显卡（12G/16G）进行GPU串行推理。详情请参见该[文档](https://github.com/OpenBMB/MiniCPM-V/blob/main/docs/inference_on_multiple_gpus.md)配置。
* [2024.05.25] MiniCPM-Llama3-V 2.5 [支持流式输出和自定义系统提示词](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5#usage)了，欢迎试用!
* [2024.05.24] 我们开源了 MiniCPM-Llama3-V 2.5 [gguf](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5-gguf)，支持 [llama.cpp](#llamacpp-部署) 推理！实现端侧 6-8 tokens/s 的流畅解码，欢迎试用！
* [2024.05.20] 我们开源了 MiniCPM-Llama3-V 2.5，增强了 OCR 能力，支持 30 多种语言，并首次在端侧实现了 GPT-4V 级的多模态能力！我们提供了[高效推理](#手机端部署)和[简易微调](./finetune/readme.md)的支持，欢迎试用！
-* [2024.04.23] 我们增加了MiniCPM-V 2.0对 [vLLM](#vllm-部署-) 的支持，欢迎体验！
+* [2024.04.23] 我们增加了对 [vLLM](#vllm) 的支持，欢迎体验！
* [2024.04.18] 我们在 HuggingFace Space 新增了 MiniCPM-V 2.0 的 [demo](https://huggingface.co/spaces/openbmb/MiniCPM-V-2)，欢迎体验！
* [2024.04.17] MiniCPM-V 2.0 现在支持用户部署本地 [WebUI Demo](#本地webui-demo部署) 了，欢迎试用!
* [2024.04.15] MiniCPM-V 2.0 现在可以通过 SWIFT 框架 [微调](https://github.com/modelscope/swift/blob/main/docs/source/Multi-Modal/minicpm-v-2最佳实践.md) 了，支持流式输出!
@@ -60,797 +48,29 @@
* [2024.03.14] MiniCPM-V 现在支持 SWIFT 框架下的[微调](https://github.com/modelscope/swift/blob/main/docs/source/Multi-Modal/minicpm-v最佳实践.md)了，感谢 [Jintao](https://github.com/Jintao-Huang) 的贡献！
* [2024.03.01] MiniCPM-V 现在支持在 Mac 电脑上进行部署！
* [2024.02.01] 我们开源了 MiniCPM-V 和 OmniLMM-12B，分别可以支持高效的端侧部署和同规模领先的多模态能力！
-
## 目录
-- [MiniCPM-V 2.6](#minicpm-v-26)
- [MiniCPM-Llama3-V 2.5](#minicpm-llama3-v-25)
- [MiniCPM-V 2.0](#minicpm-v-20)
-- [Gradio Demo 🤗](#gradio-demo-)
+- [Online Demo](#online-demo)
- [安装](#安装)
- [推理](#推理)
- [模型库](#模型库)
- [多轮对话](#多轮对话)
- - [多图理解](#多图理解)
- - [少样本上下文学习](#少样本上下文学习)
- - [视频理解](#视频理解)
- - [多卡推理](#多卡推理)
- [Mac 推理](#mac-推理)
- [手机端部署](#手机端部署)
- [本地WebUI Demo部署](#本地webui-demo部署)
- [llama.cpp 部署](#llamacpp-部署)
- - [ollama 部署](#ollama-部署)
- [vLLM 部署 ](#vllm-部署-)
- [微调](#微调)
-- [FAQs](#faqs)
+- [未来计划](#未来计划)
+- [🌟 Star History](#-star-history)
+- [引用](#引用)
-## MiniCPM-V 2.6
-
-**MiniCPM-V 2.6** 是 MiniCPM-V 系列中最新、性能最佳的模型。该模型基于 SigLip-400M 和 Qwen2-7B 构建，共 8B 参数。与 MiniCPM-Llama3-V 2.5 相比，MiniCPM-V 2.6 性能提升显著，并引入了多图和视频理解的新功能。MiniCPM-V 2.6 的主要特点包括：
-
-
-- 🔥 **领先的性能。**
- MiniCPM-V 2.6 在最新版本 OpenCompass 榜单上（综合 8 个主流多模态评测基准）平均得分 65.2，**以8B量级的大小在单图理解方面超越了 GPT-4o mini、GPT-4V、Gemini 1.5 Pro 和 Claude 3.5 Sonnet 等主流商用闭源多模态大模型**。
-
-- 🖼️ **多图理解和上下文学习。**
- MiniCPM-V 2.6 还支持**多图对话和推理**。它在 Mantis-Eval、BLINK、Mathverse mv 和 Sciverse mv 等主流多图评测基准中取得了**最佳水平**，并展现出了优秀的上下文学习能力。
-
-- 🎬 **视频理解。**
- MiniCPM-V 2.6 还可以**接受视频输入**，进行对话和提供涵盖时序和空间信息的详细视频描述。模型在 有/无字幕 评测场景下的 Video-MME 表现均超过了 **GPT-4V、Claude 3.5 Sonnet 和 LLaVA-NeXT-Video-34B**等商用闭源模型。
-
-- 💪 **强大的 OCR 能力及其他功能。**
- MiniCPM-V 2.6 可以处理任意长宽比的图像，像素数可达 180 万（如 1344x1344）。在 OCRBench 上取得**最佳水平，超过 GPT-4o、GPT-4V 和 Gemini 1.5 Pro 等商用闭源模型**。基于最新的 [RLAIF-V](https://github.com/RLHF-V/RLAIF-V/) 和 [VisCPM](https://github.com/OpenBMB/VisCPM) 技术，其具备了**可信的多模态行为**，在 Object HalBench 上的幻觉率显著低于 GPT-4o 和 GPT-4V，并支持英语、中文、德语、法语、意大利语、韩语等**多种语言**。
-
-- 🚀 **卓越的效率。**
- 除了对个人用户友好的模型大小，MiniCPM-V 2.6 还表现出**最先进的视觉 token 密度**（即每个视觉 token 编码的像素数量）。它**仅需 640 个 token 即可处理 180 万像素图像，比大多数模型少 75%**。这一特性优化了模型的推理速度、首 token 延迟、内存占用和功耗。因此，MiniCPM-V 2.6 可以支持 iPad 等终端设备上的高效**实时视频理解**。
-
-- 💫 **易于使用。**
- MiniCPM-V 2.6 可以通过多种方式轻松使用：(1) [llama.cpp](https://github.com/OpenBMB/llama.cpp/blob/minicpmv-main/examples/llava/README-minicpmv2.6.md) 和 [ollama](https://github.com/OpenBMB/ollama/blob/minicpm-v2.6/examples/minicpm-v2.6/README.md) 支持在本地设备上进行高效的 CPU 推理，(2) [int4](https://huggingface.co/openbmb/MiniCPM-V-2_6-int4) 和 [GGUF](https://huggingface.co/openbmb/MiniCPM-V-2_6-gguf) 格式的量化模型，有 16 种尺寸，(3) [vLLM](#vllm-部署-) 支持高吞吐量和内存高效的推理，(4) 针对新领域和任务进行微调，(5) 使用 [Gradio](#本地-webui-demo-) 快速设置本地 WebUI 演示，(6) 在线[demo](https://huggingface.co/spaces/openbmb/MiniCPM-V-2_6)即可体验。
-
-### 性能评估
-
-
-
-点击查看 OpenCompass, MME, MMVet, OCRBench, MMMU, MathVista, MMB, AI2D, TextVQA, DocVQA, HallusionBench, Object HalBench 上的单图评测结果详情。
-
-
-
-
-
- | Model |
- Size |
- Token Density+ |
- OpenCompass |
- MME |
- MMVet |
- OCRBench |
- MMMU val |
- MathVista mini |
- MMB1.1 test |
- AI2D |
- TextVQA val |
- DocVQA test |
- HallusionBench |
- Object HalBench |
-
-
-
-
- | Proprietary |
-
-
- | GPT-4o |
- - |
- 1088 |
- 69.9 |
- 2328.7 |
- 69.1 |
- 736 |
- 69.2 |
- 61.3 |
- 82.2 |
- 84.6 |
- - |
- 92.8 |
- 55.0 |
- 17.6 |
-
-
- | Claude 3.5 Sonnet |
- - |
- 750 |
- 67.9 |
- 1920.0 |
- 66.0 |
- 788 |
- 65.9 |
- 61.6 |
- 78.5 |
- 80.2 |
- - |
- 95.2 |
- 49.9 |
- 13.8 |
-
-
- | Gemini 1.5 Pro |
- - |
- - |
- 64.4 |
- 2110.6 |
- 64.0 |
- 754 |
- 60.6 |
- 57.7 |
- 73.9 |
- 79.1 |
- 73.5 |
- 86.5 |
- 45.6 |
- - |
-
-
- | GPT-4o mini |
- - |
- 1088 |
- 64.1 |
- 2003.4 |
- 66.9 |
- 785 |
- 60.0 |
- 52.4 |
- 76.0 |
- 77.8 |
- - |
- - |
- 46.1 |
- 12.4 |
-
-
- | GPT-4V |
- - |
- 1088 |
- 63.5 |
- 2070.2 |
- 67.5 |
- 656 |
- 61.7 |
- 54.7 |
- 79.8 |
- 78.6 |
- 78.0 |
- 87.2 |
- 43.9 |
- 14.2 |
-
-
- | Step-1V |
- - |
- - |
- 59.5 |
- 2206.4 |
- 63.3 |
- 625 |
- 49.9 |
- 44.8 |
- 78.0 |
- 79.2 |
- 71.6 |
- - |
- 48.4 |
- - |
-
-
- | Qwen-VL-Max |
- - |
- 784 |
- 58.3 |
- 2281.7 |
- 61.8 |
- 684 |
- 52.0 |
- 43.4 |
- 74.6 |
- 75.7 |
- 79.5 |
- 93.1 |
- 41.2 |
- 13.4 |
-
-
- | Open-source |
-
-
- | LLaVA-NeXT-Yi-34B |
- 34B |
- 157 |
- 55.0 |
- 2006.5 |
- 50.7 |
- 574 |
- 48.8 |
- 40.4 |
- 77.8 |
- 78.9 |
- 69.3 |
- - |
- 34.8 |
- 12.6 |
-
-
- | Mini-Gemini-HD-34B |
- 34B |
- 157 |
- - |
- 2141 |
- 59.3 |
- 518 |
- 48.0 |
- 43.3 |
- - |
- 80.5 |
- 74.1 |
- 78.9 |
- - |
- - |
-
-
- | Cambrian-34B |
- 34B |
- 1820 |
- 58.3 |
- 2049.9 |
- 53.2 |
- 591 |
- 50.4 |
- 50.3 |
- 77.8 |
- 79.5 |
- 76.7 |
- 75.5 |
- 41.6 |
- 14.7 |
-
-
- | GLM-4V-9B |
- 13B |
- 784 |
- 59.1 |
- 2018.8 |
- 58.0 |
- 776 |
- 46.9 |
- 51.1 |
- 67.9 |
- 71.2 |
- - |
- - |
- 45.0 |
- - |
-
-
- | InternVL2-8B |
- 8B |
- 706 |
- 64.1 |
- 2215.1 |
- 54.3 |
- 794 |
- 51.2 |
- 58.3 |
- 79.4 |
- 83.6 |
- 77.4 |
- 91.6 |
- 45.0 |
- 21.3 |
-
-
- | MiniCPM-Llama-V 2.5 |
- 8B |
- 1882 |
- 58.8 |
- 2024.6 |
- 52.8 |
- 725 |
- 45.8 |
- 54.3 |
- 72.0 |
- 78.4 |
- 76.6 |
- 84.8 |
- 42.4 |
- 10.3 |
-
-
- | MiniCPM-V 2.6 |
- 8B |
- 2822 |
- 65.2 |
- 2348.4* |
- 60.0 |
- 852* |
- 49.8* |
- 60.6 |
- 78.0 |
- 82.1 |
- 80.1 |
- 90.8 |
- 48.1* |
- 8.2 |
-
-
-
-
-
-
-
-
-点击查看 Mantis Eval, BLINK, Mathverse mv, Sciverse mv, MIRB 上的多图评测结果详情。
-
-
-
-
-
- | Model |
- Size |
- Mantis Eval |
- BLINK val |
- Mathverse mv |
- Sciverse mv |
- MIRB |
-
-
-
-
- | Proprietary |
-
-
- | GPT-4V |
- - |
- 62.7 |
- 54.6 |
- 60.3 |
- 66.9 |
- 53.1 |
-
-
- | LLaVA-NeXT-Interleave-14B |
- 14B |
- 66.4 |
- 52.6 |
- 32.7 |
- 30.2 |
- - |
-
-
- | Open-source |
-
-
- | Emu2-Chat |
- 37B |
- 37.8 |
- 36.2 |
- - |
- 27.2 |
- - |
-
-
- | CogVLM |
- 17B |
- 45.2 |
- 41.1 |
- - |
- - |
- - |
-
-
- | VPG-C |
- 7B |
- 52.4 |
- 43.1 |
- 24.3 |
- 23.1 |
- - |
-
-
- | VILA 8B |
- 8B |
- 51.2 |
- 39.3 |
- - |
- 36.5 |
- - |
-
-
- | InternLM-XComposer-2.5 |
- 8B |
- 53.1* |
- 48.9 |
- 32.1* |
- - |
- 42.5 |
-
-
- | InternVL2-8B |
- 8B |
- 59.0* |
- 50.9 |
- 30.5* |
- 34.4* |
- 56.9* |
-
-
- | MiniCPM-V 2.6 |
- 8B |
- 69.1 |
- 53.0 |
- 84.9 |
- 74.9 |
- 53.8 |
-
-
-
-
-
-
-
-
-点击查看 Video-MME 和 Video-ChatGPT 上的视频评测结果详情。
-
-
-
-
-
- | Model |
- Size |
- Video-MME |
- Video-ChatGPT |
-
-
- |
- |
- w/o subs |
- w subs |
- Correctness |
- Detail |
- Context |
- Temporal |
- Consistency |
-
-
-
-
- | Proprietary |
-
-
- | Claude 3.5 Sonnet |
- - |
- 60.0 |
- - |
- - |
- - |
- - |
- - |
- - |
-
-
- | GPT-4V |
- - |
- 59.9 |
- - |
- - |
- - |
- - |
- - |
- - |
-
-
- | Open-source |
-
-
- | LLaVA-NeXT-7B |
- 7B |
- - |
- - |
- 3.39 |
- 3.29 |
- 3.92 |
- 2.60 |
- 3.12 |
-
-
- | LLaVA-NeXT-34B |
- 34B |
- - |
- - |
- 3.29 |
- 3.23 |
- 3.83 |
- 2.51 |
- 3.47 |
-
-
- | CogVLM2-Video |
- 12B |
- - |
- - |
- 3.49 |
- 3.46 |
- 3.23 |
- 2.98 |
- 3.64 |
-
-
- | LongVA |
- 7B |
- 52.4 |
- 54.3 |
- 3.05 |
- 3.09 |
- 3.77 |
- 2.44 |
- 3.64 |
-
-
- | InternVL2-8B |
- 8B |
- 54.0 |
- 56.9 |
- - |
- - |
- - |
- - |
- - |
-
-
- | InternLM-XComposer-2.5 |
- 8B |
- 55.8 |
- - |
- - |
- - |
- - |
- - |
- - |
-
-
- | LLaVA-NeXT-Video |
- 32B |
- 60.2 |
- 63.0 |
- 3.48 |
- 3.37 |
- 3.95 |
- 2.64 |
- 3.28 |
-
-
- | MiniCPM-V 2.6 |
- 8B |
- 60.9 |
- 63.6 |
- 3.59 |
- 3.28 |
- 3.93 |
- 2.73 |
- 3.62 |
-
-
-
-
-
-
-
-点击查看 TextVQA, VizWiz, VQAv2, OK-VQA上的少样本评测结果详情。
-
-
-
-
-
- | Model |
- Size |
- Shot |
- TextVQA val |
- VizWiz test-dev |
- VQAv2 test-dev |
- OK-VQA val |
-
-
-
-
- | Flamingo |
- 80B |
- 0* |
- 35.0 |
- 31.6 |
- 56.3 |
- 40.6 |
-
-
- | 4 |
- 36.5 |
- 39.6 |
- 63.1 |
- 57.4 |
-
-
- | 8 |
- 37.3 |
- 44.8 |
- 65.6 |
- 57.5 |
-
-
- | IDEFICS |
- 80B |
- 0* |
- 30.9 |
- 36.0 |
- 60.0 |
- 45.2 |
-
-
- | 4 |
- 34.3 |
- 40.4 |
- 63.6 |
- 52.4 |
-
-
- | 8 |
- 35.7 |
- 46.1 |
- 64.8 |
- 55.1 |
-
-
- | OmniCorpus |
- 7B |
- 0* |
- 43.0 |
- 49.8 |
- 63.2 |
- 45.5 |
-
-
- | 4 |
- 45.4 |
- 51.3 |
- 64.5 |
- 46.5 |
-
-
- | 8 |
- 45.6 |
- 52.2 |
- 64.7 |
- 46.6 |
-
-
- | Emu2 |
- 37B |
- 0 |
- 26.4 |
- 40.4 |
- 33.5 |
- 26.7 |
-
-
- | 4 |
- 48.2 |
- 54.6 |
- 67.0 |
- 53.2 |
-
-
- | 8 |
- 49.3 |
- 54.7 |
- 67.8 |
- 54.1 |
-
-
- | MM1 |
- 30B |
- 0 |
- 26.2 |
- 40.4 |
- 48.9 |
- 26.7 |
-
-
- | 8 |
- 49.3 |
- 54.7 |
- 70.9 |
- 54.1 |
-
-
- | MiniCPM-V 2.6+ |
- 8B |
- 0 |
- 43.9 |
- 33.8 |
- 45.4 |
- 23.9 |
-
-
- | 4 |
- 63.6 |
- 60.5 |
- 65.5 |
- 50.1 |
-
-
- | 8 |
- 64.6 |
- 63.4 |
- 68.2 |
- 51.4 |
-
-
-
-
-
-
-
-### 典型示例
-
-
-
- 点击查看更多示例。
-
-
-
-
-
-我们将 MiniCPM-V 2.6 部署在iPad Pro上，并录制了以下演示视频。
-
-
-
- 
-
- 
-
-
-
-
## MiniCPM-Llama3-V 2.5
-
-
-查看 MiniCPM-Llama3-V 2.5 的详细信息
-
**MiniCPM-Llama3-V 2.5** 是 MiniCPM-V 系列的最新版本模型，基于 SigLip-400M 和 Llama3-8B-Instruct 构建，共 8B 参数量，相较于 MiniCPM-V 2.0 性能取得较大幅度提升。MiniCPM-Llama3-V 2.5 值得关注的特点包括：
- 🔥 **领先的性能。**
@@ -859,6 +79,7 @@
- 💪 **优秀的 OCR 能力。**
MiniCPM-Llama3-V 2.5 可接受 180 万像素的任意宽高比图像输入，**OCRBench 得分达到 725，超越 GPT-4o、GPT-4V、Gemini Pro、Qwen-VL-Max 等商用闭源模型**，达到最佳水平。基于近期用户反馈建议，MiniCPM-Llama3-V 2.5 增强了全文 OCR 信息提取、表格图像转 markdown 等高频实用能力，并且进一步加强了指令跟随、复杂推理能力，带来更好的多模态交互体感。
+
- 🏆 **可信行为。**
借助最新的 [RLAIF-V](https://github.com/RLHF-V/RLAIF-V/) 对齐技术（[RLHF-V](https://github.com/RLHF-V/) [CVPR'24]系列的最新技术），MiniCPM-Llama3-V 2.5 具有更加可信的多模态行为，在 Object HalBench 的幻觉率降低到了 **10.3%**，显著低于 GPT-4V-1106 (13.6%)，达到开源社区最佳水平。[数据集已发布](https://huggingface.co/datasets/openbmb/RLAIF-V-Dataset)。
@@ -868,8 +89,7 @@
- 🚀 **高效部署。**
MiniCPM-Llama3-V 2.5 较为系统地通过**模型量化、CPU、NPU、编译优化**等高效加速技术，实现高效的终端设备部署。对于高通芯片的移动手机，我们首次将 NPU 加速框架 QNN 整合进了 llama.cpp。经过系统优化后，MiniCPM-Llama3-V 2.5 实现了多模态大模型端侧**语言解码速度 3 倍加速**、**图像编码 150 倍加速**的巨大提升。
-- 💫 **易于使用。**
- MiniCPM-Llama3-V 2.5 可以通过多种方式轻松使用：（1）[llama.cpp](https://github.com/OpenBMB/llama.cpp/blob/minicpm-v2.5/examples/minicpmv/README.md) 和 [ollama](https://github.com/OpenBMB/ollama/tree/minicpm-v2.5/examples/minicpm-v2.5) 支持在本地设备上进行高效的 CPU 推理；（2）提供 16 种尺寸的 [GGUF](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5-gguf) 格式量化模型；（3）仅需 2 张 V100 GPU 即可进行高效的 [LoRA](https://github.com/OpenBMB/MiniCPM-V/tree/main/finetune#lora-finetuning) 微调；（ 4）支持[流式输出](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5#usage)；（5）快速搭建 [Gradio](https://github.com/OpenBMB/MiniCPM-V/blob/main/web_demo_2.5.py) 和 [Streamlit](https://github.com/OpenBMB/MiniCPM-V/blob/main/web_demo_streamlit-2_5.py) 本地 WebUI demo；（ 6.）[HuggingFace Spaces](https://huggingface.co/spaces/openbmb/MiniCPM-Llama3-V-2_5) 交互式 demo。
+
### 性能评估
@@ -1167,8 +387,20 @@
+我们将 MiniCPM-Llama3-V 2.5 部署在小米 14 Pro 上，并录制了以下演示视频。
-
+
+
+
+
+
+
+
+
+
+
+
+
## MiniCPM-V 2.0
@@ -1233,29 +465,9 @@
| OmniLMM-12B | [文档](./omnilmm.md) |
-## Gradio Demo 🤗
-
-我们提供由 Hugging Face Gradio 支持的在线和本地 Demo。Gradio 是目前最流行的模型部署框架，支持流式输出、进度条、process bars 和其他常用功能。
-
-### Online Demo
-
-欢迎试用 Online Demo: [MiniCPM-V 2.6](https://huggingface.co/spaces/openbmb/MiniCPM-V-2_6) | [MiniCPM-Llama3-V 2.5](https://huggingface.co/spaces/openbmb/MiniCPM-Llama3-V-2_5) | [MiniCPM-V 2.0](https://huggingface.co/spaces/openbmb/MiniCPM-V-2) 。
-
-### 本地 WebUI Demo
-
-您可以使用以下命令轻松构建自己的本地 WebUI Demo。
-
-```shell
-pip install -r requirements.txt
-```
-
-```shell
-# 对于 NVIDIA GPU，请运行：
-python web_demo_2.6.py --device cuda
-
-```
-
+## Online Demo
+欢迎通过以下链接使用我们的网页端推理服务： [MiniCPM-Llama3-V 2.5](https://huggingface.co/spaces/openbmb/MiniCPM-Llama3-V-2_5) ｜ [MiniCPM-V 2.0](https://huggingface.co/spaces/openbmb/MiniCPM-V-2).
## 安装
@@ -1285,11 +497,8 @@ pip install -r requirements.txt
| 模型 | 设备 | 资源 | 简介 | 下载链接 |
|:--------------|:-:|:----------:|:-------------------|:---------------:|
-| MiniCPM-V 2.6| GPU | 17 GB | 最新版本，提供最佳的端侧单图、多图、视频理解能力。 | [🤗](https://huggingface.co/openbmb/MiniCPM-V-2_6) [](https://modelscope.cn/models/OpenBMB/MiniCPM-V-2_6) |
-| MiniCPM-V 2.6 gguf | CPU | 6 GB | gguf 版本，更低的内存占用和更高的推理效率。 | [🤗](https://huggingface.co/openbmb/MiniCPM-V-2_6-gguf) [](https://modelscope.cn/models/OpenBMB/MiniCPM-V-2_6-gguf) |
-| MiniCPM-V 2.6 int4 | GPU | 7 GB | int4量化版，更低显存占用。 | [🤗](https://huggingface.co/openbmb/MiniCPM-V-2_6-int4) [](https://modelscope.cn/models/OpenBMB/MiniCPM-V-2_6-int4) |
-| MiniCPM-Llama3-V 2.5| GPU | 19 GB | 提供出色的端侧多模态理解能力。 | [🤗](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5/) [](https://modelscope.cn/models/OpenBMB/MiniCPM-Llama3-V-2_5) |
-| MiniCPM-Llama3-V 2.5 gguf | CPU | 6 GB | gguf 版本，更低的内存占用和更高的推理效率。 | [🤗](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5-gguf) [](https://modelscope.cn/models/OpenBMB/MiniCPM-Llama3-V-2_5-gguf) |
+| MiniCPM-Llama3-V 2.5| GPU | 19 GB | 最新版本，提供最佳的端侧多模态理解能力。 | [🤗](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5/) [](https://modelscope.cn/models/OpenBMB/MiniCPM-Llama3-V-2_5) |
+| MiniCPM-Llama3-V 2.5 gguf| CPU | 5 GB | gguf 版本，更低的内存占用和更高的推理效率。 | [🤗](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5-gguf) [](https://modelscope.cn/models/OpenBMB/MiniCPM-Llama3-V-2_5-gguf) |
| MiniCPM-Llama3-V 2.5 int4 | GPU | 8 GB | int4量化版，更低显存占用。 | [🤗](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5-int4/) [](https://modelscope.cn/models/OpenBMB/MiniCPM-Llama3-V-2_5-int4) |
| MiniCPM-V 2.0 | GPU | 8 GB | 轻量级版本，平衡计算开销和多模态理解能力。 | [🤗](https://huggingface.co/openbmb/MiniCPM-V-2) [](https://modelscope.cn/models/OpenBMB/MiniCPM-V-2) |
| MiniCPM-V 1.0 | GPU | 7 GB | 最轻量版本， 提供最快的推理速度。 | [🤗](https://huggingface.co/openbmb/MiniCPM-V) [](https://modelscope.cn/models/OpenBMB/MiniCPM-V) |
@@ -1306,40 +515,30 @@ pip install -r requirements.txt
```python
+from chat import MiniCPMVChat, img2base64
import torch
-from PIL import Image
-from transformers import AutoModel, AutoTokenizer
+import json
torch.manual_seed(0)
-model = AutoModel.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True,
- attn_implementation='sdpa', torch_dtype=torch.bfloat16) # sdpa or flash_attention_2, no eager
-model = model.eval().cuda()
-tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True)
+chat_model = MiniCPMVChat('openbmb/MiniCPM-Llama3-V-2_5')
-image = Image.open('./assets/airplane.jpeg').convert('RGB')
+im_64 = img2base64('./assets/airplane.jpeg')
# First round chat
-question = "Tell me the model of this aircraft."
-msgs = [{'role': 'user', 'content': [image, question]}]
+msgs = [{"role": "user", "content": "Tell me the model of this aircraft."}]
-answer = model.chat(
- image=None,
- msgs=msgs,
- tokenizer=tokenizer
-)
+inputs = {"image": im_64, "question": json.dumps(msgs)}
+answer = chat_model.chat(inputs)
print(answer)
# Second round chat
# pass history context of multi-turn conversation
-msgs.append({"role": "assistant", "content": [answer]})
-msgs.append({"role": "user", "content": ["Introduce something about Airbus A380."]})
+msgs.append({"role": "assistant", "content": answer})
+msgs.append({"role": "user", "content": "Introduce something about Airbus A380."})
-answer = model.chat(
- image=None,
- msgs=msgs,
- tokenizer=tokenizer
-)
+inputs = {"image": im_64, "question": json.dumps(msgs)}
+answer = chat_model.chat(inputs)
print(answer)
```
@@ -1351,130 +550,7 @@ print(answer)
"The Airbus A380 is a double-deck, wide-body, four-engine jet airliner made by Airbus. It is the world's largest passenger airliner and is known for its long-haul capabilities. The aircraft was developed to improve efficiency and comfort for passengers traveling over long distances. It has two full-length passenger decks, which can accommodate more passengers than a typical single-aisle airplane. The A380 has been operated by airlines such as Lufthansa, Singapore Airlines, and Emirates, among others. It is widely recognized for its unique design and significant impact on the aviation industry."
```
-#### 多图理解
-
- 点击查看使用 MiniCPM-V 2.6 进行多图理解的Python示例
-
-```python
-import torch
-from PIL import Image
-from transformers import AutoModel, AutoTokenizer
-model = AutoModel.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True,
- attn_implementation='sdpa', torch_dtype=torch.bfloat16) # sdpa or flash_attention_2, no eager
-model = model.eval().cuda()
-tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True)
-
-image1 = Image.open('image1.jpg').convert('RGB')
-image2 = Image.open('image2.jpg').convert('RGB')
-question = 'Compare image 1 and image 2, tell me about the differences between image 1 and image 2.'
-
-msgs = [{'role': 'user', 'content': [image1, image2, question]}]
-
-answer = model.chat(
- image=None,
- msgs=msgs,
- tokenizer=tokenizer
-)
-print(answer)
-```
-
-
-#### 少样本上下文学习
-
-
- 点击查看使用 MiniCPM-V 2.6 进行few-shot推理的Python示例
-
-```python
-import torch
-from PIL import Image
-from transformers import AutoModel, AutoTokenizer
-
-model = AutoModel.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True,
- attn_implementation='sdpa', torch_dtype=torch.bfloat16) # sdpa or flash_attention_2, no eager
-model = model.eval().cuda()
-tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True)
-
-question = "production date"
-image1 = Image.open('example1.jpg').convert('RGB')
-answer1 = "2023.08.04"
-image2 = Image.open('example2.jpg').convert('RGB')
-answer2 = "2007.04.24"
-image_test = Image.open('test.jpg').convert('RGB')
-
-msgs = [
- {'role': 'user', 'content': [image1, question]}, {'role': 'assistant', 'content': [answer1]},
- {'role': 'user', 'content': [image2, question]}, {'role': 'assistant', 'content': [answer2]},
- {'role': 'user', 'content': [image_test, question]}
-]
-
-answer = model.chat(
- image=None,
- msgs=msgs,
- tokenizer=tokenizer
-)
-print(answer)
-```
-
-
-#### 视频理解
-
- 点击查看使用 MiniCPM-V 2.6 进行视频理解的Python示例
-
-```python
-import torch
-from PIL import Image
-from transformers import AutoModel, AutoTokenizer
-from decord import VideoReader, cpu # pip install decord
-
-model = AutoModel.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True,
- attn_implementation='sdpa', torch_dtype=torch.bfloat16) # sdpa or flash_attention_2, no eager
-model = model.eval().cuda()
-tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True)
-
-MAX_NUM_FRAMES=64 # if cuda OOM set a smaller number
-
-def encode_video(video_path):
- def uniform_sample(l, n):
- gap = len(l) / n
- idxs = [int(i * gap + gap / 2) for i in range(n)]
- return [l[i] for i in idxs]
-
- vr = VideoReader(video_path, ctx=cpu(0))
- sample_fps = round(vr.get_avg_fps() / 1) # FPS
- frame_idx = [i for i in range(0, len(vr), sample_fps)]
- if len(frame_idx) > MAX_NUM_FRAMES:
- frame_idx = uniform_sample(frame_idx, MAX_NUM_FRAMES)
- frames = vr.get_batch(frame_idx).asnumpy()
- frames = [Image.fromarray(v.astype('uint8')) for v in frames]
- print('num frames:', len(frames))
- return frames
-
-video_path="video_test.mp4"
-frames = encode_video(video_path)
-question = "Describe the video"
-msgs = [
- {'role': 'user', 'content': frames + [question]},
-]
-
-# Set decode params for video
-params = {}
-params["use_image_id"] = False
-params["max_slice_nums"] = 2 # 如果cuda OOM且视频分辨率大于448*448可设为1
-
-answer = model.chat(
- image=None,
- msgs=msgs,
- tokenizer=tokenizer,
- **params
-)
-print(answer)
-```
-
-
-
-### 多卡推理
-您可以通过将模型的层分布在多个低显存显卡（12 GB 或 16 GB）上，运行 MiniCPM-Llama3-V 2.5。请查看该[教程](https://github.com/OpenBMB/MiniCPM-V/blob/main/docs/inference_on_multiple_gpus.md)，详细了解如何使用多张低显存显卡载入模型并进行推理。
### Mac 推理
@@ -1514,11 +590,11 @@ PYTORCH_ENABLE_MPS_FALLBACK=1 python test.py
### 手机端部署
-MiniCPM-V 2.0 可运行在Android手机上，点击[MiniCPM-V 2.0](https://github.com/OpenBMB/mlc-MiniCPM)安装apk使用;
+MiniCPM-V 2.0 可运行在Android手机上, 点击[2.0](https://github.com/OpenBMB/mlc-MiniCPM)安装apk使用; MiniCPM-Llama3-V 2.5 将很快推出，敬请期待。
### 本地WebUI Demo部署
-点击查看本地WebUI demo 在 NVIDIA GPU、Mac等不同设备部署方法
+点击查看本地WebUI demo 在 NVIDIA GPU, Mac等不同设备部署方法
```shell
pip install -r requirements.txt
@@ -1526,101 +602,39 @@ pip install -r requirements.txt
```shell
# For NVIDIA GPUs, run:
-python web_demo_2.6.py --device cuda
+python web_demo_2.5.py --device cuda
+
+# For Mac with MPS (Apple silicon or AMD GPUs), run:
+PYTORCH_ENABLE_MPS_FALLBACK=1 python web_demo_2.5.py --device mps
```
### llama.cpp 部署
-MiniCPM-V 2.6 现在支持llama.cpp啦! 用法请参考[我们的fork llama.cpp](https://github.com/OpenBMB/llama.cpp/tree/minicpmv-main/examples/llava/README-minicpmv2.6.md)， 在iPad上可以支持 16~18 token/s 的流畅推理（测试环境：iPad Pro + M4）。
-
-### ollama 部署
-MiniCPM-V 2.6 现在支持ollama啦! 用法请参考[我们的fork ollama](https://github.com/OpenBMB/ollama/blob/minicpm-v2.6/examples/minicpm-v2.6/README.md)， 在iPad上可以支持 16~18 token/s 的流畅推理（测试环境：iPad Pro + M4）。
+MiniCPM-Llama3-V 2.5 现在支持llama.cpp啦! 用法请参考我们的fork [llama.cpp](https://github.com/OpenBMB/llama.cpp/tree/minicpm-v2.5/examples/minicpmv)， 在手机上可以支持 6~8 token/s 的流畅推理（测试环境：Xiaomi 14 pro + Snapdragon 8 Gen 3）。
### vLLM 部署
-点击查看, vLLM 现已官方支持MiniCPM-V 2.6、MiniCPM-Llama3-V 2.5 和 MiniCPM-V 2.0
+点击查看 vLLM 部署运行的方法
+由于我们对 vLLM 提交的 PR 还在 review 中，因此目前我们 fork 了一个 vLLM 仓库以供测试使用。
-1. 安装 vLLM(>=0.5.4):
+1. 首先克隆我们 fork 的 vLLM 库:
```shell
-pip install vllm
+git clone https://github.com/OpenBMB/vllm.git
```
-3. 安装 timm 库: （可选，MiniCPM-V 2.0需安装）
+2. 安装 vLLM 库:
+```shell
+cd vllm
+pip install -e .
+```
+3. 安装 timm 库:
```shell
pip install timm=0.9.10
```
-4. 运行示例代码:（注意：如果使用本地路径的模型，请确保模型代码已更新到Hugging Face上的最新版)
-```python
-from transformers import AutoTokenizer
-from PIL import Image
-from vllm import LLM, SamplingParams
-
-MODEL_NAME = "openbmb/MiniCPM-V-2_6"
-# Also available for previous models
-# MODEL_NAME = "openbmb/MiniCPM-Llama3-V-2_5"
-# MODEL_NAME = "HwwwH/MiniCPM-V-2"
-
-image = Image.open("xxx.png").convert("RGB")
-tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME, trust_remote_code=True)
-llm = LLM(
- model=MODEL_NAME,
- trust_remote_code=True,
- gpu_memory_utilization=1,
- max_model_len=2048
-)
-
-messages = [{
- "role":
- "user",
- "content":
- # Number of images
- "(./)" + \
- "\nWhat is the content of this image?"
-}]
-prompt = tokenizer.apply_chat_template(
- messages,
- tokenize=False,
- add_generation_prompt=True
-)
-
-# Single Inference
-inputs = {
- "prompt": prompt,
- "multi_modal_data": {
- "image": image
- # Multi images, the number of images should be equal to that of `(./)`
- # "image": [image, image]
- },
-}
-# Batch Inference
-# inputs = [{
-# "prompt": prompt,
-# "multi_modal_data": {
-# "image": image
-# },
-# } for _ in 2]
-
-
-# 2.6
-stop_tokens = ['<|im_end|>', '<|endoftext|>']
-stop_token_ids = [tokenizer.convert_tokens_to_ids(i) for i in stop_tokens]
-# 2.0
-# stop_token_ids = [tokenizer.eos_id]
-# 2.5
-# stop_token_ids = [tokenizer.eos_id, tokenizer.eot_id]
-
-sampling_params = SamplingParams(
- stop_token_ids=stop_token_ids,
- use_beam_search=True,
- temperature=0,
- best_of=3,
- max_tokens=1024
-)
-
-outputs = llm.generate(inputs, sampling_params=sampling_params)
-
-print(outputs[0].outputs[0].text)
+4. 测试运行示例程序:
+```shell
+python examples/minicpmv_example.py
```
-4. [点击此处](https://modelbest.feishu.cn/wiki/C2BWw4ZP0iCDy7kkCPCcX2BHnOf?from=from_copylink)查看带视频推理和其他有关 `vLLM` 的信息。
+
@@ -1637,17 +651,25 @@ print(outputs[0].outputs[0].text)
我们支持使用 SWIFT 框架微调 MiniCPM-V 系列模型。SWIFT 支持近 200 种大语言模型和多模态大模型的训练、推理、评测和部署。支持 PEFT 提供的轻量训练方案和完整的 Adapters 库支持的最新训练技术如 NEFTune、LoRA+、LLaMA-PRO 等。
-参考文档：[MiniCPM-V 1.0](https://github.com/modelscope/swift/blob/main/docs/source/Multi-Modal/minicpm-v最佳实践.md)，[MiniCPM-V 2.0](https://github.com/modelscope/swift/blob/main/docs/source/Multi-Modal/minicpm-v-2最佳实践.md)
+参考文档：[MiniCPM-V 1.0](https://github.com/modelscope/swift/blob/main/docs/source/Multi-Modal/minicpm-v最佳实践.md), [MiniCPM-V 2.0](https://github.com/modelscope/swift/blob/main/docs/source/Multi-Modal/minicpm-v-2最佳实践.md)
+
+## 未来计划
+
+- [x] 支持 MiniCPM-V 系列模型微调
+- [ ] 实时多模态交互代码开源
-## FAQs
-点击查看 [FAQs](./docs/faqs.md)
## 模型协议
-* 本仓库中代码依照 [Apache-2.0](https://github.com/OpenBMB/MiniCPM/blob/main/LICENSE) 协议开源
-* MiniCPM-V 模型权重的使用则需要遵循 [“MiniCPM模型商用许可协议.md”](https://github.com/OpenBMB/MiniCPM/blob/main/MiniCPM%E6%A8%A1%E5%9E%8B%E5%95%86%E7%94%A8%E8%AE%B8%E5%8F%AF%E5%8D%8F%E8%AE%AE.md)。
-* MiniCPM 模型权重对学术研究完全开放，在填写[“问卷”](https://modelbest.feishu.cn/share/base/form/shrcnpV5ZT9EJ6xYjh3Kx0J6v8g)进行登记后亦允许免费商业使用。
+本仓库中代码依照 Apache-2.0 协议开源
+
+本项目中模型权重的使用遵循 “[通用模型许可协议-来源说明-宣传限制-商业授权](https://github.com/OpenBMB/General-Model-License/blob/main/通用模型许可协议-来源说明-宣传限制-商业授权.md)”。
+
+本项目中模型权重对学术研究完全开放。
+
+如需将模型用于商业用途，请联系 cpm@modelbest.cn 来获取书面授权，登记后可以免费商业使用。
+
## 声明
@@ -1664,16 +686,15 @@ print(outputs[0].outputs[0].text)
- [面壁智能](https://modelbest.cn/)
- [知乎](https://www.zhihu.com/ )
-## 🌟 Star History
+## 其他多模态项目
+👏 欢迎了解我们更多的多模态项目：
-
-
-
-
-
+[VisCPM](https://github.com/OpenBMB/VisCPM/tree/main) | [RLHF-V](https://github.com/RLHF-V/RLHF-V) | [LLaVA-UHD](https://github.com/thunlp/LLaVA-UHD) | [RLAIF-V](https://github.com/RLHF-V/RLAIF-V)
-
+
-## 支持技术和其他多模态项目
-
-👏 欢迎了解 MiniCPM-V 背后的支持技术和更多我们的多模态项目！
-
-[VisCPM](https://github.com/OpenBMB/VisCPM/tree/main) | [RLHF-V](https://github.com/RLHF-V/RLHF-V) | [LLaVA-UHD](https://github.com/thunlp/LLaVA-UHD) | [RLAIF-V](https://github.com/RLHF-V/RLAIF-V)
-
-
-
-## 引用
+## 引用
如果您觉得我们模型/代码/论文有帮助，请给我们 ⭐ 和 引用 📝，感谢！
```bib
-@article{yao2024minicpm,
- title={MiniCPM-V: A GPT-4V Level MLLM on Your Phone},
- author={Yao, Yuan and Yu, Tianyu and Zhang, Ao and Wang, Chongyi and Cui, Junbo and Zhu, Hongji and Cai, Tianchi and Li, Haoyu and Zhao, Weilin and He, Zhihui and others},
- journal={arXiv preprint arXiv:2408.01800},
+@article{yu2023rlhf,
+ title={Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback},
+ author={Yu, Tianyu and Yao, Yuan and Zhang, Haoye and He, Taiwen and Han, Yifeng and Cui, Ganqu and Hu, Jinyi and Liu, Zhiyuan and Zheng, Hai-Tao and Sun, Maosong and others},
+ journal={arXiv preprint arXiv:2312.00849},
+ year={2023}
+}
+@article{viscpm,
+ title={Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languages},
+ author={Jinyi Hu and Yuan Yao and Chongyi Wang and Shan Wang and Yinxu Pan and Qianyu Chen and Tianyu Yu and Hanghao Wu and Yue Zhao and Haoye Zhang and Xu Han and Yankai Lin and Jiao Xue and Dahai Li and Zhiyuan Liu and Maosong Sun},
+ journal={arXiv preprint arXiv:2308.12038},
+ year={2023}
+}
+@article{xu2024llava-uhd,
+ title={{LLaVA-UHD}: an LMM Perceiving Any Aspect Ratio and High-Resolution Images},
+ author={Xu, Ruyi and Yao, Yuan and Guo, Zonghao and Cui, Junbo and Ni, Zanlin and Ge, Chunjiang and Chua, Tat-Seng and Liu, Zhiyuan and Huang, Gao},
+ journal={arXiv preprint arXiv:2403.11703},
+ year={2024}
+}
+@article{yu2024rlaifv,
+ title={RLAIF-V: Aligning MLLMs through Open-Source AI Feedback for Super GPT-4V Trustworthiness},
+ author={Yu, Tianyu and Zhang, Haoye and Yao, Yuan and Dang, Yunkai and Chen, Da and Lu, Xiaoman and Cui, Ganqu and He, Taiwen and Liu, Zhiyuan and Chua, Tat-Seng and Sun, Maosong},
+ journal={arXiv preprint arXiv:2405.17220},
year={2024}
}
```
- 
- 
- 
- 
-
-