目标检测多模态大模型实践：貌似是全网唯一Shikra的部署和测试教程，内含各 ...

原文：
Shikra: Unleashing Multimodal LLM’s Referential Dialogue Magic
代码：
https://github.com/shikras/shikra
模型：
https://huggingface.co/shikras/shikra-7b-delta-v1
https://huggingface.co/shikras/shikra7b-delta-v1-0708
第一个是论文用的，第二个会有迭代。
本人的shikra论文解读，逐行解读，非常具体！
多模态大模型目标检测，精读，Shikra
部署：

• 下载GitHub工程，和shikras的模型参数，注意，还要下载LLaMA-7b的模型；
  
• 创建环境：
  

1. conda create -n shikra python=3.10
2. conda activate shikra
3. pip install -r requirements.txt

复制代码

反面我运行的时间报缺包了，又pip install了以下包，不过每个人环境差别：

1. pip install uvicorn
2. pip install mmengine

复制代码

然后还会报错：

1. File "/usr/local/lib/python3.10/dist-packages/cv2/typing/__init__.py", line 171, in <module>
2.     LayerId = cv2.dnn.DictValue
3. AttributeError: module 'cv2.dnn' has no attribute 'DictValue'

复制代码

解决方案:
修改/usr/local/lib/python3.10/dist-packages/cv2/typing/init.py
注释掉LayerId = cv2.dnn.DictValue这行即可。

• 权重下载和合并
  shikra官方提供的模型权重必要和llama1-7b合并之后才能用，然而llama1必要申请，比力贫苦，在hf上找到了平替（这一步我走了好久QwQ）：
  https://huggingface.co/huggyllama/llama-7b
  各人本身下载，然后运行官方提供的合并代码：
  

1. python mllm/models/shikra/apply_delta.py \
2.     --base /path/to/llama-7b \
3.     --target /output/path/to/shikra-7b-merge \
4.     --delta shikras/shikra-7b-delta-v1

复制代码

得到了可用的模型参数shikra-7b-merge。
注意要把参数文件夹里config里的模型路径改成merge版的。
此外还必要下载clip模型参数：
https://huggingface.co/openai/clip-vit-large-patch14
代码和配置文件中有多处调用/openai/clip-vit-large-patch14，要改成当地版本。如果不预先下载，应该会在运行时自动下载，各人看网络环境自行选择。

• 我写的demo文件，用于在命令行测试模型效果，主要是为了不消gradio和fastapi这些东西。
  

1. import argparse
2. import os
3. import sys
4. import base64
5. import logging
6. import time
7. from pathlib import Path
8. from io import BytesIO
10. import torch
11. import uvicorn
12. import transformers
13. from PIL import Image
14. from mmengine import Config
15. from transformers import BitsAndBytesConfig
17. sys.path.append(str(Path(__file__).parent.parent.parent))
19. from mllm.dataset.process_function import PlainBoxFormatter
20. from mllm.dataset.builder import prepare_interactive
21. from mllm.models.builder.build_shikra import load_pretrained_shikra
22. from mllm.dataset.utils.transform import expand2square, box_xyxy_expand2square
24. # Set up logging
25. log_level = logging.DEBUG
26. transformers.logging.set_verbosity(log_level)
27. transformers.logging.enable_default_handler()
28. transformers.logging.enable_explicit_format()
30. # prompt for coco
32. # Argument parsing
33. parser = argparse.ArgumentParser("Shikra Local Demo")
34. parser.add_argument('--model_path', default = "xxx/shikra-merge", help="Path to the model")
35. parser.add_argument('--load_in_8bit', action='store_true', help="Load model in 8-bit precision")
36. parser.add_argument('--image_path', default = "xxx/shikra-main/mllm/demo/assets/ball.jpg", help="Path to the image file")
37. parser.add_argument('--text', default="What do you see in this image? Please mention the objects and their locations using the format [x1,y1,x2,y2].", help="Text prompt")
38. parser.add_argument('--boxes_value', nargs='+', type=int, default=[], help="Bounding box values (x1, y1, x2, y2)")
39. parser.add_argument('--boxes_seq', nargs='+', type=int, default=[], help="Sequence of bounding boxes")
40. parser.add_argument('--do_sample', action='store_true', help="Use sampling during generation")
41. parser.add_argument('--max_length', type=int, default=512, help="Maximum length of the output")
42. parser.add_argument('--top_p', type=float, default=1.0, help="Top-p value for sampling")
43. parser.add_argument('--temperature', type=float, default=1.0, help="Temperature for sampling")
45. args = parser.parse_args()
46. model_name_or_path = args.model_path
47. # Model initialization
48. model_args = Config(dict(
49.     type='shikra',
50.     version='v1',
52.     # checkpoint config
53.     cache_dir=None,
54.     model_name_or_path=model_name_or_path,
55.     vision_tower=r'xxx/clip-vit-large-patch14',
56.     pretrain_mm_mlp_adapter=None,
58.     # model config
59.     mm_vision_select_layer=-2,
60.     model_max_length=2048,
62.     # finetune config
63.     freeze_backbone=False,
64.     tune_mm_mlp_adapter=False,
65.     freeze_mm_mlp_adapter=False,
67.     # data process config
68.     is_multimodal=True,
69.     sep_image_conv_front=False,
70.     image_token_len=256,
71.     mm_use_im_start_end=True,
73.     target_processor=dict(
74.         boxes=dict(type='PlainBoxFormatter'),
75.     ),
77.     process_func_args=dict(
78.         conv=dict(type='ShikraConvProcess'),
79.         target=dict(type='BoxFormatProcess'),
80.         text=dict(type='ShikraTextProcess'),
81.         image=dict(type='ShikraImageProcessor'),
82.     ),
84.     conv_args=dict(
85.         conv_template='vicuna_v1.1',
86.         transforms=dict(type='Expand2square'),
87.         tokenize_kwargs=dict(truncation_size=None),
88.     ),
90.     gen_kwargs_set_pad_token_id=True,
91.     gen_kwargs_set_bos_token_id=True,
92.     gen_kwargs_set_eos_token_id=True,
93. ))
94. training_args = Config(dict(
95.     bf16=False,
96.     fp16=True,
97.     device='cuda',
98.     fsdp=None,
99. ))
101. quantization_kwargs = dict(
102.     quantization_config=BitsAndBytesConfig(
103.         load_in_8bit=args.load_in_8bit,
104.     )
105. ) if args.load_in_8bit else dict()
107. model, preprocessor = load_pretrained_shikra(model_args, training_args, **quantization_kwargs)
109. # Convert the model and vision tower to float16
110. if not getattr(model, 'is_quantized', False):
111.     model.to(dtype=torch.float16, device=torch.device('cuda'))
112. if not getattr(model.model.vision_tower[0], 'is_quantized', False):
113.     model.model.vision_tower[0].to(dtype=torch.float16, device=torch.device('cuda'))
115. preprocessor['target'] = {'boxes': PlainBoxFormatter()}
116. tokenizer = preprocessor['text']
118. # Load and preprocess the image
119. pil_image = Image.open(args.image_path).convert("RGB")
120. ds = prepare_interactive(model_args, preprocessor)
122. image = expand2square(pil_image)
123. boxes_value = [box_xyxy_expand2square(box, w=pil_image.width, h=pil_image.height) for box in zip(args.boxes_value[::2], args.boxes_value[1::2], args.boxes_value[2::2], args.boxes_value[3::2])]
125. ds.set_image(image)
126. ds.append_message(role=ds.roles[0], message=args.text, boxes=boxes_value, boxes_seq=args.boxes_seq)
127. model_inputs = ds.to_model_input()
128. model_inputs['images'] = model_inputs['images'].to(torch.float16)
130. # Generate
131. gen_kwargs = dict(
132.     use_cache=True,
133.     do_sample=args.do_sample,
134.     pad_token_id=tokenizer.pad_token_id,
135.     bos_token_id=tokenizer.bos_token_id,
136.     eos_token_id=tokenizer.eos_token_id,
137.     max_new_tokens=args.max_length,
138.     top_p=args.top_p,
139.     temperature=args.temperature,
140. )
142. input_ids = model_inputs['input_ids']
143. st_time = time.time()
144. with torch.inference_mode():
145.     with torch.autocast(device_type='cuda', dtype=torch.float16):
146.         output_ids = model.generate(**model_inputs, **gen_kwargs)
147. print(f"Generated in {time.time() - st_time} seconds")
149. input_token_len = input_ids.shape[-1]
150. response = tokenizer.batch_decode(output_ids[:, input_token_len:])[0]
151. print(f"Response: {response}")

复制代码

这么良心，点个关注吧，会连续更新多模态大模型相关内容。

免责声明：如果侵犯了您的权益，请联系站长，我们会及时删除侵权内容，谢谢合作！更多信息从访问主页：qidao123.com:ToB企服之家，中国第一个企服评测及商务社交产业平台。