部署应用llama-2-7b模子

llama-2-7b模子下载

官网链接
   必要申请才能下载，参考怎样在国内下载llama模子的思绪，我申请了一个gmail邮箱，vpn挂澳大利亚，非常钟内通过了申请。
    也可以直接下载hf版本
    此外，迅雷云盘下载（来自中文大语言模子 Llama-2 7B（或13B） 本地化部署 （国内云服务器、GPU单卡16GB、中文模子、WEB页面TextUI、简单入门））：
  

• llama-2-7b
  
• llama-2-7b-hf
  

  下载好的llama-2-7b文件包罗：

转hf

   在huggingface申请llama权限没能通过T T，拜托同砚下了一个llama-2-7b模子，但是发现源代码使用不了，遂探索怎样转为llama-2-7b-hf
    参考下载 llama2-7b-hf 全流程【小白踩坑记录】的第一种方法
  

• 下载 convert_llama_weights_to_hf.py文件
  
• python convert_llama_weights_to_hf.py --input_dir llama-2-7b --output_dir llama-2-7b-hf --model_size 7B，其中llama-2-7b和llama-2-7b-hf分别是转换前后的文件夹路径。
  

   遇见报错 ImportError:
LlamaConverter requires the protobuf library but it was not found in your environment. Checkout the instructions on the installation page of its repo: https://github.com/protocolbuffers/protobuf/tree/master/python#installation and follow the ones that match your environment. Please note that you may need to restart your runtime after installation.

办理：pip install protobuf==3.19.0
  

• 运行成功
  
  
  
  

转换后的llama-2-7b-hf模子文件包罗：

加载模子

1. from transformers import LlamaForCausalLM, LlamaTokenizer
2. self.base_tokenizer = LlamaTokenizer.from_pretrained(model_path)
3. self.base_model = LlamaForCausalLM.from_pretrained(model_path, device_map="auto", load_in_8bit=True)

复制代码

报错与办理

SentencePiece library

报错

1. ImportError:
2. LlamaTokenizer requires the SentencePiece library but it was not found in your environment. Checkout the instructions on the installation page of its repo: https://github.com/google/sentencepiece#installation and follow the ones that match your environment. Please note that you may need to restart your runtime after installation.

复制代码

办理

1. pip install SentencePiece

复制代码

bitsandbytes

1.版本更新

报错

1. ImportError: Using `bitsandbytes` 8-bit quantization requires the latest version of bitsandbytes: `pip install -U bitsandbytes
2. `

复制代码

办理

1. pip install -U bitsandbytes

复制代码

2.CUDA Setup failed despite GPU being available.

报错

1. /home/u2023141116/.conda/envs/seqxgpt_new/lib/python3.9/site-packages/bitsandbytes/cuda_setup/main.py:167: UserWarning: /home/u2023141116/.conda/envs/seqxgpt_new did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths...
2.   warn(msg)
3. /home/u2023141116/.conda/envs/seqxgpt_new/lib/python3.9/site-packages/bitsandbytes/cuda_setup/main.py:167: UserWarning: Found duplicate ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] files: {PosixPath('/home/u2023141116/cuda-11.3/lib64/libcudart.so.11.0'), PosixPath('/home/u2023141116/cuda-11.3/lib64/libcudart.so')}.. We select the PyTorch default libcudart.so, which is {torch.version.cuda},but this might missmatch with the CUDA version that is needed for bitsandbytes.To override this behavior set the BNB_CUDA_VERSION=<version string, e.g. 122> environmental variableFor example, if you want to use the CUDA version 122BNB_CUDA_VERSION=122 python ...OR set the environmental variable in your .bashrc: export BNB_CUDA_VERSION=122In the case of a manual override, make sure you set the LD_LIBRARY_PATH, e.g.export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-11.2
4.   warn(msg)
5. /home/u2023141116/.conda/envs/seqxgpt_new/lib/python3.9/site-packages/bitsandbytes/cuda_setup/main.py:167: UserWarning: /home/u2023141116/cuda-11.3/extras/CUPTI/lib64:/home/u2023141116/cuda-11.3/lib64:/usr/local/nvidia/lib::/home/u2023141116/cuda-11.3/lib64 did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths...
6.   warn(msg)
7. The following directories listed in your path were found to be non-existent: {PosixPath('() {  eval `/usr/bin/modulecmd bash $*`\n}')}
8. CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
9. /home/u2023141116/.conda/envs/seqxgpt_new/lib/python3.9/site-packages/bitsandbytes/cuda_setup/main.py:167: UserWarning: Found duplicate ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] files: {PosixPath('/usr/local/cuda/lib64/libcudart.so.11.0'), PosixPath('/usr/local/cuda/lib64/libcudart.so')}.. We select the PyTorch default libcudart.so, which is {torch.version.cuda},but this might missmatch with the CUDA version that is needed for bitsandbytes.To override this behavior set the BNB_CUDA_VERSION=<version string, e.g. 122> environmental variableFor example, if you want to use the CUDA version 122BNB_CUDA_VERSION=122 python ...OR set the environmental variable in your .bashrc: export BNB_CUDA_VERSION=122In the case of a manual override, make sure you set the LD_LIBRARY_PATH, e.g.export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-11.2
10.   warn(msg)
11. DEBUG: Possible options found for libcudart.so: {PosixPath('/usr/local/cuda/lib64/libcudart.so.11.0'), PosixPath('/usr/local/cuda/lib64/libcudart.so')}
12. CUDA SETUP: PyTorch settings found: CUDA_VERSION=113, Highest Compute Capability: 8.0.
13. CUDA SETUP: To manually override the PyTorch CUDA version please see:https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md
14. CUDA SETUP: Required library version not found: libbitsandbytes_cuda113.so. Maybe you need to compile it from source?
15. CUDA SETUP: Defaulting to libbitsandbytes_cpu.so...
17. ================================================ERROR=====================================
18. CUDA SETUP: CUDA detection failed! Possible reasons:
19. 1. You need to manually override the PyTorch CUDA version. Please see: "https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md
20. 2. CUDA driver not installed
21. 3. CUDA not installed
22. 4. You have multiple conflicting CUDA libraries
23. 5. Required library not pre-compiled for this bitsandbytes release!
24. CUDA SETUP: If you compiled from source, try again with `make CUDA_VERSION=DETECTED_CUDA_VERSION` for example, `make CUDA_VERSION=113`.
25. CUDA SETUP: The CUDA version for the compile might depend on your conda install. Inspect CUDA version via `conda list | grep cuda`.
26. ================================================================================
28. CUDA SETUP: Something unexpected happened. Please compile from source:
29. git clone https://github.com/TimDettmers/bitsandbytes.git
30. cd bitsandbytes
31. CUDA_VERSION=113 make cuda11x
32. python setup.py install
33. CUDA SETUP: Setup Failed!

复制代码

1. RuntimeError: Failed to import transformers.integrations.bitsandbytes because of the following error (look up to see its traceback):
3.         CUDA Setup failed despite GPU being available. Please run the following command to get more information:
5.         python -m bitsandbytes
7.         Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
8.         to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
9.         and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues

复制代码

使用python -m bitsandbytes查察详细报错信息：

1. bin /home/u2023141116/.conda/envs/seqxgpt_new/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda113.so
2. False
3. /home/u2023141116/.conda/envs/seqxgpt_new/lib/python3.9/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: /home/u2023141116/.conda/envs/seqxgpt_new did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths...
4.   warn(msg)
5. CUDA SETUP: CUDA runtime path found: /home/u2023141116/cuda-11.3/lib64/libcudart.so.11.0
6. CUDA SETUP: Highest compute capability among GPUs detected: 8.0
7. CUDA SETUP: Detected CUDA version 113
8. CUDA SETUP: Loading binary /home/u2023141116/.conda/envs/seqxgpt_new/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda113.so...
9. /lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by /home/u2023141116/.conda/envs/seqxgpt_new/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda113.so)
11. CUDA SETUP: Something unexpected happened. Please compile from source:
12. git clone git@github.com:TimDettmers/bitsandbytes.git
13. cd bitsandbytes
14. CUDA_VERSION=113 make cuda11x
15. python setup.py install

复制代码

办理

报错信息中比力紧张的是

1. /lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by /home/u2023141116/.conda/envs/seqxgpt_new/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda113.so)

复制代码

  尝试降低版本bitsandbytes==0.39.0办理失败（参考deepspeed 或 bitsandbytes 遇到CUDA Setup failed despite GPU being available.）
通过添加环境变量成功（参考Potential issue with GNU C++ Library）
  

• 找到libstdc++.so.6的对应路径
  

1. sudo find / -name "libstdc++.so.6*"

复制代码

• 验证文件是否支持对应的版本，会出现很多CXXABI版本，找到和上述报错对应的谁人。
  

1. strings /anaconda/envs/py38/lib/libstdc++.so.6 | grep CXXABI

复制代码

• 添加环境变量
  

1. vim ~/.bashrc

复制代码

在最后一行参加（PATH替换成第一步找到的路径）

1. export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/anaconda/envs/py38/lib/

复制代码

激活

1. source ~/.bashrc

复制代码

3.‘LlamaTokenizerFast’ object has no attribute ‘sp_model’

报错

1. AttributeError: 'LlamaTokenizerFast' object has no attribute 'sp_model'

复制代码

办理

   尝试降低transformers版本（原4.44.2）的方式无法办理
  

• 4.33.2（参考[linux] ‘LlamaTokenizer‘ object has no attribute ‘sp_model‘）
  
• 4.30.2
  

    搜刮发现LlamaTokenizerFast现在还没有这个属性，遂不使用此tokenizer。（参考AttributeError: ‘LlamaTokenizerFast’ object has no attribute ‘sp_model’）
  将代码中的AutoTokenizer和AutoModelForCausalLM固定为LlamaTokenizer和LlamaForCausalLM。
文本天生

1. from transformers import AutoModelForCausalLM, AutoTokenizer
2. import torch
4. # 加载Llama-2-7b模型和分词器
5. model_name = "model/llama-2-7b-hf"  # 确保这是正确的模型名称
6. tokenizer = AutoTokenizer.from_pretrained(model_name)
7. model = AutoModelForCausalLM.from_pretrained(model_name)
9. # 移动模型到GPU（如果可用）
10. device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
11. model.to(device)
13. # 定义输入文本
14. input_text = "What is cat?"
16. # 将输入文本编码为模型输入
17. input_ids = tokenizer.encode(input_text, return_tensors="pt").to(device)
19. # 生成文本（定义生成的最大长度）
20. max_length = 500
21. output = model.generate(input_ids, max_length=max_length, num_return_sequences=1)
23. # 解码生成的文本
24. generated_text = tokenizer.decode(output[0], skip_special_tokens=True)
26. print(generated_text)

复制代码

免责声明：如果侵犯了您的权益，请联系站长，我们会及时删除侵权内容，谢谢合作！更多信息从访问主页：qidao123.com:ToB企服之家，中国第一个企服评测及商务社交产业平台。