构建llama.cpp并在linux上使用gpu

使用gpu构建llama.cpp

更多详情拜见https://github.com/abetlen/llama-cpp-python，官网网站会随着版本迭代更新。
下载并进入llama.cpp

地址：https://github.com/ggerganov/llama.cpp
可以下载到当地再传到服务器上

1. git clone https://github.com/ggerganov/llama.cpp
2. cd llama.cpp

复制代码

编译源码（make
）

天生./main和./quantize等二进制文件。详见：https://github.com/ggerganov/llama.cpp/blob/master/docs/build.md
使用CPU

1. make

复制代码

使用GPU

1. make
2. GGML_CUDA=1

复制代码

大概出现的报错及办理方法

I ccache not found. Consider installing it for faster compilation.

1. sudo apt-get install ccache

复制代码

Makefile:1002: *** I ERROR: For CUDA versions < 11.7 a target CUDA architecture must be explicitly provided via environment variable CUDA_DOCKER_ARCH, e.g. by running "export CUDA_DOCKER_ARCH=compute_XX" on Unix-like systems, where XX is the minimum compute capability that the code needs to run on. A list with compute capabilities can be found here: https://developer.nvidia.com/cuda-gpus . Stop.
阐明cuda版本太低，假如不是自己下载好的，参考该文章nvcc -V 显示的cuda版本和实际版本不同等更换
NOTICE: The 'server' binary is deprecated. Please use 'llama-server' instead.
提示：随版本迭代，下令大概会失效
正确效果

内容很长，只截取了一部门

调用大模型

安装llama.cpp，比力慢

1. CMAKE_ARGS="-DGGML_CUDA=on" pip install llama-cpp-python

复制代码

调用

1. from langchain_community.chat_models import ChatLlamaCpp
2. from langchain_community.llms import LlamaCpp
4. local_model = "/data/pretrained/gguf/Meta-Llama-3-8B-Instruct-Q5_K_M.gguf"
5. llm = ChatLlamaCpp(
6.     seed=1,
7.     temperature=0.5,
8.     model_path=local_model,
9.     n_ctx=8192,
10.     n_gpu_layers=64,
11.     n_batch=12,  # Should be between 1 and n_ctx, consider the amount of VRAM in your GPU.
12.     max_tokens=8192,
13.     repeat_penalty=1.5,
14.     top_p=0.5,
15.     f16_kv=False,
16.     verbose=True,
17. )
18. messages = [
19.     (
20.         "system",
21.         "You are a helpful assistant that translates English to Chinese. Translate the user sentence.",
22.     ),
23.     ("human",
24.      "OpenAI has a tool calling API that lets you describe tools and their arguments, and have the model return a JSON object with a tool to invoke and the inputs to that tool. tool-calling is extremely useful for building tool-using chains and agents, and for getting structured outputs from models more generally."),
25. ]
27. ai_msg = llm.invoke(messages)
28. print(ai_msg.content)

复制代码

正确打印中存在如下内容，阐明找到gpu

1. ggml_cuda_init: found 2 CUDA devices:
2.   Device 0: 你的gpu型号, compute capability gpu计算能力, VMM: yes
3.   Device 1: 你的gpu型号, compute capability gpu计算能力, VMM: yes
4. llm_load_tensors: offloading 32 repeating layers to GPU
5. llm_load_tensors:        CPU buffer size =   344.44 MiB
6. llm_load_tensors:      CUDA0 buffer size =  2932.34 MiB
7. llm_load_tensors:      CUDA1 buffer size =  2183.15 MiB

复制代码

构建自己的gguf

很多llm没有gguf格式，但有着自己的环境要求，llama.cpp需要举行统一，使用gguf文件格式（量化模型 quantizing model）。该过程需要再之前下载的llama.cpp文件夹中操作。

1. python convert_hf_to_gguf.py --outfile /data/pretrained/gguf/meta-llama-3.1-8b-instruct.gguf /data/pretrained/Meta-Llama-3.1-8B-Instruct

复制代码

大概出现的错误

1. AttributeError: module 'torch' has no attribute 'float8_e4m3fn'

复制代码

这是由于torch版本较低导致，可以直接在源码中举行解释。（convert-hf-to-gguf.py文件第4226行）

1. _dtype_str_map: dict[str, torch.dtype] = {
2.         "F64": torch.float64,
3.         "F32": torch.float32,
4.         "BF16": torch.bfloat16,
5.         "F16": torch.float16,
6.         # "U64": torch.uint64,
7.         "I64": torch.int64,
8.         # "U32": torch.uint32,
9.         "I32": torch.int32,
10.         # "U16": torch.uint16,
11.         "I16": torch.int16,
12.         "U8": torch.uint8,
13.         "I8": torch.int8,
14.         "BOOL": torch.bool,
15.         # "F8_E4M3": torch.float8_e4m3fn,
16.         # "F8_E5M2": torch.float8_e5m2,
17.     }

复制代码

免责声明：如果侵犯了您的权益，请联系站长，我们会及时删除侵权内容，谢谢合作！更多信息从访问主页：qidao123.com:ToB企服之家，中国第一个企服评测及商务社交产业平台。