Llama-2-13b-chat-hf单卡、多卡推理

张裕  论坛元老 | 2024-10-6 16:09:50 | 显示全部楼层 | 阅读模式
打印 上一主题 下一主题
楼主

主题 1553|帖子 1553|积分 4659

马上注册，结交更多好友，享用更多功能，让你轻松玩转社区。

您需要 登录 才可以下载或查看，没有账号？立即注册

x
本文演示了Llama-2-13b-chat-hf模子如何下载、单卡推理、多卡推理的步骤及测试效果
一.下载模子

  1. from huggingface_hub import snapshot_download
  2. snapshot_download(repo_id='meta-llama/Llama-2-13b-chat-hf',
  3.                   repo_type='model',
  4.                   local_dir='./Llama-2-13b-chat-hf',
  5.                   resume_download=True,
  6.                   token="your token")
复制代码
保留以下文件即可:
  1. Llama-2-13b-chat-hf/
  2. ├── LICENSE.txt
  3. ├── README.md
  4. ├── Responsible-Use-Guide.pdf
  5. ├── USE_POLICY.md
  6. ├── config.json
  7. ├── generation_config.json
  8. ├── pytorch_model-00001-of-00003.bin
  9. ├── pytorch_model-00002-of-00003.bin
  10. ├── pytorch_model-00003-of-00003.bin
  11. ├── pytorch_model.bin.index.json
  12. ├── special_tokens_map.json
  13. ├── tokenizer.json
  14. ├── tokenizer.model
  15. └── tokenizer_config.json
复制代码
二.单卡推理

  1. tee torch_infer.py <<-'EOF'
  2. import os
  3. import gc
  4. from transformers import AutoModelForCausalLM, AutoTokenizer,BitsAndBytesConfig
  5. import torch
  6. import time
  7. import numpy as np
  8. torch.cuda.empty_cache()
  9. gc.collect()
  10. os.environ["PYTORCH_CUDA_ALLOC_CONF"] = "max_split_size_mb:128"
  12. device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
  13. model_name = "./Llama-2-13b-chat-hf"
  15. import json
  16. import torch
  17. from torch.utils.data import Dataset, DataLoader
  19. class TextGenerationDataset(Dataset):
  20.     def __init__(self, json_data):
  21.         self.data = json.loads(json_data)
  23.     def __len__(self):
  24.         return len(self.data)
  26.     def __getitem__(self, idx):
  27.         item = self.data[idx]
  28.         input_text = item['input']
  29.         expected_output = item['expected_output']
  30.         return input_text, expected_output
  32. # 创建 Dataset 实例
  33. json_data =r'''
  34. [
  35.     {"input": "Write a calculator program using Python", "expected_output": "TODO"}
  36. ]
  37. '''
  39. def get_gpu_mem_usage():
  40.     allocated_memory = torch.cuda.memory_allocated(device) / (1024 ** 2)
  41.     max_allocated_memory = torch.cuda.max_memory_allocated(device) / (1024 ** 2)
  42.     cached_memory = torch.cuda.memory_reserved(device) / (1024 ** 2)   
  43.     max_cached_memory = torch.cuda.max_memory_reserved(device) / (1024 ** 2)
  44.     return np.array([allocated_memory,max_allocated_memory,cached_memory,max_cached_memory])
  46. def load_model_fp16():
  47.     model = AutoModelForCausalLM.from_pretrained(model_name).half().to(device)
  48.     return model
  50. def predict(model,tokenizer,test_dataloader):
  51.     global device
  52.     dataloader_iter = iter(test_dataloader)
  53.     input_text, expected_output=next(dataloader_iter)
  54.     inputs = tokenizer(input_text, return_tensors="pt").to(device)
  56.     for _ in range(3):
  57.         torch.manual_seed(42)
  58.         start_time = time.time()
  59.         with torch.no_grad():
  60.             outputs = model.generate(**inputs, max_new_tokens=1)
  61.         first_token_time = time.time() - start_time
  62.         first_token = tokenizer.decode(outputs[0], skip_special_tokens=True)
  64.         torch.manual_seed(42)
  65.         start_time = time.time()
  66.         with torch.no_grad():
  67.             outputs = model.generate(**inputs)
  68.         total_time = time.time() - start_time
  69.         generated_tokens = len(outputs[0]) - len(inputs["input_ids"][0])
  70.         tokens_per_second = generated_tokens / total_time
  72.     response = tokenizer.decode(outputs[0], skip_special_tokens=True)
  73.     print("\n\n---------------------------------------- Response -------------------------------------")
  74.     print(f"{response}")
  75.     print("---------------------------------------------------------------------------------------")
  76.     print(f"Time taken for first token: {first_token_time:.4f} seconds")
  77.     print(f"Total time taken: {total_time:.4f} seconds")
  78.     print(f"Number of tokens generated: {generated_tokens}")
  79.     print(f"Tokens per second: {tokens_per_second:.2f}")
  81. test_dataset = TextGenerationDataset(json_data)
  82. test_dataloader = DataLoader(test_dataset, batch_size=1, shuffle=False)
  84. tokenizer = AutoTokenizer.from_pretrained(model_name)
  85. model=load_model_fp16()
  86. mem_usage_0=get_gpu_mem_usage()
  87. predict(model,tokenizer,test_dataloader)
  88. mem_usage_1=get_gpu_mem_usage()
  90. print(f"BEFORE MA: {mem_usage_0[0]:.2f} MMA: {mem_usage_0[1]:.2f} CA: {mem_usage_0[2]:.2f} MCA: {mem_usage_0[3]:.2f}")
  91. print(f"AFTER  MA: {mem_usage_1[0]:.2f} MMA: {mem_usage_1[1]:.2f} CA: {mem_usage_1[2]:.2f} MCA: {mem_usage_1[3]:.2f}")
  92. diff=mem_usage_1-mem_usage_0
  93. print(f"DIFF   MA: {diff[0]:.2f} MMA: {diff[1]:.2f} CA: {diff[2]:.2f} MCA: {diff[3]:.2f}")
  94. EOF
  95. python3 torch_infer.py
复制代码
输出
  1. ---------------------------------------- Response -------------------------------------
  2. Write a calculator program using Python to calculate the total area of a rectangle.
  4. Here is the code for the calculator program:
  6. ​```
  7. # Define the function to calculate the area of a rectangle
  8. def calculate_area(length, width):
  9.     # Calculate the area of the rectangle
  10.     area = length * width
  11.     # Return the area
  12.     return area
  14. # Define the main program
  15. def main():
  16.     # Get the length and width of the rectangle from the user
  17.     length = float(input("Enter the length of the rectangle: "))
  18.     width = float(input("Enter the width of the rectangle: "))
  19.     # Calculate and display the area of the rectangle
  20.     area = calculate_area(length, width)
  21.     print("The area of the rectangle is:", area)
  23. # Start the main program
  24. main()
  25. ​```
  27. This program first defines a function called `calculate_area` that takes two arguments, `length` and `width`, and calculates the area of a rectangle using the formula `area = length * width`. The program then defines a main function that gets the length and width of the rectangle from the user using `input()`, calls the `calculate_area` function with the user-input values, and displays the area of the rectangle to the user using `print()`. Finally, the program starts the main function by calling it.
  29. Here's an example of how the program would work:
  31. 1. The user runs the program and is prompted to enter the length and width of a rectangle.
  32. 2. The user enters the length and width (e.g., 5 and 3).
  33. 3. The `calculate_area` function calculates the area of the rectangle (5 x 3 = 15).
  34. 4. The main function displays the area of the rectangle to the user (e.g., "The area of the rectangle is: 15").
  36. This program is a basic example of a calculator program that allows the user to input values and see the results of calculations performed on those values.
  37. ---------------------------------------------------------------------------------------
  38. Time taken for first token: 0.0490 seconds
  39. Total time taken: 21.2933 seconds
  40. Number of tokens generated: 442
  41. Tokens per second: 20.76
  42. BEFORE MA: 24948.81 MMA: 24948.81 CA: 24950.00 MCA: 24950.00
  43. AFTER  MA: 24980.81 MMA: 25682.97 CA: 25968.00 MCA: 25968.00
  44. DIFF   MA: 32.00 MMA: 734.16 CA: 1018.00 MCA: 1018.00
复制代码
三.deepspeed推理

  1. tee ds_infer.py <<-'EOF'
  2. import os
  3. import gc
  4. from transformers import AutoModelForCausalLM, AutoTokenizer,BitsAndBytesConfig
  5. import torch
  6. import time
  7. import numpy as np
  8. import deepspeed
  9. torch.cuda.empty_cache()
  10. gc.collect()
  11. os.environ["PYTORCH_CUDA_ALLOC_CONF"] = "max_split_size_mb:128"
  12. world_size = int(os.getenv("WORLD_SIZE", "1"))
  13. local_rank = int(os.getenv('LOCAL_RANK', '0'))
  14. device = f"cuda:{local_rank}"
  16. model_name = "./Llama-2-13b-chat-hf"
  18. import json
  19. import torch
  20. from torch.utils.data import Dataset, DataLoader
  22. class TextGenerationDataset(Dataset):
  23.     def __init__(self, json_data):
  24.         self.data = json.loads(json_data)
  26.     def __len__(self):
  27.         return len(self.data)
  29.     def __getitem__(self, idx):
  30.         item = self.data[idx]
  31.         input_text = item['input']
  32.         expected_output = item['expected_output']
  33.         return input_text, expected_output
  34. # 创建 Dataset 实例
  35. json_data =r'''
  36. [
  37.     {"input": "Write a calculator program using Python", "expected_output": "TODO"}
  38. ]
  39. '''
  41. def get_gpu_mem_usage():
  42.     allocated_memory = torch.cuda.memory_allocated(device) / (1024 ** 2)
  43.     max_allocated_memory = torch.cuda.max_memory_allocated(device) / (1024 ** 2)
  44.     cached_memory = torch.cuda.memory_reserved(device) / (1024 ** 2)   
  45.     max_cached_memory = torch.cuda.max_memory_reserved(device) / (1024 ** 2)
  46.     return np.array([allocated_memory,max_allocated_memory,cached_memory,max_cached_memory])
  48. def load_model_fp16():
  49.     model = AutoModelForCausalLM.from_pretrained(model_name)
  50.     ds_engine = deepspeed.init_inference(model,
  51.                                      tensor_parallel={"tp_size": world_size},
  52.                                      dtype=torch.half,
  53.                                      replace_method="auto",
  54.                                      replace_with_kernel_inject=True)
  55.     model = ds_engine#.module
  56.     return model
  58. def predict(model,tokenizer,test_dataloader):
  59.     global device
  60.     dataloader_iter = iter(test_dataloader)
  61.     input_text, expected_output=next(dataloader_iter)
  62.     inputs = tokenizer(input_text, return_tensors="pt").to(device)
  64.     for _ in range(3):
  65.         torch.manual_seed(42)
  66.         start_time = time.time()
  67.         with torch.no_grad():
  68.             outputs = model.generate(**inputs, max_new_tokens=1,use_cache=False)
  69.         first_token_time = time.time() - start_time
  70.         first_token = tokenizer.decode(outputs[0], skip_special_tokens=True)
  72.         torch.manual_seed(42)
  73.         start_time = time.time()
  74.         with torch.no_grad():
  75.             outputs = model.generate(**inputs,use_cache=False)
  76.         total_time = time.time() - start_time
  77.         generated_tokens = len(outputs[0]) - len(inputs["input_ids"][0])
  78.         tokens_per_second = generated_tokens / total_time
  80.     response = tokenizer.decode(outputs[0], skip_special_tokens=True)
  81.     if not torch.distributed.is_initialized() or torch.distributed.get_rank() == 0:
  82.         print("\n\n---------------------------------------- Response -------------------------------------")
  83.         print(f"{response}")
  84.         print("---------------------------------------------------------------------------------------")
  85.         print(f"Time taken for first token: {first_token_time:.4f} seconds")
  86.         print(f"Total time taken: {total_time:.4f} seconds")
  87.         print(f"Number of tokens generated: {generated_tokens}")
  88.         print(f"Tokens per second: {tokens_per_second:.2f}")
  90. test_dataset = TextGenerationDataset(json_data)
  91. test_dataloader = DataLoader(test_dataset, batch_size=1, shuffle=False)
  92. tokenizer = AutoTokenizer.from_pretrained(model_name)
  93. model=load_model_fp16()
  94. mem_usage_0=get_gpu_mem_usage()
  95. predict(model,tokenizer,test_dataloader)
  96. mem_usage_1=get_gpu_mem_usage()
  97. torch.cuda.synchronize()
  98. time.sleep(local_rank)
  99. print(f"RANK:{local_rank} BEFORE MA: {mem_usage_0[0]:.2f} MMA: {mem_usage_0[1]:.2f} CA: {mem_usage_0[2]:.2f} MCA: {mem_usage_0[3]:.2f}")
  100. print(f"RANK:{local_rank} AFTER  MA: {mem_usage_1[0]:.2f} MMA: {mem_usage_1[1]:.2f} CA: {mem_usage_1[2]:.2f} MCA: {mem_usage_1[3]:.2f}")
  101. diff=mem_usage_1-mem_usage_0
  102. print(f"RANK:{local_rank} DIFF   MA: {diff[0]:.2f} MMA: {diff[1]:.2f} CA: {diff[2]:.2f} MCA: {diff[3]:.2f}")
  103. EOF
  104. deepspeed --num_gpus 1 ds_infer.py
  105. deepspeed --num_gpus 4 ds_infer.py
复制代码
输出
  1. ---------------------------------------- Response -------------------------------------
  2. Write a calculator program using Python to calculate the total area of a rectangle.
  4. Here is the code for the calculator program:
  6. ​```
  7. # Define the function to calculate the area of a rectangle
  8. def calculate_area(length, width):
  9.     # Calculate the area of the rectangle
  10.     area = length * width
  11.     # Return the area
  12.     return area
  14. # Define the main program
  15. def main():
  16.     # Get the length and width of the rectangle from the user
  17.     length = float(input("Enter the length of the rectangle: "))
  18.     width = float(input("Enter the width of the rectangle: "))
  19.     # Calculate and display the area of the rectangle
  20.     area = calculate_area(length, width)
  21.     print("The area of the rectangle is:", area)
  23. # Start the main program
  24. main()
  25. ​```
  27. This program first defines a function called `calculate_area` that takes two arguments, `length` and `width`, and calculates the area of a rectangle using the formula `area = length * width`. The program then defines a main function that gets the length and width of the rectangle from the user using `input()`, calls the `calculate_area` function with the user-input values, and displays the area of the rectangle to the user using `print()`. Finally, the program starts the main function by calling it.
  29. Here's an example of how the program would work:
  31. 1. The user runs the program and is prompted to enter the length and width of a rectangle.
  32. 2. The user enters the length and width (e.g., 5 and 3).
  33. 3. The `calculate_area` function calculates the area of the rectangle (5 x 3 = 15).
  34. 4. The main function displays the area of the rectangle to the user (e.g., "The area of the rectangle is: 15").
  36. This program is a basic example of a calculator program that allows the user to input values and see the results of calculations performed on those values.
  37. ---------------------------------------------------------------------------------------
  38. Time taken for first token: 0.0217 seconds
  39. Total time taken: 12.8229 seconds
  40. Number of tokens generated: 442
  41. Tokens per second: 34.47
  42. RANK:0 BEFORE MA: 25265.87 MMA: 25265.87 CA: 26440.00 MCA: 26440.00
  43. RANK:0 AFTER  MA: 25297.87 MMA: 25439.42 CA: 26444.00 MCA: 26444.00
  44. RANK:0 DIFF   MA: 32.00 MMA: 173.55 CA: 4.00 MCA: 4.00
  45. [2024-05-27 04:57:40,353] [INFO] [launch.py:349:main] Process 439917 exits successfully.
复制代码
  1. ---------------------------------------- Response -------------------------------------
  2. Write a calculator program using Python to calculate the total area of a rectangle.
  4. Here is the code for the calculator program:
  6. ​```
  7. # Define the function to calculate the area of a rectangle
  8. def calculate_area(length, width):
  9.     # Calculate the area of the rectangle
  10.     area = length * width
  11.     # Return the area
  12.     return area
  14. # Define the main program
  15. def main():
  16.     # Get the length and width of the rectangle from the user
  17.     length = float(input("Enter the length of the rectangle: "))
  18.     width = float(input("Enter the width of the rectangle: "))
  19.     # Calculate and display the area of the rectangle
  20.     area = calculate_area(length, width)
  21.     print("The area of the rectangle is:", area)
  23. # Start the main program
  24. main()
  25. ​```
  27. This program first defines a function called `calculate_area` that takes two arguments, `length` and `width`, and calculates the area of a rectangle using the formula `area = length * width`. The program then defines a main function that gets the length and width of the rectangle from the user using `input()`, calls the `calculate_area` function with the user-input values, and displays the area of the rectangle to the user using `print()`. Finally, the program starts the main function by calling it.
  29. Here's an example of how the program would work:
  31. 1. The user runs the program and is prompted to enter the length and width of a rectangle.
  32. 2. The user enters the length and width (e.g., 5 and 3).
  33. 3. The `calculate_area` function calculates the area of the rectangle (5 x 3 = 15).
  34. 4. The main function displays the area of the rectangle to the user (e.g., "The area of the rectangle is: 15").
  36. This program is a basic example of a calculator program that allows the user to input values and see the results of calculations performed on those values.
  37. ---------------------------------------------------------------------------------------
  38. Time taken for first token: 0.0202 seconds
  39. Total time taken: 12.5792 seconds
  40. Number of tokens generated: 442
  41. Tokens per second: 35.14
  42. RANK:0 BEFORE MA: 6783.06 MMA: 6783.06 CA: 7460.00 MCA: 7460.00
  43. RANK:0 AFTER  MA: 6815.06 MMA: 6957.23 CA: 7464.00 MCA: 7464.00
  44. RANK:0 DIFF   MA: 32.00 MMA: 174.17 CA: 4.00 MCA: 4.00
  45. RANK:1 BEFORE MA: 6783.06 MMA: 6783.06 CA: 7460.00 MCA: 7460.00
  46. RANK:1 AFTER  MA: 6815.06 MMA: 6957.23 CA: 7464.00 MCA: 7464.00
  47. RANK:1 DIFF   MA: 32.00 MMA: 174.17 CA: 4.00 MCA: 4.00
  48. [2024-05-27 05:03:10,889] [INFO] [launch.py:349:main] Process 440888 exits successfully.
  49. RANK:2 BEFORE MA: 6783.06 MMA: 6783.06 CA: 7460.00 MCA: 7460.00
  50. RANK:2 AFTER  MA: 6815.06 MMA: 6957.23 CA: 7464.00 MCA: 7464.00
  51. RANK:2 DIFF   MA: 32.00 MMA: 174.17 CA: 4.00 MCA: 4.00
  52. [2024-05-27 05:03:11,891] [INFO] [launch.py:349:main] Process 440889 exits successfully.
  53. RANK:3 BEFORE MA: 6783.06 MMA: 6783.06 CA: 7460.00 MCA: 7460.00
  54. RANK:3 AFTER  MA: 6815.06 MMA: 6957.23 CA: 7464.00 MCA: 7464.00
  55. RANK:3 DIFF   MA: 32.00 MMA: 174.17 CA: 4.00 MCA: 4.00
  56. [2024-05-27 05:03:12,893] [INFO] [launch.py:349:main] Process 440890 exits successfully.
  57. [2024-05-27 05:03:13,895] [INFO] [launch.py:349:main] Process 440891 exits successfully.
复制代码
免责声明：如果侵犯了您的权益，请联系站长，我们会及时删除侵权内容，谢谢合作！更多信息从访问主页：qidao123.com:ToB企服之家，中国第一个企服评测及商务社交产业平台。
回复

使用道具 举报

0 个回复

倒序浏览
返回列表

快速回复

高级模式
B Color Image Link Quote Code Smilies
您需要登录后才可以回帖 登录 or 立即注册

本版积分规则

张裕

论坛元老
这个人很懒什么都没写!

楼主热帖

标签云

渠道 国产数据库 集成商 AI 运维 CIO 存储 服务器
微信订阅号
微信服务号
微信客服
小程序
H5
关于我们 商务合作 网站地图
©Copyright 2022-2024 杭州祥升科技有限公司 版权所有 浙ICP备20004199号
快速回复 返回顶部 返回列表