Python 动态网页Fetch/XHR爬虫——以获取NBA球员信息为例

Python 动态网页Fetch/XHR爬虫——以获取NBA球员信息为例

动态网页抓取信息，一般利用F12开发者工具-网络-Fetch/XHR获取信息，实现难点有：

• 
  
  • 动态网页的加载方式
    
  • 获取请求Url
    
  • 编排处理Headers
    
  • 分析返回的数据Json
    
  • pandas DataFrame的处理
    
  
  

 
我们本次想获取的信息如下：
 
成功获取到的csv一共506位球员，具体如下：

 
实现代码：

1.  import requests
2.  import pandas as pd
3.  ​
4.  def get_headers(header_raw):
5.     return dict(line.split(": ", 1) for line in header_raw.split("\n") if line != '')
6.  ​
7.  # 设置headers
8.  headers_str = '''
9.  accept: application/json, text/plain, */*
10.  accept-encoding: gzip, deflate, br
11.  accept-language: zh-CN,zh;q=0.9
12.  referer: https://china.nba.cn/playerindex/
13.  sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="96", "Google Chrome";v="96"
14.  sec-ch-ua-mobile: ?0
15.  sec-ch-ua-platform: "Windows"
16.  sec-fetch-dest: empty
17.  sec-fetch-mode: cors
18.  sec-fetch-site: same-origin
19.  cookie: sensorsdata2015jssdkcross=%7B%22distinct_id%22%3A%22182d0029f842fc-0d281a685dd4e08-4303066-2400692-182d0029f85406%22%2C%22first_id%22%3A%22%22%2C%22props%22%3A%7B%22%24latest_traffic_source_type%22%3A%22%E7%9B%B4%E6%8E%A5%E6%B5%81%E9%87%8F%22%2C%22%24latest_search_keyword%22%3A%22%E6%9C%AA%E5%8F%96%E5%88%B0%E5%80%BC_%E7%9B%B4%E6%8E%A5%E6%89%93%E5%BC%80%22%2C%22%24latest_referrer%22%3A%22%22%7D%2C%22identities%22%3A%22eyIkaWRlbnRpdHlfY29va2llX2lkIjoiMTgyZDAwMjlmODQyZmMtMGQyODFhNjg1ZGQ0ZTA4LTQzMDMwNjYtMjQwMDY5Mi0xODJkMDAyOWY4NTQwNiJ9%22%2C%22history_login_id%22%3A%7B%22name%22%3A%22%22%2C%22value%22%3A%22%22%7D%2C%22%24device_id%22%3A%22182d0029f842fc-0d281a685dd4e08-4303066-2400692-182d0029f85406%22%7D; privacyV2=true; i18next=zh_CN; locale=zh_CN
20.  user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36
21.  '''
22.  headers = get_headers(headers_str)
23.  # print(headers)
24.  ​
25.  # requests请求
26.  param = {'locale': 'zh_CN'}
27.  url = 'https://china.nba.cn/stats2/league/playerlist.json'
28.  response = requests.get(url=url, headers=headers, params=param)
29.  ​
30.  print('返回状态码：', response.status_code)
31.  print('编码：', response.encoding)
32.  ​
33.  # json解码成字典
34.  myjson = response.json()
35.  ​
36.  # 保存为pandas DataFrame
37.  # print(players_dicts['playerProfile'])
38.  # print(players_dicts['teamProfile'])
39.  ​
40.  # 遍历选手信息
41.  players_info = []
42.  for players_dicts in myjson['payload']['players']:
43.      players_info.append(pd.DataFrame([players_dicts['playerProfile']]))
44.  ​
45.  # 遍历队伍简介信息
46.  teams_info = []
47.  for players_dicts in myjson['payload']['players']:
48.      teams_info.append(pd.DataFrame([players_dicts['teamProfile']]))
49.  ​
50.  # 得到两个DataFrame
51.  players_pandas = pd.concat(players_info)
52.  teams_pandas = pd.concat(teams_info)
53.  ​
54.  # 合并得到最终DataFrame
55.  result = pd.concat([players_pandas, teams_pandas], axis=1)
56.  result.to_csv(r'C:\Users\WeiRonbbin\Desktop\NBA_Players1.csv')

复制代码

免责声明：如果侵犯了您的权益，请联系站长，我们会及时删除侵权内容，谢谢合作！