PyTorch MNIST Dataset

1. MNIST Dataset

https://pytorch.org/vision/main/generated/torchvision.datasets.MNIST.html

1. torchvision.datasets.MNIST(root: Union[str, Path], train: bool = True, transform: Optional[Callable] = None, target_transform: Optional[Callable] = None, download: bool = False)

复制代码

Parameters:

• root (str or pathlib.Path) - Root directory of dataset where MNIST/raw/train-images-idx3-ubyte and MNIST/raw/t10k-images-idx3-ubyte exist.
  
• train (bool, optional) - If True, creates dataset from train-images-idx3-ubyte, otherwise from t10k-images-idx3-ubyte.
  
• download (bool, optional) - If True, downloads the dataset from the internet and puts it in root directory. If dataset is already downloaded, it is not downloaded again.
  
• transform (callable, optional) - A function/transform that takes in a PIL image and returns a transformed version. E.g, transforms.RandomCrop
  
• target_transform (callable, optional) - A function/transform that takes in the target and transforms it.
  

Special-members:

1. __getitem__(index: int) -> Tuple[Any, Any]

复制代码

• Parameters: index (int)
  
• Returns: (image, target) where target is index of the target class.
  
• Return type: tuple
  

2. Source code for torchvision.datasets.mnist

https://pytorch.org/vision/main/_modules/torchvision/datasets/mnist.html

1.     mirrors = [
2.         "http://yann.lecun.com/exdb/mnist/",
3.         "https://ossci-datasets.s3.amazonaws.com/mnist/",
4.     ]
6.     resources = [
7.         ("train-images-idx3-ubyte.gz", "f68b3c2dcbeaaa9fbdd348bbdeb94873"),
8.         ("train-labels-idx1-ubyte.gz", "d53e105ee54ea40749a09fcbcd1e9432"),
9.         ("t10k-images-idx3-ubyte.gz", "9fb629c4189551a2d022fa330f9573f3"),
10.         ("t10k-labels-idx1-ubyte.gz", "ec29112dd5afa0611ce80d1b7f02629c"),
11.     ]
13.     training_file = "training.pt"
14.     test_file = "test.pt"
15.     classes = [
16.         "0 - zero",
17.         "1 - one",
18.         "2 - two",
19.         "3 - three",
20.         "4 - four",
21.         "5 - five",
22.         "6 - six",
23.         "7 - seven",
24.         "8 - eight",
25.         "9 - nine",
26.     ]

复制代码

可以通过下面的链接在欣赏器中下载，复制到 data/MNIST/raw/ 目次下。

1. https://ossci-datasets.s3.amazonaws.com/mnist/train-images-idx3-ubyte.gz
2. https://ossci-datasets.s3.amazonaws.com/mnist/train-labels-idx1-ubyte.gz
3. https://ossci-datasets.s3.amazonaws.com/mnist/t10k-images-idx3-ubyte.gz
4. https://ossci-datasets.s3.amazonaws.com/mnist/t10k-labels-idx1-ubyte.gz

复制代码

2.1. mnist-dataset.py

/home/yongqiang/llm_work/ggml_25_02_15/ggml/examples/mnist/mnist-dataset.py

1. import torch
2. import torchvision
3. import torchvision.datasets
4. import torchvision.transforms
5. import numpy as np
6. import matplotlib.pyplot as plt
8. print(torch.__version__)
10. train_data = torchvision.datasets.MNIST(root='./data', train=True, transform=torchvision.transforms.ToTensor(), download=True)
11. test_data  = torchvision.datasets.MNIST(root='./data', train=False, transform=torchvision.transforms.ToTensor(), download=True)
13. assert len(train_data) == 60000
14. assert len(test_data)  == 10000
15. print("len(train_data) =", len(train_data))
16. print("len(test_data) =", len(test_data))
18. print("type(train_data[0]):", type(train_data[0]))
20. print("train_data[0].shape:", train_data[0][0].shape)
22. classes = train_data.classes
23. print("train_data.classes:", classes)
24. print("train_data.class_to_idx: ", train_data.class_to_idx)
26. def ImShow(sample_element, shape = (28, 28)):
27.     plt.imshow(sample_element[0].numpy().reshape(shape), cmap='gray')
28.     plt.title('Label = ' + str(sample_element[1]))
29.     plt.show()
31. ImShow(train_data[0])

复制代码

1. (base) yongqiang@yongqiang:~/llm_work/ggml_25_02_15/ggml/examples/mnist$ python mnist-dataset.py
2. 2.5.1
3. Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz
4. Failed to download (trying next):
5. HTTP Error 404: Not Found
7. Downloading https://ossci-datasets.s3.amazonaws.com/mnist/train-images-idx3-ubyte.gz
8. Downloading https://ossci-datasets.s3.amazonaws.com/mnist/train-images-idx3-ubyte.gz to ./data/MNIST/raw/train-images-idx3-ubyte.gz
9. 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████| 9.91M/9.91M [00:03<00:00, 3.20MB/s]
10. Extracting ./data/MNIST/raw/train-images-idx3-ubyte.gz to ./data/MNIST/raw
12. Downloading http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz
13. Failed to download (trying next):
14. HTTP Error 404: Not Found
16. Downloading https://ossci-datasets.s3.amazonaws.com/mnist/train-labels-idx1-ubyte.gz
17. Downloading https://ossci-datasets.s3.amazonaws.com/mnist/train-labels-idx1-ubyte.gz to ./data/MNIST/raw/train-labels-idx1-ubyte.gz
18. 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████| 28.9k/28.9k [00:00<00:00, 117kB/s]
19. Extracting ./data/MNIST/raw/train-labels-idx1-ubyte.gz to ./data/MNIST/raw
21. Downloading http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz
22. Failed to download (trying next):
23. HTTP Error 404: Not Found
25. Downloading https://ossci-datasets.s3.amazonaws.com/mnist/t10k-images-idx3-ubyte.gz
26. Downloading https://ossci-datasets.s3.amazonaws.com/mnist/t10k-images-idx3-ubyte.gz to ./data/MNIST/raw/t10k-images-idx3-ubyte.gz
27. 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████| 1.65M/1.65M [00:02<00:00, 656kB/s]
28. Extracting ./data/MNIST/raw/t10k-images-idx3-ubyte.gz to ./data/MNIST/raw
30. Downloading http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz
31. Failed to download (trying next):
32. HTTP Error 404: Not Found
34. Downloading https://ossci-datasets.s3.amazonaws.com/mnist/t10k-labels-idx1-ubyte.gz
35. Downloading https://ossci-datasets.s3.amazonaws.com/mnist/t10k-labels-idx1-ubyte.gz to ./data/MNIST/raw/t10k-labels-idx1-ubyte.gz
36. 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████| 4.54k/4.54k [00:00<00:00, 3.05MB/s]
37. Extracting ./data/MNIST/raw/t10k-labels-idx1-ubyte.gz to ./data/MNIST/raw
39. len(train_data) = 60000
40. len(test_data) = 10000
41. type(train_data[0]): <class 'tuple'>
42. train_data[0].shape: torch.Size([1, 28, 28])
43. train_data.classes: ['0 - zero', '1 - one', '2 - two', '3 - three', '4 - four', '5 - five', '6 - six', '7 - seven', '8 - eight', '9 - nine']
44. train_data.class_to_idx:  {'0 - zero': 0, '1 - one': 1, '2 - two': 2, '3 - three': 3, '4 - four': 4, '5 - five': 5, '6 - six': 6, '7 - seven': 7, '8 - eight': 8, '9 - nine': 9}
45. (base) yongqiang@yongqiang:~/llm_work/ggml_25_02_15/ggml/examples/mnist$

复制代码

3. SSL: CERTIFICATE_VERIFY_FAILED

1. <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)>
3. <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1129)>
5. <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)>
7. <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:833)>
9. <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)>

复制代码

取消证书验证：

1. import ssl
2. ssl._create_default_https_context = ssl._create_unverified_context

复制代码

References

[1] Yongqiang Cheng, https://yongqiang.blog.csdn.net/

免责声明：如果侵犯了您的权益，请联系站长，我们会及时删除侵权内容，谢谢合作！更多信息从访问主页：qidao123.com:ToB企服之家，中国第一个企服评测及商务社交产业平台。