How to do data preprocessing with PyTorch on CentOS

Efficiently processing PyTorch data on CentOS system requires the following steps:

1. Dependency installation: First update the system and install Python 3 and pip:

    sudo yum update -y
   sudo yum install python3 -y
   sudo yum install python3-pip -y

   Then, download and install CUDA Toolkit and cuDNN from the official NVIDIA website according to your CentOS version and GPU model.

2. Virtual Environment Configuration (recommended): Use conda to create and activate a new virtual environment, for example:

    conda create -n pytorch python=3.8
   conda activated pytorch

3. PyTorch installation: In the activated virtual environment, use conda or pip to install PyTorch. The version that supports CUDA is as follows:

    conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch # Adjust cudatoolkit version number to match your CUDA version

   Or use pip (you may need to specify the CUDA version):

    pip install torch torchvision torchaudio

4. Data preprocessing and enhancement: Use the torchvision.transforms module for data preprocessing and enhancement. The following examples show image resizing, random horizontal flip, conversion to tensor, and normalization:

    import torch
   import torchvision
   from torchvision import transforms
   
   transform = transforms.Compose([
       transforms.Resize((224, 224)),
       transforms.RandomHorizontalFlip(),
       transforms.ToTensor(),
       transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
   ])
   
   dataset = torchvision.datasets.ImageFolder(root='path/to/data', transform=transform)
   dataloader = torch.utils.data.DataLoader(dataset, batch_size=32, shuffle=True)

5. Custom dataset: For custom datasets, inherit the torch.utils.data.Dataset class and implement __getitem__ and __len__ methods. For example:

    import os
   from PIL import Image
   from torch.utils.data import Dataset
   
   class MyDataset(Dataset):
       def __init__(self, root_path, labels):
           self.root_path = root_path
           self.labels = labels # list of labels for corresponding image self.image_files = [f for f in os.listdir(root_path) if f.endswith(('.jpg', '.png'))] # Assume that the image is in jpg or png format def __getitem__(self, index):
           img_path = os.path.join(self.root_path, self.image_files[index])
           img = Image.open(img_path)
           label = self.labels[index]
           return img, label
   
       def __len__(self):
           return len(self.image_files)

6. Data loading: Use torch.utils.data.DataLoader to load and batch data:

    from torch.utils.data import DataLoader
   
   my_dataset = MyDataset('path/to/your/data', [0,1,0,1, ...]) # Replace 'path/to/your/data' and tag list data_loader = DataLoader(dataset=my_dataset, batch_size=64, shuffle=True, num_workers=0) # num_workers Adjust based on your CPU core number

   Remember to replace the placeholder path and label with your actual data. The num_workers parameter can be adjusted according to the number of CPU cores to improve data loading speed.

Through the above steps, you can complete the data preprocessing of PyTorch on CentOS. If you have any questions, please refer to the official PyTorch documentation or seek community support.

The above is the detailed content of How to do data preprocessing with PyTorch on CentOS. For more information, please follow other related articles on the PHP Chinese website!