PyTorch

Introduction

SORT

basic creation and training of deep nn. section on convolution. section on transformers. also for keras and tensorflow. and jax.

From NumPy to PyTorch

Basic functions are the same.

import numpy as np
import torch
a_np = np.array([[0,1],[2,3]])
b_np = np.array([[4,5],[6,7]])
a_torch = torch.tensor([[0,1],[2,3]])
b_torch = torch.tensor([[4,5],[6,7]])

a+b
a-b
a*b
a/b
a@b

Tensors can also be defined using NumPy arrays

tensor = torch.from_numpy(a_np)

Where @ is elementwise multiplication and @ is matrix multiplication.

As with NumPy we can define arrays like this.

torch.eye(n)
torch.ones(i, j, k, ...)
torch.full(i, j, k, ...)
torch.rand(i, j, k, ...)
torch.zeroes(i, j, k, ..)

We can also define arrays of specific dtypes, as in NumPy.

a_torch = torch.tensor([[0,1],[2,3]], dtype =torch.float32)

Attaching CUDA cores to PyTorch tensors

If tensors are on CUDA cores then the CUDA cores will be used rather than the CPU.

Can check if CUDA is available:

torch.cuda.is_available()

Can move a tensor to the GPU.

tensor_cpu = torch.randn(3,3)
tensor_gpu = tensor_cpu.cuda()
# or tensor_gpu = tensor.cpu.to("cuda")
tensor_back_to_cpu = tensor_gpu.cpu()
# or tensor_back_to_cpu = tensor_gpu.to("cpu")

The .cuda() function can take an integer if there are multiple GPUs available.

Whole models can also be moved to the GPU. model.cuda()

We can be dynamic

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
tensor = torch.randn(3, 3).to(device)  # This will be on GPU if CUDA is available, otherwise CPU
model = SomeNeuralNetwork().to(device)  # Move model to the same device

Neural networks in PyTorch

Defining feedforward networks

nn.Module

nn.sequential

nn.Linear

nn.ReLU

nn.Softmax

Backpropagation

Data

torch.utils.data.DataLoader

torch.utils.data.Dataset

torchvision (images)

torchtext

datasets available

torchaudio

PyTorch and autograd

JIT compiling on PyTorch

XLA and TPUs with PyTorch

fast.ai

fast.ai is a thing that sits on top of pytorch