Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pytorch parallelization #5915

Open
wants to merge 25 commits into
base: main
Choose a base branch
from
Open
Changes from 17 commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
6744158
.md file and updated title
parkersarahl Jan 1, 2025
3818880
updated title
parkersarahl Jan 1, 2025
1b1cd4f
changed to correct template
parkersarahl Jan 1, 2025
7f1b7a0
added description
parkersarahl Jan 2, 2025
791f4a0
first section added
parkersarahl Jan 2, 2025
412ab91
added import section
parkersarahl Jan 2, 2025
c33a971
Merge branch 'Codecademy:main' into pytorch_parallelization
parkersarahl Jan 2, 2025
723b593
added example
parkersarahl Jan 2, 2025
29720aa
Merge branch 'pytorch_parallelization' of https://github.com/parkersa…
parkersarahl Jan 2, 2025
50a0aa6
added syntax
parkersarahl Jan 3, 2025
f655f99
added output section
parkersarahl Jan 3, 2025
3cc574b
added CatalogContent
parkersarahl Jan 3, 2025
dc18047
changed syntax to psuedo shell and output to shell
parkersarahl Jan 3, 2025
7413f97
changed subjects to add data science
parkersarahl Jan 3, 2025
3ca3ccc
updated the description
parkersarahl Jan 3, 2025
fff2b66
Merge branch 'main' into pytorch_parallelization
parkersarahl Jan 6, 2025
8342c28
Merge branch 'main' into pytorch_parallelization
parkersarahl Jan 10, 2025
534210c
Merge branch 'main' into pytorch_parallelization
parkersarahl Jan 14, 2025
8db9043
updated the description per PR recommendation.
parkersarahl Jan 14, 2025
911a6e6
changed the wording under the model parallelization paragraph
parkersarahl Jan 14, 2025
a6cb305
Merge branch 'Codecademy:main' into pytorch_parallelization
parkersarahl Jan 14, 2025
e12ae9a
removed the section Setting Up the Environment
parkersarahl Jan 14, 2025
c039a97
changed the location of model defintion in example
parkersarahl Jan 14, 2025
e83e5c3
Update content/pytorch/concepts/parallelizing-models/parallelizing-mo…
parkersarahl Jan 14, 2025
30c8b2d
changed the code example per recommendations in PR
parkersarahl Jan 14, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
---
Title: 'parallelizing-models'
Description: 'Model parallelization is used to train models that require more memory than what is available on a single GPU.'
Subjects:
- 'Computer Science'
- 'Machine Learning'
- 'Data Science'
Tags:
- 'Algorithms'
- 'PyTorch'
- 'Machine Learning'
CatalogContent:
- 'intro-to-py-torch-and-neural-networks'
- 'paths/build-a-machine-learning-model'
---
parkersarahl marked this conversation as resolved.
Show resolved Hide resolved

**Model parallelization** trains deep learning models that require more memory than what is available on a single graphic processing unit (GPU). It is used within the pyTorch library. The model is separated into parts (i.e. layers or modules) and assigned to different GPUs. The GPUs perform the computations simultaneously, allowing for faster processing of large models. The GPUs communicate with each other and share the data to ensure the data output from one GPU is used in another GPU as needed.
parkersarahl marked this conversation as resolved.
Show resolved Hide resolved

## Setting up the environment
```pseudo
import torch
import torch.nn as nn
```

parkersarahl marked this conversation as resolved.
Show resolved Hide resolved
## Syntax
To utilize model parallelization the model should be wrapped using the following syntax:

```shell
class ModelParallel(nn.Module)
parkersarahl marked this conversation as resolved.
Show resolved Hide resolved
```

## Example
The layers or modules should then be assigned to a specified GPU. The following example demonstrates how this can be accomplished.

```py
# Define a model split across two GPUs
class ModelParallel(nn.Module):
def __init__(self):
super(ModelParallel, self).__init__()
self.layer1 = nn.Linear(1000, 500).to('cuda:0') # First GPU
self.layer2 = nn.Linear(500, 100).to('cuda:1') # Second GPU

def forward(self, x):
x = x.to('cuda:0') # Input to first GPU
x = self.layer1(x)
x = x.to('cuda:1') # Output of first layer to second GPU
x = self.layer2(x)
return x

model = ModelParallel()
x = torch.randn(64, 1000)
output = model(x)
parkersarahl marked this conversation as resolved.
Show resolved Hide resolved
```
## Output
parkersarahl marked this conversation as resolved.
Show resolved Hide resolved

The output of the above code would result in a tensor. The exact values would depend on the initialization of the model weights and the input data, but could be expected to look similar to the following output:
```shell
tensor([[ 0.1324, -0.2847, ..., 0.5921], # First sample in the batch
[-0.0412, 0.4891, ..., -0.2345], # Second sample in the batch
...
[ 0.2347, -0.1011, ..., 0.4567]]) # 64 rows, each with 100 values
```