Code

Autotp training#6922

Code

Open

inkcherry wants to merge 74 commits intomicrosoft:mastermicrosoft/DeepSpeed:masterfrom inkcherry:autotp_traininginkcherry/DeepSpeed:autotp_trainingCopy head branch name to clipboard

+1,646-164

Commits

Commits on Apr 22, 2024

• auto tp training

  inkcherry

  committed
  674a873

  Copy full SHA for 674a873

Commits on Apr 23, 2024

• update parallel_states

  inkcherry

  committed
  a2e4c47

  Copy full SHA for a2e4c47

Commits on Nov 19, 2024

• Merge branch 'master' into HEAD

  inkcherry

  committed
  f4eb142

  Copy full SHA for f4eb142

• WA skips assertions, the loss remains exactly consistent with the low-precision version before the rebase, but the grad norm differs (display issue)

  inkcherry

  committed
  dd081ed

  Copy full SHA for dd081ed

Commits on Nov 22, 2024

• save/load ckpt & save/load hf model basic POC

  inkcherry

  committed
  cdaed2f

  Copy full SHA for cdaed2f

Commits on Nov 27, 2024

• finish all the basic functionalities

  inkcherry

  committed
  9aad0e7

  Copy full SHA for 9aad0e7

Commits on Nov 28, 2024

• update

  inkcherry

  committed
  2bb11fd

  Copy full SHA for 2bb11fd

Commits on Dec 2, 2024

• use groups for parallel_states

  inkcherry

  committed
  e75c1c2

  Copy full SHA for e75c1c2

• enable bwd allreduce, enable scale loss by gas

  inkcherry

  committed
  840a5f2

  Copy full SHA for 840a5f2

Commits on Dec 4, 2024

• add dataloader check

  inkcherry

  committed
  60bd6ab

  Copy full SHA for 60bd6ab

• refactor autoTP step1

  inkcherry

  committed
  9266383

  Copy full SHA for 9266383

Commits on Dec 5, 2024

• rm parallel_states

  inkcherry

  committed
  07174a9

  Copy full SHA for 07174a9

• refactor autoTP step2

  inkcherry

  committed
  ee6323e

  Copy full SHA for ee6323e

Commits on Dec 10, 2024

• update ut step1

  inkcherry

  committed
  6461b84

  Copy full SHA for 6461b84

Commits on Dec 11, 2024

• update

  inkcherry

  committed
  4d73011

  Copy full SHA for 4d73011

• add uts

  inkcherry

  committed
  c79c3bb

  Copy full SHA for c79c3bb

Commits on Dec 12, 2024

• finished all ut code base

  inkcherry

  committed
  97e659c

  Copy full SHA for 97e659c

• addllr scheduler test

  inkcherry

  committed
  a15905b

  Copy full SHA for a15905b

• refine ut

  inkcherry

  committed
  e9802b0

  Copy full SHA for e9802b0

Commits on Dec 15, 2024

• fix bcast_objlist

  inkcherry

  committed
  88b8acf

  Copy full SHA for 88b8acf

• refine layers.py

  inkcherry

  committed
  868be0b

  Copy full SHA for 868be0b

• refine gather

  inkcherry

  committed
  3788e07

  Copy full SHA for 3788e07

Commits on Dec 16, 2024

• pass codegen350M +TP2 ut

  inkcherry

  committed
  27b24f6

  Copy full SHA for 27b24f6

• add mode choice

  inkcherry

  committed
  3d7b89f

  Copy full SHA for 3d7b89f

• fix chatglm

  inkcherry

  committed
  47a6b0b

  Copy full SHA for 47a6b0b

• fix chatglm2 with transformers=4.40 version

  inkcherry

  committed
  3a23997

  Copy full SHA for 3a23997

• uneven

  inkcherry

  committed
  e3ec46e

  Copy full SHA for e3ec46e

• fix uneven

  inkcherry

  committed
  9685879

  Copy full SHA for 9685879

• fix training

  inkcherry

  committed
  7b99b03

  Copy full SHA for 7b99b03

Commits on Dec 17, 2024

• refine code

  inkcherry

  committed
  570645f

  Copy full SHA for 570645f

• remove skip bcase&reduce

  inkcherry

  committed
  3729b64

  Copy full SHA for 3729b64

• fix typo

  inkcherry

  committed
  62d8858

  Copy full SHA for 62d8858

• format

  inkcherry

  committed
  dd17313

  Copy full SHA for dd17313

Commits on Dec 18, 2024

• refine code

  inkcherry

  committed
  93cf6f5

  Copy full SHA for 93cf6f5

• refine code

  inkcherry

  committed
  87c4bc2

  Copy full SHA for 87c4bc2

• refine

  inkcherry

  committed
  1714bb5

  Copy full SHA for 1714bb5

Commits on Dec 19, 2024

• update yuan

  inkcherry

  committed
  dadf915

  Copy full SHA for dadf915

• optimize usage of move function

  inkcherry

  committed
  86c9399

  Copy full SHA for 86c9399

• refine args usage

  inkcherry

  committed
  2526dc6

  Copy full SHA for 2526dc6

• format

  inkcherry

  committed
  c9fd699

  Copy full SHA for c9fd699

• zero1 compatible

  inkcherry

  committed
  797e71f

  Copy full SHA for 797e71f

Commits on Dec 22, 2024

• remove wa

  inkcherry

  committed
  86ae65e

  Copy full SHA for 86ae65e

• fix cpu device name

  inkcherry

  committed
  3e40024

  Copy full SHA for 3e40024

Commits on Dec 23, 2024

• fix lm-head

  inkcherry

  committed
  7d94b77

  Copy full SHA for 7d94b77

• add detach

  inkcherry

  committed
  b297950

  Copy full SHA for b297950

• fix ipex intergration

  inkcherry

  committed
  67ce220

  Copy full SHA for 67ce220

Commits on Dec 25, 2024

• fix tied_embedding

  inkcherry

  committed
  f818be9

  Copy full SHA for f818be9

Commits on Jan 2, 2025

• Merge remote-tracking branch 'origin/master' into autotp_training

  inkcherry

  committed
  11c98f6

  Copy full SHA for 11c98f6

• format

  inkcherry

  committed
  e22b625

  Copy full SHA for e22b625

Commits on Jan 6, 2025

• Merge branch 'master' into autotp_training

  tjruwase

  authored
  8531b64

  Copy full SHA for 8531b64

• Merge branch 'master' into autotp_training

  loadams

  authored
  8d19e01

  Copy full SHA for 8d19e01

Commits on Jan 13, 2025

• remove outdated comments

  inkcherry

  committed
  060d48b

  Copy full SHA for 060d48b

• Enhance unit test coverage

  inkcherry

  committed
  6667ba1

  Copy full SHA for 6667ba1

• update ut

  inkcherry

  committed
  84c9335

  Copy full SHA for 84c9335

• sequential some tests

  inkcherry

  committed
  cb29d7c

  Copy full SHA for cb29d7c

• format

  inkcherry

  committed
  a49e77e

  Copy full SHA for a49e77e

• use parameterized save path

  inkcherry

  committed
  0ef5274

  Copy full SHA for 0ef5274

• Merge remote-tracking branch 'my/autotp_training' into autotp_training

  inkcherry

  committed
  481088d

  Copy full SHA for 481088d

Commits on Jan 15, 2025

• refactor infer/training path

  inkcherry

  committed
  f740de0

  Copy full SHA for f740de0

• format

  inkcherry

  committed
  726004d

  Copy full SHA for 726004d

• remove empty line

  inkcherry

  committed
  bd8de77

  Copy full SHA for bd8de77

• remove autotp_size config from zero scope

  inkcherry

  committed
  c334da0

  Copy full SHA for c334da0

• update

  inkcherry

  committed
  29eef07

  Copy full SHA for 29eef07

• format

  inkcherry

  committed
  ba47ed1

  Copy full SHA for ba47ed1

• fix layer typo and rename

  inkcherry

  committed
  bbde63f

  Copy full SHA for bbde63f

• fix python3.9

  inkcherry

  committed
  bdca62c

  Copy full SHA for bdca62c

• refine code

  inkcherry

  committed
  5d89422

  Copy full SHA for 5d89422

• refine

  inkcherry

  committed
  0a9caff

  Copy full SHA for 0a9caff

Commits on Jan 16, 2025

• refine config

  inkcherry

  committed
  c923a3b

  Copy full SHA for c923a3b

Commits on Jan 17, 2025

• improve ut coverage for save

  inkcherry

  committed
  92be193

  Copy full SHA for 92be193

• fix process exit early

  inkcherry

  committed
  23bd0fc

  Copy full SHA for 23bd0fc

• improve ut coverage

  inkcherry

  committed
  358f395

  Copy full SHA for 358f395

• Merge remote-tracking branch 'origin/master' into autotp_training

  inkcherry

  committed
  cdfb54c

  Copy full SHA for cdfb54c

• fix zero1 regression

  inkcherry

  committed
  6d030c4

  Copy full SHA for 6d030c4