请教，为什么PipelineP2PCommunication不使用send/recv API #4924
Replies: 3 comments 5 replies
-
|
我也发现了，我测下来1F1B的GPU利用率不高，经常会低于50%。用send/recv实现的Megatron和Deepspeed则会一直维持在99%以上 |
Beta Was this translation helpful? Give feedback.
-
|
broadcast_object会先将数据做序列化，再通过broadcast的方式传递数据，好处可能是可以传递除了tensor之外的其他anyting 数据，但是多了很多冗余操作，而且对于PP来说似乎也是用不上的，所以不清楚为啥colossal-ai要做这样的设计 |
Beta Was this translation helpful? Give feedback.
-
|
这是deepspeed早期的实现方式，我猜是直接拿来用了。不过deepspeed现在也改成send/recv了 |
Beta Was this translation helpful? Give feedback.
-
|
legacy下面好像有一点send/recv的影子，不知道这些代码是后面要用的还是啥 |
Beta Was this translation helpful? Give feedback.
-
|
我把P2P代码改成send/recv后，速度提升了20% |
Beta Was this translation helpful? Give feedback.
-
|
legacy中的内容为旧版本代码，保留了之前的一些特性。关于P2P的问题，我们会进一步做profile |
Beta Was this translation helpful? Give feedback.
-
|
Hi @lichenlu and @imgaojun Thanks for the feedback, we are testing this. You are welcome to contribute by submitting a PR. |
Beta Was this translation helpful? Give feedback.
-
|
We are happy to announce that the latest code in the repo has supported send/recv API. Thanks to all the participants in this conversation as well as the contributors. 😃 |
Beta Was this translation helpful? Give feedback.
-
请教，为什么PipelineP2PCommunication里的send_forward/recv_forward等用的是broadcast_object 而不是 send/recv API
Beta Was this translation helpful? Give feedback.
All reactions