Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[VSphere] 64bitMMIOSizeGB should be configurable #48269

Open
juulie opened this issue Oct 25, 2024 · 3 comments · May be fixed by #48740
Open

[VSphere] 64bitMMIOSizeGB should be configurable #48269

juulie opened this issue Oct 25, 2024 · 3 comments · May be fixed by #48740
Labels
bug Something that is supposed to be working; but isn't core Issues that should be addressed in Ray Core core-clusters For launching and managing Ray clusters/jobs/kubernetes P2 Important issue, but not time-critical

Comments

@juulie
Copy link

juulie commented Oct 25, 2024

What happened + What you expected to happen

Hi all,

ray/python/ray/autoscaler/_private/vsphere/gpu_utils.py

Line 272 in fc34890

vim.option.OptionValue(key="pciPassthru.64bitMMIOSizeGB", value="64"),

Sets the 64bitMMIOSizeGB paramater to 64, but this is depended on the GPU being passed through:
https://docs.nvidia.com/ai-enterprise/latest/release-notes/index.html#tesla-p40-large-memory-vms
I think we should add a configuration for this in the vsphere gpu_config

Versions / Dependencies

Ray-2.38.0

Reproduction script

Launch cluster on vsphere with a GPU requiring more than 64gb MMIO space

Issue Severity

High: It blocks me from completing my task.
@juulie juulie added bug Something that is supposed to be working; but isn't triage Needs triage (eg: priority, bug/not-bug, and owning component) labels Oct 25, 2024
@jcotant1 jcotant1 added core Issues that should be addressed in Ray Core core-clusters For launching and managing Ray clusters/jobs/kubernetes labels Oct 25, 2024
@jcotant1
Copy link
Collaborator

jcotant1 commented Nov 4, 2024

Hey @kevin85421 @jjyao - any update on this one? Thx

@jcotant1 jcotant1 added core Issues that should be addressed in Ray Core and removed core Issues that should be addressed in Ray Core labels Nov 8, 2024
@rynewang
Copy link
Contributor

@JingChen23 can you take a look at this?

@rynewang rynewang added P2 Important issue, but not time-critical and removed triage Needs triage (eg: priority, bug/not-bug, and owning component) labels Nov 12, 2024
@juulie
Copy link
Author

juulie commented Nov 14, 2024

I have a fix for this, just need to do a pull request (which ive never done before on github, so might take a bit to get right)

@juulie juulie linked a pull request Nov 14, 2024 that will close this issue
Open
7 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something that is supposed to be working; but isn't core Issues that should be addressed in Ray Core core-clusters For launching and managing Ray clusters/jobs/kubernetes P2 Important issue, but not time-critical
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Added PCI MMIO size configuration, needed for when GPU PCI passthroug… juulie/ray
3 participants
@juulie @rynewang @jcotant1