ArcVRArthur

6 ms·

1.
Lessons learned scaling LLM training and inference with RDMA (2024)(vgpu.io)
1 points by ArcVRArthur 2y ago|0 comments
2.
by ArcVRArthur 2y ago
Hey YC, I helped Cohere scale GPUs to a 10k+ total GPU count for hybrid training and inference - if you’re curious about anything I wrote in the article I think there’s a good opportunity to read / reply to your comment. This article i
3.
by ArcVRArthur 2y ago
Hey YC, I helped Cohere scale GPUs to a 10k+ total GPU count for hybrid training/inference computer - if you’re curious about anything I wrote in the article I think there’s a good opportunity to read / reply to your comment here.
4.
by ArcVRArthur 2y ago
I had the opportunity to help Cohere work on scaling transformers in the last year. If you have any questions about the article leave them below and I’ll do my best to answer openly. :)
5.
Scaling Transformers at Cohere: What I Learned(vgpu.io)
1 points by ArcVRArthur 2y ago|1 comments
6.
by ArcVRArthur 2y ago
I had the opportunity to help Cohere work on scaling transformers in the last year. If you have any questions about the article leave them below and I’ll do my best to answer openly. :)
7.
by ArcVRArthur 2y ago
I had the opportunity to help Cohere work on scaling transformers in the last year. If you have any questions about the article leave them below and I’ll do my best to answer openly. :)
8.
by ArcVRArthur 4y ago
Hey all, I'm the the co-author of the GPU Virtual Machine (GVM project), and LibVF.IO. We just announced our enterprise product based on GVM called GVM Server. I'd love to hear what you all think of the work we've done and gi
9.
GVM Server: A complete virtualization solution based on GPU Virtual Machine(GVM)(youtube.com)
1 points by ArcVRArthur 4y ago|1 comments
10.
GPU Virtual Machine (GVM) at QubesOS Summit(youtube.com)
18 points by ArcVRArthur 4y ago|2 comments
11.
by ArcVRArthur 4y ago
I was thinking this over in the past couple days and I think the words 'that they are aware' is really key here. Ideally if GPU virtualization were sufficiently widespread as is support today for Intel VT-d, and AMD-v (IOMMU APIs
12.
by ArcVRArthur 4y ago
Thanks!! We do our best to keep the code as clean/readable as possible. The first version was a bit of a mess but we rewrote it again clean slate to improve over the first implementation. :)
13.
by ArcVRArthur 4y ago
That depends on your use-case. In general I would recommend you consider purchasing Nvidia's GPUs for the best price/performance and GVM support. Intel's Xe architecture is currently improving but the performance isn't q
14.
by ArcVRArthur 4y ago
We have Ampere support working on some devices (and it is in development on other Ampere devices) as well as 11th & 12th generation Intel Xe. :) Here's the GPU Support page if you'd like to take a look: https://open
15.
by ArcVRArthur 4y ago
Ya, that's accurate. The precise driver implementation matters a lot. Having said that there are some good 'best practices' that seem to make a difference. In my opinion 'IOMMU Aware Mediated Device' could also make
16.
by ArcVRArthur 4y ago
GVM uses IOMMU for compartmentalization: This page has a comparison of the various IO assistance modes GVM can make use of (see comparison of assistance modes, the Mdev Mode section, and the SR-IOV Mode section): https://openmdev
17.
by ArcVRArthur 4y ago
Ya! You can use VMs that use X11 in the guest without issue. X11 also works on the host. Wayland is also working on the host - I haven't tested yet with Wayland guests yet so that's something to try.
18.
by ArcVRArthur 4y ago
Ya, LibVF.IO & GVM are built for things like this! For example I have a friend who uses it for various Adobe programs which also don't work well on Linux.
19.
by ArcVRArthur 4y ago
For sure! They can reach me at arthur@arccompute.io I'll also be attending KVM Forum this year so I'd love to chat with folks there as well! :)
20.
by ArcVRArthur 4y ago
Thanks!! We'll do our best to keep improving things for everyone. Hopefully security by compartmentalization folks benefit from our work as well. I'll be going to QubesOS Summit so hopefully there will be more good conversations t
21.
by ArcVRArthur 4y ago
If they would consider helping with the things I mention here we'd do our best to support them! https://news.ycombinator.com/item?id=32585950
22.
by ArcVRArthur 4y ago
Unfortunately there is limited support on AMD. I would like to try to make it better though! https://openmdev.io/index.php/GPU_Support
23.
by ArcVRArthur 4y ago
I hope I can work with them some time to improve support! I outlined a few things they would need to do to help support GVM using the amdgpu driver on this page: https://openmdev.io/index.php/AMDGPU Ideally some folks
24.
by ArcVRArthur 4y ago
There are also some benefits in load balancing GPUs. For instance processes can be controlled with controlled groups (cgroups) and niceness/thread affinity to make sure scheduling / resource allocation is done equitably between pr
25.
by ArcVRArthur 4y ago
For my use I tend to run a high performing Windows VM (very good with most games and other GPU accelerated programs - even chrome/adobe these days uses the GPU) inside Linux. This can also be used for various high assurance use cases s
26.
by ArcVRArthur 4y ago
It is slightly different - this runs an unmodified GPU driver in the guest (no para-virtualization) and in our observations our performance is close to native which I believe is a fair improvement over the overhead of Virgl. Also there&#x27
27.
by ArcVRArthur 4y ago
Ya, you can use multiple VMs. Here is a video I took of a laptop with LibVF.IO and an early pre-release version of GVM installed - at the start of the video you can see a fullscreen Windows VM and then I back out and open another compartmen
28.
by ArcVRArthur 4y ago
That depends on the driver implementation and what exactly the features are that you are looking for. In general if you're using virtualization tools like KVM/Xen you can configure a computer to make virtual machines that can be a
29.
by ArcVRArthur 4y ago
I would love to work with folks like the people at RedHat! We're working on some new tools in C (GPLv2) which we think will make it easier to work with existing kernel/driver maintainers. We'll announce some of those things e
30.
by ArcVRArthur 4y ago
According to https://docs.linux-gvm.org/ "GVM ... may be used in combination with KVM or other platform hypervisors such as Xen* to provide a complete virtualization solution via both central processing (CPU) and graph

More ›