8 ms·

So, this is my project! Was somewhat hoping to wait until there was a bit more content up on the site before it started doing the rounds, but here we are! :) T

So, this is my project! Was somewhat hoping to wait until there was a bit more content up on the site before it started doing the rounds, but here we are! :) To answer what seems to be the most common question I get asked about this, I am intending on open-sourcing the entire stack (PCB schematic/layout, all the HDL, Windows WDDM drivers, API runtime drivers, and Quake ported to use the API) at some point, but there are a number of legal issues that need to be cleared (with respect to my job) and I need to decide the rest of the particulars (license, etc.) - this stuff is not what I do for a living, but it's tangentially-related enough that I need to cover my ass. The first commit for this project was on August 22, 2021. It's been a bit over two and a half years I've been working on this, and while I didn't write anything up during that process, there are a fair number of videos in my YouTube FuryGpu playlist (https://www.youtube.com/playlist?list=PL4FPA1MeZF440A9CFfMJ7F6YoZ1Jvxu5j https://www.youtube.com/playlist?list=PL4FPA1MeZF440A9CFfMJ7...) that can kind of give you an idea of how things progressed. The next set of blog posts that are in the works concern the PCIe interface. It'll probably be a multi-part series starting at the PCB schematic/layout and moving through the FPGA design and ending with the Windows drivers. No timeline on when that'll be done, though. After having written just that post on how the Texture Units work, I've got even more respect for those that can write up technical stuff like that with any sort of timing consistency. I'll answer the remaining questions in the threads where they were asked. Thanks for the interest!

rustybolt 2y ago
I have seen semi-regular updates from you on discord and it is awesome to see how far this project has come (and also a bit frustrating to see how relatively little progress I have made on my FPGA projects in the same time!). I was hoping you'd do a writeup, can't wait!
michaelt 2y ago
Googling the Xilinx Zynq UltraScale+ it seems kinda expensive. Of course plenty of hobbies let people spend thousands (or more) so there's nothing wrong with that if you've got the money. But is it the end target for your project? Or do you have ambitions to go beyond that?
- kanetw 2y ago
  The Kria SOM in use here is like $300.
- PfhorSlayer 2y ago
  Let's be clear here, this is a toy. Beyond being a fun project to work on that could maybe get my foot in the door were I ever to decide to change careers and move into hardware design, this is not going to change the GPU landscape or compete with any of the commercial players. What it might do is pave the way for others to do interesting things in this space. A board with all of the video hardware that you can plug into a computer with all the infrastructure available to play around with accelerating graphics could be a fun, if extremely niche, product. That would also require a *significant* time and money investment from me, and that's not something I necessarily want to deal with. When this is eventually open-sourced, those who really are interested could make their own boards. One thing to note that is that while the US+ line is generally quite expensive (the higher end parts sit in the five-figures range for a one-off purchase! No one actually buying these is paying that price, but still!), the Kria SOMs are quite cheap in comparison. They've got a reasonably-powerful Zynq US+ for about $400, or just $350ish the dev boards (which do not expose some of the high-speed interfaces like PCIe). I'm starting to sound like a Xilinx shill given how many times I've re-stated this, but for anyone serious about getting into this kind of thing, those devboards are an amazing deal.
  - belter 2y ago
    "...I'm doing a (free) operating system (just a hobby, won't be big and professional like gnu) for 386(486) AT clones..."
    - Rinzler89 2y ago
      Yeah you're referring to the Linux kernel but software is much cheaper to design, test, build, scale and turn profitable than HW, especially GPUs. Open source GPUs won't threat Nvidia/AMD/Intel anytime soon or ever. They're way too far ahead in the game and also backed by patents if any new player were to become a thereat.
ruslan 2y ago
How much it depends on hard IP blocks ? I mean, can it be ported to FPGAs of other vendors, like Lattice ECP5 ? Did you implement PCIe in HDL or used vendor specific IP block ? Please, provide some resource utilization statistics. Thanks.
- PfhorSlayer 2y ago
  Implementing PCIe in the fabric without using the hard IP would be foolish, and definitely not the kind of thing I'd enjoy spending my time on! The design makes extensive use of the DSP48E2 and various BRAM/URAM blocks available in the fabric. I don't have exact numbers off the top of my head, but roughly it's ~500 DSP units (primarily for multiplication), ~70k LUTs, ~135k FFs, and ~90 BRAMs. Porting it to a different device would be a pretty significant undertaking, but would not be impossible. Many of the DSP resources are inferred, but there is a lot of timing stuff that depends on the DSP48E2's behavior - multiple register stages following the multiplies, the inputs are sized appropriately for those specific DSP capabilities, etc.
- alexforencich 2y ago
  The GPU uses https://github.com/alexforencich/verilog-pcie https://github.com/alexforencich/verilog-pcie + the Xilinx PCIe hard IP core. When using the device-independent DMA engine, that library supports both Xilinx and Intel FPGAs.
pocak 2y ago
In the post about the texture unit, that ROM table for mip level address offsets seems to use quite a bit of space. Have you considered making the mip base addresses a part of the texture spec instead?
- PfhorSlayer 2y ago
  The problem with doing that is it would require significantly more space in that spec. At a minimum, one offset for each possible mip level. That data needs to be moved around the GPU internally quite a bit, crossing clock domains and everything else, and would require a ton of extra registers to keep track of. Putting it in a ROM is basically free - a pair of BRAM versus a ton of registers (and the associated timing considerations), the BRAM wins almost every time.
billconan 2y ago
this is very awesome! Could you recommend me some books if I want to do something similar? for example, on how to design a pcb board for a PCIE pluggable hardware. Thank you very much!

rustybolt 2y ago

I have seen semi-regular updates from you on discord and it is awesome to see how far this project has come (and also a bit frustrating to see how relatively little progress I have made on my FPGA projects in the same time!). I was hoping you'd do a writeup, can't wait!

michaelt 2y ago

Googling the Xilinx Zynq UltraScale+ it seems kinda expensive. Of course plenty of hobbies let people spend thousands (or more) so there's nothing wrong with that if you've got the money. But is it the end target for your project? Or do you have ambitions to go beyond that?

kanetw 2y ago
The Kria SOM in use here is like $300.
PfhorSlayer 2y ago
Let's be clear here, this is a toy. Beyond being a fun project to work on that could maybe get my foot in the door were I ever to decide to change careers and move into hardware design, this is not going to change the GPU landscape or compete with any of the commercial players. What it might do is pave the way for others to do interesting things in this space. A board with all of the video hardware that you can plug into a computer with all the infrastructure available to play around with accelerating graphics could be a fun, if extremely niche, product. That would also require a *significant* time and money investment from me, and that's not something I necessarily want to deal with. When this is eventually open-sourced, those who really are interested could make their own boards. One thing to note that is that while the US+ line is generally quite expensive (the higher end parts sit in the five-figures range for a one-off purchase! No one actually buying these is paying that price, but still!), the Kria SOMs are quite cheap in comparison. They've got a reasonably-powerful Zynq US+ for about $400, or just $350ish the dev boards (which do not expose some of the high-speed interfaces like PCIe). I'm starting to sound like a Xilinx shill given how many times I've re-stated this, but for anyone serious about getting into this kind of thing, those devboards are an amazing deal.
- belter 2y ago
  "...I'm doing a (free) operating system (just a hobby, won't be big and professional like gnu) for 386(486) AT clones..."
  - Rinzler89 2y ago
    Yeah you're referring to the Linux kernel but software is much cheaper to design, test, build, scale and turn profitable than HW, especially GPUs. Open source GPUs won't threat Nvidia/AMD/Intel anytime soon or ever. They're way too far ahead in the game and also backed by patents if any new player were to become a thereat.

ruslan 2y ago

How much it depends on hard IP blocks ? I mean, can it be ported to FPGAs of other vendors, like Lattice ECP5 ? Did you implement PCIe in HDL or used vendor specific IP block ? Please, provide some resource utilization statistics. Thanks.

PfhorSlayer 2y ago
Implementing PCIe in the fabric without using the hard IP would be foolish, and definitely not the kind of thing I'd enjoy spending my time on! The design makes extensive use of the DSP48E2 and various BRAM/URAM blocks available in the fabric. I don't have exact numbers off the top of my head, but roughly it's ~500 DSP units (primarily for multiplication), ~70k LUTs, ~135k FFs, and ~90 BRAMs. Porting it to a different device would be a pretty significant undertaking, but would not be impossible. Many of the DSP resources are inferred, but there is a lot of timing stuff that depends on the DSP48E2's behavior - multiple register stages following the multiplies, the inputs are sized appropriately for those specific DSP capabilities, etc.
alexforencich 2y ago
The GPU uses https://github.com/alexforencich/verilog-pcie https://github.com/alexforencich/verilog-pcie + the Xilinx PCIe hard IP core. When using the device-independent DMA engine, that library supports both Xilinx and Intel FPGAs.

pocak 2y ago

In the post about the texture unit, that ROM table for mip level address offsets seems to use quite a bit of space. Have you considered making the mip base addresses a part of the texture spec instead?

PfhorSlayer 2y ago
The problem with doing that is it would require significantly more space in that spec. At a minimum, one offset for each possible mip level. That data needs to be moved around the GPU internally quite a bit, crossing clock domains and everything else, and would require a ton of extra registers to keep track of. Putting it in a ROM is basically free - a pair of BRAM versus a ton of registers (and the associated timing considerations), the BRAM wins almost every time.

billconan 2y ago

this is very awesome! Could you recommend me some books if I want to do something similar? for example, on how to design a pcb board for a PCIE pluggable hardware. Thank you very much!