6 ms·
It needs to be a little less than double the overall price for 256GB option, otherwise it's better to get 2 256GB and link them.
by notnullorvoid 22d ago
It needs to be a little less than double the overall price for 256GB option, otherwise it's better to get 2 256GB and link them.
- Zylokloto 22d agoyou can't link onchip memory.
- notnullorvoid 22d agoTrue, but you can link them up over thunderbolt or Ethernet. If your goal is to run local LLMs, not all weights need to live on the same computer. You can segment the workload by layers and pass the activations along the lower bandwidth interconnect with a small perf penalty. Also you get double the CPU/GPU cores allowing for better multi user/agent performance.
- Zylokloto 22d agoAh sorry you are right I missread what you meant by it.
- nowittyusername 22d agoI was thinking the same as you as far as price per value, it does make seance to get 2x of these things IMO, but what throughput hit would you see in linking versus one machine? latency does matter, and there must be a trade off no?