Hello friends,
I’m pretty deep into self-hosting - especially on the home automation side. I’ve got a couple of options for self-hosted AI, but I don’t think they’ll meet my long term goals:
-
Coral TPUs: I have 2x processing my Frigate data. These seem fine for that purpose, but not useful for generative AIs?
-
Jetson Nano: Near as I can tell nothing supports these things except DeepStack, which appears to be abandoned. Bummed these haven’t gotten broader support in the community.
I’ve got plenty of rack space and my day job is managing thousands of machines, so not afraid of a more technical setup.
The used NVIDIA rack mounted Tesla GPU servers look interesting. What are y’all using?
Requirements:
- Rack mounted
- Supports local LLM and GenAI
- Linux-based
- Works with Docker
Don’t have direct experience with either, but:
It’s my understanding that a corral tpu is exclusively an inference accelerator, no training or more generative applications. Also, corral TPUs are a little bit unobtainium, with the only options I’ve seen behind scalped about as much as a pi, to basically the same result.
I think you’re overthinking the nano a bit. I’m not sure that you’d need explicit support for the nano, because it’s just a cuda gpu and so it should^TM just run anything cuda, as long as the arm cpu doesn’t trip the software up . For example, I’ve seen people running blender renders across a cluster of jetsons, just because, and I doubt that blender has any explicit support for jetsons.
If you’re coming at it from the sense that you have rack space to spare, a used Tesla / Quadro gpu would probably be better value than a jetson nano OG, because those were I think 2GB/4GB and 256 Kepler era cuda cores. You’d almost have to go out of your way to find a worse PCIe card, plus a normal PCIe card in a normal x86 server wouldn’t have arm software restrictions. Although as the other commenter mentioned, cooling/power draw is a more serious consideration for a PCIe card, plus the risks of buying used.
I totally agree on the Coral TPUs. Great for Frigate, but not much else. I’ve got 2x of the USB ones cranking on half a dozen 4K stream - works wonderfully.
And I agree in theory these Nanos should be great for all sorts of stuff, but nothing supports them. Everything I’ve seen is custom one offs outside of DeepStake (though CodeProject.AI purports there’s someone now working on a Nano port).
Sounding like a decent gaming GPU and a 2-3U box is the ticket here.
Point of pedantry- the Nano uses a Tegra X1 as its SoC. It has a Maxwell generation GPU, not Kepler.
The new Jetson Orin Nano uses an Ampere GPU.