Personal LLM Use

With the advent of LLMs small enough to run on consumer-grade hardware, I was hoping to find advice on building a computer that can run an advanced personal-use LLM. Of course, this technology is changing rapidly, especially with today's release of DBRX. Hopefully though, we can try to find some future-proof advice. So essentially what I'm asking here is: what's the best hardware to buy for a personal LLM? GPUs, racks, cases, anything you might know would be useful. Also, let's talk about uses for personal LLMs (besides the obvious)

I’ve heard of this: but obviously it may not be in your budget: https://tinygrad.org/ There’s also Ollama that helps people get started with LLMs on local hardware

My understanding is you only need to care about gpu memory size and bandwidth because the limitation is getting the model into the gpu. As an example private copilot llvm: Code llama smallest model is ~2gb and then it goes up to 8gb in stages. Might have bigger ones by now too. If you look at more memory for fancier models there is only 16gb on the most modern consumer gpus and 32gb requires enterprise gpus which cost $1.5k+. tldr: you can run a small model on midgrade 8gb gpu, if you have money obviously could buy fancier cards for bigger models

>>76484 Llamafiles on Huggingface

>>76484 You can pick up an older M6000 24GB Quadro for around £350 uk - prob less in USA... This is like a 980Ti speed GPU. Or you can get a P5000 Quadro (1080 gpu) with 16GB for about 250-300 UK squid. I think 16GB gives access to more LLMs, 8GB is min as has been stated above. I've not personally seen any spec 24GB... Grok wants 380GB!!! You can get a M4000 Quadro with 8GB for about 70 uk squid! And finally a 1080 8Gb for about 100 squid!

>>76533 you seem to know more. Seen any benchmarks old quadro vs modern consumer? I know the enterprise cards always had way better cuda performance due more focus on compute units but im unsure if they can maintain that advantage with 2 generation gap. Another consideration is watt - cheaper card might cost you more longterm if its more power hungry.

>>76543 A M4000 is significantly worse than a 3060.

>>76558 >>76543 Yep these Quadros are nothing special - they just have a fat amount of RAM meaning they can probably run things as opposed to not run things... I personally would not be concerned if it took a few sec or so to answer questions... rather than not run it due to a lack of ram... To note; non of these cards have AI cores (the 2080 series was the first to specifically have tensor cores -Which equates to the Quadro RTX series e.g. Quadro RTX4000 (currently 300squid UK for 8GB). I was giving info on cheep path to enough GPU RAM not a path to high speed compute... but all these cards run CUDA so will compute the LLMs and should be only say 10x slower than state of the art.... So instead of a 1sec delay you might have to wait 10sec... not too problematic! If you want compute and RAM I think a 2080Ti 11GB is a good bet for the money today. Although this will not let you run the 16GB stuff and for these type of bucks the Quadro p5000 gets you that majic 16GB at 1080 speed compute. The first card that gives 16GB + Tensor cores and hence fast-ish compute for AI is the Quadro RTX 5000 costing about 500 squid UK. Unfortunately no 30xx series cards have 16GB, although they have fast compute! And the 40xx series with 16GB are currently about £1k! Hope this helps - unless your processing loads of AI stuff I would not be too worried about speed. Most of the need for speed is training AI... if your trying to train AI then yes you need speed to bring times down from months to days! But then your not scrounging around for a bargain! Depending if you want 8 or 16 GB LLm's will determine you path to bargain AI. I know some will run on CPU and if you got an old Xeon you can stuff in 48GB or more for very little $... but this will be SLOW!

>>76573 I'm just a bit lost. I have a Lenovo Tiny just sitting on my desk with 16GB of ram. I might host like my own LLM but I'd somehow want it to have internet access if that's even a thing. Might just dedicate the box for a LLM (if there is a way to connect it to the net)

>>76610 For qd1 inference - which is what you're doing - _the_ thing that matters is RAM bandwidth. And GPUs have silly-high amounts of RAM bandwidth compared to CPUs. (Because that's what they are optimized for...) What this means in practice is CPU-attached ram is (IMO) unusably slow, and even the slowest GPU runs rings around a CPU, _so long as the model fits in VRAM_. Yes, there are some GPUs that are faster than others, if you're careful, but normally just going for the largest amount of VRAM you can afford is a fairly decent approach.

>>76484 2x 7900xtx here. Works fine in Linux with exllama2, but be careful with ROCm. A lot of motherboards run most of the pcie ports through the chipset, and most AM4 chipsets don't support PCIe atomics, and ROCm needs PCIe atomics, read silently locking up otherwise. The second card is attached via an M.2 adapter on the direct-connect pcie lanes. Runs 70B models with large contexts at decent quantization depths just fine.

>>76610 >Lenovo Tiny So biggest issue here is they only can fit slim GRFX cards not full size... also I doubt the PSU could handle more than 100W ish of GRFX card.... This massively limits you. Non of the cards I listed will work for you. ====Your only options are to run a CPU model and suffer horrible slowness (n.b. LLMs that can go online are even slower!) Or you only option I can think of to accelerate an LLM is a Quadro RTX A2000, these have only 6GB ram but some LLMs can run with that I think... The Quadro A2000 does not need any extra power leads and can fir on a SFF case. Cost ~ £250 uk on ebay. This card while not a game monster does have some tensor cores so will accelerate AI as well as have ram for it to run... but it is the entry card and speed equivalent in games to a 1070. (with the tensor cores it should be quicker at LLMs). Bottom line is its a relatively expensive card for the bang... But your only option to accelerate LLMs... I dont know AMD stuff so maybe there is something there but I cant help with that.

>>76484 For 80% of people interested in this as a hobby, a used 3090/4090 (24gb vram) will go a long way for toying around with local models. Best paired with 64gb+ of ram so you don't have to hit swap while loading things into vram. if you're giga autistic and/or rich you can follow this guy's guide to take advantage of the influx of cheap-ish V100s from data centers that are hitting the market: https://rentry.org/V100MAXXING