Your cart is currently empty!
Tag: megabucks
Today I learned I can run my very own DeepSeek R1 chatbot on just $6,000 of PC hardware and no megabucks Nvidia GPUs required
Got the impression that a bazillion dollar’s worth of GPUs are required to run a cutting-edge chatbot? Think again. Matthew Carrigan, an engineer at AI tools outfit HuggingFace, claims that you can run the hot new DeepSeek R1 LLM on just $6,000 of PC hardware. The kicker? You don’t even need a high-end GPU.
Carrigan’s suggested build involves a dual-socket AMD EPYC motherboard and a couple of compatible AMD chips to go with it. Apparently, the spec of the CPUs isn’t actually that critical. Instead, it’s all about the memory.
Complete hardware + software setup for running Deepseek-R1 locally. The actual model, no distillations, and Q8 quantization for full quality. Total cost, $6,000. All download and part links below:January 28, 2025
“We are going to need 768GB (to fit the model) across 24 RAM channels (to get the bandwidth to run it fast enough). That means 24 x 32 GB DDR5-RDIMM modules,” Carrigan explains.
Links are helpfully provided and the RAM alone comes to about $3,400. Then you’ll need a case, PSU, a mere 1 TB SSD, some heatsinks and fans.
Indeed, Carrigan says this setup gets you the full DeepSeek R1 experience with no compromises. “The actual model, no distillations, and Q8 quantization for full quality,” he explains.
From there, simply “throw” on Linux, install llama.cpp, download 700 GB of weights, input a command line string Carrigan helpfully provides and Bob’s your large language model running locally, as they say.
Notable in all this is a total absence of mention of expensive Nvidia GPUs. So what gives? Well, Carrigan provides a video of the LLM running locally on this setup plus a rough performance metric.
Nvidia’s H100: You won’t be needing one of these. (Image credit: Nvidia) “The generation speed on this build is 6 to 8 tokens per second, depending on the specific CPU and RAM speed you get, or slightly less if you have a long chat history. The clip above is near-realtime, sped up slightly to fit video length limits,” he says.
The video shows the model generating text at a reasonable pace. But that, of course, is for just one user. Open this setup out to multiple users and the per-user performance would, we assume, quickly become unusable.
In other words, that’s $6,000 of hardware to support, in effect, a single user. So, this likely isn’t an approach that’s practical for setting up an AI business serving hundreds, thousands or even millions of users. For that kind of application, GPUs may well be more cost effective, even with their painful unit price.
Carrigan suggests a build relying on GPUs might run up to triple-digits pretty quickly, albeit with better performance.
And if you got this far: Yes, there’s no GPU in this build! If you want to host on GPU for faster generation speed, you can! You’ll just lose a lot of quality from quantization, or if you want Q8 you’ll need >700GB of GPU memory, which will probably cost $100k+January 28, 2025
But it is intriguing to learn that you don’t actually need a bazillion dollar’s worth of GPUs to get a full-spec LLM running locally. Arguably, it also provides insight into the true scale of intelligence implied by the latest LLMs.
As an end user experiencing what can seem like consciousness streaming out of these bots, the assumption is that it takes huge computation to generate an LLM’s output. But this setup is doing it on a couple of AMD CPUs.
So, unless you think a couple of AMD CPUs is capable of consciousness, this hardware solution demonstrates the prosaic reality of even the very latest and most advanced LLMs. Maybe the AI apocalypse isn’t quite upon us after all.
Today I learned I can run my very own DeepSeek R1 chatbot on just $6,000 of PC hardware and no megabucks Nvidia GPUs requiredI have always been fascinated by chatbots and the technology behind them, but I never thought I could actually run my own without breaking the bank. However, after some research and experimentation, I discovered that I can create and run a DeepSeek R1 chatbot on just $6,000 of PC hardware – and I don’t need any expensive Nvidia GPUs.
The DeepSeek R1 chatbot is a powerful and versatile chatbot that can be customized to suit a variety of applications. With its advanced natural language processing capabilities and deep learning algorithms, it can provide intelligent and engaging conversations with users.
To run the DeepSeek R1 chatbot, all you need is a decent PC with at least 16GB of RAM, a modern CPU (such as an Intel Core i7 or AMD Ryzen processor), and a good internet connection. You can also use a GPU for faster processing, but it is not necessary – the chatbot can run perfectly fine on CPU alone.
By utilizing cost-effective hardware and open-source software, I was able to set up my own DeepSeek R1 chatbot without breaking the bank. Now, I can explore the exciting world of conversational AI and create engaging chatbot experiences for users without spending a fortune.
So, if you’ve always wanted to run your own chatbot but thought it was out of reach, think again! With just $6,000 of PC hardware and some creativity, you can bring your chatbot dreams to life.
Tags:
- DeepSeek R1 chatbot
- PC hardware
- Chatbot technology
- Affordable chatbot solution
- AI chatbot development
- Cost-effective chatbot setup
- Low-cost chatbot hardware
- Nvidia GPU alternatives
- Chatbot on a budget
- DIY chatbot implementation
#Today #learned #run #DeepSeek #chatbot #hardware #megabucks #Nvidia #GPUs #required
