How I run my Lab
文章目录
My lab is a massive cluster of compute spanning commodity hardware all the way to state of the art blackwell systems.
It takes a bit to organize and set it up, but now that it is setup it runs every experiment and research I'd want to, with ease, and with secure setups.
Here's a horrible write up of how things are managed, probably more confusing than helpful but some folks were asking so whipped this up in a few minutes.
-
The Lab
-
The Enclave
-
The AutoResearch
-
Jilly
-
Frontier Research
The Lab
High level view of the lab.
The lab consists of the following high level hardware:
-
3x DGX Sparks
-
GPUs (4x P100 GPUs, 4x V100 GPUs, 1x Radeaon VII, 1x BC-160, 1x Intel b70).
-
GPU server (P100/V100) + 128 GB Ram, 16TB RAID5 Disk)
-
Virtualization Server (80 Cores, 512 GB Ram, 16TB RAID5 Disk)
-
4x Mini PCs (Enclave, Compute Node, IoT Controller, GPU Swapper Bench)
-
Microcontroller Bench PC
-
Old Gaming computer used to swap out and test GPUs.
-
2 Routers, 2 10gbps Switches, Battery Backups, etc...
The Enclave
The main brain of the entire cluster is called "The Enclave". It is the command and control center of the entire cluster and is hidden from the rest of the cluster via a Slate 7 gl.inet router. The Enclave can see and touch everything in the lab, but nothing in the lab can see or touch it.
The Enclave
The enclave is a beast of 128gb ram with an Intel ultra cpu, mirrored 4tb NVMe drives and two external storage (5tb the, 2TB NVMe), and an Intel Arc b70.
The enclave runs my entire agent orchestration factory and knowledge base. The GPU here is for running embedding and clef.
Clef runs the guardrails and permissions checks, as well as prompt injection protections (As much as we can). Every request to the enclave, every command it runs, every decision it makes on where to place something in the KB etc runs through @cloudflare Clef.
For embeddings it runs Gemma Embedding 2 model. All models and podman images are saved on the external 2tb NVMe to leave the mirrored 4TB NVMe for knowledge base storage.
Everything in the enclave is in real time synced to Cloudflare R2 as an offsite backup.
The enclave host itself runs proxmox and splits the enclave into two VMs. Agent Host and KB Host. With strict isolation between the two.
High level enclave layout
Everything I do starts in the Enclave and sprawls out from there. I run hermes as the agent harness there, with multiple profiles that are finetuned for the various workloads. Back with the KB, shared team memory, and systems they are really effecient.
I've also recently upgraded to use the bot mode in hermes desktop and hermex app. This has taken some getting used to, but I'm getting better flows with multi agent collaborations going.
Snapshot of recent bot conversations.
When I need hands on under the hood access I reach for Herdr, which is setup across the entire lab. In fact my bots actually have access to herdr to manage agent execution across the lab (Especially ledger who manages the autoresearch agents).
Old snapshot of herdr machine list
So one thing with the Enclave is I wanted to have a Claude Artifacts / ChatGPT style setup. For this I use something called Coolify, hosted on my big virtualization server. It allows the Enclave to build and publish any kind of app, site, page, html and expose it to me over apps.<tailscale-address>.net/<appid>. Which then only I can access and view. Its great for creating progress dashboards on projects and one off little visualizations etc.
The AutoResearch
I do a lot of research on older cards and systems. For this I have 6 GPUs with active auto research at a time. (4x P100's, 1x Radeon VII 7, 1x BC-160) These are all ancient by modern standards. But have great potential for local AI at e-waste costs.
Radeon VII 7. Autoresearching optimal engines and runtimes for prefill and decode speeds.
Autoresearch works on my custom research harness called Thermite (Which I'll be releasing open source soon). It is a refinement of what I value most in the hillclimbs and how I believe student + teacher + valley breaking should be.
The Main loop of thermite.
You've seen the raw Pi screenshots before of pi-autoresearch running, and that was used in the previous versions of thermite. But now I have a system that automatically spins up a container on my GPU clusters, attaches a GPU, starts an expirement. Then my "ledger" bot runs period checks and steering on the experiments. All results are piped to prometheus which I can view status of autoresearch campaigns in Grafana or custom build dashboards in hermes desktop.
Jilly
Jilly is a service I offer on the Hackers in the Loop discord, which is free coding agent powered by deepseek flash v4.1. It runs a full agent connect setup with its own isolated vlan and sandboxes. Tag her in the discord and work with her in a thread and do cool things. She's free until her tokens run out.
Backend of jilly showing various sessiosn ran with her in the HACK Labs discord.
Frontier Research
Outside of running the Hackers in the Loop community, Doing open research on P100 and older hardware for local AI, and just setting up lab stuff I do a lot of unspoken research quietly in the background.
This is what we use the DGX Sparks for mostly, the modern blackwell architecture and up to date CUDA drivers etc let us test bed ideas that can translate into production scales on larger hardware like the B200/B300 systems etc.
As a result these are the most locked down systems in the lab, only the Enclave or specific machines like my personal mac and other researchers devices can reach them.
At a high level this research spans across latent space thinking, KV Cache improvements, and agent swarm vs large models.
Thats all folks
For now thats all I got, but later might share more specifics around different areas if people are interested.
