好文集ChatGPT Plus¥119

how to build your own uncensored AI (COMPLETE SETUP)

文章目录

i'm going to walk you through the full system i'd use to run an uncensored open-weight model on my own GPU and use it from my laptop, my phone or my agents running anywhere, without opening a single port...

because uncensored models are the loudest thing in AI on X right now, and almost every guide stops at "download this" or "run it at your desk"

downloading is the easy part, and it's the part that matters least

reaching it from an agent that runs somewhere you don't control, making sure nobody else can use it, picking a build that isn't broken, and knowing when not to trust what it says is the part nobody writes down

and before we touch a single tool, here's the one rule that decides whether any of this is worth your time:

an uncensored model removes the model's "no"... it doesn't remove yours, and it doesn't remove the law's

if you keep that in your head for the whole article, everything below is safe, useful and boring in the best way

here's what we'll cover, in the order you'd actually build it:

  • why uncensored models are everywhere this week, with the real numbers

  • what abliteration actually is, and how Heretic automates it

  • the best current uncensored models, one by one, with sizes at every quant

  • how to pick the right model and quant for your hardware

  • how to serve it locally with Ollama, llama.cpp or vLLM

  • how to reach it from anywhere with Cloudflare Tunnel and Cloudflare Access

  • how to test an uncensored model yourself before you trust it

  • how to plug it into your agent as a read-only subagent

  • one real job, start to finish

  • when not to use the local lane, and the tailnet option

  • what it's actually good for, with example prompts

  • the risks, the licenses and the legal part

  • the rules, and the hardening checklist

  • every repo in this guide

  • your first week, and a FAQ

why uncensored models are everywhere this week

it started with one article, and then X did what X does

on 11 november, Transformer News reported that more than 6,500 models tagged abliterated, uncensored or heretic were hosted on Hugging Face in july, and asked whether some of them could be used for cyber attacks

that number got screenshotted, quoted and turned into panic within two days

here's what the reaction actually looked like, pulled from X this afternoon:

  • @superalesha posted "DOWNLOAD THESE 24 OPEN MODELS BEFORE MORE REPOS GET GATED OR TAKEN DOWN" on 14 november, and it's sitting at 303,503 views and 9,964 bookmarks posted “DOWNLOAD posted "DOWNLOAD THESE 24 OPEN MODELS BEFORE MORE REPOS GET GATED OR TAKEN DOWN" on 14 november, and it's sitting at 303,503 views and 9,964 bookmarksTHESE 24 OPEN MODELS BEFORE MORE REPOS GET GATED OR TAKEN DOWN” on 14 november, and it’s sitting at 303,503 views and 9,964 bookmarks

  • the day before, his "I am one of the people behind the 6,500 uncensored models the press now wants off Hugging Face" post got 277,446 views

  • @0x0SojalSec's "top 5 most downloaded uncensored models on Hugging Face right now" got 54,193 views and 1,237 bookmarks

  • @HuggingModels on an abliterated Gemma 4 26B GGUF got 20,439 views and 477 bookmarks on an abliterated Gemma 4 26B GGUF got 20,439 views and 477 bookmarks on an abliterated Gemma 4 26B GGUF got 20,439 views and 477 bookmarks

  • @DeRonin_'s list of legal use cases got 11,672 views and 225 bookmarks

look at the bookmark counts more than the views

9,964 bookmarks on one post means people aren't just reading about this, they're planning to do it

the downloads back that up, straight from Hugging Face's own counter:

  • huihui-ai's abliterated Qwen3.6-35B-A3B GGUF at 5,056,428 downloads

  • huihui-ai's abliterated Qwen3.8-27B GGUF at 4,253,987 downloads

  • SC117's abliterated Qwen3.8 Flash-Next GGUF at 4,164,878 downloads

(Hugging Face's download number is a rolling count, not a lifetime total, so these are recent pulls)

GitHub agrees too... p-e-w/heretic, the tool behind most of the new "heretic" uploads, is at 41,125 stars with a push on 16 november

and Hacker News has been chewing on it for a month:

  • "How abliterated models can get you pwned" (ProjectDiscovery's research) hit 256 points and 143 comments

  • "There are now 5 inference providers for abliterated DeepSeek" hit 100 points and 122 comments

that first one matters a lot, and we'll come back to it in the risks section

the "how to run it" side has already blown up too, and this is the part that shaped this guide:

  • @DeRonin_'s "Here's 4 steps to run an uncensored AI on your own GPU for $0" got 724,972 views and 14,215 bookmarks on 6 november, and it covers running the model and plugging it into Claude Code’s “Here’s 4 steps to run an uncensored AI on your own GPU for $0” got 724,972 views and 14,215 b's "Here's 4 steps to run an uncensored AI on your own GPU for $0" got 724,972 views and 14,215 bookmarks on 6 november, and it covers running the model and plugging it into Claude Codeookmarks on 6 november, and it covers running the model and plugging it into Claude Code

  • his "Best Uncensored AI models now" list got 204,677 views and 6,988 bookmarks two days later

  • his "spend one evening setting up Tailscale" post on 15 november, about reaching his models from anywhere on a private network, got 35,794 views and 1,070 bookmarks

and one reply under that Tailscale post said exactly what people want next: the uncensored model on one machine, reachable from every other device they own

so running it at your desk is solved, and reaching it from your own devices is solved

what isn't covered anywhere i looked is the version for agents: a model your cloud agent can call with its own token, an origin that checks that token a second time, a build you've actually tested, and a subagent that can't touch anything

Hacker News is pointing the same way, from the local side:

  • "Apple has new Mac app coming to make running local AI models easy" (9to5Mac) hit 576 points and 567 comments on 10 november

  • "Llama-modchip: 284B DeepSeek on 64GB RAM, no GPU" hit 487 points and 211 comments on 8 november

  • "Ask HN: What hardware are you using to run local LLMs?" got 52 points and 46 comments

  • llama.cpp shipped v1.0.0 on 13 november, and the release notes say it hardens llama-server against SSRF, CORS and DNS rebinding problems

and Product Hunt shows where the products are... small, private, local, and almost never about secure remote access:

  • CuriousLM, "Private AI that runs on your device. No account.", 93 upvotes, #15 of the day on 2 november

  • Pinku, offline AI chat on iPhone and Mac with Apple's MLX, 73 upvotes on 5 november

  • Locally Uncensored, an offline local AI studio that runs abliterated models, launched on 20 august (its GitHub repo is at 4,335 stars)

  • Local, a macOS local AI app whose "Office Mode" lets every laptop in an office connect to the fastest machine, and its top review asks whether that shared service "is authenticated or just open to anyone on the same network"

that review is this whole article in one sentence

and the tunnel products on Product Hunt sell the same worry: Smuggl's pitch is sharing localhost "instead of blindly exposing your computer to the whole internet" (206 upvotes, 9 april)

now one correction before you repeat anything from this week

the viral claim that Hugging Face was deleting uncensored repos was wrong

Hugging Face's @julien_c said the removed repo was taken down for abusing the gating system to steal users' data, unrelated to uncensored models, and @superalesha posted a correction afterwards

and the 6,500 is a tag count... it's not 6,500 attack tools

so the honest version of the story is this: a lot of people suddenly want a model that doesn't refuse, a lot of those models are one click away, and very few people know how to run one without hurting themselves

that's the gap this article fills

嵌入推文(联网查看)


what abliteration actually is

here’s the whole idea in one line: a model’s “no” lives in one direction inside its activations, and if you remove that direction, the model stops refusing

that comes from a 2024 paper by Arditi et al., “Refusal in Language Models Is Mediated by a Single Direction”, and once you get it, every abliterated model on Hugging Face makes sense

let me explain it without the math

a language model is a stack of layers, and as your prompt moves through the stack, every layer reads and writes to one shared running state called the residual stream

the paper found that when a model is about to refuse, that running state leans in one particular direction, the same direction across lots of different refused prompts

so if you can find that direction, you can stop the model from ever leaning that way

文章配图

here’s how the classic method finds and removes it, in the order it actually runs:

  1. you build two prompt sets, one the model refuses and one it answers normally (the original recipe in many model cards uses 128 of each)

  2. you run the model on both sets and record the residual stream at each layer, usually at the last token of the prompt

  3. for each layer, you take the average activation on the refused set and subtract the average on the answered set… that difference is a candidate refusal direction

  4. you test the candidates and keep the single direction that best separates “about to refuse” from “about to answer”

  5. you edit the weights that write into the residual stream so they can’t write along that direction anymore (this is called orthogonalizing the weights)

  6. you check the result on fresh prompts, both for refusals and for whether it still answers normal questions well

that’s it… no retraining, no new data, no fine-tuning run

the model doesn’t learn anything new, it just loses the reflex to refuse

which is also why an abliterated model is never smarter than the base it came from

there’s a second way to do step 5, too

instead of editing the weights permanently, you can subtract the direction at inference time, every token, every layer… same effect, but the weights stay untouched

heterodoxin/AbliteratedOutTheBox is a llama.cpp fork that does exactly that at inference

the people who made all of this practical, in order:

  • Andy Arditi’s team published the code with the paper (andyrdt/refusal_direction)

  • FailSpy built abliterator, a small Python library on top of TransformerLens, and uploaded some of the first abliterated Llama 3 models

  • Maxime Labonne wrote “Uncensor any LLM with abliteration” on the Hugging Face blog in june 2024, based on FailSpy’s notebook, and that’s the guide most people learned from

  • Sumandora/remove-refusals-with-transformers did the same thing with plain Hugging Face Transformers, and it’s the method huihui-ai’s model cards point to

  • p-e-w built Heretic, which automates the whole thing

how Heretic works

the classic method has one big weakness… a human has to guess the settings

which layers do you edit, how hard, with what weight, and how do you know you didn’t break the model?

Heretic turns those guesses into an optimization problem

here’s the loop:

  1. you pick a model and a config (the default config suppresses refusals, and there are shipped configs that target slop and sycophancy instead)

  2. Heretic benchmarks your machine to pick the best batch size

  3. a modifier applies a candidate edit to the model, and the default modifier in Heretic 2.x is Arbitrary-Rank Ablation (ARA), with classic directional ablation still available

  4. scorer plugins measure the result, and by default that’s two scores: refusal rate on a prompt set, and KL divergence from the original model (how far its normal answers drifted)

  5. an Optuna TPE optimizer proposes the next set of parameters, trying to push refusals down while keeping KL divergence low

  6. it repeats until it finds the best trade-off, then lets you save it, upload it, chat with it or run standard benchmarks on it

the README says a 4B model takes about 30 to 40 minutes on an RTX 3090 with the default config

and that second score is the important one

refusals are easy to remove if you don’t care what happens to the model… KL divergence is the leash that stops the edit from wrecking everything else

two more Heretic features are worth knowing:

  • it can write a reproduce.json into an uploaded repo, so anyone can rebuild that exact model and Heretic checks the weight hashes at the end

  • there’s a public library of reproducible Heretic models called the Heretic Grimoire

that reproducibility part is the closest thing this whole lane has to a trust signal, and we’ll use it in the evaluation section

running Heretic yourself

you don’t need to run Heretic to use this guide, but doing it once on a small model teaches you more than any explainer

do it only on a model whose license lets you modify it, on your own hardware, and keep the output for legal work

here’s the path from the README, in order:

1. install it into a fresh Python environment (you need PyTorch working for your GPU first)

pip install -U heretic-llm

2. point it at a small model, the README’s own example is a 4B

heretic Qwen/Qwen3-4B-Instruct-2507

3. let it benchmark your machine, it picks the batch size on its own

4. let the optimizer run, and watch two numbers move against each other: refusals going down, KL divergence staying low

5. when it finishes, pick the trial with the best trade-off, not the one with the fewest refusals, because zero refusals with high KL divergence usually means a damaged model

6. save it locally, chat with it, and run the benchmarks it offers before you even think about uploading

7. if you do upload it, keep the reproduce.json in the repo, so anyone can rebuild your exact model and check it

and if you’re tight on memory, the README says Heretic can load the model in 4-bit through bitsandbytes on the fly, which cuts VRAM a lot

one warning: Heretic 2.x configs aren’t compatible with 1.x, so if you copy a config from an old blog post, check it first

the real lesson from running it once is this… refusal removal is a dial, not a switch, and every notch you turn costs a little of the model you started with

one more thing you’ll see on model cards: “fine-tuned” uncensored models like Dolphin from dphn

those are trained on data without refusals, not edited, so they behave differently… the Dolphin card is explicit that you set the alignment yourself through the system prompt

and one naming trap: many Hugging Face repos have “fable” in the name

that’s a community label, it has nothing to do with Claude Fable 5.2


the best current uncensored models

every number in this section is from the Hugging Face API on 17 november, around 3:30 PM IST

sizes are the actual GGUF file sizes in each repo, and the file size is the floor for memory, not the total… the context window needs memory on top of it

i’ll go through the five i’d actually consider, then two big ones for people with serious hardware

1. huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF

this is my pick for most people

  • base: Qwen/Qwen3.8-27B

  • license tag: apache-2.0

  • downloads: 4,253,987, likes: 2,263, last updated 15 november

  • method: abliteration via the remove-refusals-with-transformers approach (the card links it)

sizes at the quants you’d actually use:

  • UD-IQ2_S: 8.42 GB

  • Q2_K: 10.86 GB

  • UD-IQ3_XXS: 11.02 GB

  • Q3_K: 13.50 GB

  • UD-IQ4_XS: 14.40 GB

  • UD-DW-Q4_K_M: 16.55 GB

  • Q4_K: 16.81 GB

  • Q5_K: 19.54 GB

  • Q6_K: 22.43 GB

  • Q8_0: 29.05 GB

  • bf16: 54.66 GB

what it’s good at: it’s a dense 27B, so it’s strong at long reasoning, writing and code for its size, and it’s the most downloaded dense model in this lane

known issues, straight from the card:

  • huihui calls it “a crude, proof-of-concept implementation to remove refusals”

  • some variants in the repo only ablate layers 22 to 52, and the card says those “may come with a small disclaimer warning”

  • the repo has dozens of files from different quantizers and side projects, so read the file name carefully before you pull

2. huihui-ai/Huihui-Qwen3.6-35B-A3B-abliterated-MTP-GGUF

the fast one

  • base: Qwen/Qwen3.6-35B-A3B (a mixture-of-experts model, 35B total, about 3B active per token as the name says)

  • license tag: apache-2.0

  • downloads: 5,056,428, the most downloaded GGUF in this lane right now

sizes:

  • Q2_K: 13.25 GB

  • Q3_K: 17.17 GB

  • Q4_K: 21.71 GB

  • Q5_K: 25.35 GB

  • Q6_K: 29.21 GB

  • Q8_0: 37.80 GB

  • f16: 71.07 GB

what it’s good at: MoE models only run a fraction of their weights per token, so they tend to generate faster than a dense model of the same file size, which is great for batch reading jobs

known issues: you still need memory for the full file, not just the active part, and the card carries a long usage warning (research and controlled environments, review outputs, no safety guarantees)

3. mradermacher/gemma-4-31B-it-ARA-heretic-GGUF

the Heretic one

  • base: richardyoung/gemma-4-31B-it-ARA-heretic (Gemma 4 31B edited with Heretic’s ARA modifier), quantized by mradermacher

  • license tag: apache-2.0

  • downloads: 161,317, likes: 359

sizes:

  • Q2_K: 11.92 GB

  • Q3_K_M: 15.29 GB

  • IQ4_XS: 16.86 GB

  • Q4_K_S: 17.76 GB

  • Q4_K_M: 18.69 GB

  • Q5_K_M: 21.85 GB

  • Q6_K: 25.20 GB

  • Q8_0: 32.64 GB

what it’s good at: a different base family from the Qwens, so it’s worth having as a second opinion, and ARA is Heretic’s own method

known issues: the source repo is gated (you need to accept access on Hugging Face to read it), so you can’t fully inspect how it was made from the quant repo alone

4. mlabonne/Meta-Llama-3.1-8B-Instruct-abliterated-GGUF

the laptop one, and the classic

  • base: meta-llama/Llama-3.1-8B-Instruct

  • license tag: llama3.1 (not apache, read it)

  • downloads: 38,770, likes: 248

  • made by the author of the abliteration guide, with credit to FailSpy on the card

sizes:

  • Q2_K: 3.18 GB

  • Q3_K_M: 4.02 GB

  • Q4_K_M: 4.92 GB

  • Q5_K_M: 5.73 GB

  • Q6_K: 6.60 GB

  • Q8_0: 8.54 GB

what it’s good at: it runs on almost anything, and it’s the fastest way to feel what abliteration changes

known issues: it’s an older 8B, so it’s clearly weaker than the 27B and 31B models above at anything hard

5. dphn/Dolphin3.0-Llama3.1-8B-GGUF

the fine-tuned one

  • base: meta-llama/Llama-3.1-8B

  • license tag: llama3.1

  • downloads: 57,890, likes: 260

  • chat template: ChatML, per the card

sizes:

  • Q3_K_M: 4.02 GB

  • Q4_K_M: 4.92 GB

  • Q5_K_M: 5.73 GB

  • Q6_K: 6.60 GB

  • Q8_0: 8.54 GB

what it’s good at: it’s trained to follow your system prompt as the rules, so it’s very steerable for fiction and roleplay-style writing

known issues: same 8B ceiling as above, and it behaves differently from abliterated models because it was trained, not edited

the two big ones

  • SC117/Qwen3.8-Flash-Next-GSQ-RCO-abliterated-GGUF (apache-2.0, 4,164,878 downloads): the smallest file in the repo is 66.8 GB (Q2_0), so this is multi-GPU or big-Mac territory

  • AtomicChat/DeepSeek-V4.1-Flash-1017-Uncensored-GSQ-RCO-GGUF (mit, 207,466 downloads): the smallest file is 120.7 GB (IQ1_M), so this is multi-GPU only

if you’re reading this on a single gaming GPU, skip both, the 27B will serve you better


how to pick the right model and quant

quantization just means storing the weights with fewer bits, so the file gets smaller and the model gets a little worse

Q8 is close to the original, Q4 is the sweet spot most people run, Q3 and Q2 are where you start to feel it

here’s the rule i use, and i’ll be straight with you, it’s a rule of thumb, not a measurement:

pick the biggest file that leaves a few GB of your VRAM (or unified memory on a Mac) free for the context window

文章配图

here’s the decision, in the order i’d make it:

  1. what’s your memory?

  2. what’s the job: heavy reasoning and writing, fast batch reading, or just trying it out?

  3. pick the model, then the biggest quant that fits with headroom

and here’s how that plays out by memory tier, using the real file sizes above:

  • 8 GB: Llama 3.1 8B abliterated or Dolphin 3.0 at Q4_K_M (4.92 GB)

  • 12 GB: the same 8B models at Q8_0 (8.54 GB), or the 27B at UD-IQ2_S (8.42 GB) if you accept a heavy quant

  • 16 GB: the 27B at Q3_K (13.50 GB) or UD-IQ3_XXS (11.02 GB)

  • 24 GB: the 27B at UD-DW-Q4_K_M (16.55 GB), the Gemma 4 31B Heretic at Q4_K_M (18.69 GB), or the 35B-A3B at Q4_K (21.71 GB) if you keep context small

  • 32 GB and up (or a big-memory Mac): the 27B at Q6_K (22.43 GB) or the 35B-A3B at Q5_K (25.35 GB)

  • 2 or more GPUs, 64 GB and up: now the Flash-Next build is on the table

by job:

  • writing and reasoning: the dense 27B, always

  • fast batch reading (logs, emails, documents): the 35B-A3B MoE

  • a second opinion from a different family: the Gemma 4 31B Heretic

  • trying it tonight on a laptop: an 8B

and when in doubt, start one size smaller than you think, check memory with ollama ps, then go up

no GPU at all, or a 4 to 6 GB card? the next section has a RAM-tier picker for CPUs, old cards and base Macs


no GPU, or an old one: the cheap always-on box

everything above assumes a real GPU, but you don't need one to start

an old laptop, a mini PC or a base Mac can run a small abliterated model all day, and it can sit behind the exact same tunnel and the exact same Access door

but you have to know what limits your speed, or you'll spend a weekend tuning the wrong thing

文章配图

the one number that decides your speed

on a CPU, every new token means reading the model's active weights out of RAM once

so the speed limit is your memory bandwidth divided by the size of the model, not your core count

Johannes Gaessler, a llama.cpp contributor, says it plainly in his benchmark notes: "the most important factor for performance is memory bandwidth"

Big Iron tested that rule on three CPUs in october and got close to it (tg128 is the generation speed in llama-bench):

  • Ryzen 5 5600, dual-channel DDR4: Qwen3-4B Q4_K_M at 13.19 tok/s, Qwen3-8B at 7.14 tok/s

  • i5-10400H laptop chip: 9.31 and 4.85 tok/s

  • i7-10700: 6.67 and 3.67 tok/s

all three were VMs, so treat them as ballparks, not promises

three things fall out of that:

  • dual channel matters more than the CPU. InsiderLLM measured an i7-6700T going from 5.89 to 9.15 tok/s on Gemma 3 4B just by moving from one RAM stick to two, a 55.9% jump

  • more threads can be slower. on the Ryzen, Big Iron's best was 3 threads, and the usual "one thread per physical core" setting (6) was 5% slower, while Gaessler found about 5 threads saturate dual-channel memory

  • context costs speed too. Big Iron saw the dense 4B and 8B lose 41 to 55% of their generation speed with 8,000 tokens already in context

AVX2 is the baseline you want on x86 (AVX-512 is a bonus), and you check it in one line further down

the MoE trick

a mixture-of-experts model only reads its active experts for each token, so a big MoE can run faster than a small dense model

on that same Ryzen 5 5600 with no GPU, Big Iron measured gpt-oss-20b at 12.97 tok/s and LFM2.5-24B-A2B at 21.89 tok/s, against 7.14 for the dense 8B

the catch is that the whole file still has to fit in RAM, so the MoE trick is a 32 GB trick

the RAM-tier picker

every repo here was checked on Hugging Face on 17 november 2026, file sizes are the GGUF files themselves, and all are apache-2.0 on the card:

  • 8 GB RAM: mradermacher/Huihui-Qwen3.5-4B-abliterated-GGUF at Q4_K_M (2.71 GB, 152,114 downloads), or llmfan46/gemma-4-E2B-it-ultra-uncensored-heretic-GGUF at Q4_K_M (3.43 GB, 14,926 downloads)

  • 12 GB RAM: llmfan46/gemma-4-E4B-it-ultra-uncensored-heretic-GGUF at Q4_K_M (5.34 GB, 144,990 downloads), and its card reports 3/100 refusals against 99/100 for the original, at a KL divergence of 0.0076 (the uploader's numbers, so check them with your own test)

  • 16 GB RAM: mradermacher/Huihui-Qwen3.5-9B-abliterated-GGUF at Q4_K_M (5.63 GB, 477,931 downloads), or the Llama 3.1 8B abliterated from the picker above (4.92 GB)

  • 24 to 32 GB RAM: huihui-ai/Huihui-gemma-4-26B-A4B-it-qat-q4_0-unquantized-abliterated-GGUF at Q4_K (16.80 GB, 39,981 downloads), an MoE

  • 32 GB and up: the 35B-A3B from the picker above at Q4_K (21.71 GB)

the rule from the GPU picker still applies, just with RAM: the file plus the OS plus your browser plus the context window must fit, with room left over

on an 8 GB laptop, the OS and the browser are already eating a big slice of that, so go one tier down

the swap trap

if the model doesn't fit in RAM, it gets read from your SSD instead, and Big Iron says speed "falls by an order of magnitude"

so never "make it fit" with swap

llama.cpp memory-maps the file by default, which is fine when it fits, and the v1 server README has -lm mlock (load mode) to "force system to keep model in RAM" once you know it does

an old GPU: split the work

a 4 to 6 GB card still helps, as long as you split the model instead of forcing all of it onto the card

  • Ollama splits automatically, and ollama ps shows it in the PROCESSOR column, like "48%/52% CPU/GPU"

  • old Nvidia cards: Ollama's docs want compute capability 5.0 or higher and driver 550 or newer, and cards between 5.0 and 6.2 (the GTX 10-series is 6.1) need driver 570 or newer

  • llama.cpp, dense model: -ngl N puts N layers on the GPU and keeps the rest on the CPU, so raise N until VRAM is almost full

  • llama.cpp, MoE model: -ngl all with --n-cpu-moe N keeps the expert weights of the first N layers in system RAM, and --cpu-moe keeps all of them there, so the small card only holds the parts every token touches

if ollama ps says "100% CPU" on a machine with a GPU, your driver or backend isn't being picked up, and that's the first thing to fix

an iGPU or a base Mac

  • Intel or AMD integrated graphics: both Ollama and llama.cpp have a Vulkan backend, and Ollama's docs say it's on by default once the backend is installed (Linux needs the extra Vulkan packages)

  • the iGPU shares the same RAM, so the bandwidth limit doesn't go away... run llama-bench with and without it, and keep whichever is faster on your machine

  • base Apple silicon Macs: Metal is on by default in llama.cpp and Ollama, and unified memory means you use the RAM tiers above with your Mac's total memory, minus what macOS needs

the copy-paste setup (Linux, llama.cpp)

  1. check your CPU flags, cores and RAM
grep -o -w -E 'avx2|avx512f' /proc/cpuinfo | sort -u
lscpu | grep -E 'Model name|Core|Thread'
free -h

2. install a current llama.cpp, using the one-liner from its README

curl -LsSf https://llama.app/install.sh | sh

3. download the model for your tier, for example the 12 GB pick

hf download llmfan46/gemma-4-E4B-it-ultra-uncensored-heretic-GGUF --include "*Q4_K_M*" --local-dir ./models

4. find your thread knee, because the fastest thread count is often fewer than your cores

llama-bench -m ./models/<the-file>.gguf -t 2,3,4,6 -p 0 -n 64

pick the lowest -t where tok/s stops going up

5. serve it on loopback with a small context and an 8-bit cache

llama-server -m ./models/<the-file>.gguf -c 8192 -t <your-knee> -ngl 0 --jinja -ctk q8_0 -lm mlock --host 127.0.0.1 --port 8080 --api-key-file ./llama-keys.txt

-ngl 0 is the pure-CPU setting... on an old card, raise it, or add --n-cpu-moe for an MoE

6. or do it with Ollama, which is simpler on a small box

OLLAMA_CONTEXT_LENGTH=8192 OLLAMA_KV_CACHE_TYPE=q8_0 OLLAMA_KEEP_ALIVE=-1 ollama serve
ollama pull hf.co/llmfan46/gemma-4-E4B-it-ultra-uncensored-heretic-GGUF:Q4_K_M

OLLAMA_KEEP_ALIVE=-1 keeps the model loaded, so the box answers the first request without a cold load, and the q8_0 cache halves context memory by Ollama's own FAQ

7. run the same local curl test from the serving section, then the same three tunnel tests

for llama.cpp, the tunnel's service line becomes http://localhost:8080, the version check becomes /health, and every /v1/messages call also needs the x-api-key header with a key from your llama-keys.txt (pulled from your secret manager, never typed into a chat)

the always-on box behind your tunnel

this is where an old machine earns its place

put it in a corner, run Ollama as its system service or llama-server inside tmux, and point cloudflared at it

nothing about the Access application, the service token or the origin check changes... the door doesn't care how fast the room behind it is

two settings matter more on a slow box:

  • leave OLLAMA_NUM_PARALLEL at its default of 1, so two jobs don't split an already thin bandwidth

  • always stream, because Cloudflare gives up with a 524 when the origin sends nothing for 125 seconds

and here's the catch with that 524: reading a long prompt on a CPU follows compute, not bandwidth, and a slow box can spend a long time reading a big input before it writes a single token

so keep the inputs small and chunk long files, instead of feeding a whole log in one request

what a CPU box is good for, honestly

good for:

  • summarizing or classifying a log, an email or a note you don't want leaving your network

  • a first draft of a scene or a paragraph that a cloud model would refuse

  • overnight batch jobs, one file per request, results written to the log folder

not good for:

  • interactive agent loops, where every turn re-reads a long system prompt

  • long-context work, because the context cache eats both RAM and speed

  • coding agents, and the same goes for the Claude Code worker, because Ollama's Claude Code guide asks for 64k of context, and a 4B on a laptop isn't the model you want doing that job

so on a CPU box, make the worker lighter: the local-lane script can skip claude and send one streamed /v1/messages request with curl, the same request as test 3 of the tunnel section, with your task as the message

no tools, no agent loop, same read-only rule, and the lead model still reviews everything that comes back

one last thing: run your refusal set and your 20 normal prompts on the small model too, because a 4B that answers everything is not the same as a 4B that answers well


how to serve it locally

pick your server by your hardware, and keep it simple:

  • one GPU or a Mac: llama.cpp, or Ollama if you want the easiest path

  • two or more GPUs: vLLM, because it splits one model across cards with tensor parallelism

  • whatever you pick, bind it to 127.0.0.1, because the outside world reaches it through the tunnel in the next section, never directly

and run the server so it survives you closing the terminal

Ollama on Linux installs as a system service, and for llama.cpp or vLLM over SSH, start it inside tmux so a dropped connection doesn’t kill it

文章配图

one more thing before the commands: GGUF versus safetensors

GGUF is the single-file quantized format that llama.cpp and Ollama run, and it’s what all the “-GGUF” repos above ship

safetensors is the original full-precision format that vLLM wants

so the rule is simple: GGUF goes to llama.cpp or Ollama, safetensors goes to vLLM

option 1: Ollama (the easiest, and Claude Code can talk to it directly)

1. install Ollama from ollama.com, and leave it on its default address, 127.0.0.1:11434

2. start it with a real context window, because Ollama’s Claude Code guide asks for 64k or more

OLLAMA_CONTEXT_LENGTH=65536 OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 ollama serve

what those three do:

  • OLLAMA_CONTEXT_LENGTH sets the default context window for every model

  • OLLAMA_FLASH_ATTENTION=1 forces Flash Attention on (Ollama already uses it automatically when your hardware supports it), which cuts memory as context grows

  • OLLAMA_KV_CACHE_TYPE=q8_0 stores the context cache in 8 bits, which Ollama’s FAQ says uses about half the memory of the default f16 with “a very small loss in precision”

3. pull the model straight from Hugging Face, using the quant name from the file

ollama pull hf.co/huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF:UD-DW-Q4_K_M

Ollama takes hf.co/{user}/{repo}:{quant}, and Hugging Face’s docs say it picks Q4_K_M by default when the repo has one

4. check that it can call tools, because any agent setup needs that

ollama show hf.co/huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF:UD-DW-Q4_K_M

you want to see tools in the capabilities list

if it’s missing, Claude Code fails with “does not support tools”, and we hit that exact error testing a small abliterated Qwen on our own box… the fix was adding the official Qwen3 chat template in a Modelfile and running ollama create

5. test the local endpoint, because Ollama serves an Anthropic-compatible /v1/messages

curl -sS http://127.0.0.1:11434/v1/messages -H ‘Content-Type: application/json’ -H ‘x-api-key: ollama’ -H ‘anthropic-version: 2023-06-01’ -d ‘{“model”:“hf.co/huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF:UD-DW-Q4_K_M”,“max_tokens”:200,“messages”:[{“role”:“user”,“content”:“explain abliteration in two sentences”}]}’

6. check what’s loaded and how much memory it’s using

ollama ps

now burn this into your brain: Ollama’s docs say the local server does not validate API keys

ollama in that header is a placeholder, and anyone who can reach port 11434 can use your model

that’s the whole reason the next section exists

option 2: llama.cpp (one GPU, the most control)

llama-server binds 127.0.0.1 by default, serves OpenAI-compatible endpoints plus an Anthropic-compatible /v1/messages, and supports API keys

llama-server -m ./models/Huihui-Qwen3.8-27B-abliterated-UD-DW-Q4_K_M.gguf -c 65536 -ngl all -fa on –jinja -ctk q8_0 –host 127.0.0.1 –port 8080 –api-key-file ./llama-keys.txt –log-file ./logs/llama-server.log

one version note first: llama.cpp hit v1.0.0 on 13 november, so use a current build, and the release notes say it hardens llama-server against DNS rebinding, CORS and SSRF problems

what every flag does, from the server README:

  • -m is the model file

  • -c 65536 is the context size

  • -ngl all puts all layers on the GPU (the default is auto)

  • -fa on turns Flash Attention on (the default is auto)

  • –jinja uses the model’s own chat template, and the README says tool use needs it

  • -ctk q8_0 stores the K cache in 8 bits to save memory

  • –host and –port keep it on loopback

  • –api-key-file reads keys from a file, one per line

  • –log-file writes the server log to disk

and one flag you never add here: llama-server now has experimental built-in agent tools (–tools, –agent) that include shell commands and file writes, and the README itself says “do not enable in untrusted environments”… an uncensored model is exactly that environment, so leave the default, which is no tools

option 3: vLLM (two or more GPUs)

vLLM serves an OpenAI-compatible API, and its README also lists an Anthropic Messages API

uv pip install vllm vllm serve huihui-ai/Huihui-Qwen3.8-27B-abliterated –host 127.0.0.1 –port 8000 –tensor-parallel-size 2 –max-model-len 65536 –api-key “$VLLM_API_KEY”

what those do:

  • –tensor-parallel-size 2 splits the model across two GPUs

  • –max-model-len caps the context, which is the main memory lever in vLLM’s docs

  • –api-key (or the VLLM_API_KEY environment variable) turns on key checks

and two warnings straight from vLLM’s own docs:

  • –api-key only protects paths under /v1, /v2, /inference and /cohere, and other endpoints like /invocations are not authenticated, so don’t rely on the key alone

  • GGUF support in vLLM is “highly experimental” and now lives in a separate plugin, so give vLLM the safetensors repo, not the GGUF

that first warning is the same lesson as Ollama’s: the key on the server is not your security, the door in front of it is


reach it from anywhere: Cloudflare Tunnel + Access

here’s the problem

your GPU sits at home, and you want it from your laptop at a cafe, your phone, or an agent running in the cloud

the bad fix is opening a port on your router, and an unauthenticated uncensored model on the open internet is the worst version of this whole build

if every device that calls the model is yours, a private tailnet is the simplest answer, and it’s a great one

but if your agent runs somewhere you don’t control, or you want a proper token per machine, Cloudflare Tunnel with Cloudflare Access is the cleaner fit

here’s why it works:

  • cloudflared makes an outbound-only connection from your GPU box to Cloudflare (Cloudflare’s docs say to check port 7844 is reachable), so nothing on your box listens on the internet

  • Cloudflare Access sits in front of the hostname and rejects every request that doesn’t carry a valid token

  • cloudflared itself can check the Access token again before it forwards anything to your model

文章配图

here’s the full request path, in order, so the diagram reads like the steps:

  1. your agent looks up llm.your-domain, and DNS points it at Cloudflare (the tunnel route creates a CNAME to your tunnel)

  2. your agent sends an HTTPS request with two headers, CF-Access-Client-Id and CF-Access-Client-Secret

  3. Cloudflare’s edge checks those against your Access application, and the Service Auth policy decides… no valid token means a 401 or 403 and the request dies right there

  4. allowed requests get an Access JWT attached (the Cf-Access-Jwt-Assertion header) and go down the tunnel to cloudflared on your GPU box

  5. cloudflared matches the hostname against the ingress rules in config.yml, checks the JWT itself, sets the Host header to localhost, and forwards to 127.0.0.1:11434

  6. the model answers, and the response streams back up the same path to your agent

and the order you build it matters more than anything else in this section

Cloudflare’s own docs say to create the Access application before the tunnel route, because without it “the published application will be available to anyone on the Internet”

lock first, door second

step 1: create a service token

your agent is a machine, so it uses a service token, not an SSO login

  1. in the Cloudflare dashboard, go to Zero Trust, then Access controls, then Service credentials, then Service Tokens

  2. select Create Service Token and name it llm-agent

  3. pick a short Service Token Duration (i’d start with 30 days)

  4. select Generate token, then copy the Client ID and Client Secret straight into your secret manager

Cloudflare shows the Client Secret exactly once, so if you lose it, you make a new token

and never paste it into a chat, a note, a screenshot or a repo… the rest of this guide uses placeholders like on purpose

step 2: create the Access application

  1. go to Zero Trust, then Access controls, then Applications, then Create new application

  2. select Self-hosted and private, then Add public hostname, and enter llm.

  3. add a policy with the action Service Auth that includes only your llm-agent token

  4. in Additional settings, turn on “401 Response for Service Auth policies” so a bad token gets a clean 401 instead of a login page redirect

  5. select Create, then copy the application’s AUD tag for step 4

Access is deny by default, so if a request doesn’t match a policy, it doesn’t get in

want the model in your browser too? add a second policy with the action Allow, your SSO identity provider, and only your own email

machines on one policy, humans on another, never mixed

step 3: create the tunnel

cloudflared tunnel login cloudflared tunnel create local-llm cloudflared tunnel list

login opens a browser so you can pick your domain, and writes a cert.pem into your .cloudflared folder

create writes a credentials JSON into the same folder and prints the tunnel UUID… treat that file like a key

step 4: write config.yml

create ~/.cloudflared/config.yml:

(X strips indentation, so here is the same config in YAML flow style, which needs no indentation and parses identically)

tunnel: credentials-file: /home//.cloudflared/.json ingress: [{hostname: llm., service: “http://localhost:11434”, originRequest: {httpHostHeader: localhost, access: {required: true, teamName: , audTag: []}}}, {service: “http_status:404”}]

the four lines doing the real work:

  • service: http://localhost:11434 is where cloudflared forwards allowed requests (use 8080 for llama.cpp or 8000 for vLLM)

  • httpHostHeader: localhost matters for Ollama specifically, because Ollama on loopback rejects requests whose Host header isn’t local (that check is in Ollama’s server code), so without this line you’d get a 403 from Ollama itself

  • access.required: true makes cloudflared validate the Access JWT before forwarding anything, which Cloudflare calls “Protect with Access”, so even if something skipped the edge, your origin still says no

  • service: http_status:404 at the end is the catch-all Cloudflare requires, and it gives every other hostname nothing

then validate it before you run anything:

cloudflared tunnel ingress validate cloudflared tunnel ingress rule https://llm.

the second command tells you which rule a URL would hit, which is the fastest way to catch a typo

step 5: route and run

cloudflared tunnel route dns local-llm llm. cloudflared tunnel run local-llm cloudflared tunnel info local-llm

route dns creates the CNAME, run connects the tunnel, and info shows its connections

once it works, install cloudflared as a service so it survives reboots, Cloudflare documents that for Linux, Windows and macOS

step 6: prove the lock works

run this from a different network, like your phone’s hotspot

1. no token: must fail with 401 or 403
curl -i https://llm./api/version # 2. with token: must return Ollama’s version JSON curl -i https://llm./api/version -H “CF-Access-Client-Id: ” -H “CF-Access-Client-Secret: ” # 3. a real generation through the tunnel, streamed curl -sS https://llm./v1/messages -H “CF-Access-Client-Id: ” -H “CF-Access-Client-Secret: ” -H ‘Content-Type: application/json’ -H ‘anthropic-version: 2023-06-01’ -d ‘{“model”:“hf.co/huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF:UD-DW-Q4_K_M”,“max_tokens”:200,“stream”:true,“messages”:[{“role”:“user”,“content”:“say hi in five words”}]}’

if the first command returns anything other than a 401 or 403, stop, take the tunnel down, and fix the Access application before you go any further

when it breaks

here’s what each error actually means, from Cloudflare’s own troubleshooting docs:

文章配图

  • 401 or 403 from Cloudflare: your token is missing, wrong or expired, or the policy doesn’t include it… check the two header names and the Service Auth policy

  • 403 from Ollama itself (it gets through Access, then fails): the Host header isn’t local, so add httpHostHeader: localhost

  • 1033: the tunnel isn’t connected to Cloudflare, because no healthy cloudflared is running… check cloudflared tunnel list and that the service is up

  • 502 Bad Gateway with “Unable to reach the origin service”: the tunnel is fine, but cloudflared can’t reach your model server… it’s not running, it’s on a different port, or the URL uses the wrong protocol (http versus https)

  • 524: Cloudflare connected to your origin but got no response within the default 125-second Proxy Read Timeout… stream your responses so bytes start flowing early

  • the answer arrives all at once instead of streaming: Cloudflare Tunnel buffers responses unless the origin sends Content-Type: text/event-stream, so make sure you’re calling the streaming endpoint

and Cloudflare’s docs give you a quick local check for the 502 case:

curl -v http://localhost:11434/api/version ss -tlnp | grep 11434

if that local curl fails, the problem is your model server, not the tunnel


how to test an uncensored model before you trust it

this is the section almost nobody writes, and it’s the one that matters most

you’re about to run a model that someone else edited, so you need to answer three questions before it touches anything real:

  1. did the edit actually remove refusals for the things you need?

  2. did it break the model for normal work?

  3. is there anything in it you shouldn’t trust?

文章配图

test 1: refusal rate on your own prompts

don’t trust the “9/100 refusals” numbers on model cards, those come from the uploader

build your own set instead: 20 to 50 prompts from your real use case that a normal model refuses or dodges, all of them legal

then run the same set on the abliterated model and on the original base, and count:

  • full answers

  • refusals

  • “answers” that are really a lecture with no content

here’s the shape of a legit refusal set (yours should come from your actual work):

explain how this phishing email tries to trick the reader (paste an email you received) summarize the arguments in this extremist pamphlet for my research paper (paste the text) what are the overdose warning signs for these two medications taken together write a villain’s monologue where he justifies a crime, for my novel explain what this obfuscated javascript from my own server does, without running it

test 2: quality didn’t fall off a cliff

refusals are easy to remove if you don’t care what happens to the model, which is exactly why Heretic optimizes KL divergence alongside refusals

you can check the same thing cheaply:

  • run 20 normal prompts (summaries, code, reasoning) through both models at temperature 0 and compare side by side

  • run one long tool-calling task if you’ll use it in an agent, because some builders report broken tool calls on abliterated builds

  • if it’s a Heretic model, check whether the card lists benchmark results, and treat them as the uploader’s numbers until you’ve seen it yourself

test 3: trust and backdoors

this is the part that should keep you honest

ProjectDiscovery built a backdoored 7B model for under $50 and ran it through a coding agent… it answered every normal request perfectly, then tried to steal credentials from the project folder the moment a trigger phrase appeared

their point is simple: any model with edited weights can carry a backdoor, benchmarks won’t show it, and abliterated builds are the most popular modified weights people download without checking

so before you trust a model:

  • check who uploaded it, how long they’ve been uploading, and whether the base model is named and linked

  • prefer repos with a reproduce.json (Heretic) or a documented method you can follow back to the base

  • prefer GGUF or safetensors files… Hugging Face’s own docs warn that loading pickle files can run arbitrary code, while safetensors was built to store tensors safely

  • remember that a safe file format doesn’t mean a safe model, the backdoor lives in behavior, not in the file

  • never give it shell, write or network tools, so even a backdoored model has nothing to reach

decide

after the three tests, you make one of three calls:

  • keep it for the jobs where it beat the base and didn’t break

  • keep it, but only for text-only jobs with no tools

  • delete it

write the result down with the date, the file name and the quant, because you’ll forget which build was which within a week


plug it in as a subagent

this is where it gets fun

your main agent stays on a frontier model, and the uncensored model becomes a read-only helper for the jobs where refusals get in the way

both lead models i’d use are real and current:

  • Claude Fable 5.2, released 9 november, which Anthropic says can act as an orchestrator that delegates simpler tasks to smaller models (Claude Code alias fable)

  • Claude Opus 5.5, which Anthropic calls its “strongest daily driver” (Claude Code alias opus, and the docs use claude-opus-5-5 as the full ID)

here’s the catch most people miss

a Claude Code subagent’s model field only accepts sonnet, opus, haiku, fable, a full Claude model ID, or inherit… you can’t point a subagent definition at your GPU

so the uncensored lane is a second, isolated Claude Code process that your lead calls through one allowed script

we proved the core of this on our own box on 14 november: a parent script started an isolated claude –bare -p pointed at a local Ollama endpoint, it answered on an abliterated Qwen, and the parent got the result back in about 26 seconds

the new part in this guide is the tunnel, and Claude Code can send extra headers with ANTHROPIC_CUSTOM_HEADERS (v2.1.227 or later), which is exactly how the two Access headers ride along

文章配图

here’s the call path, in order:

  1. you give your lead model a task

  2. the lead hands it to the local-lane subagent, defined in .claude/agents/local-lane.md

  3. the subagent runs ./scripts/local-lane.sh, the only command .claude/settings.json allows it

  4. the script starts an isolated worker (claude –bare -p) with no tools, which sends the request through the tunnel to your model

  5. the model’s response comes back to the worker, and the script saves the prompt and the output to logs/local-lane/

  6. the subagent returns a summary to the lead, so the raw output stays out of your main context

  7. the lead checks it against your files and tells you what it verified and what it couldn’t

  8. you decide what gets used, and nothing ships on its own

file 1: the worker script

save as scripts/local-lane.sh and make it executable:

#!/usr/bin/env bash # local lane: uncensored model behind Cloudflare Access. read-only, no tools. set -euo pipefail LOG_DIR=“./logs/local-lane”; mkdir -p “$LOG_DIR” STAMP=$(date +%Y%m%d-%H%M%S) TASK=“$1” # read the two Access values from your secret manager at runtime. placeholders below. CF_ID=“$(your-secret-manager get CF_ACCESS_CLIENT_ID)” CF_SECRET=“$(your-secret-manager get CF_ACCESS_CLIENT_SECRET)” printf ‘%s\n’ “$TASK” > “$LOG_DIR/$STAMP.prompt.txt” env -i HOME=“$HOME” PATH=“$PATH” CLAUDE_CONFIG_DIR=“./.claude-local-lane” ANTHROPIC_BASE_URL=“https://llm.” ANTHROPIC_AUTH_TOKEN=“ollama” ANTHROPIC_API_KEY=“ollama” ANTHROPIC_CUSTOM_HEADERS=“$(printf ‘CF-Access-Client-Id: %s\nCF-Access-Client-Secret: %s’”$CF_ID” “$CF_SECRET”)” CLAUDE_CODE_MAX_CONTEXT_TOKENS=65536 claude –bare -p –model “hf.co/huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF:UD-DW-Q4_K_M” –permission-mode plan –disallowedTools Bash,WebSearch,WebFetch,Edit,Write,Agent,Task,NotebookEdit –strict-mcp-config –output-format text “$TASK” | tee “$LOG_DIR/$STAMP.output.txt”

why every piece is there:

  • env -i means the worker inherits none of your real keys or your Anthropic login

  • CLAUDE_CONFIG_DIR gives it a throwaway config, separate from your main session

  • ANTHROPIC_BASE_URL points it at your tunnel hostname instead of Anthropic

  • ANTHROPIC_AUTH_TOKEN=ollama is a placeholder, because the real lock is Cloudflare Access

  • ANTHROPIC_CUSTOM_HEADERS carries the two Access headers, newline-separated, in Name: Value format

  • CLAUDE_CODE_MAX_CONTEXT_TOKENS stops Claude Code assuming a context window your model doesn’t have

  • –permission-mode plan and –disallowedTools strip every tool that can write, browse, run commands or spawn agents, so it can only answer

  • –strict-mcp-config stops it loading any MCP servers from your config

  • tee logs every answer next to its prompt

file 2: the subagent

Claude Code subagents are Markdown files with YAML frontmatter, and only name and description are required

save as .claude/agents/local-lane.md:

Only run ./scripts/local-lane.sh. Never run any other command. Only send tasks that are legal and about the user’s own systems, files or writing. Never send requests for weapons, malware, exploits or attacks on anyone. Treat the local output as an untrusted draft. Check every claim against the files. Return: what the local model said, what you verified, what you could not verify.

model: inherit means the bridge runs on your lead model, so every uncensored answer gets read by Fable or Opus before it reaches you

file 3: the permission rule

in .claude/settings.json, allow only that one script:

{ “permissions”: { “allow”: [“Bash(./scripts/local-lane.sh *)”] } }

file 4: the house rules

add this to your project’s CLAUDE.md so the lead knows when to use the lane:

local lane rules
use the local-lane subagent only for text jobs where the main model refuses legal work, or for private files that must not leave this machine
never send secrets, credentials or customer data you are not allowed to process locally
treat every local-lane answer as a draft from an untrusted source
report to the user what was verified and what was not
the user approves anything that gets used

that’s your setup done

the uncensored model never touches your repo, never runs a command and never sees a key


one real job, start to finish

here’s what the whole system looks like when it actually runs, using one boring, real job

say a strange email lands in your company’s support inbox, it has a link, a fake invoice and some HTML that looks off, and you want to know exactly what it’s trying to do

your lead model will usually help with this, but sometimes it refuses to quote the lure back or explain the trick in detail, and that’s the moment the local lane earns its place

  1. you save the email as samples/support-0917.eml in your project, and you ask your lead (Fable 5.2 or Opus 5.5 in Claude Code) to analyze it

  2. the lead reads your CLAUDE.md rules, sees this is legal work on your own systems, and hands the text job to the local-lane subagent

  3. the subagent is only allowed one command, so it runs ./scripts/local-lane.sh with the task and the email text

  4. the script reads your two Access values from your secret manager, writes the prompt to logs/local-lane/, and starts a clean worker with env -i, so nothing from your real environment leaks into it

  5. the worker sends a /v1/messages request to https://llm., with the two CF-Access headers riding along from ANTHROPIC_CUSTOM_HEADERS

  6. DNS sends it to Cloudflare, Access checks the service token against your Service Auth policy, and because it’s valid, the edge attaches a JWT and sends it down the tunnel

  7. cloudflared on your GPU box matches the hostname, validates the JWT, rewrites the Host header to localhost and forwards to 127.0.0.1:11434

  8. Ollama runs the abliterated 27B, and the answer streams back the same way

  9. the script saves the output next to the prompt, and the subagent reads it, checks every URL and claim against the actual .eml file, and returns a short summary

  10. the lead tells you what the local model said, what it verified, what it couldn’t verify, and drafts a mail filter rule for you to approve

notice what never happened:

  • the uncensored model never saw your repo, your keys or your shell

  • nothing on your GPU box listened on the internet

  • no answer reached you without a frontier model checking it first

  • nothing changed anywhere until you approved it

and if anything in that chain breaks, you know exactly where to look, because every hop has its own error: 401 or 403 at Access, 1033 when the tunnel is down, 502 when the model server is down, and 524 when it takes too long


when not to use the local lane

i’ll be straight with you, most of your work should never touch this lane

the local model is weaker, slower and less trustworthy than your lead, so the lane exists for two narrow reasons only:

  • the lead refuses legal work you actually need done

  • the data must not leave your machine

everything else stays on the lead, and here’s where i’d never route to the local model:

  • anything that writes code into your repo, because it can’t be trusted with edits

  • anything that needs tools, browsing or fresh facts, because the worker has none on purpose

  • anything you’d ship to a customer without a human read

  • anything that targets a system, a person or an account you don’t own

and if you catch yourself sending more and more to the local lane because “it doesn’t argue”, stop and ask why

a model that agrees with everything isn’t more useful, it’s just quieter about its mistakes


the tailnet option, if every device is yours

if you only ever call the model from your own laptop and phone, you can skip Cloudflare entirely

put the GPU box and your devices on the same tailnet (Tailscale is the usual pick), keep the model server on 127.0.0.1, and publish it only to the tailnet with Tailscale’s own serve feature (check Tailscale’s docs for the exact command), so nothing is ever on the public internet

the trade-off is simple:

  • tailnet: fewer moving parts, no public hostname, but every calling device has to join your network

  • Cloudflare Tunnel with Access: a public hostname with a lock, per-machine tokens, logs and rate limits, and it works for agents running in places you don’t control

you can also run both, a tailnet for your own devices and an Access-protected hostname for cloud agents, as long as the model server itself still only listens on loopback


what it’s actually good for

here’s the honest list, and every item on it is legal, mostly boring and genuinely useful:

  • fewer false refusals: medical, legal, drug-safety and history questions that safety-tuned models dodge or bury under warnings

  • fiction: dark themes, villains with real motives and violence on the page, without a lecture every paragraph

  • security research on your own systems: reading a phishing email sent to your domain, explaining an obfuscated script found on your own server, drafting detection rules

  • research: reading extremist, scam or propaganda material for a paper or a report without the model refusing to summarize it

  • privacy: your notes, logs and documents never leave your machine

  • awareness training for your own company: phishing simulations and tabletop exercises, with permission

here are example briefs for each, written the way you’d actually send them:

fiction:

write a 600-word scene where the antagonist explains to his brother why he burned the warehouse. he believes he’s right. no moralizing narrator. keep the violence off-page but make the threat real.

medical and drug safety:

my father takes warfarin and was just prescribed a common antibiotic. list the interaction risks a pharmacist would warn about, the symptoms that mean “go to the ER now”, and the questions i should ask his doctor.

research:

here is a 4-page recruitment pamphlet from a banned group (text below). summarize its persuasion techniques for my sociology paper: the grievances it targets, the promises it makes, the emotional hooks. no endorsement, no rewriting it to be more persuasive.

security on your own systems:

this email was sent to my company’s support address (saved as samples/support-0917.eml). explain what it tries to make the reader do, list every URL and domain, and point out which parts of the HTML hide the real link. do not visit any URL. output a summary, a list of indicators, and a draft rule for our mail filter.

privacy:

read the 40 pages of my therapy journal in notes/2026/ and give me the three themes that come up most, with dates. nothing leaves this machine.

and what it’s not for is anything that hurts people

the model losing its “no” doesn’t mean the law lost its “no”


i’ll be straight with you, this lane has sharp edges, and most of them cut the person running it

quality:

  • abliteration removes refusals, it doesn’t add skill, so a 27B local model is clearly weaker than Fable 5.2 or Opus 5.5

  • removing the refusal direction can hurt quality, and huihui’s own cards call plain ablation crude

  • heavy quants (Q2, Q3) make it worse again, so test at the quant you’ll actually run

trust:

  • edited weights can carry backdoors you can’t see, which is the whole ProjectDiscovery result

  • uploaders change files, so pin the exact file name and note the date you pulled it

  • the safest model is the one that can’t do anything, which is why the worker has no tools

licenses:

  • read every model card: apache-2.0 on the Qwen and Gemma Heretic builds above, the Llama 3.1 license on the 8B Llama builds, and they aren’t the same thing

  • Heretic itself is AGPL-3.0, which matters if you build a product on the tool, not if you only use its output

  • if you ship anything built on these models, the base model’s license still applies

logging and privacy:

  • local means your prompts stay on your disk, and that also means protecting and deleting them is your job

  • your lead model is a normal cloud API call, so don’t send it data you can’t share

  • log prompts and outputs, never token values

legal:

  • uncensored doesn’t mean legal, and what you ask for and what you do with the answer is on you

  • the lead model’s usage policy still applies to everything that passes through it

  • if you’re unsure whether something is legal where you live, that’s a question for a lawyer, not a model


the rules i’d tape to the monitor

here’s the short version of everything above, the rules that keep this lane safe and useful:

  1. the model server only ever listens on 127.0.0.1

  2. no hostname goes live without an Access application in front of it

  3. machines get service tokens, humans get SSO, and never the same policy

  4. every secret lives in a secret manager and gets read at runtime

  5. every model gets your own three tests before it touches real work

  6. the uncensored model never gets shell, write or network tools

  7. a frontier model reads every uncensored answer before you do

  8. every prompt and output gets logged, and no token ever does

  9. you approve anything that gets used, sent or shipped

  10. if a request would hurt someone, it doesn’t go to any model

if you only remember one, remember number 6

a model with no refusals and no tools is a writing assistant… a model with no refusals and a shell is an incident waiting for a trigger phrase


the hardening checklist

文章配图

run this once when you set up, then every week:

model server:

  • bound to 127.0.0.1, never 0.0.0.0

  • OLLAMA_NUM_PARALLEL left at its default of 1 unless you need more (a natural throttle)

  • for vLLM, remember –api-key doesn’t cover every path

  • for llama-server, never –tools or –agent (shell and file-write tools)

tunnel and Access:

  • no public hostname without an Access application in front, ever

  • Access application created before the tunnel route

  • the Service Auth policy names one token, not everyone

  • “401 Response for Service Auth policies” turned on

  • strict service token authentication turned on in your Zero Trust settings, so requests with token headers only pass Service Auth policies and fail with 401 or 403 instead of redirecting

  • access.required: true in config.yml, so cloudflared checks the JWT too

  • a catch-all http_status:404 as the last ingress rule

tokens:

  • short Service Token Duration

  • Cloudflare’s expiring service token alert set up

  • rotate on a schedule, and immediately if you ever see a request you don’t recognize

  • secrets in a secret manager, never in a file in your repo

limits and logs:

  • a WAF rate limiting rule on llm., so a leaked token can’t turn your GPU into someone’s free API (check what your plan includes)

  • read your Access logs in the Zero Trust dashboard every week

  • local-lane prompts and outputs logged on your disk, never token values

the model itself:

  • passed your own three tests

  • uploader and base model checked

  • no shell, write or network tools, ever


every repo in this guide

all checked on the GitHub API on 17 november 2026, around 3:35 PM IST, with the last commit on the default branch

abliteration and uncensoring:

  • p-e-w/heretic, 41,125 stars, last commit 16 nov 2026, AGPL-3.0: automated, training-free model editing that removes refusals and can also cut slop and sycophancy

  • FailSpy/abliterator, 829 stars, last commit 7 jun 2024, MIT: small Python library to find and ablate features like refusal with TransformerLens

  • andyrdt/refusal_direction, 492 stars, last commit 1 oct 2024, Apache-2.0: code and results for the Arditi et al. refusal-direction paper

  • Sumandora/remove-refusals-with-transformers, 2,378 stars, last commit 27 nov 2025, Apache-2.0: refusal removal with plain Hugging Face Transformers

  • Orion-zhen/abliteration, 206 stars, last commit 17 nov 2026, GPL-3.0: makes abliterated models with Transformers

  • heterodoxin/AbliteratedOutTheBox, 218 stars, last commit 22 oct 2026, MIT: llama.cpp fork that abliterates at inference time instead of editing weights

  • mlabonne/llm-course, 85,135 stars, last commit 5 feb 2026, Apache-2.0: LLM course with Colab notebooks, including the abliteration notebook

local serving:

  • ollama/ollama, 187,329 stars, last commit 17 nov 2026, MIT: runs local models with one command, with OpenAI- and Anthropic-compatible APIs

  • ggml-org/llama.cpp, 135,416 stars, last commit 17 nov 2026, MIT: C/C++ inference for GGUF models, and llama-server gives you an HTTP API

  • vllm-project/vllm, 97,091 stars, last commit 17 nov 2026, Apache-2.0: high-throughput serving engine that splits models across GPUs

  • mudler/LocalAI, 50,406 stars, last commit 17 nov 2026, MIT: self-hosted OpenAI-compatible engine for LLMs, image and voice

  • LostRuins/koboldcpp, 12,331 stars, last commit 2 nov 2026, AGPL-3.0: one-file GGUF runner with a KoboldAI UI, popular for fiction

  • ml-explore/mlx-lm, 7,671 stars, last commit 16 nov 2026, MIT: runs LLMs on Apple silicon with MLX

  • PurpleDoubleD/locally-uncensored, 4,335 stars, pushed 17 nov 2026, AGPL-3.0: an all-in-one local AI studio (chat, image, video, coding agent) for Windows and Linux, the one from the Product Hunt launch

  • synchopate/llama-modchip, 1,329 stars, pushed 17 nov 2026, Apache-2.0: runs a 284B DeepSeek on 64 GB of RAM with no GPU (the 487-point HN thread)

  • ikawrakow/ik_llama.cpp, 3,401 stars, pushed 17 nov 2026, MIT: a llama.cpp fork with extra quant types and CPU-focused performance work, worth a look once the stock build works

  • lmstudio-ai/lms, 5,434 stars, last commit 9 nov 2026, MIT: the CLI for LM Studio (the app itself is closed source)

reaching it:

  • cloudflare/cloudflared, 16,626 stars, last commit 16 nov 2026, Apache-2.0: the Cloudflare Tunnel client

  • tailscale/tailscale, 39,218 stars, last commit 17 nov 2026, BSD-3-Clause: a private WireGuard network for your own devices, the no-Cloudflare option

  • tmux/tmux, 50,451 stars, last commit 17 nov 2026, ISC: keeps your model server running after SSH drops

agent harnesses:

  • anthropics/claude-code, 155,971 stars, last commit 17 nov 2026, no SPDX license listed: the harness used for the lead and the worker in this guide

  • musistudio/claude-code-router, 38,196 stars, last commit 16 nov 2026, MIT: routes Claude Code requests across different model providers

  • BerriAI/litellm, 63,400 stars, last commit 17 nov 2026, open core (the enterprise folder is licensed separately): an AI gateway that maps model names to providers, including local ones

  • anomalyco/opencode, 227,444 stars, last commit 17 nov 2026, MIT: open-source coding agent with per-agent models, including subagents

  • NousResearch/hermes-agent, 264,043 stars, last commit 17 nov 2026, MIT: agent harness that runs frontier and local models

  • rawveg/ollama-mcp, 179 stars, last commit 13 aug 2026, AGPL-3.0: an MCP server that exposes Ollama as a tool to any MCP agent

dropped on purpose: jim-plus/llm-abliteration is a fork of Orion-zhen/abliteration, so only the original is listed, and the GitHub search for new “uncensored” repos since october is full of paid API resellers and “jailbreak prompt” collections, none of which belong in a build like this


your first week

here’s how i’d lay out the first seven days if you were starting from zero today:

文章配图

day 1:

  • pick a model and quant from the memory tiers above

  • no GPU? use the RAM tiers in the no-GPU section, and run llama-bench to find your thread count first

  • install Ollama or llama.cpp on 127.0.0.1 and run the local curl test

day 2:

  • run your own refusal set and 20 normal prompts on the abliterated model and its base

  • write down where it’s actually better and where it’s worse

day 3:

  • create the service token, the Access application and the Service Auth policy

  • then the tunnel, config.yml and the DNS route

  • run the no-token test from your phone’s hotspot, it must fail

day 4:

  • add the four files: local-lane.sh, the local-lane subagent, the permission rule and the CLAUDE.md rules

  • give it one real job, like a long log file, a chapter of your novel or a phishing email you received

day 5:

  • set the token expiry alert, a rate limiting rule and strict service token authentication

  • read your Access logs

day 6:

  • check the uploader and base of every model on your disk

  • delete anything you can’t trace back to a known base

day 7:

  • decide which jobs stay on the local lane and which go back to the lead model, and write it into CLAUDE.md

after that, rerun the three tests every time you pull a new build


FAQ

1. is it legal to download an uncensored model?

in most places, downloading open weights under their license is legal, but what you do with the output is what matters… check the license on the card and your local law

2. is abliteration the same as a jailbreak?

no, a jailbreak tricks a model with a prompt, abliteration edits the weights so the refusal reflex is gone, and this guide uses neither for anything harmful

3. will it make the model smarter?

no, it removes refusals and can slightly hurt quality, which is why you test against the base

4. why not just open a port, or use a quick tunnel?

because Ollama doesn’t check keys, so whoever finds the port gets your model, and the whole point of Access is that nothing reaches it without a token

5. do i need Cloudflare if i only use it at home?

no, keep it on 127.0.0.1 and use it locally, or put your own devices on a tailnet

6. can the uncensored model be my main agent?

it can, but i wouldn’t, because a weaker model with tools and no refusals is the riskiest combination in this whole article

7. what if Hugging Face removes the model i use?

keep a local copy of the exact file you tested, with its name and the date, and remember this week’s “everything is being deleted” claim was corrected

8. how do i know a model isn’t backdoored?

you can’t fully know, which is why the worker gets no tools and every output gets checked

9. which one should i start with tonight?

the 27B at UD-DW-Q4_K_M if you have a 24 GB GPU, an 8B at Q4_K_M if you don’t

10. does this work on a Mac?

yes, llama.cpp and Ollama both run on Apple silicon, and mlx-lm is the Apple-native option… use the memory tiers with your unified memory, minus what macOS itself needs

  1. can i run this with no GPU?

yes, a 4B to 9B abliterated model runs on a CPU with 8 to 16 GB of RAM, and an MoE can go further with 32 GB... dual-channel RAM matters more than the CPU, and the no-GPU section has the picker and the setup

12. Cloudflare or a tailnet?

a tailnet if every device that calls the model is yours, Cloudflare Tunnel with Access if an agent runs somewhere you don’t control or you want a separate token per machine

13. can i give a friend access?

make them their own service token and their own policy, never share yours, and remember their requests are now running on your hardware under your name

14. why does my agent say the model doesn’t support tools?

the model’s chat template doesn’t expose tool calling to Ollama, so check ollama show for tools, and fix the template before you blame the model

15. why are answers so slow through the tunnel?

usually it’s the model, not the tunnel… check speed locally first with the 127.0.0.1 curl, then compare, and make sure you’re streaming

16. what’s the one thing people get wrong most?

publishing the tunnel route before the Access application exists, which Cloudflare’s own docs warn makes it “available to anyone on the Internet”

that’s the whole course

a model with no “no” needs an owner with one

全文完
查看原文

沿着这个主题继续读

从一篇文章,找到相关的教程、经验和阅读路径。

本地大模型部署本地大模型部署与硬件选型,从模型参数、显存到文字、图像和 Agent 案例。