好文集ChatGPT Plus¥119

Your AI Acts in Your Company’s Name 24/7. Who Authorizes Every Move?

文章目录

Satya Nadella is right that AI must be governed from outside the model. Netomi has already proven that architecture at Fortune 500 scale, and it is the unlock enterprise AI has been waiting for.


Earlier this month, Satya Nadella published "Models as Insider Risks in the Super Intelligence Era," arguing that companies should treat frontier AI models like insider risks. Not because models are necessarily malicious, he wrote, but because any capable actor with access to important systems can make mistakes or be compromised.

His prescription was to "separate the supply of intelligence from the authority over it." The controls on what a model can access and do belong outside the model, where it cannot bypass them, and every meaningful model action should leave "tamper-proof human readable evidence." His conclusion is the standard every enterprise should hold itself to: "The most trustworthy Super Intelligence system will not be the one with the model we trust most. It will be the one that enables us to trust the model the least."

This week, Satya said Microsoft was putting that principle into practice, expanding its Project Perception security system to continuously assess each agent's risk and adjust what it is allowed to do.

This is the right architecture for the enterprise. We built our platform on it and have run it in production for some of the world's largest companies, in the moments where failure costs the most. For more than two years, we have argued that AI must be governed from outside the model. We call this approach sanctioned superintelligence: the model is free to reason, but it acts only within authority the enterprise has explicitly granted, enforced outside the model.

Satya framed this moment as the "Super Intelligence Era," and the word matters. Superintelligence you can't sanction is superintelligence you can't deploy.

The most visible insiders talk to your customers

Satya's essay and Microsoft's launch focus on agents running inside an organization: giving each one an identity, limiting its permissions and containing it when its risk rises. That is where the security industry naturally starts. The agents the public actually sees work somewhere else.

Picture an airline on the night a storm grounds hundreds of flights. Thousands of travelers ask for the same thing at once: get me home. An AI agent can read an itinerary, search the alternatives and propose a new route in seconds.

Then a traveler asks for an exception. Can the agent waive the change fee? Issue a voucher? Upgrade a frequent flier who missed a connection? The model may understand exactly what the customer wants. Whether it has the authority to grant it is a different question, and the answer belongs to the airline, not the model.

Agents like this act in a company's name, with its customers, around the clock. Every refund, rebooking and loyalty benefit is a brand promise, kept or broken in public. When an agent tells a policyholder a claim is approved, the question is not whether the model is aligned. It is who authorized that sentence, on what terms, and who can stop the next one.

Brand policy is now code

For these agents, the controls that matter most are business policies: refund limits, fare restrictions, identity checks, loyalty commitments, the things a brand will never say. They can no longer live in a policy binder or in a prompt the model is asked to remember. A prompt can tell an agent not to issue an unauthorized refund. Only an enforced policy can stop the refund from going through.

That makes authority a leadership question, not just a technical one. The CIO and CTO must make sure the architecture can enforce the company's policies. The CMO must make sure every action keeps the brand's promises. Operations, risk and legal leaders must set the boundaries of what the business will do. And the chief executive owns the result. As Satya wrote, a model provider's assurances do not relieve a company of that responsibility.

You don't have to see inside the model to govern it

There are two ways to make powerful AI safe to use. One works from the inside: train the model to behave, and build tools to see what it is thinking. The other works from the outside: let the model reason, and hold it accountable through a system it cannot override.

The frontier labs work on both, with heavy emphasis on the inside. Anthropic's Dario Amodei, who last month urged AI companies and governments to pace the frontier, compares interpretability research to an fMRI for AI. He also calls for stronger monitoring, sandboxing and outside evaluators, and he is candid that "we still only understand a tiny fraction of what goes on inside these models."

That science will take years to mature, and enterprises don't have to wait for it. Even a model we understood perfectly would still need permission to open a customer's account, waive a fee or move money. Authority is not a property of the model. It is a decision the business makes.

I learned this before generative AI existed. Before founding Netomi, I built trading systems on Wall Street. A trading algorithm can spot an opportunity and execute faster than any person, but it never decides whether its own risk limits apply. The exposure caps, the monitoring and the mechanisms that halt trading sit outside the system making the decisions. Enterprise AI needs the same discipline. Intelligence can be probabilistic. Authorization must be enforceable.

Proven at 40,000 requests and millions of micro authorizations a second

When OpenAI profiled Netomi's architecture in January, it described models running inside a governed execution layer, with governance "woven directly into the runtime—not added as an external layer." In our design, that runtime sits outside the model itself, where the model cannot override it. The model proposes an action; the system decides whether it is permitted before anything happens. Every tool call is checked against its expected contract before it runs. Brand restrictions and compliance checks apply inline, during reasoning and tool use. When the system is uncertain, it falls back to known-safe paths. Reasoning traces and tool calls are logged for inspection.

That storm scene is exactly the moment the platform was built for. United Airlines runs on it, and OpenAI's profile walks through that exact case: a traveler rebooking a canceled flight, with fare and loyalty policies applied and the response validated before it reaches the customer.

The architecture has held up under real load. At DraftKings, traffic during major sporting events has spiked above 40,000 concurrent customer requests per second, and Netomi has kept responses under three seconds. One insurance customer handles close to two million provider requests a year across all 50 states, where a wrong answer can create regulatory risk.

The lesson is simple. You don't need a model you can fully explain to deploy AI responsibly. You need a system that never lets the model have the final word.

Authority should grow with the evidence

The industry is now debating whether to pace the frontier. That debate concerns the next generation of models, and the risks it raises deserve serious work. It should not stall the responsible deployment of this one.

For most large-enterprise work, model capability is no longer the bottleneck. Today's models can already reason through a rebooking, a claim or a billing dispute. What holds enterprises back is the lack of an architecture that lets them trust those models with real authority. Without it, the problem stays unsolved, however capable the next model is.

With it, the question changes. Leaders can ask not only whether the next model is smarter, but how much more their current models could do: complete a refund rather than recommend one, or resolve an exception without escalating it. The answer is not unlimited freedom. It is a principle any leadership team can adopt:

Authority should expand as evidence of reliability builds, narrow when risk rises and stop when a control fails.

Built this way, governance is what makes autonomy possible. Whether or not the frontier is paced, it comes first.

That is what sanctioned superintelligence means: AI trusted with real authority because it never decides on its own. The question for every chief executive is no longer whether the intelligence is good enough. It is whether that intelligence has earned a badge at your company: who issued it, on what terms, and who can take it back.

全文完
查看原文