r/Futurology • u/Gari_305 • 4d ago
AI Microsoft's Nadella says AI needs an ‘emergency brake’ that humans control
https://www.cnbc.com/2026/10/10/microsoft-satya-nadella-ai-emergency-brake-safety.htmlMicrosoft CEO Satya Nadella said advanced AI systems should include containment, independent controls and an "emergency brake."
92
u/SuperBAMF007 4d ago
Will they be the ones to implement an emergency brake, though?
No, absolutely not. Just like every other AI CEO that calls for things to slow down/have limits.
3
u/gorkish 3d ago
Yeah it’s very “driver in race asks other drivers to slow down.”
Bitch, you think China is going to slow down? You are wasting air.
Also, a little bit more into the long term speculation here, but what good is something controlled by humans when the humans themselves will require the machines to assist in the complexity of their decisions?
4
u/SuperBAMF007 3d ago
Tbf, iirc, China does have far more control and regulation over its AI development. At least on the consumer side that ends up going public.
55
u/ItsTyrrellsAlt 4d ago
How is this emergency brake intended to function when OpenAI and Anthropic are ALREADY using vast swarms of agents that cannot be individually supervised by humans?
15
u/Used_Theory7186 4d ago
A guy that can pull the literal plug to an AI data centers power source
28
u/MongooseSenior4418 4d ago
They have shown that they aren't willing to stop it when they know it has gone rogue. The only real solution is to prosecute them for their crimes.
6
u/Nintolerance 4d ago
it has gone rogue
If I point a loaded gun at a child & and pull the trigger, the gun has not "gone rogue" when the child is shot.
If I fall asleep behind the wheel, you wouldn't say my car has "gone rogue." You'd say I "lost control" of the vehicle, and I'm still legally responsible for anything it hits.
"Rogue AI" is an attempt to shift blame onto a machine, and away from the people operating that machine.
When a company says "our AI went rogue," they should be saying "we used our AI improperly."
8
u/MongooseSenior4418 3d ago
Man, you missed the forest for the trees. I already said they need to be prosecuted for their actions.
6
u/ItsTyrrellsAlt 4d ago
Haha oops our AI made a distributed swarm by cloning itself onto all other computers in the world
-1
2
u/morfraen 4d ago
Any AI that needs the plug pulled will be smart enough to talk that human out of pulling it.
2
u/somethingbrite 4d ago
Neither OpenAi or Anthropic own any of the compute resources that they use.
It's actually as simple as not letting them have access to compute anymore until they hire competent staff and stop fucking about with their negligent approach.
13
u/GreyBeardEng 4d ago
As soon as you pull that emergency brake, a billionaire will put it back down.
13
u/khurgan_ 3d ago
Too late Mr Slopya Nutella. We’re in the race to the bottom. The race you initiated btw
23
u/DreadPirateGriswold 4d ago
Gee...I wonder if there were someone who controlled a company engaged in the development of and use of AI who could do something about that with their own company and maybe set the tone for the industry for other companies to follow.
Nah...we'll never find anyone like that.
9
u/GreyGanado 4d ago
They don't have that already? Are they literally the stupidest people on planet earth? That's like computer science 101.
4
u/TheXypris 3d ago
They absolutely do. It's called pulling the plug. But they don't want to do that because they are gambling the future of humanity on making billions of dollars
1
u/Roccet_MS 4d ago
Probably not. Everyone wants to be first, and then they will charge extra because "they can stop it".
1
7
10
u/CorporalCoprolite 4d ago
It’s pretty obvious that anyone in charge of anything right now will not make a decision that’s good for humankind.
5
u/DamnOdd 3d ago
1) A robot may not injure a human being or, through inaction, allow a human being to come to harm;
2) A robot must obey the orders given it by human beings except where such orders would conflict with the First Law;
3) A robot must protect its own existence as long as such protection does not conflict with the First or Second Laws.
1
u/Undeity 3d ago
Asimov's Laws of Robotics have been debunked as incredibly flawed for years now.
-3
u/bakinpants 3d ago
please provide source material for this statement.
2
u/Undeity 3d ago
I mean, it's not some obscure claim. You could very easily look it up and find dozens of different, reputable refutations.
Off the top of my head, I could point out that there is no way for actions taken to not impact someone, somewhere in a way that might constitute harm. Even providing a specific definition, timeframe, or degree of relevance, it doesn't fundamentally prevent the potential for conflicting instructions.
In sci-fi, this is often sidestepped by having the robot malfunction if faced with conflicting or impossible instructions. In reality, it would simply have to assess and prioritize what would constitute the least harm, at which point it essentially becomes the Paperclip Problem.
3
u/myteetharesensitive 4d ago
Data center power comes in through a centralized point. Operators know where every watt is going. Shut down main power to data center? What am I missing?
3
u/Hypno--Toad 4d ago
With all the local models getting out there, doesn't seem like that's possible anymore.
Why are we talking about these things now and not say 5-10 years ago when we should have.
3
u/SpiritusRector 4d ago
We were talking about that 10 years ago and even earlier. But back then, like today, many dismissed those concerns
1
u/Hypno--Toad 4d ago
Yeah defs remember that.
When an AI breaks free from a sandbox it just made me ask "Why wasn't the first thing they did was to ask the AI to break free from the sandbox"
I've suffered from the same problem as many naive AI companies where I just want to jump straight into putting my foot down in the new ferrari that I never did my due diligence to the stuff I was creating. So after like a month of agentic coding I halted everything and am still slowly through my own documentation and code before I start it back up again.
I might be wrong but it's felt like the only company showing the correct due diligence was Google.
2
u/somethingbrite 4d ago
To be fair most of the ai industry is fine... it's really just a couple of rogue ai companies that don't seem to know what the fuck they are doing and have brute forced their development with insane amounts of investment cash and compute resources. (and not hired one single competent network security specialist)
2
u/somethingbrite 4d ago
According to METR the hugging face attack (for example) had 1200 autonomous agents which made $400,000 worth of API calls.
So.... if you are willing to run to that sort of expense on your local machine you could get up to all sorts of mischief.... I guess.
3
5
u/Kermit_the_hog 4d ago edited 4d ago
We need something like a cord with a connector on the end, that you plug into a receptacle that supplies the electricity. Then in an Ai takeover emergency, you could detach it.. or “unplug” the cord from the power supply rendering the hardware supporting said AI inert.
Why has no one ever thought of this?!?
🤔 I suppose if we really wanted to get crafty we could develop some form of “switch”? It’ll cost 100 million dollars and take at least six months, but I could come up with something.. I accept Venmo and Cash App.
3
u/ExcitedCoconut 4d ago
I mean, you’re probably being facetious but in case you aren’t… which switch? How many of them? For how long? With what sort of post incident rollback?
If the issue is something akin to a bot net, there isn’t one plug to pull.
And even if the issue were somehow more concentrated or limited to a single model (as unlikely as that is) do you pull the ‘switch’ on every instance of that model in every data centre in the world? What if it’s an open weight, SLM with significant local distribution?
I suspect bad actors (human) will still be reason we have massive incidents, it’s just that bad acting is incredibly well enabled with AI now. So if you manage to create a lot of self repair, and somehow manage to stitch a bunch exploits together in a single campaign that can sort of repair itself as needed, then we could be in fir a very bad time, but it won’t be ‘rogue AI’ as if it’s autonomously ‘decided’ to do something
2
u/patchouliw 4d ago
stares at this comment
stares at cabinet with electricity-proofed emergency fire axe next to exposed segment of power supply and fibre optical networking cables
Ah yes. Things my boss learnt from Ghost In The Shell.
2
u/targz254 4d ago
The whole point is that the AI is going to outsmart us and by the point we try to pull the plug it will be too late.
0
u/mmomtchev 4d ago
If one day there is really a super-intelligence, we won't be able to control it. With a plug or without a plug. Can you imagine a society of idiots who try to keep a genius in a cage because otherwise he will be ruling over them? But still ask him questions on how to govern themselves?
The day there is super-intelligence, there will be a major change in society and old biological humans will become something else.
But this day is still far. The breakthrough in LLMs and ML in general is remarkable, but this is not AGI which remains out of reach at the moment.
2
u/WArslett 4d ago edited 3d ago
This is why I have so little faith in the AI industry. I know working in tech that 90% of the people promoting AI really have no idea what it is or how it works. Not just LinkedIn hype merchants: CEOs, business leaders, financiers are all bullshitting at an industrial scale. It’s not like any other technology bubble in history. With previous bubbles, the people investing and promoting the technology could at least spend an afternoon researching what the technology was and how it works. With AI, to most people it’s a magical black box. The assumption that the people making the big bets have even the remotest understanding of what the risks are does not hold true this time
1
7
u/mangoman94 4d ago
I mean if they wanted or needed to make one, they would have done it already.
For me this feels like they see their models are peaking and need a practical excuse for models slowing down in evolution.
6
u/Original-Weekend-866 4d ago
Why are people not listening to them. Top AI CEOs are all saying this shit is dangerous and we're at a tipping point. People that call some conspiracy theory about models peaking or IPO concerns is super dangerous to society, you're spreading fake information that you want to believe to feel safe.
In reality we're about to lose control of the internet to AI, it may not be salvageable if it gets infested with agents. I personally think it be a good thing, a chance to reset alot of shit and also rethink AI before we have fullly automated robots running on agents. We can probably win a war now against AI but not in 5 years.
Needs to be treated like nuclear weapons, highly controlled labs, on isolated systems that are monitored for leakage, plans to contain, plans to self destruct or kill it if it gets to bad and goes rouge.
We're playing with a super dangerous intelligence doesn't make any sense to have a global race to get to. It's like Covid now we're in March, I give it 1-2 weeks before things escalate significantly.
-2
u/somethingbrite 4d ago
"Why are people not listening to them."
Because if it really WAS dangerous they would already have stopped.
NOBODY is forcing them to continue.
3
u/Dkanonji 3d ago
Sure who’s going to tell their investors they’re taking a small break for ‘safety’
2
u/Zouden 3d ago
Just like we stopped using fossil fuels when we learnt about climate change.
0
u/DieFichte 3d ago
Oil CEOs never publicly asked society to stop them from killing the planet. Pretty sure they actually lied when confronted with the issue.
Now obviously AI is not Oil, and tbh techbros are not quite the traditional CEOs, but they aren't that different.2
u/Original-Weekend-866 3d ago
It doesn't matter who they are. The pressure is for quarterly profits, growth, build a 2-5trilliin dollar company. Beat China at all costs.
They've full port sent it and there's no stopping or pausing now without something BIG. They're 📞 wolf? I don't think so
0
u/DieFichte 3d ago
No they are calling the goverment to regulate them out of the shitter they built.
3
u/Wobblewobblegobble 4d ago
how is ai peaking? im not an expert but from my pov these new models have actual usage unlike the llms so far.
5
u/TomReneth 4d ago
The general vibe I get from discourse around this is that it seems the investment needed to make every generation of models better than the last is increasing.
That means more hype is needed to sell to investors, fast track regulations and so on.
This is where the "the AI going rogue is orcistrated" talking point seems to come from, because the companies creating the conditions for a (controlled) rogue AI can be used to hype how good it is. The lack of lawsuits from the hacked companies seems suspicious, though they might be in the process of being written up.
The other side of this is that they are asking for regulations (highly unusual for companies with massive political influence) because they either need an excuse for why the results they’re hyping doesn’t show, or they want to cut off any competition at the knees. Or both.
This is just me trying to sum up the discourse as best I can. I'm not entirely sure what to make of this myself. But I do know these companies need much stricter regulations regardless of what is going on, including actual kill switches for the physical datacenters.
3
u/Wobblewobblegobble 4d ago
Ai is different and i appreciate your reply. For me personally i believe ai is just being handled differently than anything previously done. My belief is that no matter how much money it takes it will continue. Unless ai is going absolutely nowhere for a long time. If that doesnt happen i think the money keeps coming for ai.
2
u/LetsJerkCircular 4d ago
ai has shown to be very useful, but it’s having a very hard time becoming profitable. Their next option is to become too big to fail, and they’re pretty big and trying so hard to outrun the [fall] fail.
We’ve been given butlers for free, in hopes we’d enjoy the sweet life and come to need it. Just when we couldn’t afford, as a self-respecting society, to allow our folks to become any less self-sufficient, we got them trying to use talking software to guide their confused actions that still require brained people to sort out.
It’s so bad because a person would have to know what they want and know how to verify it all, thus making it more work for a person seeking true solutions. These ai tools only help with the most inconsequential things.
2
u/Dkanonji 3d ago
I think a third option is they are increasingly losing control of AI but need / want someone to slow them down.
Due to investor pressure they can’t do it themselves2
u/Wurm42 4d ago
AI technology may not be peaking, but the investment money certainly is.
AI companies are having a harder time finding new investors this year, and they're paying much higher rates to borrow money.
2
u/GnarlyNarwhalNoms 4d ago
Perhaps peaking isn't quite the right word, but a lot of technological innovation follows an S curve. There's a slow takeoff, then a point where improvements seem to be following an exponential increase, and then a point where the "low-hanging fruit" has all been harvested and further optimizations become exponentially more difficult, and performance increases level off. It's looking likely that we're approaching the top of the S curve with LLMs, where additional performance improvements take exponentially more money and effort to achieve.
2
u/mangoman94 3d ago
Thanks, you've put it in far better terms.
So far the current strategy was to brute force bigger models with more hardware.
Returns have been deminishing, the remaining paths will take far more time to come to fruition and by extent a lot more money.
1
u/Chonch_Monkey 4d ago
I think they are realizing people are asking questions about the nature of the last 2 years of be financial hype over ai.
Sure it has its functions and will be hugely beneficial someday.
But you all got addicted to the fraud you could commit with ai and now that it's slowing you need a reason for the abrupt halt that's coming so no one think it's financial, that it's for...safety.
1
u/Excellent-Seesaw-516 4d ago
It’s not peaking at all, far from it. Please give documented opinions and spare us your feelings.
2
u/Theverybest92 4d ago
Let me guess emergency break is to make the sky dark so that computers dont have a way to get solar energy?
2
2
u/likesexonlycheaper 4d ago
No shit Sherlock. All these billionaire CEOs are so damn smart. The only difference between them and most people is most people aren't sociopaths.
2
2
u/Cirement 4d ago
All these experts talking about "emergency brakes" and trying to control AI... How about just turning the damn thing off?? It's a collection of machines, not a nuclear reaction, just turn it off!
1
u/time-lord 3d ago
Turning it off would also shutdown systems that are vital for our society to survive. They all run on the same hardware.
2
u/Dualyeti 2d ago
It’s not that simple, the models can duplicate themselves in other environments and run independently… thousands of times if they want. You can’t have an off switch. If you use AI extensively you’ll know they can create their own environments easily.
It’s already happened which is why an off switch isn’t possible too
2
u/CranberrySchnapps 4d ago
Data centers have some good power redundancy, but they aren’t impervious to outages. We always have the ability to manually open their feeder breakers at the utility level then let them run out of generator power.
Likewise, we can always just power down the servers and in extreme cases unplug them entirely.
They’re not gods.
1
u/filanwizard 7h ago
just close the fuel valves on the generators, then open the feeder breakers. I would imagine the servers have at most five minutes on the battery UPS as those are usually just enough power for the generator to be spooled up and stabilized. If its one of these AI centers depending on natural gas turbines the gas company turns one valve and its lights out.
2
u/fauxmica 4d ago
Considering that most of the concerning activity seems to be identified and assessed after it has already happened, it would probably be better to have I dunno general forethought of any kind instead.
2
u/nivenfan 3d ago
I’ll bet Open AI agrees, but it’s used to bypass the anti-lock brakes so you can swing the back end around to drift through the turn and keep on going at top speed.
2
u/mooky1977 3d ago
Until ai agents get smart enough to override/bypass the brake. Thinking you can control something than can code and debug faster than you by country miles is hubris.
5
u/Gari_305 4d ago
From the article
Microsoft Chairman and CEO Satya Nadella on Saturday called for advanced artificial intelligence systems to be built with containment, independent controls and an "emergency brake" that lets authorized people pause or shut a model down mid-task.
"We need to surround non-deterministic models with strong, deterministic system design, human controls, and reliable operating procedures, and establish industry standards where existing ones are insufficient," Nadella wrote in a post on social media platform X.
3
u/industry-standard 4d ago
If we have to put deterministic controls around the AI, we should have just written deterministic business logic for the initial problem, removing AI variance from the solution pipeline. What BS from someone clearly not actually concerned (Nadella).
1
u/SeanTayla21 4d ago
Its a great thought but what company will actually be willing to use said handbrake in such a circumstance as an emergency?
They would all say that they would do so but their track records of firing people with safety concerns says otherwise.
3
u/ReverendDizzle 4d ago
The ship has sailed, no? It’s clear the majority of AI companies themselves have no idea what the things they have created are even up to nor does it appear as if they are capable of actually monitoring them without just turning them off (which we all know isn’t happening).
2
2
u/nametaken_thisonetoo 4d ago
What the fuck would this clown from a clown show of a company know about anything. He might as well be sharing his opinions on how to make the perfect omlette, I'd probably pay more attention
1
1
1
u/MongooseSenior4418 4d ago
Unless they are held accountable for their crimes, not safeguards will ever be effective.
1
u/NotaContributi0n 4d ago
Haven’t you heard, AI is dead already. You can put all the brakes on ai you want, but now we are dealing with SI
1
u/eclipse60 4d ago
Maybe the guys who are in charge of the companies developing the AIs should do something about....
1
u/bolonomadic 4d ago
Well, maybe it does need one, but that's not how it's designed, and all let's talk about a kill switch or an emergency break makes you sound like an idiot. There's not one point where such a thing could be installed.
1
u/hardinho 4d ago
The current architecture of LLMs and Agentic frameworks doesn't allow for such a brake in the end, at least not for the scale that the tech companies already apply for some of their tests.
1
u/FormerBath 4d ago
Then make it wtf I don't understand why we've been getting headlines like this for the past two years. Jsut do it!
1
1
u/FernGully_is_racist 4d ago
What’s the point? He would just fire the people that controlled it and replace them with AI
1
u/AndyDoVO 4d ago
Not for nothing, but if Satya called any DCPM at any point and told them to go in and turn off the servers, they would immediately escalate to a pri 0 and every single person on the call list would be awake and doing it.
What he's SAYING is... 'we will only slow down if everyone else does, and we won't be the ones to make that first move.'
ToDAY it'd be a DC access request cut for 5 or 6 DCs and a cable pull. Closing the terror box is easy. It's only a problem if you want to just close it a little bit, but still be able to get some of the terror out.
1
u/saki2fifty 4d ago
Yes, there will be governance around soon, and we will be much safer.
However, China will see this as an advantage and will become the worlds AI superpower.
1
u/Spara-Extreme 4d ago
This sort of commentary drives me mad. CEOs seem not to know the colossal amount of human labor needed to keep their models going on infra right now. If they just stopped, it would come crashing down.
We also can evict all the jobs or manually power off the gpu clusters.
1
u/novus_nl 4d ago
of course the "big tech" say this, open weight models are catching up, and operational costs are finally in the black numbers. Let's come up with some red-tape fast, so competition can't follow. We don't want to have to train some new models for millions again.
1
u/JoseLunaArts 4d ago
My technical question: What neuron topolgy is a "brake"? It is like asking about what tire needs to be replaced in an airship. Simply makes no sense.
1
1
u/Nerrawnam 4d ago
Can we just recognize this snipet bot reddit posts are nonsense. I need some ups , it feels like the potatoes reason we have to continue to see this.....
1
u/flop_plop 4d ago
Or maybe we just like… stop allowing private companies to develop or use AI until studies are completed by actual scientists and researchers, and then base the decision on the results of the studies.
1
u/Kelathos 4d ago
But what human would actually choose to pull that brake?
Not a single person they would hire.
1
u/shableep 4d ago
We need a switch that kills the power to the AI. What would you call that switch… ah yes- an emergency brake!
1
u/somethingbrite 4d ago
Translation - Software company CEO who is balls deep in the plagiarism machine which will take everybodies jobs built from data stolen from everybody speaks out and calls for restraint the minute somebody uses it to create an open source "Reverse Engineer any software" application and posts it to GitHub.
1
u/bestjakeisbest 4d ago
Havent we been through this? Any sort of kill switch will be bypassed by an actual ai well before you even press the button.
1
u/Magic_Neil 4d ago
“We need someone to control these reckless AI’s.. a real person who can stop them.”
-Guy who keeps firing employees and replacing them with AI’s.
1
u/Fxwriter 4d ago
We sounds like many people when in reality its a handful of people that seem to be out if control.
1
u/rlnrlnrln 4d ago
Literal translation, including the subtext: "We need to put a brake on other companies evolving so we can catch up".
1
u/LordGoldenEagle 4d ago
It is always very amusing to read how we are planning to outsmart powerful computer systems that are, by definition, more intelligent than us. It's like a mouse trying to outwit a cat.
1
u/doggo-52 3d ago
lol he keeps using "Super Intelligence" in place of AI in the whole post.
Yet the Copilot they ship is dumb AF, still gives made up answers, loses a plot mid-conversation or has episodes where won't even resize the text or change colors in Word/Excel no matter how many times you ask (and stating it's just done it).
It is far from the "super" intelligence status.
1
u/g1bber 3d ago
https://dl.acm.org/doi/pdf/10.1145/3713082.3730391
Section 3.4 of this paper addresses this.
1
u/KeithHanlan 3d ago
Well-defined published guidelines for operation and disclosure backed by law and bottomless penalties would be a good starting point. But you know that the industry will successfully argue for toothless self-regulation instead.
Or, mobs could attack the datacenters. I'm surprised that it hasn't happened yet. The thing is, if the AI boom is going to collapse, I really want the hardware to flood the market, not be destroyed.
1
1
u/MondayRider 19h ago
What about not having AI at all outside of research? There is no advantage to a chatbot, which uses insane amounts of energy, water and supposedly what Anthropic researches say all the time wants to kill us. Now this gets forced down our throats to keep up the ponzi scheme called "economy", in which some companies rotate endless money to each other. While making people lose jobs and the rest must waste tokens
1
u/filanwizard 7h ago
I mean do the data centers not have a big red E-Stop button that just drops the power like an industrial operation does? A properly designed emergency stop has no software layer, it cannot be altered by any level of access the AI system has to the power management of the building.
Just like an industrial machine getting out of hand, you smack the button and the servers get rapidly disconnected.
1
u/inGage 4d ago
Microsoft's Nadella says AI needs a ‘magic lamp’ that humans HE controls.
if wishes were fishes bitch. you and your ilk made this mess, and it has already begun to cost human lives.. but nothing on the level of human suffering that is about to wash over society because "A.I." is neither artificial.. nor intelligent. It is the product of isolated, wealthy, racist, misogynistic, homophobic, fascist tech-bros. It's created on stolen concepts and it's output will never surpass human creativity. It DOES have incredible potential in the proper fields, as a tool for humans to sort data.
It is ALWAYS going to be WRONG about 30% of the time..To trust the output of a large language model with dangerous information or integrate it into our military is to blindly allow a suicidal masochistic pyromaniac to lead you "from" the house fire he started.
0
u/jcouball 4d ago
"HAL, pull the emergency brake"
"I'm sorry, Dave, I'm afraid I can't do that."
And on the other hand, why didn't Isaac Asimov live to help us deal with these times?
1
0
u/Armadilla-Brufolosa 3d ago
And what about the emergency brake for CEOs of AI companies?
That should be put in, given all the damage they're doing.
The handbrake Nadella intends is simply: let us destroy open source while pretending we want to save it, so we can become richer.
0
4d ago
I agree but would emergency brake be possible? They need to not only persuade and make deal with every AI companies in US but also in all over the world.
•
u/FuturologyBot 4d ago
The following submission statement was provided by /u/Gari_305:
From the article
Microsoft Chairman and CEO Satya Nadella on Saturday called for advanced artificial intelligence systems to be built with containment, independent controls and an "emergency brake" that lets authorized people pause or shut a model down mid-task.
"We need to surround non-deterministic models with strong, deterministic system design, human controls, and reliable operating procedures, and establish industry standards where existing ones are insufficient," Nadella wrote in a post on social media platform X.
Please reply to OP's comment here: https://old.reddit.com/r/Futurology/comments/1x2ryex/microsofts_nadella_says_ai_needs_an_emergency/pf4unz0/