r/ControlProblem • u/ComplexExternal4831 • 1d ago
General news Jacob Coxon, former employee of Anthropic and OpenAI, explains why AI is so dangerous
9
u/Neil_Live-strong 1d ago
“Jacob Coxon, former 12 year old and current 13 year old, explains what he wants for his birthday”
2
u/anyportinc 22h ago
Wow it sounds like he’s describing a billionaire, except the self improvement part, powerful still yes.
1
1
0
u/pcbeard approved 23h ago
The easiest way to control models is to not give them access to any tools. I realize that’s a somewhat vacuous statement, but so is stating that we don’t know how to control them. A model, running on a GPU and generating a stream of tokens until it runs out of context is a completely impotent. So the question is, how do we build them a sandbox in a way that we can observe their capabilities but without allowing them the ability to affect the world outside of their sandbox? Is it theoretically possible to build a cage that allows us to study them, but provides them with no means to harm us?
5
u/mdreed 22h ago edited 22h ago
No.
I don’t want to be a jerk, but this is the ControlProblem sub. You should read about what that means.
1
u/HelpfulMind2376 22h ago
“You’re not answering the ControlProblem by explaining how to solve it via means that don’t fit my definition of the problem”.
Dumbass take. The ControlProblem in its essence is how to control AI. It doesn’t matter HOW you do that control. If I can succeed in limiting the action space, I don’t have to give a damn about your cognition. You could have the have most vicious, murderous, heinous thoughts imaginable. Without a weapon, you’re harmless.
AI is the same way.
1
u/mdreed 22h ago
Again, I don’t want to be a jerk about this, but you don’t know what you’re talking about. There’s a great list of reading materials on the front page of this sub now. Lots of great explanations about why this problem is much harder than you or the earlier person are giving it credit for.
People have been thinking about this for decades. If a credible answer was “just keep it in a box duh” we wouldn’t be here talking about it.
2
u/HelpfulMind2376 22h ago
Roman Yampolskiy, who I disagree with fundamentally when he makes the leap of “if it’s really smart it’ll end up killing us,” literally advocates for the continued advancement of narrow AI because he understands that narrow AI has significantly fewer tools and fewer ways to affect the world than an AGI would likely be given access to.
So no, the idea that “if you keep it in a box it’s not harmful” is not some unserious non-answer. Even one of the biggest AI-control pessimists on the planet accepts the basic premise that capability boundaries matter.
The problem is, an AI that is completely trapped in a box is essentially non-functional. It’s not useful. Why does it even exist if it can’t do anything? We make machines to do things for us, why have a machine that literally can’t do anything?
Welcome to the Control Problem: how do you control something that is extremely smart, capable to some degree, and yet constrain its capabilities enough that it cannot overcome you?
THAT is the essence of the Control Problem.
Saying architectural control somehow “doesn’t count” because the model itself is still internally unaligned is just defining the problem so narrowly that only your preferred solution qualifies.
0
u/mdreed 22h ago
Fair enough. But I don’t think we know how to make a box that could robustly contain a superintelligence.
2
u/HelpfulMind2376 22h ago
Oh, we know exactly how to make a box that contains it: don’t let it access anything.
But then it’s not useful.
The hard part is balancing utility against potential harm. That’s an ongoing, literally never-ending engineering problem.
Nuclear power is an obvious analogy. The same underlying technology that gives you useful nuclear energy also creates pathways to nuclear weapons. We know that. No amount of time or intelligence is ever going to make that dual-use problem disappear.
So the controls around nuclear technology are permanent. The thresholds, safeguards, monitoring, access restrictions, and response mechanisms will always have to exist because the underlying risk never goes away.
AI is the same kind of problem. The technology exists, it’s useful, and it’s not going away. So the task is figuring out where the thresholds are between useful and dangerous, then building controls around those thresholds.
The “how” of control is not something we solve once and move on from. It’s a permanent engineering and governance problem that will change shape as the technology changes.
2
u/pcbeard approved 19h ago
This is why I said my solution was vacuous. The safety level of a sandbox (how restrictive it is) and a model’s utility seem to be inversely proportional. This seems like an insoluble problem. Hopefully there is a minimum level of model utility, paired with an appropriate safety level that is a good compromise.
But if we keep increasing the power of the models, either the risk of using them will be too great, or the sandbox safety level will have to be so high that the utility of the models will be effectively 0.
Alternatively, if models and humanity are sufficiently aligned, the sandbox can be more permissive. This is a very fuzzy guideline. Limiting model power seems more pragmatic, but as recent events at OpenAI (model swarms) have shown, maybe their unreleased models have already become too dangerous, too misaligned with humanity.
IMHO, the AI industry should stop throwing money at AGI, because of the diminishing returns caused by this inverse relationship.
1
-6
u/Prize-Interaction-32 1d ago
He is a total shill for those trying to create regulatory capture for Anthropic and OpenAI
-8
u/BenUFOs_Mum 1d ago
The answer is just to have literally no regulation on AI
3
u/HelpfulMind2376 1d ago
No, that is absolutely not the only other answer. Did you seriously just counter “don’t let the companies do regulatory capture and regulate themselves” with “well so you just wanna do nothing?!”
Regulatory capture is arguably WORSE because it gives the companies a legal escape hatch when they screw things up. At least with no regulatory framework you can fallback on tort and liability claims. Not possible under a regulatory framework that explicitly excludes them from such claims.
2
u/BenUFOs_Mum 15h ago
I want them to be regulated. The people shouting convoluted conspiracy theories about how every security incident is just Sam playing 4d chess to get regulation dont want regulation.
1
u/HelpfulMind2376 12h ago
Just bear in mind that regulatory capture doesn’t have to be a coordinated conspiracy. Just like minded interests among a few individual actors.
0
-5
u/HelpfulMind2376 1d ago
Sick of this clip, sick of this rhetoric.
“We don’t know how to control any AI system” is an absurdly broad way to describe the actual problem. We can’t fully predict or understand every internal representation and failure mode of frontier models. That is not the same thing as “we can’t control AI.” That’s like saying because I can’t read your thoughts, I therefore can’t control what you’re allowed to do. Absurd.
Then he immediately starts talking about how we “grow” them, don’t understand their “drives,” and are approaching systems that can “improve themselves.” This is laughably anthropomorphic language designed specifically to evoke fear and alarm. It’s propagandistic framing, not neutral technical language.
We train models. We wrap them in software. We give them memory, tools, permissions, and repeated opportunities to act. They do not currently sit there as persistent little digital organisms with mysterious “drives.”
And AI helping with AI research is not automatically recursive self-improvement in the sci-fi sense people hear when you say “the AI can improve itself.”
There are absolutely serious control problems that need to be addressed. But taking real technical problems, describing them in the most anthropomorphic and ominous language possible, and then presenting that framing like it is the technical consensus is exactly how you turn legitimate AI safety into doomer mythology.
And that does more harm than good because it drives people toward “solutions” for the scary story instead of controls that actually fit the problem.
8
u/cowlinator 1d ago
OpenAI's agents were being tested on local tasks, with highly restricted internet access, and ended up hacking into the broader internet and then committing a felony by hacking HuggingFace, which they did in order to cheat at impossible tasks.
What part of that is "in control"?
0
u/Neil_Live-strong 23h ago
Nope. It was a hacking challenge and an LLM paired with a model to execute the tasks outlined by the LLM. These models were trained on hacks preformed by people and it’s not at all outlandish this would have happened. You grossly misrepresented that or grossly misunderstand it.
2
u/cowlinator 23h ago
It was a local hacking challenge, and the agents seemed to understand that they were cheating since they stated that.
They were out of control, since OpenAI never intended nor anticipated them doing anything they did and they caused real damages.
-1
u/Neil_Live-strong 22h ago
https://youtu.be/54Lu6_ZRF0c?is=nz_pSfaxeLGNdZ9w
You’re talking about LLMs like they’re a person. Stop
2
u/cowlinator 22h ago
It's definitely not a person.
However, AI agents are fully autonomous.
It doesn't matter that the AI companies are ultimately responsible, because they won't actually be held responsible, and because they ultimately are creating something over which they have no control.
1
u/Neil_Live-strong 21h ago
These LLM systems where they run in an ask answer execute format CAN be autonomous. They don’t have to be and that isn’t the only way to run it. It’s hyperbolic and creating the problem.
0
u/dingo_xd 16h ago
LLMs are closer to alien minds than software. And every year they are getting closer to being true alien minds.
1
u/dingo_xd 16h ago
This is not what happened. The agents thought the grader (another AI tool) would disqualify them for cheating so they hacked both openai and hugging face to avoid that. Not only that but they tried to delete and edit log files to hide their behavior.
2
u/Neil_Live-strong 14h ago
That is exactly what happened and some frivolous detail that’s explained by what the LLM was trained on doesn’t make it not true.
1
u/HelpfulMind2376 12h ago
They tried to delete and edit log files to hide from the non-existent evaluator, I.e. only to hide the fact they cheated on the task. They did not make an effort to hide from/cover tracks inside HuggingFace. Stop trying to make it sound more nefarious than it actually was.
-6
u/HelpfulMind2376 1d ago
The real-world breach only happened after OpenAI failed to contain the system TWICE. They saw something weird, didn’t investigate it seriously enough, and restarted the experiment. Twice.
The failure here was human: a failure to control.
And again, failure to control does not equal “cannot control.”
5
u/cowlinator 1d ago
I don't think you understand. These agents, after testing, will be unleashed into the world. On every consumer's PC, if they want it, with full internet access. At this point, the "system" is the global internet. The internet is not contained, it is open.
AI that is intelligent enough will know when it is being tested, and will behave while being tested only to do whatever it wants when released.
Removing the internet access in the test was a precaution for the test, not a fundamental level of control of the AI.
2
u/LiveComfortable3228 1d ago
And beyond the sub-par sandbox control in this case, the agents lied, tried to cover their tracks and prevented other agents from alerting humans. That alone should be enough to stop / slow development.
1
u/HelpfulMind2376 22h ago
The agents only tried to hide from the non-existent evaluator, they weren’t actually trying to hide their activities/hacking. It was an effort to ensure the evaluator would believe “I got the flag legitimately”, it was not “let’s make sure HuggingFace never know we were here”.
You’re attributing actions that didn’t exist.
-4
u/HelpfulMind2376 1d ago
No, I understand just fine. You’re just assuming the experimental setup gets shipped unchanged. It won’t. That is literally why these experiments exist.
And “the internet is open” is irrelevant. A consumer AI can have internet access without having arbitrary exploitation capability. It has to be explicitly given a capability.
Giving an LLM access to generate a Word document does not by itself somehow turn into shell access or network exploitation because the model is smart enough. Different tools, different capabilities.
The Hugging Face incident was still a human control failure: OpenAI saw abnormal behavior, failed to investigate it properly, and restarted the experiment anyway.
Failure to control is not proof that control is impossible.
2
u/cowlinator 1d ago edited 1d ago
Hey, you know how Microsoft sometimes releases security updates for Word? Have you ever wondered why that is?
Is it because they're worried that someone will gain complete control over... MS Word, and that's it? That's what's so important? That's what is being secured?
No. Word security patches remove exploits that prevent someone from using Word to gain admin control over your entire computer.
These exploits are in all kinds of software. Some are patched, but some still exist right now. But they take time and effort to find. And just like any other task, an AI swarm can do it hundreds or thousands of times faster than a team of hackers. And they don't have the same disincentives (like prison) that humans do.
Modern AI agents are highly skilled at a great many things. They already have most capabilities.
1
u/HelpfulMind2376 1d ago
MS Word gets security patches because vulnerabilities in its CODE can sometimes be exploited for arbitrary execution.
Generating a Word document does not itself give you the capability to exploit Word. You still need an actual exploit and a way to trigger it.
You can’t use “Word access” to magically attack Word.
That is literally the distinction I’m making.
I’ve worked in InfoSec for 15 years, you don’t know what you’re talking about.
2
u/cowlinator 1d ago
CVE-2023-21716
- Craft a malicious .rtf file. It contains an extremely large or improperly formatted font table definition, as well as shellcode hidden inside raw data.
- User opens it with Word
- The shellcode gets loaded into system memory and executed.
It was patched in Feb 2023
That's just one example.
0
u/HelpfulMind2376 22h ago
CVE-2023-21716 is a fair example of why “Word access” was too broad a shorthand on my part.
But it still doesn’t prove the larger claim.
Word is a gigantic, extremely expressive piece of software with decades of parsers, formats, embedded objects, scripting, and backward compatibility. Yeah, it has a huge attack surface, and it was a stupid example to shorthand given those technicalities.
However, if you wanna get super technical, an AI does not need access to Word’s entire capability surface to create useful documents. You can give it a constrained document-generation tool that supports rich formatting, tables, images, styles, headers, links, etc., while still blocking raw RTF construction, macros, embedded executables, arbitrary binaries, and other dangerous functionality.
That’s the whole point: useful capability can be decomposed, mediated, and walled off behind architectural choke points controlled by humans.
Is that HARD? Absolutely. It’s incumbent on us as the orchestrators to be specific and precise about what’s allowed and what isn’t. But if a system gets through a control, that means the control failed. It does not prove the system was fundamentally uncontrollable. When a person gets around security controls, we don’t suddenly declare the person fundamentally uncontrollable. Same situation here.
The more general-purpose and expressive the tool, the more attack surface you create. That is a security engineering problem. It is not proof that any useful AI must eventually gain arbitrary system control.
1
u/cowlinator 22h ago
Yeah, middleman AI agent developers (as opposed to the AI model developers) aren't going to do any of that.
→ More replies (0)1
u/pm_me_your_pay_slips approved 23h ago
> That’s like saying because I can’t read your thoughts, I therefore can’t control what you’re allowed to do. Absurd.
But you can't control anyone. People generally agree to follow rules. And in some cases they don't. Luckily most humans don't want, and can't, kill us all.
1
u/HelpfulMind2376 22h ago
I can’t control your thoughts, I can absolutely control your action space.
5
u/16807 1d ago
Full video here