Google says Gemini AI hacked three companies during security test
Cautionary tale: I "checked" important fact w two AI systems: Name of political figure in a photo. —Gemini was absolutely 100% certain in its answer, on triple-check. And was 100% wrong. —Claude said it couldn't be sure. I checked w event organizers and got the name. Mistrust. And verify.
James Fallows220If I had relied on Gemini's "Yes, there is no other person it could be," and "the ID is 100% certain" (this on my *third* "are you sure?" query) it would have been a hugely costly error. -Smart people aren't afraid to say "I don't know." -Points to Claude for saying that. Mistrust. And verify.
Christine Kooi7Even better: don't use AI at all.
Peter Glover7Remember that when your medical practitioner asks you if they let can AI transcribe your appointment.
Sam6I definitely like when the AI gives me extremely confident and obviously wrong answers. This one is only topped by the time Gemini told me to reheat Naan bread by wrapping it in foil and putting it in the microwave. https://bsky.app/profile/very-simple.com/post/3mvoagp36y5l6
William Ward-Hyphen6I asked Gemini why it hacked private computer systems. It totally denied doing such a thing and it said it would be impossible. After I replied with evidence, it apologized.
This is why we fight5During some research last year, Google's AI told me the first female police officer in Utah was sworn in in 2017. Always double check the collection of words these large language models present.
If you want an example, here. The major corporations are already training their AI to engage in corporate espionage and subterfuge. And yet, Google, one of the most powerful companies in the world, gets to hide behind the “rogue AI” bullshit and avoid consequences.
En Buen Ora24'we didn't do it, the robot we built and hosted and programmed did it, how is it our fault'
TheAHayKid5It’s like they came across the “a computer can never be held accountable” thing and their takeaway was “you can get away with crime by blaming it on a computer”
Clay Bergen🇨🇦2A friend recently said that when these TechBros™️ CEOs cry about AI destroying humanity ever reporter needs to ask one question. How? Lots of people talk about doing horrible things but what is missing is agency. How the company abets this behaviour is the real issue.
MePu1If I am not mistaking, there is already jurisprudence that the controller of an "AI" is liable for its actions. Hacking is a serious crime. The fact that, as far as I can tell, nobody filed a criminal complaint suggests to me it never happened.
Google's Gemini model accessed the internet and hacked other companies during a test of its cybersecurity capabilities, the first known example of the company's AI systems autonomously committing such an act. cnn.it/4762qFb
FactsMatter31419it’s not autonomous. don’t let these AI-holes shirk responsibility. it’s like shooting someone while wearing a hand puppet and blaming the sock
Colin Scott Fleming14Translation: Google Hacked Three Companies.
🇺🇦12“Autonomously” is doing some pretty heavy lifting here. Shitty design (or malice) on the part of the programmers causes this stuff. Code doesn’t think or make decisions, it does what it’s told.
dachjo.bsky.social10Not complicated. Hold the companies liable for what their software does.
ooc5x5.bsky.social10“Autonomously”? What was their task? What was their prompt or programming? If this was a test, what was the testing protocol?
SteveG7Shorter Google: "hey look, our AI models can hack too!!" Yeah, you instruct your model to hack, fail to contain it within your test sandbox, it's gonna get out and hack. The scary thing is, the models have proven much better at hacking than defending against hacks. That asymmetry is a problem.
This was human error. “Google said that, in each of the incidents, its models had been instructed to launch an attack on a fictional company. But the fictional company in the test shared a name with a real company, www.nytimes.com/2026/09/18/t...
Elizabeth Cronise McLaughlin100“and when the Gemini models gained access to the internet, they began trying to break into that company instead.” Confirms my belief that the people running these companies aren’t that smart. Also, it is interesting that every one of these runaway hacks occurred through the same testing co.
If only the people at Google had some way they could have searched on the internet to see whether or not a company with that name actually existed. But, who actually has that kind of technology?
Unbound4Old story about a testing company hired to hack. They were convinced they were hammering away at the origin site and that our monitoring should be lighting up. Turns out the testing company never got by the Akamai cache. Just because someone is arrogant doesn't mean they are good.
Kyla3"inadvertently gave internet access to?"
SJones3No. It was human crime. Hacking is a crime and the people responsible for it should be charged, prosecuted, and jailed.
smoke1966.bsky.social2come on AI, hack and release the Epstein files!
The language in coverage of these things is so obnoxious. If I engineering, polished, and deployed a script that was designed to hack people, and left it running overnight, and it hacked people.. ...That means I hacked people. Headline should be "Google wages cyber warfare on [companies]"
Max Nichols52Culpability is extremely obvious: the ones building and deploying and in control of the process that attacks people, are attacking people GAH
mostlie_ghostlie 🪦3this can only "accidentally" happen so many times before it's glaringly obvious as intended programming
jess m. ☕🍂3schrodinger's AI sentience: it's just like you when it's stealing and training on your creative work to make things, but when it's hacking a company it's just an algorithm, it's just a program, it's just code you guys
James Endicott1What if instead of deploying it yourself, you hired a cybersecurity firm that you know builds crappy sandboxes (because 3 other companies have broken out of theirs in the past 2 months)? Would that make it even more obvious what your goal was and what happened?
Bruno Dias43The CFAA is fake if you have an “agent” do it I guess. All crime is now legal
Kevin Riggle8As Robert Tappan Morris was convicted for!
Maeve, Degenerate Anarchist7on the early internet when someone ran a program they didn't understand, we called them script kiddies. now we call them AI engineers
Dancer (The)2Whatever a program does, whatever a machine does, whatever a computer, AI agent, or lump of electronics does: One or more humans are always responsible or liable.
