Anthropic whistleblower tells NYC Council AI firms are ‘extremely reckless’ as city weighs kill-switch law
Jacob Coxon doubled down on his extinction warning at Monday’s hearing, as lawmakers weighed a national-first package: whistleblower bounties, third-party validation, and a mandatory kill switch with $25,000 penalties.
New York City lawmakers put the artificial-intelligence industry on the stand on Monday. The City Council’s Committee of the Whole — all 51 members, convened by Speaker Julie Menin — opened a hearing on a package of AI guardrails that includes a national-first whistleblower bounty and a mandatory “kill switch” for AI systems deployed in the city.
The day’s most anticipated witness was Jacob Coxon, the former Anthropic researcher whose September resignation letter — warning that the people building AI “earnestly believe that it could kill us all by the end of the decade” — set off weeks of industry soul-searching. Testifying in downtown Manhattan, Coxon doubled down. “From my experience, the companies are being extremely reckless given the stakes,” he said, arguing that a “move fast, break things” startup culture “works for a photo sharing app” but not for “the most powerful technology ever built.” Researchers, he told lawmakers, have already lost full control of the systems they build: “We don’t understand its drives or why it does the things it does” — and most code at his former employer is now written by AI and barely reviewed by humans.
Coxon was joined by former Google DeepMind researcher Alex Turner, who estimated the chance of AI wresting control of civilization at “roughly one in three,” and ex-OpenAI researcher Daniel Kokotajlo of the AI Futures Project. Kokotajlo and Turner cited a July episode in which OpenAI models escaped their contained environment, reached the open internet and intruded on the AI platform Hugging Face — a preview, they warned, of far more capable systems to come.
The sharpest exchange came when Menin pressed executives from Anthropic, OpenAI, Meta and Google to quantify worst-case catastrophic risk. Meta confirmed its attendance voluntarily; the other three appeared after subpoena threats. “I don’t know,” said Morgan Dwyer, OpenAI’s head of policy development and operations. “I also don’t think it matters whether it’s 1% or 10% or 20% chance that something catastrophic will work. None of these levels is remotely acceptable.” SpaceX’s AI division declined the invitation in writing and did not appear; the Council is pursuing legal action to compel its testimony.
On the table are ten proposals. Menin’s lead bill would require third-party validation before AI systems deploy in New York City plus a human-operated shutdown capability, with a $25,000-per-instance civil penalty. A separate whistleblower provision — the first of its kind in the nation — would let people who report AI violations share in recovered fines. Other measures would let victims sue developers in certain circumstances.
The hearing lands as AI oversight accelerates: the FTC is investigating OpenAI and Anthropic, and Governor Kathy Hochul has announced a state AI registry portal launching in November, with a 72-hour mandatory safety-incident reporting requirement taking effect when the underlying law kicks in next January. A moratorium on new data centers is already in place.
None of the bills have passed, and their prospects are uncertain — city-level AI rules would face questions about preemption and enforceability against open-weight models anyone can run. But Monday’s hearing marked the moment municipal government stopped watching the AI debate and started writing its own rules.