California AI regulations mandates reporting of serious incidents and protects whistleblowers

Governor Gavin Newsom signs new Bill to regulate the work of AI developers

The Governor of California, Gavin Newsom, has signed a new Bill to regulate the operations of AI developers operating in the state. In addition to protecting whistle blowers, it aims to force greater transparency in AI development and reduce the risk of harm.

The new Bill makes AI developers responsible for reporting incidents carried out by their technology, among other measures.

Those incidents might relate to crimes or critical safety issues – “post-deployment impacts”, in the Bills vernacular – which must be reported to California’s Office of Emergency Services.

Moreover, the new laws also include protections for whistleblowers.

“Unless they are developed with careful diligence and reasonable precaution, there is concern that advanced artificial intelligence systems could have capabilities that pose catastrophic risks from both malicious uses and malfunctions, including artificial intelligence-enabled hacking, biological attacks, and loss of control,” the Bill claims.

It continues: “With the frontier of artificial intelligence rapidly evolving, there is a need for legislation to track the frontier of artificial intelligence research and alert policymakers and the public to serious risks and harms from the very most advanced artificial intelligence systems.”

However, while developers have put together frameworks of industry best practice, reporting remains inconsistent “to ensure necessary transparency and protection of the public”.

It continues: “Mandatory, standardized, and objective reporting by frontier developers is required to provide the government and the public with timely and accurate information.”

While the Bill is currently focused on the major AI developers, including Google, OpenAI, Meta and X, formerly Twitter, the Bill also opens the prospect of being extended to include smaller developers that “may pose significant catastrophic risk”.

The Bill expects developers to publish how they “write, implement, comply with” AI frameworks, incorporating national and international standards, defining their thresholds for “whether a frontier model has capabilities that could pose a catastrophic risk” and how they respond to safety-critical incidents. They are also expected to outline their governance practices.

Catastrophic risk is defined as a foreseeable risk that could result in the death or serious injury of 50 or more people, or more than $1bn in damage.

The Bill also includes a series of protections for whistleblowers, including internal processes by which anonymous disclosures can be made, and damages for staff who believe that they have been discriminated against for raising concerns over AI frameworks.

The industry in the US has been split over the new laws. Some lobbied to have the measures watered down, possibly on the grounds that new regulations often make the emergence of new competition more costly and difficult, while others were broadly supportive.

The California Bill mirrors similar initiatives in the UK and EU. Earlier this month, the-then Technology Secretary Peter Kyle announced a £2.7m fund for AI regulation reforms, aiming to speed up innovation while ensuring oversight and boosting the UK’s tech competitiveness.

In addition, the Artificial Intelligence (Regulation) Bill Private Members Bill was introduced to the House of Lords in March by Lord Holmes. If passed, the Bill would establish a new AI Authority in the UK, which would seek to ensure regulators from different sectors of the economy were aligned, and that any regulatory gaps are identified.

That Bill had first been introduced in 2023 but fell following the dissolution of Parliament for the General Election last year.

The EU, meanwhile, introduced its AI Act earlier this year, offering a comprehensive framework on artificial intelligence using a risk-based AI classification system.