Cracks, hacks, attack: California's vulnerable water system faces many threats (2023)

A February morning in 2021 noted an operator of water treatment plants in Oldsmar, Fla., Something unusual: An unidentified user had remote access to the system's computer system and moved the mouse around the screen.

The operator watched as the intruder clicked into various software programs before landing on a function that controls the amount of sodium hydroxide, or lye, in the plant's water system. The hacker then increased the amount of lye - a potentially dangerous substance used to control acidity - from 100 parts per Million to 11,100 parts per Million.

The plant operator almost immediately reversed the change, and officials said there was never any threat to public safety. But the incident has highlighted the threats facing major drinking water systems across the country.

"Water systems, like other public utility systems, are part of the nation's critical infrastructure and can be vulnerable targets when someone wants to adversely affect public safety," said Sheriff Bob Gualtieri of Pinellas County, Florida.said then.

In California, where epic Sierra Nevada -snowpack and"the great melt"Has significantly increased efforts for reservoir managers, officials say they are taking steps to protect the state's water systems from hackers, terrorist attacks and natural disasters, such as the flooding that temporarilycut off the Los Angeles Aqueduct- The city's water lifeline, which connects to the Owens Valley.

But experts say the challenges are many. Many of the systems in California and nationwide still operate with outdated software, bad passwords, aging infrastructure and other weaknesses that can leave them at risk.

"We have seen a steady increase in both the occurrence and effect of cyberin rusions as well as an extraordinary increase in ransomware -attacks, which has become more destructive and more expensive," said Joe Oregon, Head of CyberSecurity for Region 9 IFEDERAL CYBERSECURITY & INFRASTRUCTURE SECURITY AGENCY or CISA.

Andrew Reddie, an assistant professor in practice in cyber security at UC Berkeley's School of Information, said much of the problem is "driven by the fact that the infrastructure is really, truly old and ultimately precedes the era we findus in now that we are actually baking cyber security into these ... systems by design. ”

"You can point to any number of critical infrastructure, including things like dams and water treatment plants, that are not very well protected in terms of passwords," he said.

Crews install earthquake-resistant ductile iron pipe along the Metropolitan Water District's Colorado River Aqueduct in San Jacinto in 2022.

(Irfan Khan / Los Angeles Times)

A lot of older infrastructure is not "air gap" from the Internet, he said, referring to a separation between operational technology and internet technology.It could enable a bad actor to do things like changing chemical levels or open locks to manipulate currents in water channels or dams.

Agating the problem is a lack of central regulation or uniform protocols.More Agencies - including Environmental Protection Agency, National Institute of Standards and Technology, American Water Works Assn.And the Department of Homeland Security and CISA - provides a certain degree of risk management monitoring or offering frameworks and recommendations.But many of the daily decisions are left to the individual operator.

"Much of the responsibility certainly falls on the stakeholders' shoulders to manage their own information systems effectively to prevent any kind of cyber -comprom or cyber events," Oregon of CISA said.

The agency estimates that approx. 63% of the nation's 91,000 dams are privately owned. Federal, state and local governments and utilities own 35%, and the remaining 2% have "indeterminate ownership."

Despite the risk, experts said it is important for water systems to become a network to speed up maintenance and monitoring. In California, reservoirs often spread intentionally far apart to maximize rainwater trapping and other benefits, so sending physical herds to respond to respondOn any potential problem would be time-consuming and expensive, said Ethan Schmertzler, CEO of Dispel, a cyberdefense company.

"It all depends on how water systems are connected and most water systems in the United States are not - it's not a national water system," he said. "The good news is that each community is divided into their own command and control systems.The downside is that they are all divided into their own command and control systems."

Although most standards are not mandatory, cyber security recommendations - and expenses - have been significantly improved in recent years, he said.Recent legislation through the Defense Authorization Act willSoon, utilities will be forced to report cybersecurity threats to CISA, which will help the federal agency better see trends, share information and provide a response.

John Rizzardo, security coordinator with the State Water Project at the California Department of Water Resources, said the agency operates with an ethos of "layer upon layer of security," for both physical and cyber threats. Because the agency is also an energy supplier in the state, "we probably employ more security functions than a lot of just the water industry," he said.

An air image of water flowing out of the compromised Oroville Dam emissions on February 21, 2017.

(Marcus Yam / Los Angeles Times)

However, it does not mean that it is immune.cisapointed to the Oroville dam crisisfrom 2017 as an example of the country's need for "extensive supervision and guidance on dam resistance."During this incident, erosion on the hillside on the dam's emergency spillthreatened with a major floodAnd asked the evacuation of about 200,000 people, although the disaster was ultimately averted.

Rizzardo said the agency has since supported the spill and made significant safety upgrades and is working to implement the same standards across all State Water Project facilities. The Department of Homeland Security runs national security exercises for the dam sector every two years, he said, in which the agency also participates.

But even with the best protocols in place, "there will still be a risk of a cyber or physical attack," Rizzardo said."It can happen - we do our best to prevent it - but if that happens, we will practice our emergency action plans regularly, so we are prepared if there is some kind of attack that we can try to mitigate, to reduce the consequences."

In fact, the Oldsmar incident was not a single.A few months later, aransomware attack on Colonial Pipeline- A vital US oil lines between the Gulf of Mexico and the East Coast - spurred for fuel deficiency, flight cancellations and an exceptional state declaration from President Biden.

Earlier this year, Biden unveiled anational strategy for cyber securitythat calls for a "more conscious, more coordinated and more resourceful approach to cyber defense."

Similar attacks have threatened other water systems, including aIranian attack on a New York damIn 2016, when hackers tried but could not take control of a sluice gate.

In January 2021, an unnamed water treatment plant in San Francisco Bay Area was also exposed to a cyber attack,NBC News first reported.Hackers gained access to the system's system through a Remote Access TeamViewer account and deleted programs used to treat drinking water. The programs were reinstalled the next day and no errors were reported (Northern California's regional intelligence center was being preparedA report on the incident said it could not give more details as an investigation is in progress.)

One of the largest water providers in the country is the Metropolitan Water District of Southern California, a massive regional wholesaler that supplies 26 agencies serving 19 million people, including the Los Angeles Department of Water and Power.

General Manager Adel Hagekhalil said in an E -mail that itAmerica's Water Infrastructure Act fra 2018served as a "catalyst for utilities to evaluate their resistance to risks and create emergency plans to respond to all dangers."

"We are constantly taking steps to ensure the security of our water supplies against physical and cyber security threats," Hagekhalil said. He noted that local water systems serving more than 3,300 people are required to actively update their risk and resilience assessment and emergency preparedness plans every five years.

In addition, MWD Cyber Security experts employs and constantly monitors network and computer activity to "detect unusual events quickly so they can be resolved," he said.Computer and networking access is tightly controlled and employees are also required to take an annual cyber security education.The agency also performs periodic emergency management exercises at various facilities to simulate reactions to physical threats such as earthquakes, floods, fires and terrorist attacks, which include first aids and law enforcement authorities, he said.

But the United States is home to more than 55,000 public water systems and 16,000 wastewater systems, said Jennifer Lyn Walker, director of infrastructure cyberdefense at Water Information Sharing and Analysis Center. One of her primary concerns was that there is often a "lack of awareness" about the potentialfor cyber threats and other such vulnerabilities.

"Physical threats are so much more top of mind, or more easily identified, or more easily understood than the cyber threat," she said. "The concern is a lack of preparedness."

But most major systems in California "do what needs to be done" when it comes to cyber security, she said.However, small and medium -sized systems that often have fewer resources than larger providers may need assistance and may benefit from guidance from major operators.

"A smaller system that just barely serves 5,000 people - it is still 5,000 people's lives that may be in danger if something should happen, and it is from physical or cyber [threats]," she said.

Reddie from Berkeley said that more revision would provide a better understanding of which systems are networking, as well as which systems follow the best practice. He also recommended to train work forces on proper cyberhygia.

However, even with such steps in place, there are still vulnerabilities.Ongoing investigationsinto the Oldsmar incident indicates that it may not have been the work of an external hacker at all, but may have been caused by an internal employee. Should that prove to be the case, it would highlight that insider threats can also be a cause for concern, Reddie said.

"These individual companies have to think about what's their model for the type of threat actor that they're likely to see," he said. "Like, is this going to be a state actor? Is it going to be a disgruntled employee? Is it going to be, you know, a manuscript kiddie in a basement? "