The unprecedented wave of high-profile cyberattacks on US water utilities over the past year has just kept flowing.
In one incident, pro-Iranian hackers penetrated a Pittsburgh-area water utility’s PLC and defaced the touchscreen with an anti-Israel message, forcing the utility to revert to manual control of its water pressure-regulation system. A water and wastewater operator for 500 North American communities temporarily severed connections between its IT and OT networks after ransomware infiltrated some back-end systems and exposed its customers’ personal data. Customer-facing websites and the telecommunications network at the US’s largest regulated water utility went dark after an October cyberattack.
Those were just some of the more chilling stories that have recently sparked fear over the security and physical safety of drinking water and wastewater systems. The cyberattacks have spurred warnings and security guidelines from the Cybersecurity and Infrastructure Security Agency (CISA), the White House, the FBI and the Office of the Director of National Intelligence (ODNI), the Environmental Protection Agency (EPA), and the Water ISAC (Information Sharing and Analysis Center).
Most of the attacks landed on the softest of targets, small water utilities without security expertise and resources, in mainly opportunistic attacks. Meanwhile, cyberattacks on large utilities like Veolia and American Water hit IT, not OT, systems — none of which actually disrupted water services. Overall, the cyberattacks on water appeared to be mainly about “poking around and eroding confidence,” says Gus Serino, president of I&C Secure and a former process control engineer for the Massachusetts Water Resources Authority.
The race is now on to secure the water sector — especially the smaller more vulnerable utilities — from further cyberattacks. Many larger water utilities already have been “stepping up their game” in securing their OT networks, and others started building out their security infrastructures years ago, notes Dale Peterson, president of ICS/OT security consultancy Digital Bond. “My first client in 2000 was a water utility,” he recalls. “Some [large utilities] have been working on this for a very long time.”
The challenge lies in securing smaller utilities, without overprescribing them with unnecessary and high-overhead security infrastructure. Tools that require expertise and overhead are a nonstarter at sites where there isn’t even dedicated IT support, much less cyber know-how. Peterson argues that government recommendations for sophisticated security monitoring systems are just plain overkill for most small utilities. These tiny outfits have bigger and more tangible priorities, he says, like replacing aging or damaged pipes in their physical infrastructure.
ICS/OT Cyber-Risk: Something in the Water?
Like other ICS/OT industries, water utilities of all sizes have been outfitting once-isolated programmable logic controller (PLC) systems and OT equipment with remote access, so operators can more efficiently monitor and manage plants from afar — to control water pumps or check alarms, for instance. That has put traditionally isolated equipment at risk.
“They are starting and stopping pumps, setting changes, responding to alarms or failures [in] a system. They remote in to look at SCADA/HMI screens to see what’s wrong or to take corrective action,” explains I&C Secure’s Serino, who works closely with water utilities. He says it’s rare for those systems to be properly segmented, and VPNs are “not always” used for secure remote access.
PLC vendors such as Siemens are increasingly building security features into their devices, but water plants don’t typically run this next-generation gear.
“I have yet to see any secure PLCs deployed” in smaller water sites, Serino says. “Even if there are new PLCs, their security features are not ‘on.’ So if you [an attacker] can get in and get access to the device on that network, you can do whatever you are capable of doing to a PLC.”
Because many ICS/OT systems integrators that install OT systems traditionally do not also set up security for the equipment and software they install in water utility networks, these networks often are left exposed, with open ports or default credentials. “We need to help integrators making [and installing] SCADA equipment for these utilities make sure they are secured” for utilities, says Chris Sistrunk, technical leader of Google Cloud Mandiant’s ICS/OT consulting practice and a former senior engineer at Entergy.
Default credentials are one of the most common security weaknesses found in OT networks, as well as industrial devices sitting exposed on the public Internet. The Iranian-based Cyber Av3ngers hacking group easily broke into the Israeli-made Unitronics Vision Series PLCs at the Aliquippa Municipal Water Authority plant (as well as other water utilities and organizations), merely by logging in with the PLCs’ easily discoverable factory-setting credentials.
The good news is that some major systems integrators such as Black & Veatch are working with large water utilities on building security into their new OT installations. Ian Bramson, vice president of global industrial cybersecurity at Black & Veatch, says his team works with utilities that consider security a physical safety issue. “They are looking to build [security] in and not bolt it in,” he explains, to prevent any physical safety consequences from poor cybersecurity security controls.
Cybersecurity Cleanup for Water
Meanwhile, there are plenty of free cybersecurity resources for resource-strapped water utilities, including the Water-ISAC’s top 12 Security Fundamentals and the American Waterworks Association (AWWA)’s free security assessment tool for water utilities that helps them map their environments to the NIST Cybersecurity Framework. Kevin Morley, manager of federal relations for the AWWA and a utility cybersecurity expert, says the tool includes a survey of the utility’s technology and then provides a priority list of the security controls the utility should adopt and address, focusing on risk and resilience.
“It creates a heat map” of where the utility’s security weaknesses and risks lie, he says. That helps arm a utility with a cybersecurity business case in the budget process. “They can go to leadership and say ‘we did this analysis and this is what we found,'” he explains.
There’s also a new cyber volunteer program that assists rural water utilities. The National Rural Water Association recently teamed up with DEF CON to match volunteer cybersecurity experts to utilities in need of cyber help. Six utilities in Utah, Vermont, Indiana, and Oregon encompass the initial cohort for the bespoke DEF CON Franklin project, where volunteer ICS/OT security experts will assess their security posture and help them secure and protect their OT systems from cyber threats.
Mandiant’s Sistrunk, who serves as a volunteer cyber expert for some small utilities, points to three main and basic security steps small (and large) utilities should take to improve their defenses: enact multifactor authentication, especially for remote access to OT systems; store backups offline or with a trusted third party; and have a written response plan for who to call when a cyberattack hits.
Serino recommends a firewall as well. “Get a firewall if you don’t have one, and have it configured and locked down to control data flows in and out,” he says. It’s common for firewalls at a water utility to be misconfigured and left wide open to outgoing traffic, he notes: “If an adversary can get in, they could establish their own persistence and command and control, so hardening up the perimeter” for both outgoing and ingoing traffic is important.
He also recommends centralized logging of OT systems, especially for larger water utilities with the resources to support logging and detection operations: “Have the ability to detect a problem so you can stop it before it reaches the end goal of causing an impact.”
