Water Resources’ Email Server Breached

A breach of the Geauga County Department of Water Resources’ email server last week caused tempers to flare during an emergency Automatic Data Processing board meeting April 13.

“On Wednesday, April 12, around 4 a.m., a product called CrowdStrike Falcon … began noticing possible nefarious scripts and command line activity on a critical water resource server,” said Geauga County Auditor and ADP Chief Administrator Chuck Walder, addressing Geauga County Commissioners, Prosecutor Jim Flaiz and Water Resources representatives who were present for the meeting.

“Shortly before 8 a.m., ADP staff began receiving a series of serious high-priority alerts through ADP Cybersecurity Center, from CrowdStrike, indicating what appeared to be a significant and persistent threat attack on this water resource server,” he continued.

CrowdStrike Falcon — an endpoint cybersecurity product ADP installed on all servers and workstations on the county’s network — alerted ADP to possible nefarious activity attempting to access and control the server, Walder said. Given the persistent nature of the critical attack, CrowdStrike automatically blocked access to the server and put in motion a series of procedures and instructions for ADP to further isolate and protect the county’s network infrastructure.

“ADP personnel immediately notified water resources of the attack, blocked all inbound water resource domain traffic, removed water resources from all shared ISP switches and began a deep scan of all county systems to ensure that the county’s environment under ADP control was secure and not affected,” he said.

Walder said the server in question is an “end-of-life, end-of-support server,” meaning it is running an operating system from 2012 and software from 2016 that has not been properly service-patched.

That vulnerability likely allowed an outside actor to penetrate the server through Exchange — an email program — and attempt to run a series of commands.

“The server was ultimately powered off by water resources staff, preventing further analysis by ADP or CrowdStrike,” Walder said.

CrowdStrike and ADP were successful in containing the attack with no disruption to other county services or systems under ADP control, he added.

However, the infected email server is one of five servers the water resources department operates without ADP’s oversight.

Walder said the department has also neglected to keep its other vulnerable servers patched and up to date.

Fingers Pointed

Water Resources Director Steve Oluic said he had no email access and had not seen any information regarding the server.

“I got an email at eight o’clock and then shortly thereafter, we were shut down. No phone calls or anything, so I don’t know (what happened) because we haven’t got any report,” Oluic said.

“It’s your server,” Flaiz responded. “Water resources is running it. I would expect water resources to come to this meeting and explain to us what’s going on.”

Flaiz asked water resources Network Administrator Michael Kurzinger what he knew about the attack.

“When (CrowdStrike) sees an attack, it will go in and shut that server off from the network or from any other access … so that threat ended at that point until remediation can take place,” Kurzinger said.

Kurzinger said he made many attempts to contact ADP Chief Deputy Administrator Frank Antenucci but did not receive an answer.

“You guys were busy,” he said. “They said to call back in a couple of minutes. I called back. I was told by your help desk, ‘Call Frank at this extension.’ I cannot give you any more information.”

Flaiz asked Kurzinger why water resources was running an Exchange server and hadn’t yet switched to Microsoft 365 — an update of their current software that is still covered by service patches.

Kurzinger said he was instructed not to switch to the new software by County Administrator Gerry Morgan.

Flaiz presented a Feb. 2 email in which Morgan told Kurzinger not to move forward with a switch to the more secure Microsoft 365 until mediation between the commissioners and ADP was finalized.

(See related story – Lawsuit dropped)

Walder asked what mediation had to do with the department running its email server — Morgan replied commissioners were under a “gag order” and could not talk about what happened.

“If you had migrated to Microsoft 365, would you be having this problem right now?” Flaiz asked.

“I don’t know,” Oluic quickly responded.

In response to a comment from Commissioner Tim Lennon, who said the county has been in discussions regarding interactions between water resources and ADP, Flaiz said the ADP board is used to the department blaming them.

“This is Gerry’s fault,” Flaiz said.

Flaiz asked Morgan if he would allow the migration to Microsoft 365 and Morgan agreed to move forward with it, saying he already spoke to water resources about implementation.

“Well, you didn’t talk to ADP or the board, did you? Apparently, you run the whole county,” Flaiz shot back, adding Morgan has only given his blessing now that there’s been a cyber-attack.

“Gerry, you lie to me all the time,” Flaiz said. “When you sat in this meeting almost two years ago and told us that water resources was going to completely come under ADP, do you remember sitting there telling us that?”

Morgan responded the department was working toward that goal.

“When you haven’t done it in two years, I will say that you are lying,” Flaiz said. “You stopped the migration and now they don’t have email. I know you’re anxious to blame Frank and Chuck, but it’s your fault.”

Walder said ADP has installed Microsoft 365 everywhere in the county except for water resources because of the difficulty ADP has had in dealing with the department, especially after an issue last year at the McFarland Wastewater Plant.

Morgan asked if water resources could transfer to Microsoft 365 themselves without ADP.

“ADP does IT,” Flaiz responded. “We are not doing these games anymore.”

Morgan pushed his point, saying the department could solve the current issue by moving to Microsoft 365 on its own, then converting to ADP administration of IT services at a later date.

“Why would we do a later date, Gerry? It’s been two and a half years migrating water resources supposedly into ADP,” Walder said.

Resolution

After a lengthy discussion, the ADP board passed a motion to migrate the water resources email server to Microsoft 365 and perform any other services necessary to get the department operational.

Water resources will cover any costs incurred and ADP will attempt to recover historical email data if possible.

After the motion, Lennon asked about the possibility of designating a person to serve as a liaison between ADP and water resources.

“It doesn’t have to be the directors or whatever, but whoever is comfortable being able to communicate together and talk to each other,” Lennon said. “Is that possible?”

Antenucci suggested designating Budget and Finance Manager Adrian Gorton because of his experience with water resources.

Oluic quickly disagreed while Walder added it is difficult to volunteer someone for a position which could make them subject to being named in a lawsuit.

“If any of you walk in those shoes for a minute, I get to pick the next person who could be subject to have his name put on a lawsuit. I’ve never in my life experienced a county naming a person (in a lawsuit),” Walder said, referring to a lawsuit filed last September by county commissioners personally naming him, Antenucci and an ADP employee.

Morgan left the room as Flaiz said that lawsuit was “classless” and uncalled for.

“(The lawsuit) was vindictive by the guy who just walked out of the room,” said Flaiz. “It is what it is.”