The only two Telstra engineers with deep expertise in the system that failed in July had been sent home on mandatory rest by the time the national mobile outage they had unwittingly triggered took hold, an external investigation has found.
The outage cut hundreds of people off from Triple Zero, halted trains in Victoria and NSW, and brought down eftpos systems across the country. It has triggered a Senate inquiry and an investigation by the communications regulator that could result in Telstra being fined up to $30 million.
The pair had replaced a faulty power supply in a timing chassis in Melbourne in the early hours of July 8. When it came back online at 2.50am, a GPS card inside reset and began telling the mobile network the date was 2006. The card was missing a firmware update for a predictable fault the supplier had already flagged in bulletins, confirming this masthead’s July reporting.
The report by Technology Audit Partners found alarms on the timing servers were checked only during business hours by a small number of staff, and did not appear in the monitoring tools used by round-the-clock support teams. When things went wrong, nobody on duty could find records of the servers, the work done that night, or the people to call.
The outages have heaped fresh scrutiny on Australia’s Triple Zero system and the reliability of the nation’s telecommunications networks more broadly.
Chief executive Vicki Brady said the company accepted every finding.
“At its core, this came down to the network timing system not being given the priority that a critical network capability requires,” she said on Wednesday. “That is a miss on our side.”
Brady also revealed Telstra had missed eight failed Triple Zero calls during the outage and never carried out the welfare checks required to find out whether the people who made them were safe. She blamed a manual processing fault and a monitoring fault.
The checks have since been done. One caller got through on another network, and another found a different way to connect. Five either needed no help or had misdialled. Police visited the home of the unidentified eighth caller.
“It shouldn’t have happened. We should have picked those eight up,” Brady said. “We know how important Triple Zero is.”
The eight are on top of the hundreds of unsuccessful Triple Zero calls Telstra has already disclosed.
The telecommunications company also revised the start of the outage to 2.50am on July 8, almost 50 minutes earlier than it had said. The previous time reflected when a maintenance ticket was closed, rather than when the fault began.
Brady said the failure was “not due to infrastructure failing” but to an undocumented design change made in October last year and a software update that was never applied to a GPS card.
Despite the auditors finding the team responsible had been constrained by budget priorities, Brady rejected suggestions of underinvestment, saying Telstra employed about 2800 network engineers and had spent more than $19 billion on capital works and spectrum over five years.
“At its core, this came down to the network timing system not being given the priority that a critical network capability requires.”
Telstra chief Vicki Brady
“It’s not about not enough resources. It’s not about not enough money,” she said. “It’s about the prioritisation.”
The report also faulted a lack of curiosity among staff who saw warning signs and did not chase them, a criticism that Brady accepted.
“That is a lesson we’ve got to take,” she said. “We have to be always open to challenging the way we’re doing things.”
Asked by this masthead whether anyone had lost their job over the outage, Brady said it was not the fault of an individual.
“Clearly, for us to have an outage like this, our processes, our controls were not sufficient,” she said. “It’s not about anyone losing their job.”
The board has already docked her more than $600,000 in pay, with just under $2 million cut across her senior leadership team.
More than 30,000 customers have contacted Telstra for compensation, and it has paid just under $1 million in credits, an average of about $15 for consumers. Brady urged anyone still out of pocket to come forward.
“Modern networks are complex but complexity is not an excuse,” she said.
The Business Briefing newsletter delivers major stories, exclusive coverage and expert opinion. Sign up to get it every weekday morning.
David Swan is the technology editor for The Age and The Sydney Morning Herald. He was previously technology editor for The Australian newspaper.Connect via X or email.From our partners
About the Author
