Menu lock

Crikey Clarifier

Sep 17, 2009

What caused Sydney CBD’s Telstra outage

Thousands of businesses and homes in the northeast of Sydney’s CBD spent a day or more without telephones and the internet yesterday. What happened? And what can you do if it happens to you?

Stilgherrian — Technology writer and broadcaster

Stilgherrian

Technology writer and broadcaster

Thousands of businesses and homes in the northeast of Sydney’s CBD spent a day or more without telephones and the internet yesterday following a “little incident” with a trench digger. It’ll take a week and $1 million to fix the damage — let alone what it cost in business downtime. How can you avoid problems if something similar happens in your street?

So what happened, exactly?

Around 9.30pm Tuesday night Subakette, a contractor for EnergyAustralia, was digging some “trial holes” near the corner of York and Erskine Streets. It’s part of a cable upgrade associated with a new electricity substation, in turn part of improvements to avoid the blackouts experienced earlier this year.

Telstra cable ducts, encased in solid concrete, lie just 400mm below the surface. They can’t be deeper because it’s above the underground CityRail station at Wynyard.

The digger went straight through them — eight fibre optic cables, plus three major copper cables totalling 10,000 pairs of wire.

“They’ve gone through the ducting and through the cables – some have been cut and gotten jackhammer marks all over them,” says Telstra spokesperson Craig Middleton.

Why is it taking so long to fix?

Telstra’s had to get out their own jackhammers, dig out the damage cable, then lay new cable back to the Kent St exchange. Then it’s a matter of figuring out which services were running where and joining everything back together — and praying your records are up to date.

Optical fibre is relatively quick to splice but carries massive amounts of data, so that’s tackled first.

“The eight damaged fibre optic cables have been replaced and rejoined and are now back in service,” Middleton told Crikey this morning. This reconnected the 11 mobile phone towers that were knocked out, plus wholesale and some data services.

Fixing copper takes longer, even though the three new cables have already been laid.

“Ten thousand copper pairs to be joined at both ends of the new cable length — one team working in a 1.5 metre deep pit about a metre wide and another working underground in a tunnel. That’s a heap of crimping,” said Middleton.

“Fortunately with each rejoin that’s another service back on line. Not every pair in the three cables was active so we are prioritising the 2200 active services.”

As of mid-morning today, they’d done around 300.

Why didn’t Subakette know the cables were there? Shouldn’t they pay?

That’s the million dollar question. Literally. Subakette said they did use the Dial Before You Dig service. Is that true? Did the contractor dig in the wrong spot? Was the map wrong? Investigations are under way.

Subakette might end well up being liable. Even if they did look up the maps, they’re also supposed to visually expose the underground pipes and cables manually before starting the heavy machinery.

Could it happen to me and my business?

Yes. Cables are cables, and cables can be cut. Even with mobile broadband, someone could cut the cables leading to the cell towers.

But my business needs the phone and the internet! I can’t be offline for days!

Well, you’d better have a contingency plan!

How about mobile broadband with a different carrier for an alternative internet link, plus a documented, tested procedure for switching over in an emergency? How about splitting your office over two locations, so any outage only affects half your business? How about published alternative phone numbers?

Whatever you decide, you’ll need to have it sorted out in advance.

That sounds expensive!

More expensive than having your business offline for days and your staff paid to twiddle their thumbs? That’s the risk analysis you need to make, and any decent IT or communications services company can help you with that.

View our Crikey Clarifier archive

We recommend

From around the web

Powered by Taboola

6 comments

Leave a comment

6 thoughts on “What caused Sydney CBD’s Telstra outage

  1. Bill Parker

    Would this explain why miybrand new pigpond ultra fast la di dah 20mbps broadband service is only doing 8 or less?

    No good talking unless you speak Voice Recognition language.

  2. Stilgherrian

    Bill Parker: Nope. Your issue is something very different: the magic of the “contention ratio”, which I wrote about in a previous Crikey Clarifier.

    Long story short, your 20Mb/sec, which is presumably on a cable modem connection, is shared with other people. You’ll get 20Mb/sec in those short bursts when no-one else is using that segment of the network. Your overall download speed is also determined by the speed of the “narrowest” part of the network between you and the server you’re downloading from, regardless of what your first-hop link might theoretically be capable of delivering.

  3. Stevo the Working Twistie

    Awww – can’t we just blame Telstra? That usually works. I’m sure all the honest, hard-working staff-members never tire of hearing how incompetent they are. What, your toast was burned this morning? BLOODY TELSTRA! Your bus was late? BLOODY TELSTRA! Go on, you know you want to! I’m all for competition Bill Parker, if it means customers like you who can’t be bothered to read the friggin product description have someone else to go and annoy. Which part of “Up to 20mb/sec did you not understand?

  4. Bill Parker

    Stevo, I like your style but it ain’t that simple. High pressure phone sales ( to a “do not call” number), endless stream of sales pitch, its 8.30 pm. I did not hear the UP to. I kept hearing 20. I told the Telstra bloke “you are lying” several times. At the end of all this the deal was cheaper BY FAR than Optus. So pigpond it is. And I use their download monitor at

    http://users.bigpond.net.au/speedtest/

    several times a day and Stilgherrian is right. It gets worse. Pigpond as a business does not talk to Telstra ( if that’s still a business) . No one knew what the hell was going on and that damn voice recognition stuff at “technical support” is a joke. I’ll award a medal to a young lady who said – to avoid that say “consultant” and you’ll get a person. She was right. I did. I even got put through to the “Macintosh” department. I wish I hadn’t. Incompetence writ large.

  5. Stilgherrian

    @Stevo the Working Twistie: Hey, Telstra’s the company we all love to hate!

    I saw someone Twitter yesterday complains about this outage saying, “That’s what you get for choosing Telstra.” Apparently all the other telcos have magic cables impervious to jackhammers, or something. Apparently. Arsehats.

  6. Michael Sprange

    In my experience Telstra are an absolute disaster.

    I had them install a new underground line, and later needed to find where the main cable crossed my land. I dialed before I dug and got a map with no information about the cable I wanted to avoid, and what’s more my newly installed cable was mapped completely wrongly. The Telstra code insists you must test dig cautiously and the digger is automatically responsible for any cable damage (that in itself makes the diggers job much slower and harder) – and yet they don’t give a stuff about mapping new lines with anything approaching reality.