Technical problems continue to plague Aunty’s Ultimo
studios as this internal email suggests. Whilst the content is very
detailed, it certainly paints a vivid picture of Sydney’s technical
This is the chain of events that happened on my last shift at Ultimo (Saturday
morning, 17 July). All times are only approximate and order of
events may differ slightly.
Mark Nealon came in to see me before he went home to inform me about
the after hours arrangement with Tech Services support (which was for
me to ring David Stead in case of emergencies). He also mentioned that
Peter G was still here, and wished me well for the weekend.
At 1.57am Tony Delroy buzzed on the stentofon seeking confirmation from
me that he was still on air. I switched my monitor to 702 (noticing
that it was slow in switching) and told him that he still had program
going to air. He told me that he had lost control of his studio.
Get Crikey FREE to your inbox every weekday morning with the Crikey Worm.
1.59am the stations failed to preset and I waited anxiously for the 2am
switch. As the clock ticked over to 2am, the stations failed to switch
and the alarms went off marking the beginning of a rather eventful
night – or should I say nightmare? (Now I know exactly how the
others must have felt).
I immediately asked St 222 to continue with programming as the stations
was still switched to them. The reply was that they were unable to do
so as the whole 221/222 studio complex had crashed. I then proceeded to
patch up the NLR stations. I used MSM1 from the old NRS rack 41 to rack
27. I rang Mark Nealon and David Stead to inform them that I had lost
Rang Peter G for help on the MCR mobile – after about a minute into the
conversation with Peter the mobile turned itself off for no apparent
reason (this fault should be looked at). I then borrowed TV MC’s mobile
which was also unusable as the battery was too low. Looking into my
bag, I was very lucky I had my mobile with me. It should be noted that
it is very noisy where the switcher computers are located. It is
imperative that we get a telephone installed near those computers.
With Peter’s help I proceeded to reboot the switcher and after about 25
minutes the switcher was still not responding. I was informed by
MCTV that the network was down and that the IT communications duty
manager was on his way in to fix the problem. He said that maybe the
network had affected our switcher. During my training I was concerned
about the impact of the network on the new switcher and asked what
would happen to the switcher if the network went down. The reply was
that the switcher would be safe and that only the clients would be
affected. If this is not the case, then something needs to be done to
ensure that the MCR’s interfaces to the switcher are not completely
reliant on the network being available. This brings me to another
point, that putting all your eggs in the one basket was not a very
That night I lost the NSS, PSS, NRS and monitoring. I would have had
more functionality if I had a portable radio! The switcher has had
three major crashes since it has been in service. Considering that it
has only been online since 20/6/04, we have to start thinking about
getting a more robust and reliable system in place.
Anyhow, back to that night. Had a quick look at the new switcher
patchfield, but I was unable to patch RNAU to Melbourne (so that
Melbourne can take control of the switching of RN stations). I did
manage to patch PSV1 to MSV1 for Melbourne to switch in the news for
NLR stations. Rang MCRM to inform them about the 3am news. It was
Dallas who was on duty, what a coincidence (or is it luck) that we were
both on duty when the old switcher went down.
I then decided to have another go at rebooting the switcher before the
3am news for RN, with no luck. Lost 5 mins of news to RN stations.
Managed to find a way of getting RNAU to Melbourne by patching RNAU
from the old switcher rack to UC1L and Melbourne looping it back to us
on MSM2. Then I patched MSM2 to RN stations. Melbourne now had control
over NLR, RN and JJJ (MSS3), three down three to go.
News and News Radio still complained about the loss of their DCART,
iNews, etc, etc. With the seriousness of the situation I rang Mark and
David again requesting urgent Tech services support. At about
3.30am I recalled Eddy to come in early to give me a hand before the
early morning rush.
Peter rang and suggested we close down the s/by Vadis computer because
it may have taken over. Did that, still no joy. A sleepy Eddy
At 4.30am, rang Simon Marnie to see if he could come in early so that
he could check out ST 221 as Tony was having problems. I was surprised
to hear that he was doing an OB that morning. “Oh great … this is
going to be a challenge”. We decided to take the easy way out and have
Tom at the OB dialling directly into Melbourne’s ISDN and switching it
to MSM1 at 6.10am.
With no news broadcast able to come from Sydney at 5am (due to the
failure of… well everything), Dallas decided to repeat the 4am news
in it’s place. Not sure about the 5.30 news.
All news at this point were from Melbourne.
CFM stations are still switched to RDT1. No news for CFM at 5am, so in
it’s place I planned to play s/by fill prog by patching s/by cd to the
input of RDT1. Unfortunately I fell into a trap as the RDT1 input
switch was switched to
digital (instead of analogue), resulting in 30 seconds of dead air.
Spooled DAT 5 mins in for the 5.05am cross back. The 5.30 crosspatch to
MSS2 was seamless. We also lost about 2 mins of news for CFM at 7am due
to change of a shift in Melbourne who wasn’t aware of this requirement.
By 5.45am Melbourne had control over NLR, RN, CFM and JJJ. Brisbane had control of PNN and Orange had control of SERR.
At 6.30am IT duty coms manager finally came to work on MCR’s rack 42.
This is the first external help that we saw for the night from anyone
except for Peter outside MCR. With the seriousness of the situation, we
did not see anyone come to our help until NOW? Needless to say, we need
to urgently sort out a satisfactory solution for when problems such as
these arise and immediate external help (such as tech services) is
By 6.40am rebooted switcher’s MCPs, RJE switch and monitors now all ok,
but still having problems with the NRS, Eddy with Peter’s help solved
the problem. And by 6.55am we had control over NLR, RN and PNN. No AM
for RN at 7.05am St 242M fill.
First news out of Sydney was from 8am. AM for NLR was ok at 8am.
That’s about it.