S'pore 'has resilient telecom networks'

Enough back-up in place to cope with events like SingTel fire: IDA

By Irene Tham, The Straits Times, 24 Oct 2013

SINGAPORE has "resilient" telecommunication networks - despite the absence of a second set of connections to all homes and offices - the Infocomm Development Authority (IDA) said yesterday.

The IDA assured the public that enough back-up is in place to cope with events such as the Oct 9 fire at SingTel's Internet exchange at Bukit Panjang, which took down critical services across the island, from banking to health records retrieval.

Home broadband users including those of SingTel, StarHub and M1 were also cut off from the Internet, as the fire damaged national fibre broadband network builder OpenNet's cables housed in the same premises.

The IDA's comments came after the public criticised the country's lack of a back-up plan. It took more than a week to fully restore broadband services for home users, although all affected business customers had their connections restored after two days.

The IDA declined to comment on the speed of service recovery pending an investigation, but a spokesman said the diversity of operators here contributes to resiliency.

He said business and home users have more than 20 Internet service providers to choose from, and can select ADSL, cable and fibre fixed-line broadband platforms as well as 3G and 4G mobile networks.

Internet exchanges are also designed with different paths. Fibre cables come into the exchange via three to four lead-in pipes. When one lead-in pipe is damaged, as was the case in the SingTel fire, the cut-off cables can be physically reconnected to other cables in unaffected pipes without laying new cables.

The IDA spokesman said, however, that these factors cannot guarantee there will never be service failures and what matters is "a good recovery plan".

Addressing criticism that Singapore lacks a duplicate broadband infrastructure, the IDA said this is a balance between risk and cost management. For instance, there is no second set of water pipes supplying all homes here.

He added: "When we finish our investigations, there may be lessons we can learn."

Responding to public concerns that OpenNet's proposed sale to a SingTel-owned business trust would make Singapore even more reliant on one telco, he said: "Ownership and resiliency are two separate issues. Regardless of who owns the networks, there must be diversity and redundancy built into the networks to enhance resiliency."

* 'Series of safety lapses' caused SingTel blaze
Inquiry cites 'outdated' maintenance, misuse of blowtorch, undetected fire
By Grace Chua, The Straits Times, 17 Dec 2013

IT WAS a blowtorch that heated materials to a searing 800 deg C - twice as hot as it needed to be.

And in the end, it sparked a slow-burning fire that was undetected as a result of further fire safety lapses and human error.

That was the most likely cause of the blaze in the SingTel Internet-exchange room in October which resulted in a days-long stoppage of its services to many parts of Singapore, SingTel's board committee of inquiry concluded yesterday evening.

Comprising three independent members of the company's board, the committee also sharply criticised the telco's "outdated" cable maintenance system, which uses lead sealant to protect the exit points for cables.

This sealant needs to be heated and shrunk each time the cables need to be maintained.

On Oct 9, the SingTel employee who needed to heat a seal in SingTel's underground cable chamber at its Bukit Panjang exchange for cable diversion works borrowed a blowtorch from a contractor who was with him.

That torch was twice as hot as it needed to be, and it caused the sealant and plastic sheaths on the cables to overheat and smoulder, eventually catching fire.

All this went undetected because the employee - a veteran who had been with SingTel for 30 years - neglected to turn the smoke detectors back on when he went for lunch.

On top of that, the cable chamber was not equipped with an automatic fire suppression system, so the fire could not be put out when it was eventually discovered an hour later.

These were the conclusions of fire investigation experts, who ruled out four other possible causes: an electrical fire, a flammable gas explosion, a fire caused by a lit cigarette, and sabotage.

The committee's report slammed the use of lead sealants and open flames, calling it an unnecessary fire risk. It also called the use of lead "outdated" and a "health hazard".

The committee also engaged consultants to assess SingTel's network resiliency and public relations. The telco did much better on both these fronts, and Bell Labs praised its use of "innovative processes" such as multiple teams to splice cables in carrying out repairs.

SingTel chairman Simon Israel said the company's management accepted all of the committee's findings and recommendations.

The telco will switch - by the end of next year - to alternative sealing methods that do not require hot works and use flame-retardant PVC instead of polyethylene.

It also pledged to strengthen fire monitoring and work with the telecommunications industry to collaborate on disaster recovery, among other initiatives.

The worker concerned has been suspended with pay, and SingTel management will now decide on appropriate disciplinary action.

Meanwhile, the Infocomm Development Authority said its own investigations into the fire would wrap up in the next two months. The regulator sets and imposes fines for service lapses.

SINGTEL FIRE

Telco tightens up fire safety protocols

It stops use of blowtorches, plans switch to 'no hot works' maintenance

By Grace Chua, The Straits Times, 17 Dec 2013

SINGTEL has begun improvements to its cable maintenance system and methods, fire safety protocols and training requirements, following a board committee of inquiry into a fire at an Internet exchange in October.

It has stopped using blowtorches on lead seals, which are in 14 of its 22 exchanges, and plans to switch them all to an alternative that requires no hot works by the end of next year. Till then, it will use hot-air blowers with no open flames to soften the seals.

The seals are used to prevent water and gas leaking into underground cable chambers channelling network cables out to users.

The seals must be heated to be loosened and on Oct 9, a worker in the Bukit Panjang exchange used a non-standard issue blowtorch that was too hot, resulting in the sealant and plastic cable sheaths smouldering undetected for up to an hour.

SingTel has since also instituted a 24-hour fire watch after hot works, up from half an hour before, and now uses handheld thermal imaging cameras to check for hidden fires.

The committee, comprising non-executive independent board members Bobby Chin, Fang Ai Lian and Low Check Kian, also recommended changing the polyethylene plastic cable sheaths to flame-retardant PVC (polyvinyl chloride), and tightening access to the cable chambers.

They recommended installing automatic fire suppression systems which were previously placed only in spots with electrical or heat-generating equipment. Cable chambers had only handheld fire extinguishers.

They said fire-retardant materials, beyond the current use of fire-stop pillows, should be added to stop the spread of flames between floors and walls.

Though Mr Chin noted that the cause of the fire was due to human error and not a systemic failure in fire safety protocol, the committee suggested reinforcing safety practices.

It suggested SingTel check equipment regularly and make sure only standard-issue equipment is used for works, and ensure that those performing high-risk activities are aware of and meet safety protocols.

It should give staff regular, formal safety training and make sure they understand the need for safety checks and balances.

What happened on day of the fire

Oct 9, 2013, 12.12pm to 1.05pm: A SingTel employee with some contractors carried out hot works in the cable chamber of SingTel's Internet Exchange in Bukit Panjang. This involved using a blowtorch to heat a sleeve seal which was made of lead. They left for lunch but forgot to take steps to turn the smoke detectors back on.

2pm: Other SingTel staff in the building detected a fire in the chamber and the Singapore Civil Defence Force (SCDF) was alerted. Some bystanders grabbed the fire extinguishers but could not enter the room because of the smoke.

2.20pm: SCDF arrived and took about 20 minutes to locate and extinguish the fire.

5pm: SCDF allowed SingTel to access the cable chamber.

Human error to blame for fire at SingTel facility
Worker used unauthorised blowtorch during maintenance work, says telco
By Irene Tham, The Straits Times, 7 Nov 2013

HUMAN error was behind last month's fire at a major SingTel Internet exchange which crippled essential services across Singapore, the telco said yesterday.

Its preliminary investigations showed that the Oct9 fire at its Bukit Panjang facility was caused by a SingTel employee's use of an "unauthorised" blowtorch during "maintenance work" - which did not follow strict procedures.

A SingTel spokesman declined to reveal more on the blowtorch and why it was needed, pending an on-going review by the telco's committee of inquiry (COI).

The COI, comprising members of SingTel's board, is expected to recommend steps to prevent a similar incident, and will also decide on how to discipline the employee, who has been suspended, added the spokesman.

SingTel, which will be announcing its second-quarter financial results next Thursday, said it treats the incident "very seriously".

It added that it has always ensured that its contingency plans and safety measures meet global benchmarks.

Since the incident, the telco has taken several measures, including reinforcing training on safety rules and limiting any maintenance work which requires the use of high temperatures.

Yesterday, SingTel group chief executive Chua Sock Koong also apologised to its customers, saying it recognised the importance of reliable communication services, and Singapore's status as a leading financial and business hub. "We are committed to learning from this experience and welcome recommendations from the COI and independent experts to enhance our operations."

Acknowledging SingTel's preliminary findings, an Infocomm Development Authority spokesman said it was conducting its own investigation and will make its findings known in due course.

The blaze at the facility - one of SingTel's nine major exchanges - affected services at DBS branches and ATMs, SingHealth polyclinics, Singapore Pools branches and AXS payment machines, along with 60,000 SingTel home broadband users, the bulk of whom are also mioTV users.

StarHub and M1 home fibre broadband services were alsodisrupted as they buy wholesale fibre links from national fibre broadband network builder OpenNet.

Two-thirds of the 149 fibre optic cables which were damaged in the fire belonged to OpenNet.

Some StarHub cable TV and cable broadband users were also affected by the SingTel fire, as StarHub leases SingTel fibre links for portions of its cable network.

It took more than a week to get OpenNet's services running again, although all of SingTel's corporate customers were reconnected almost two days after the fire.

SingTel fined record $6m for outage caused by fire
October blaze at Internet exchange could have been avoided, says IDA
By Irene Tham, The Straits Times, 7 May 2014

SINGTEL has been fined $6 million - the largest penalty ever for a telco - over a blaze at its Bukit Panjang Internet exchange last year that took down phone lines, banking, payment and Internet services across Singapore.

The Infocomm Development Authority (IDA) yesterday upbraided SingTel for not having "specific contingency plans to address serious outage situations of such a scale as the Bukit Panjang incident".

"This is a very serious service outage of a magnitude that is unprecedented, but more importantly, that could have been avoided," said Mr Leong Keng Thai, IDA deputy chief executive and director-general (telecoms and post).

IDA said the disruption affected close to 270,000 subscribers, including DBS branches and ATMs, SingHealth polyclinics, Singapore Pools branches and AXS payment machines.

Home fibre broadband users of SingTel, StarHub and M1 were also cut off from the Internet, as the fire damaged the cables of national fibre broadband network builder OpenNet, which were housed in the same premises.

The hour-long fire on Oct 9 last year was caused by the use of an unauthorised blowtorch by a SingTel staff member during maintenance works.

It sparked a slow-burning fire that went undetected as a result of further safety lapses and human error, said the IDA, which conducted an independent investigation of the incident.

The IDA criticised SingTel's "outdated" cable maintenance system, which uses lead sealant to protect the exit points for cables. This sealant needs to be heated and shrunk each time the cables need to be maintained.

It also called the use of open flames "hazardous".

IDA has also fined OpenNet and CityNet - the trustee manager of a SingTel unit that owns OpenNet - $200,000 and $300,000 respectively.

The regulator found that City-Net had failed to comply with safety procedures as the legal owner of the Bukit Panjang exchange, while the fine on OpenNet was for the "unnecessarily prolonged" restoration of services to broadband users.

Mr Leong said a "strong signal" must be sent to the operators to take network resilience seriously and invest in "the necessary infrastructure, processes and training to prevent and minimise service outages".

IDA said it will see that operators take corrective measures.

SingTel has promised to switch - by year end - to alternative sealing methods that do not require hot works. CityNet said it would install gas suppression and fire warning alarm systems in all its seven Internet exchanges.

SingTel, CityNet and OpenNet said they accepted the IDA's findings and have taken steps to prevent a repeat. SingTel said its fine will not affect its financials.

SingTel's fine eclipsed the previous record fine of $1.5 million on rival M1 for a mobile outage in January last year.

Analysts said the fines - although "insignificant" relative to the telcos' revenue - will push the telcos to invest in better systems and procedures. "Their reputation is at risk if they don't pull up their socks," said Mr Ramakrishna Maruvada of the Daiwa Institute of Research.

Maybank Kim Eng analyst Gregory Yap said: "Something still needs to be done to ensure enough diversity and choice for critical infocomm infrastructure."

IDA said it has started reviewing the need for more back-up in fixed-line, broadband and submarine cable networks, with the review slated to be done by the second half of this year.

Related
Burning issues after the SingTel fire
SingTel Board releases Board Committee of Inquiry Report on the Bukit Panjang fire