===================================================================== Ticket Title : THNY and other services Ticket No : 10083 Ticket Status : Closed Ticket Source : brandon Ticket Opened : 00:20 12-Sep-2001 Ticket Closed : 08:21 21-May-2009 Outage Start Time : 22:41 12-Sep-2001 Outage End Time : Length of outage : Cause of outage : THNY power failure Reported by : 00:20 12-Sep-2001 by brandon IN CONFIDENCE - FOR INTERNAL USE ONLY ===================================================================== Description: As you know it's been a hectic afternoon with events in the USA The resulting load has stressed various parts of the infrastructure and it's quite likely that this will continue through tomorrow Hence it may be necessary to actively manage load throughout the day which will involve prioritising applications resource usage which the webmasters will look after. Our THNY site, 10 blocks away from the World Trade Center, is unharmed but operating on generator power (there is no utility mains in Lower Manhatten. This could become a problem tomorrow, particularly as evacuation of the area means staff access is an issue. As all of this is going to demand ongoing attention meetings are suspended and non essential staff will not be permitted access to KW or other facilities (THDO access may be an issue too we'll check their status in the morning). 11-Sep-2001 ----------- 13:48 jet crashed into World Trade Center, fairly immediate jump in traffic 14:05 2nd jet hits 2nd tower 14:35 NOL UK and NY servers extremely busy. Simon added www[5,6].thny to pool (still as built for temporary Election use) 14:40 Simon informed Tim @ NOL that www[5,6].thny are live Tim mentioned they were having trouble with their UK servers thought to be their content switch which was running at 76% CPU OPS worked with various people on load managment for the rest of the day Traffic graphs on the support site were performing poorly, we may need to restrict access to essential people to avoid extra server load from generating them degrading them 12-Sep-2001 ----------- 01:10 traffic is now down to normal daily (late afternoon) peak level (approx 305M/sec, peak was ~780M) 07:38 contacted news to agree plan for today, Colin & TIm won't be available until later 08:20 informal THNY contact suggests 2 days fuel for their generator 08:50 Colin@NOl agreed we can add more THDO servers from our post election pool if demand is high NOL just starting to strip down content to save resoruces 09:50 added www[11-15].thdo to NOL THDO. These are sharing one NFS content server as there is insufficient time to replicate a full set of content to each 11:32 dropped www[11-15].thdo, performace being limited by NFS server some adjustment of the server 12:18 added just www11.thdo and performance monitored 13:08 added www8 (one NFS server switched to direct http server) 13:55 www8 and www13 removed, NOL happy load with reduced weight pages OK, these can now have start to be switched to local content in case they need to cover THNY power loss (will lose 6 NOL and 2 BOL http servers + lots of real media) 14:25 Offical update from THNY "The commercial power supply to 25 Broadway was lost at about 16:35 EDT yesterday, Sept. 11. Telehouse emergency generators picked up the load and as far as we have been able to determine, there was no impact on our customers' loads. As of 07:35 EDT this morning, our emergency power system has furnished 15 hours of uninterrupted backup power. It continues to operate normally, providing support for our clients' equipment. We are conservatively stating that we have sufficient fuel for 24 hours or more, but actual fuel usage seems to indicate that could possibly continue for as long as 70 hours. Unfortunately, if the commercial power outage continues for a longer period, we may not be able to replenish the fuel given the state of the streets, etc. in the disaster area. We'll watch this situation closely. Further complicating issues, governmental authorities have evacuated the entire area of lower Manhattan (in which the 25 Broadway facility is located) and are barring anyone from entering that part of the city. We have kept essential staff on-site who are continuing to monitor the plant and its performance. We must advise you, however, that authorities may eventually force our staff to leave for safety reasons. Should an evacuation be enforced, our ability to continue to provide emergency power would be impacted." 15:00 World Service requested Arabic stream be switched to 24 hour preferably around 20:50 as this entails an interruption of the currently scheduled scheduled programme 15:20 status generally OK, DS3 to THNY is logging errors (reported to telco), load steady at ~550MBit/s. 21:12 Offical update from THNY "33 Whitehall Street Continues normal operation on commercial power (building access is limited to those with prior approval) 25 Broadway Continues to operate on generator power with on site support available through normal procedures. Primary concern to all is certainly the restoration of commercial power and our generators and fuel supply. It is our understanding that the substation serving 25 Broadway failed when debris from WTC 7 fell on top of it. It appears likely that there will be no commercial power before this weekend. At the present rate of use, we expect our fuel supply to last at least until Friday noon. We have arranged for additional fuel delivery from a supplier BUT need approval for the actual delivery from the police and emergency teams. We are actively pursuing this through several channels but do not anticipate quick approval. We will keep you informed of our progress. Other concern is access to the site. As of today, we have been restricted from sending any staff to the site. However, we anticipate that changing tomorrow based on news reports that indicate personnel will be permitted into lower Manhattan tomorrow. If you need to visit the site, please notify our CAC and will give you the latest status. As always, we are available to assist with our technical staff." 13-Sep-2001 ----------- 10:30 Overnight, we have had a local engineer in New York visit all three of our sites (111 8th & 60 Hudson - Network sites; 25 Broadway - Server site) and although access is possible, it is not easy. We are also remaining in contact with another engineer who has remained in 25 Broadway throughout this whole incident. At one time overnight, it was feared that 60 Hudson may collapse, and was evacuated. This may cause major network issues, as although we don't currently have any equipment in that building, it is a major telco site, and could cause some of our circuits to fail. We have heard since that the building has been declared structurally sound. There are still concerns about power at 25 Broadway, and Internet Services are currently instigating measures to ensure that we have sufficient capacity to serve entirely from THDO. This should be complete later today. We are working with the Webmaster team to ensure that load-shedding procedures are in place to ensure that we can still remain serving core services in the event of reduced capacity. 15:18 New Media Ops continue to contact content owners to arrange contingency lightweight versions 15:30 Offical update from THNY "Since our last report we have continued to focus on fuel delivery. At this time a delivery is in progress. By this I mean that the delivery truck is now preparing to leave from Staten Island and, using the Holland Tunnel, will be heading to 25 Broadway. We have received assurances from various organizations that we will be able to complete this delivery. However, we do anticipate that that it will be a rather slow progress. At current consumption rates, we believe the existing fuel supply will last into Friday. As a further backup, Con Edison has committed to providing fuel should our delivery efforts fail. Con Edison continues to be unable to give us any estimate as to when we can expect return of commercial power." 16:05 informal THNY contact says - Japans Head of the Chancellery and Political Director (Fukuichiro Tanaka) fixed it for the fuel tanker to be allowed to get to THNY. There are 900 gals left in their tank, tanker on Manhattan, en route. At current 20-30 gall/hr burn rate can hold out for about 30 hours. Tanker is rumoured to have 4500 gals 16:43 New Media Ops supplied event stream priorities 1 - imperative 2 - nice to have if possible 3 - lowest priority Friday 10.00pm - Live Streaming - Patrick Kielty Show (NIreland) JK- 3 Saturday 2.15pm - Live Audio - Bath v Saracens (Five Live)JK - 3 3.00pm - Live Audio - Aberdeen v Dundee Utd (Scotland) - 1 3.00pm - Live Audio - St Johnstone v Livingston (Scotland) - 1 3.00pm - Live Audio - Dundee v Celtic (Scotland) - 1 2.50pm - Live Audio - Gloucester v Leicester (Leicester Online) -3 3.30pm - Live Audio - Leeds v Northampton (Sport Online) - 2 7.15pm - Live Audio - Proms in the Park (Newcastle) - 2 7.30pm - Live Streaming - Last Night of the Proms (R2) - 2 Sunday 9.45am - Live Audio - Great North Run (Newcastle) - 3 2.30pm - Live Audio - Newcastle v Quins (Sport Online) -2 3.00pm - Live Audio - Bristol v Lon Irish (Sport Online) - 2 3.00pm - Live Audio - Wasps v Sale (Sport Online)- 2 3.00pm - Live Audio - Kilmarnock v Hearts (Scotland)-1 3.00pm - Live Audio - Rangers v Motherwell (Scotland)-1 3.00pm - Live Audio - Hibs v Dunfermline (Scotland)-1 Monday 11.45 - Live Audio of the Opening of Millennium Bridge (Newcastle) - 3 17:17 Offical update from THNY "Good news ... our fuel delivery is now underway. The tanker truck arrived at the building several minutes ago and is now refilling our tank. Once done we will have fuel on site for at least 4 more days. We are also beginning preparations to get replacement staffing into Manhattan later today. We can not guarantee client access into the building at this time but will keep you informed as we get additional information." 22:23 Alert from THNY "We regret to tell you that within the past few minutes the temperature on the critical generator feeding client equipment has experienced a rapid increase in temperature. We may shut down the generator within then next few minutes. UPS should provide about 10 minutes additional power. We can try to shut down critical equipment if you email us immediately. More info will be coming shortly" 22:25 Contacting webmasters, left message on voicemail 22:32 Alert from THNY "We are now shutting down the 25 Broadway critical generator. We also will lose internet connectivity. Pls contact by phone" 22:39 THDO load dist now not sending users to THNY servers 22:41 THNY power gone, lost contact with THNY now using contingency plan transit via LINX 22:55 work on some resulting connectivity issues continue. Due to the time of day load is lower so it's not likely to be necessary to deploy further measures. Will monitor the situation 14-Sep-2001 ----------- 02:25 THNY power back 02:45 THNY power gone again, though they got the generator going again it was still faulty 03:05 Telehouse New York update Looks like power may be restored soon. After the generator overheated, Telehouse managed to get an engineer onto site to attempt repairs. It would appear that these may be working, as we regained power at 02:25. Unfortunately the power was lost again at 02:45, and has not yet returned. Telehouse advise that there may be more short periods of power before the generator is running stable again. (Hopefully these bursts of power won't have too much of an adverse affect on the servers!) As it looks like we will not be getting reliable power in the next few hours (and it'll take at least an hour to bring back the servers after power is restored), the KW engineers are taking an opportunity to get some sleep. 05:30 Offical update from THNY "The generator failed because of the water pump. Estimated repair time is late tomorrow (Friday, 9/13). In the meantime, Con Ed has committed to provide generator from the street and we expect delivery tonight. Estimated connection is sometime tomorrow morning. We are sorry for the inconvenience this is causing you. Despite the circumstances we were hoping to provide you with uninterrupted power and are disappointed in the turn of events. We are doing our best to insure that this interruption is kept to a minimum. You probably won't hear from us until power is resumed." 08:30 Overnight update The generator isn't fixed yet. It's possible we may get power in the next few hours, but we will not rely on this. Efforts will continue this morning to bolster the capacity at THDO (Content has been copying to new servers overnight) to cope with today's demand. We are continuing to work with our circuit providers to bring up any pending network capacity which doesn't go through THNY (e.g. we have a circuit to MAE-East which goes from 111 8th, rather than THNY, and this will remove some load on our emergency transit at THDO). Webmasters are prepared to switch to "slimline" homepages if load become too high, as this will improve responsiveness to those users accessing the BBC website. 11:15 www8.thdo complete and tested, added to news.bbc.co.uk cluster 12:50 www9.thdo complete and tested, added to www.bbc.co.uk cluster 14:10 rm[30-39].thdo event servers added to live G2 realmedia cluster because to the number of sporting events due this weekend we have re purposed this bank of Real Media event servers 15:10 unofficial THNY update We received this email. Although it's not an official notice from Telehouse, we believe the contents to be accurate. Date: Fri, 14 Sep 2001 07:09:05 -0700 (PDT) Subject: [EVENT UPDATE] Telehouse Power Notification 20010914@10:07 EDT We have received another update from Telehouse regarding the power situation: o A Con Edison generator arrived on site at 09:30 EDT this morning o Telehouse is expecting the generator relief crew to arrive between 10:00-10:30 EDT o Once they are onsite the relief crew will be testing the generator and bringing it on line. If all goes well, the testing and power up process will take 4 - 6 hours. However, this is a best case scenario estimate. While we remain optimistic we would also like to be cautious in our expectations. 16:35 www10.thdo complete and tested, added to www.bbc.co.uk cluster Server update - Throughout today, we have been working to bring online additional servers at THDO, to bolster our serving infrastructure. The following servers have been brought online: 16:30 - www10.thdo added to www.bbc.co.uk cluster 14:10 - rm[30-39].thdo added to live G2 realmedia cluster 12:50 - www9.thdo added to www.bbc.co.uk cluster 11:15 - www8.thdo added to news.bbc.co.uk cluster We continue to work on loading content onto servers to increase serving capacity, in particular for the news.bbc.co.uk, which is understandably very busy during this tragic period. 17:10 A hectic day, most services have held up with the aid of a lot of attention all round. There have been some change requests which we'd not normally approve in these circumstances, some will be done later today when the risk is lower Work continues to bring more server capacity online. We are also working on some new connectivity that we've been building, (via a different NY site intended to cover losing THNY, why couldn't they wait until we'd finished) which had been pending some work by telcos who due to the circumstances have pulled out all stops to get this up sooner. More details later if we get a result. 23:00 worked ongoing with Level 3 eventually got San Jose circuit up 15-Sep-2001 ----------- 01:10 unofficial THNY update - "ConEd is almost there, another few hours" 02:21 left THNY to complete generator 03:58 THNY power back, servers start to boot 04:40 bulk of telco circuits start to come back 04:58 Offical update from THNY "We are very happy that finally Con Edison has finished the temporary generator installation work and their generator is now operating. All of our UPS'S are back on line and power again available to all of your equipment. We expect the Con Edison generator to give us continuous power for the duration of the Con Ed outage. Also, we can report that the replacement water pump for our critical generator has been received and the maintenance crew is expected to proceed with replacement and repair over night." 05:21 Offical update from THNY "TELEHOUSE 25 Broadway is back up as of September 14 23:10 in New York time running on the Con Ed provided generator refeeding the site from the street. Repair analysis of the failed generator are on going (once repaired it acts as backup to the Con Ed generator)." 05:27 rm[1-4].thny content resync started 05:50 www1.thny content resync started 07:06 requested THNY staff to power cycle bay A15 terminal server and & sw0.thny that 07:23 www[5-8] into single users 08:00 to 08:15 called THLS staff to talk through power cycling terminal server and switch that didn't come up cleanly. 08:15 www2.thny data/cgi resync done, www content started 08:18 some inter switch links reset, NYIIX connection up 08:51 load distribution name server reconfigured to match THDO set to ignore THNY server that are coming online & started 09:11 brought up www[5-8] but left apache off to avoid delivering old content 09:19 rm8.thny did not restart. fixed 11:22 on site @ KW, work carries on clearing backlog of content updated missed whilst down. Due to queue site these were dropped out of rapidly growing borg queues so requiring manual intervention to reinstate Whilst various services are ready to go live on restored THNY servers they aren't needed to cover lower weekend loads. Hence they are being held in reserve rather than brought back online 16:26 Offical update from THNY "Operations 25 Broadway continue "normal" under these very un-normal conditions. Basic power for 5th floor continues to be provided by the Con Ed generator and for the 6th floor by our new generator. Status of the various generators is as follows: * Con Ed: 24x7 watch is provided by ConEd personnel; fuel is also provided by ConEd. The capacity of the generator tank is at least 15 hours at the current rate of usage. * Critical Generator (with failed water pump): the new water pump has been replaced and unit run off line for about 30 minutes. We have arranged for further servicing including changing lubricating oil, filters and power washing the radiator. We anticipate that this work and further testing will be completed by Sunday, and the generator will again be available as "backup" to the ConEd generator. * Fuel supply: at current usage rates, on-site fuel supply will last to next Thursday. We are arranging for another delivery on Monday. Access to the area appears to be near normal with limited subway services into the nearby areas. Cars are not permitted. Access to the building is possible with some procedural restrictions. We recommend you first notify our CAC if you plan to access the site and get a ticket number. If you are stopped at nearby, call our 25 Broadway security and we will dispatch someone to "escort" you to the building. Note that building elevators are not in service; the building owner has a commitment from ConEd to provide another generator to power the rest of the buiding shortly. We understand phone access into New York continues to be difficult and suggest use of email where possible to request support." 18:43 THNY power failed again, unofficially - "InterNAP, and possibly others have lost power again. Word is that the ConED generator ran out of fuel, the replacement water pump on the generator was installed, but the generator failed again, and a fuel truck 6 blocks away isn't being allowed through by the authorities." 20:50 THNY power restored 21:51 Offical update from THNY "We lost Con Ed generator power at 12:46 p.m. EST Saturday September 15th. The cause for this failure was a lack of diesel fuel. The lack of fuel was due to the fact that the fuel truck was held up at a National Guard check point because truck driver did not have the proper permit. We switched to UPS power and finally to our critical generator at 1:06 p.m. This is the same generator that had its water pump replaced early this morning. Our generator experience a high temperature alarm again which necessitated a shut down at 1:33pm. Currently our vendor is in the process of dispatching a engineering to reassess this problem. Con Ed finally acquired the proper permit and the fuel truck was then allowed to travel through to 25 Broadway. The fuel truck arrived at 2:57pm, refueling began at 3:10pm and refueling was completed at 3:30pm. The generators were started and power was restored 3:45pm. All of our clients were powered up at 4:00pm We are continuing to work with Con Ed to insure availability of fuel on site and are now arranging for the next fuel delivery. We understand the hardship that you are enduring during this trying and difficult time and we appreciated your patience and understand." 22:14 requested THNY staff attention for bay A15 terminal server 22:25 Engineer on route to Level 3 San Jose to bring forward comissioning some new circuits 22:30 KW unmanned 16-Sep-2001 ----------- 00:18 PAIX connection now live after San Jose work 01:15 ticket updated for issue, no service affecting changes today. Lots of work was done to validate and prepare THNY for service but held off from making live as THDO operating happily and, as we found out, the power situation wasn't considered suitable at THNY. This took us backwards. 04:34 Offical update from THNY "A fuel delivery was received at 25 Broadway at 22:00 EST and the next fuel delivery is scheduled for tomorrow 9/16 at 8:00am. Our generator vendor, who is extremely busy, called at 22:20 and assured us that their technicians are scheduled to service our generator at 25 Broadway tonight." 12:42 through to 20:30 background content validation runs done on THNY www & rm servers 12:46 Offical update from THNY "A fuel delivery was received at 25 Broadway at 07:30 EST. The next scheduled fuel delivery should be later today the 16th of September, 2001 at approximately 17:30 EST." 15:52 Offical update from THNY "Here is the update as of September 16 10:45 NY time. 1) Con Ed Generator Another 2,000 gallon fuel tank is on the way to 25 Broadway. We are tying this tank in to the existing fuel tank in order to give us longer refill interval. 2) Con Ed Fuel Delivery System It seems they've developed the system. Now they have fuel trucks on the street refilling fuel to buildings around the area. So each refilling work is going smoothly. 3) 5th Floor Critical Generator The manufacturer's engineers are on site and working on the problem." 15:57 fixed rm20.thny for VillageWorld Inc (reported at 15.39) 23:10 Offical update from THNY "Here is the update as of September 16 5:55pm NY time. 1) Access to 25 Broadway At this time, the only personnel that will be granted access to the center are those with photo IDs indicating they work for TELEHOUSE. The National Guard WILL NOT allow access to the area for people who are not able to demonstrate proof that they are TELEHOUSE employees. The reason for this added security is that Con Ed has lined the streets with numerous cables and generators as it tries to restore power to the area, creating a dangerous situation for people who do not work for Con Ed. As soon as the National Guard allows non-TELEHOUSE employees access to the site, we will let you know. 2) Con Ed Generator The additional 2,000 gallon fuel tank has arrived at the 25 Broadway center. TELEHOUSE staff is in route to tie this tank in to the existing fuel tank in order to give us longer refill interval. 3) Con Ed Fuel Delivery System The fuel trucks are being driven around the affected area by the NYPD. Once the generator gage reads approximately 50% full, a truck is called in to refill. This process has been working smoothly. 4) 5th Floor Critical Generator The manufacturer's engineers are on site and continuing to work on the problem." 17-Sep-2001 ----------- 05:43 Offical update from THNY "Here is the update as of September 17th 12:36am NY time. 1) Con Ed Generator & Fuel Delivery A fuel truck with enough fuel to fill the 2,000 gallon tank and the Con Ed generator tank is en route to the Broadway center. In the meantime, the Con Ed generator fuel tank has been filled with approximately 9 hours worth of fuel. 2) Access to 25 Broadway At this time, the only personnel that will be granted access to the area where our center is located are those with photo IDs indicating they work for TELEHOUSE. The National Guard WILL NOT allow access to the area for people who are not able to demonstrate proof that they are TELEHOUSE employees. However, we have been able to escort customers to our facility. Please do not request access to the site unless it is absolutely necessary. So far the National Guard has been allowing these escorts, however, if they feel there are too many personnel in the area that do not work for the city, they may put an end to any further escorted visitors to our center. 3) 5th Floor Critical Generator The manufacturer's engineers left the Broadway Center to retrieve a necessary part from their warehouse and have not yet returned. Also, as an FYI, the building at 25 Broadway did not sustain any structural damage as a result of the WTC terrorist attacks on 9/11." 08:15 through to 10:20 rm9.thdo rebuilt (sys disk died, now replaced) 10:20 through to 14:50 chat1.thdo rebuilt (sys disk died, now replaced) 15:14 rm9.thdo added to active load pool 16:18 Offical update from THNY "Here is an update regarding the situation at 25 Broadway as of September 17th 10:00 EDT. 1) Con Ed Generator & Fuel Delivery: We were successful in adding approximately 1,860 gallons to the Con Ed generator's fuel tanks during the course of the night. This is enough fuel to continue through 21:00 Tues. 9/18 at current usage rates. 2) TELEHOUSE Diesels: We were also able to add 2,160 gallons to the fuel reserve supplying the Telehouse diesels, bringing the total to about 3,300 gallons. This amount is estimated to be enough to last until Thurs. 9/20, 02:00. 3) H.O. Penn technicians are on-site to continue work to restore the 5th Floor Critical Power Generator to service. In addition, since emergency back-up power generators require maintenance when subjected to the kinds of extended runtimes that this disaster has imposed, we will also have them examine our 5th Floor Essential Power Generator and perform any maintenance which may be necessary to assure its continued operation. 4) The building now has generator power; elevator service is available. 5) Access to 25 Broadway is still restricted; however, we have been able to escort customers to our facility." 16:38 chat1.thdo config file fixed, now running and ready for service 22:13 Offical update from THNY "Current Operation: We continue to operate on the Con Ed generator for the critical power for equipment and on our essential generator for air conditioning. Fuel for the Con Ed generator has been refilled as of 1530 today; we expect the fuel for the Telehouse generators to be "topped off" sometime later today or tomorrow. Critical Generator Repair: Based on an examination of the 4 regulators (thermostats) removed from the generator earlier today, we are very confident that the overheating problem was due to "frozen" regulators which inhibited the flow of coolant. The regulators were replaced and generator run with no load for a short period. Based on this evidence, we are highly confident that the generator will now operate for an extended period without overheating. Upcoming Plan: Tonight at approximately 1830 edt, we will transfer the critical equipment load from the Con Ed generator to the repaired critical generator. The purpose of this transfer is to absolutely confirm that the critical generator operates properly. This transfer should occur without any impact to client equipment with the UPSs providing for a smooth transfer. After a period of about one hour, and assuming the critical generator operation is normal with no signs of overheating, we will begin transferring the essential (air conditioning) load from the Telehouse essential generator to the Con Ed generator. This will be done to permit us to perform the same maintenance on the essential generator as has been done on the critical generator. (While not focused on, this essential generator has been operating at a higher than normal temperature which we are assuming is the result of "frozen" regulators also). We anticipate this work will be completed either overnight or early tomorrow morning depending on availability of the mechanics. During this time, and if required due to failure of the critical generator, the critical load can be transferred back to the Con Ed generator without interruption. Upon completion of the work on the essential generator, we be maintaining an operational scenario of using one of the Telehouse generators (either critical or essential) plus the Con Ed generator. This scenario will provide optimum use of the two available fuel tanks while at the same time providing for automatic backup should any one of the generators fail. Other information: Site Access...appears to be reasonably easy but must have two photo IDs. Please continue to contact us if you intend to go to the site thru the CAC. Miscellaneous....we are told there will 17 fuel tanker trucks in the city by tonight and that there 30 additional generators staged at the Fulton Fish Market. Seems our support infrastructure is stabilizing!" 18-Sep-2001 ----------- 03:37 Offical update from THNY "The switch of the critical load from the Con Ed generator was accomplished as scheduled at 1830; however, after approximately 45 minutes the generator again began overheating under the full load and we returned the load to the Con Ed generator. These transfers were completed without impact to the client equipment. After further consultation with the generator mechanics, it was decided the most probable cause of the continuing overheating was scale buildup in the radiator. They recommend use of a biodegradable acid wash to remove the scale. The wash works while the generator operates under load and would not need to be flushed for up to 3 months. We are now planning to do this work, including transfer of critical load tomorrow (Tuesday) at 1830edt following the same procedure as outlined in our previous report. Again, this work should not result in interruption of critical power. Summarizing current status: - critical load continues on Con Ed generator; current fuel sufficient until Wednesday am but expecting new delivery later tonight. - essential load continues on TELEHOUSE essential generator; fuel sufficient until Thursday pm Future Considerations: We are now developing a plan to bring in another temporary generator to bypass the critical generator in the event we are unable to maintain consistent operation on the existing machine...we will keep you informed as we pursue this alternative." 16:42 Offical update from THNY "Operations and plans continue as reported late last night. Fuel tanks on Con Ed generator have been topped off within the past hour. Our plan for critical generator maintenance and testing with critical load also remains as reported at 1100edt yesterday EXCEPT, if possible, we will be starting this work one hour earlier at 1730edt. Final timing will depend on arrival of crews and materials. We will confirm the schedule later in the day." 18:35 THNY has been stable on generator for a while now and we've been getting by on the extra servers/connectivity we brought up at THDO. As mentioned previously we've been working on bringing forward some new connectivity, over the weekend the San Jose link was made live along with the Palo Alto Internet Exchange connection. This connects into 111 8th Ave (see diagram at http://support.bbc.co.uk/support/network/ it's an alternative site we use in NY for diversity avoiding having absolutely everything in THNY) Today an alternate fibre path round New York was lit (at OC48) bypassing the one between THNY and 111 that has been having problems and also helping cover for the 2 * OC3 and one DS3 that are down at THNY (all WTC related). This means that though THNY isn't 100% restored we will send some load there (just Real Media initially). 23:24 Offical update from THNY "Sorry for short notice, but only recently have we been able to confirm our schedule regarding the critical generator work and testing. We are basically proceeding as previously stated but with a somewhat later test time. Mechanics have now arrived on site and are proceeding with the preliminary work necessary prior to testing. This included reflushing the radiator several more times and adding the acid wash to the radiator. We anticipate transferring the critical load to the generator beginning at approximately 1930 edt. As previously noted, this transfer should be without interruption in critical power." 19-Sep-2001 ----------- 02:32 Offical update from THNY "We have transferred the critical load successfully to the "repaired" critical generator. The generator has been operating normally with the load since approximately 2025edt; temperature is in normal range. If necessary, the ConEd generator remains available for backup of the Telehouse critical generator at any time. We are now in process of transferring the essential load to the ConEd generator. We will be proceeding with doing maintenance (similar to what was done on the critical generator) on the essential generator. Fuel supplies are ok for all generators; additional fuel for the Telehouse generators is scheduled for delivery late Wednesday morning." 14:45 THNY power has been stable, though on generators, so load moved back to THNY servers - real media live and stored, chat, www except NOL 15:38 Offical update from THNY "25 Broadway operation continues under the same conditions as last reported. The critical generator operation under load has been completely normal since the transfer last evening. Crews are now on site performing maintenance on the essential generator; we expect to have this generator available for service later today. Our basic operating plan is to continue to support the critical load with the Telehouse generator and the essential load with the Con Ed generator. Fuel supplies for both systems are near maximum. Looking a bit to the future, Con Ed has informed us that they are now expecting to re-energize the grid serving 25 Broadway sometime late this week. We are working with Con Ed for a controlled deactivation of their generator and return to commercial power within approximately 24 to 48 hours of return of power. Obviously, the schedule for this work is subject to change and we will keep you informed as this transition develops. 25 Broadway access continues to be restricted; please request support from the CAC." 16:40 THDO contingency transit shut down, all network traffic now using our regular routes 20-Sep-2001 ----------- 02:57 Offical update from THNY "25 Broadway operation continues under the same conditions as last reported. The critical generator operation under load continues normal. Crews completed maintenance on the essential generator. It will continue to run without load until tomorrow morning. A visual inspection will be conducted at that time." 16:04 Offical update from THNY "25 Broadway operation continues stable. The critical generator operation continues normal. With the essential generator maintenance completed and the generator operating normal under no load since last night, we have now transferred the essential load to our generator. Our plan is to return this load to the Con Ed generator after confirming normal operation of the Telehouse essential generator. Fuel supplies for both systems continue to be refilled regularly. Information obtained from the Con Ed web site indicates that basic power has now been restored to the grid covering 25 Broadway. Actual activation of power into 25 Broadway has not occurred. We will continue monitoring progress. As indicated previously, our plan is to make the transition from the Con Ed generator to the grid power within 24 to 48 hours after grid power is restored. We will provide as much advance notice of this activity as possible. The transfer should occur without interruption in critical service." 21-Sep-2001 ----------- 03:27 Offical update from THNY "25 Broadway operation continues to be stable. The critical generator operation remains normal. Earlier today the essential load was run on the TELEHOUSE essential generator to confirm proper operation. After more than two hours, the load was returned to the Con Edison generator, where it remains." 15:21 Offical update from THNY "25 Broadway operation continues to be stable. Fuel supplies for both systems continue to be refilled regularly. As of this morning, Con Ed is estimating that power will be available "SUNDAY" in the portion of the grid feeding 25 Broadway. We have scheduled a Monday 1800edt time slot for Con Ed to disconnect their generator and restore our feed from the grid. At this time, we consider the Con Ed schedules to be subject to change (either earlier or later). We are told the landlord has the same timeslot for re-connecting the rest of building (which includes our 6th floor power). The timing of our actual return to use of the Con Ed feeders will be under our control. After Con Ed completes their work to reconnect our feeders to the grid, we will continue to operate on our generators while we monitor the quality of the power. Actual transfer to Con Ed feeders will be made by Telehouse based on time of day and quality of the commercial power. We will provide as much advance notice of this activity as possible. The transfer should occur without interruption in critical service." 22-Sep-2001 ----------- 03:03 Offical update from THNY "25 Broadway operation continues to be stable." 16:04 Offical update from THNY 25 Broadway operations continue to remain stable. All viable fuel tanks have been top off as of 10:30am this morning and we continue to monitor future fuel delivers. 23-Sep-2001 ----------- 02:57 Offical update from THNY "25 Broadway Operations continue to remain stable" 15:13 Offical update from THNY "25 Broadway operation continues to be stable" 24-Sep-2001 ----------- 03:22 Offical update from THNY "25 Broadway operation continues to be stable. Fuel supplies for both GENERATOR systems continues to be refilled regularly. At this time, the latest we have heard from Con Edison regarding the restoration of commercial power is that they expect to have it restored sometime tomorrow (9/24) evening, after 6:00pm EDT. As indicated previously, TELEHOUSE will wait 24 to 48 hours before switching from generator power back to commercial power to ensure that the commercial power is stable and reliable." 14:34 NOL, via email, reported problems with their THDO servers "our machines have been working very hard over the last 2 hours, working much harder than last week, yet the bandwidth is much less". 16:00 Offical update from THNY "25 Broadway operation continues to be stable. Based on discussions with Con Ed, it appears that the previously reported schedule to re-connect the building feeders to the Con Ed grid tonight remains valid. We have been given a 1800edt time slot for this work to begin. We understand that Con Ed will have numerous work crews in lower Manhattan reconnecting the dozens of buildings on on generator power. At this time there is no way to be certain of the actual schedule for this work. The procedure we will follow is as follows: 1. Prior to start of work by Con Ed, we will transfer the 5th floor essential load back to the Telehouse essential generator. 2. During the actual work period (Con Ed says work may take up to 10 hours), the three Telehouse generators will provide power for the 5th and 6th floors. 3. Upon availability of the Con Ed grid power, we will begin to transfer our loads to the commercial power source. The sequence will b: 1) 6th floor critical and essential; 2) 5th floor essential and 3) 5th floor critical. The timing for each step will depend on stability of the Con Ed power. Upon completion of the transfer to commercial power, the Telehouse UPSs and generators will continue to be available to provide uninterrupted power to all equipment." 17:40 Tim @ NOL phoned, still trying to track their problem down. They've limited the number of apaches to 800 per machine and they're sitting there all used. our investigation has looked for evidence of DOS attack, all the signs point to a connectivity issue leading to longer connection duration or hanging connections increasing the number of occupied apache processes 18:27 LINX confirm they've fixed a problem - "We have just had to perform an emergency reload of switch 26 at Telehouse. Some members have been experiencing intermittent packet loss problems to some destinations on the LINX during the course of the day, and we have also been seeing "unfair" load balancing on inter-switch links outbound from this switch which are using aggregated links, which seems to be exacerbating this issue too. We're monitoring the situation now, and have opened a ticket with Foundry regarding this issue." 21:22 Offical update from THNY "During the day today, Con Ed has confirmed that work will begin at around 1800 today on restoring grid power to 25 Broadway. They have told us they first will remove the two generators on Broadway and restore the Broadway feeders (provides power to most of the building including Telehouse 6th floor). Then they will move to the side of the building and remove the generator that has been supporting the Telehouse 5th floor and restore the associated feeder. We anticipate this work will be completed sometime overnight. Our plans to transfer our loads back to the Con Ed grid power remain as outlined this morning. Except, I must clarify that the transfer of the 5th floor critical load WILL NOT BE DONE until we have monitored the grid power for at least 24 hours. We will continue to operate the critical load on the Telehouse generator until we are satisfied that the Con Ed grid power is stable. Also, we will provide you with advance notice of the transfer (which will be done outside of local business hours)." 22:00 THDO<>THNY DS3 went down totally. We have been living with some errors as it's the only REITH path to NY, we're providing emergency cover for the proper REITH circuits which went down on the 11th. Fault reported to Level 3 23:20 DS3 came back up, but loads of errors. Disabled it's use on our content network, we have another circuit that covers this but not REITH 25-Sep-2001 ----------- 00:12 Level3 say DS3 is ok and handed back to us 03:59 Offical update from THNY "After starting work at about 1800, ConEd is close to its work to re-establish power to the Broadway feeder. At approximately 2115, they began to disconnect the generator we have been using for the 5th floor. We expect them to complete this connection to the grid by approximately 0000. We will monitor the ConEd grid power and follow the transfer plan previously outlined after they complete their work." 07:01 DS3 handed back to Level3, error rate still high 08:15 Offical update from THNY "At approximately 0200, ConEd completed its work to re-establish commercial power to our 25 Broadway facility. As described in the previously outlined transfer plan, 6th floor loads and 5th floor Essential power (lights and air conditioning) have been transferred to Con Ed. While commercial power is available, the 5th floor Critical power continues to be supplied by the Telehouse Critical power generator. As stated in earlier messages, the 5th floor critical load WILL NOT BE TRANSFERRED until we have monitored the grid power for at least 24 hours. We will continue to operate the critical load on the Telehouse generator until we are satisfied that the ConEd grid power is stable. Also, we will provide you with advance notice of the transfer (which will be done outside of local business hours)." 15:49 Offical update from THNY "Status report on the restoration of commercial power at 25 Broadway: As reported in our 0300 update, ConEd completed its work to re-establish commercial power to our 25 Broadway facility at approximately 0200 this morning. As planned, the 6th floor loads and 5th floor Essential power (lights and air conditioning) were transferred to Con Ed. Although there is commercial power available, the 5th floor Critical power continues to be supplied by the Telehouse Critical power generator. Once again we emphasize that the 5th floor critical load WILL NOT BE TRANSFERRED until we have monitored the grid power for at least 24 hours. We will continue to operate the critical load on the Telehouse generator until we are satisfied that the ConEd power grid is stable. (Observations of the 6th floor UPS's indicate that ConEd power is presently stable.) We will provide you with advance notice of the transfer (which will be done outside of local business hours)." 22:27 Offical update from THNY "Operation continues as reported earlier today. As of now, the Con Ed grid power has been stable and within specifications since we began monitoring at approximately 2000edt yesterday (9/24/01). Assuming this stable condition continues, we are now planning to transfer the critical load back to the Con Ed grid power at approximately 0100 edt Thursday (9/27/01). At that time we will have been monitoring the power for more than 48 hours. This schedule is subject to change based on a significant change in the Con Ed power stability." 26-Sep-2001 ----------- 10:30 Brandon & Simon on site at THDO. besides some installation work for SMS and NM BBCi (added a couple more modperl boxes) some follow up work on the network was done to enable the problem Level3 DS3 to be ceased by moving the REITH connection to the OC12 16:13 Offical update from THNY "NOTE: CHANGED TRANSFER TIME - 0300EDT THURSDAY (9/27/01) Operation continues as previously reported. The Con Ed grid power has, as of the above time, provided stable for over 38 hours. Assuming this stable condition continues, we are now planning to transfer the critical load back to the Con Ed grid power at approximately 0300 edt Thursday (9/27/01). At that time we will have been monitoring the power for more than 60 hours. This schedule remains subject to change based on a significant change in the Con Ed power stability." 20:30 Brandon & Simon left THDO 23:50 NY end of REITH service moved to new circuit, service now restored 27-Sep-2001 ----------- 03:02 Offical update from THNY "ConEd grid power continues to be stable. The transfer of our 5th floor critical load from our generator to the ConEd power remains as scheduled for 0300edt Thursday (9/27/01). This transfer should occur without interruption of power to your equipment. As noted earlier, this schedule is subject to change based on a significant change in the Con Ed power stability." 03:29 update from LINX "Well, we seem to have had a successful maintenance slot this evening: We have enabled extra bandwidth to Telecity Millharbour - we now have 2x 2Gig from Telehouse North to Telecity Millharbour. Also, we have changed the software on switch26 at Telehouse North. The newer software does seem to have cracked the load-sharing issue. As this had worked well, we decided that we would also upgrade switch28 at Telehouse East, which was also running the older code as used on swicth26. That has also been done, successfully. The new software seems to have sorted our unfair load sharing issue, but the real proof of this will be when the traffic ramps up again in the morning. Overnight traffic patterns are a bit different, and us bouncing fibres and switches has probably dampened things a bit." 08:10 Contact on site at THNY says main changover about to happen 08:30 Offical update from THNY "Power available from Con Ed's restored grid has been monitored since its restoration on 9/24 and has been found to be stable. Therefore, Telehouse transferred the the 5th floor critical loads from the our critical power emergency generator to Con Ed commercial power at 03:15 this morning in accord with our previously announced plan. The transfer was completed without impact on our customers' equipment. The 6th floor loads and 5th floor essential power (lights and air conditioning) have been supplied by Con Ed since restoration work was completed 9/24-9/25. All of our backup generators and UPS units are available to respond automatically in the event any Con Ed commercial power problems should occur. We will also continue to maintain emergency level staffing at the facility through the night and the coming days to help assure maximum protection for our customers' equipment." 11:55 THNY has been stable a number of hours, ticket updated and issued --------------------------------------------------------------------- Circuit(s) / Equipment affected: most services