I documented all the BIOS settings for the EY computers. The h1hwsey BIOS was modified to prevent auto-power-on on power restoration. All the front ends were then restarted. We had to reconnect h1seih45 to the Dolphin network (its cable had been removed during initial roll out) to complete the set and get every Dolphin'ed front end going.

No major changes other than: the overflowing h1pslfss DAC channels as before (Rick said it would be ok to continue running this model); the h1susmc3 model would not auto-start, but we could manually start it; and h1susquadtst is showing timing errors (nothing connected to this system).

The DAQ was restarted. The EDCU is missing the PSL Beckhoff controller channels, presumably this computer is powered down.

The dust monitors and weather stations have been restarted and burt restored to the files in /ligo/cds/lho/h0/burt/2013/04/01/00:00.
I also burt restored FMCS to the same time.

The dust monitor at end Y location 2 has been swapped out for the one labeled 'S'. The previous one was removed on Tuesday (calibration failure). I removed the dust monitor in the LVEA at location 1 (calibration failure for a long time). The dust monitor at location 5 in the LVEA was moved out from under the clean room over BSC2 and into the beer garden.

The dust monitors in the PSL enclosures still need to be restarted.

- Apollo installed a couple of viewports for OL, and removed a couple of doors from HAM chambers, see Bubba's entry for more info.
- outside Tech serviced the FM200 system, system is back now.
- Dave continued working with bringing back the CDS system, more green now than this morning.
- Hertz in to take back their generators, all gone now.
- Patrick bringing back dust monitors.
- PSL still down, no light.
- We still have a 2 CCs (VE) down (red) from the power outage, one less than this morning, yes they were toggled now we wait (424B & 524B).

We removed the east door from ham 3 and both doors from ham 2. We are ready to remove the west door on ham 3, however C 3 ham door covers are rare, therefore we subsituted with dome covers and used all of those. We will replace the subsitute covers with the appropriate covers when they arrive, hopefully Monday. We also installed viewports (serial #s) 23&26 on the H 1 MC input tube.

In addition to other work that went on during the power outage.  We cleaned the Switchgear cubicles and cycled the breakers in key areas.  The MCC in the Carpenter shop. The MCC that feeds the chillers. This unit looks as if condensation in forming in the bottom a fair amount of rust has formed.  It also had dead mice in the gear.  The main switches in the Mechanical Room.  The MCC in the LSB.  The main panels in the End Stations.  We did not get to the Mid stations as we can do them at anytime and not bother the IFO.

Robert ran some seismic tests in the LVEA.  One problem was the Generator running by the carpenter shop.  Large peak in the data.

We also checked the status of the Emergency lights across the site.  We had a few immediate failures and some that did not last the full 90minutes required.  We will be addressing these issues.  A few actually lasted 3 hours.

Of particular concern were the exit lights in the OSB.  I will get with John to discuss a path forward.
Also the FM200 panel did not survive the outage.  We lost a power board around 8am

All in all the outage went smoothly and the return to normal only had some minor glitches.

Summary of Thursday CDS recovery:

All networking switches were started. hanford1 and hanford2 were started. hanford1 needed several restart attempts.

The vacuum controls workstation in the control room was started. The powered down Vacuum controls VME systems were restarted (h0veex, h0veey, h0velx, h0vely, h0vemr) along with their local workstations. All VE systems were burt restored to 23:00 Monday 1st April.

DC power was restored to the H1CER. h1psl0 was started to permit startup of the DAQ system. h1dc0 was started. We noticed a check led light on the Fujitsu 10GE switch, this was cleared before h1fw1 was started. A problem with the SATABOYs was tracked to solaris 11 not auto NFS sharing. Dan upgraded the MSR ldasgw machines to sol11.1 and patched, but still had to manually share the file systems. Also the MSR DAQ SATABOY performed an auto parity scan.

The cdswiki machine took several restarts before it booted. The MEDM snapshot program was started on script0 and the internal EPICS gateways were started. The FMCS EPICS IOC was started. The NAT router was started. At this point the VE and FMCS MEDM screens are available on the CDS web page.

The DAQ system was restarted, the VE and FMCS channels are being trended.

The autoburt system was started.

We noticed that the PSL FSS model was overdriving some DAC channels, so we stopped the PSL models. We started the h1 susaux front ends.

The front end and DAQ computers' BIOS settings were documented.

Following the site wide power outage from 9am Wed 3rd to 11am Thurs 4th April we are presently recovering the CDS systems.

At this point, we have started all the vacuum controls racks, the networking systems and the FMCS systems. A full recovery procedure will be posted in the DCC when we have completed the startup.

I'll be manually topping 80K pump levels and checking pressures etc. regularly during the outage.

Last week Chris and Kiwamu told me the slow controls weren't working for the low noise VCOs.  They are now working. 

Some of the issues were:
The corner 3 beckhoff chassis middle rail was not powered correctly, Filiberto fixed this.  
The inputs labeled RF mon on the tester go to pins that the wiring diagram indicates are temp mons, and the rf mon inputs go to pins the wiring diagram indicates are RF mons.  If the tester is right and the w 
The cable for the comm vco was unplugged at the beckhoff chassis
there were no links in the system manager for the diff vco

The Beckhoff modules used as ADCs on the IMC VCO (EL3102) are different from the ones used for the ALS VCOs (EL3104) but the conversion is identical for the different modules, so no changes the TwinCAT library are required

The RF detector used in the VCO (Minicurcuits ZX47-50LN-S+) has a temperature sensor, which the data sheet says has a linear slope of 2mV/degree C, and gives 0.6V at 20C, with the factor of ten gain in the schematic D100064, it seems like the equation used by the library (T=20+50*(V-6)) is right although the values displayed for temperatures do not make sense.  Looking at the test procedure for the VCO it doesn't seem like this part was tested.  

I turned the HEPAs and AC units off in both the H1 and H2 rooms. I've left the make up air on at 100%.

Last full day of power - tomorrow the site will have it's power shut down - DOE making changes.

Site activity:
- lots of activity with shutting things off in every building
- crane activity this morning - all are currently locked out
- two loads of Praxair
- crate inspection at MY for PSL parts
- people at EY to get the fiber working
- people looking at storage
- dust monitors moved around and staged in strategic places - some will be recording data and people will be coming in to swap out batteries
- cryopump work shimming on LN2 transfer lines
- all optics have their Master Switches set to "OFF"
- doors are on the HAM chambers
- class-A hardware in the H2 PSL is covered
- a snapshot of the vacuum overview medm is attached - unusual with all the red

"ISC Table" has been covered up.  iLIGO SUS Towers have been covered.  I saw Greg address TCS stuff on a flow bench. 

On Patrick's request, I powered off the three Dust Monitors in these rooms.

Cleanroom lights are OFF, and fans are ON.

XBM pressure as indicated by PT170B will be increasing -> No action in required

Want MTP rotor to be stopped and parked before loss of power tomorrow

Prep work continued yesterday in the West Bay. Signage was generated and attached to pallets and storage boxes, including the BS/FM at the end of the shelving units. BS/FM container ratchet was sealed. I found that the ICS paperwork for many parts could not be completed because of shipments not being received so I worked on that for a while this morning. I also participated in a telecon with the PSL folks WRT storage issues. We made some progress in giving the PSL crates at Y Mid unique identifiers and correlating existing info (Benno's sketch and spreadsheet, RS and MR inspection info) with the new identifiers. Please see attached documents for further details. I applied a few signs in the LVEA. I will continue to work ICS issues this afternoon.

[Sheila, Kiwamu]

 For some reason the issue of the picomotor controller box #3 [1] disappeared spontaneously.
We have no idea of what exactly was going on. Keep your eyes on it.

Picomotor Tests
  As planned [2] we tested the slow digital system this morning by plugging a spare controller box directly to the Beckoff chassis with short DB25 cables. With this setup we could drive an in-air picomotor and it didn't show the problem. Then we thought the original long cables that connects the Beckoff chassis and the controller box might be doing something funny. So we tested it by hooking up the in-air picomotor at the front of the original controller box (which is on the top of the IOT2L enclosure) and driving it from the Beckoff through the original cables and controller box. So this is the same configuration as at the time when we had discovered the issue [1] (except we inserted extension DB25 cables to reach the box as ITO2L is put afar). However today we didn't see the issue at all even in this configuration. We tried to reproduce the problem by loosening the DB25 cables (CAB_178 and CAB_179) intentionally but it only made the drive either completely functioning or completely not-moving and hence we were unable to reproduce the issue.

So the issue is gone spontaneously and we quit investigating it for now.

[1] LHO alog 5939 Comments on "HAM3 ISC IR QPDs are likely broken, in-vac cabling error, IO QPD should be OK"
[2] LHO alog 5960 "Test of picomotor controller box #3 : healthy"

New vacuum workstations have been installed in all of the outbuildings and in the LVEA, replacing the aging Sun workstations in those locations.  These workstations are not dependent on any other systems, and may be rebooted at any time without a loss of functionality.  All of the files necessary to monitor the vacuum system are included on the local disk.  The exception is DataViewer, which requires the NDS server (and the /ligo filesystem to run the binaries).  Note that the vacuum VME crates themselves still require hanford1/2 to boot, and should NOT be restarted during a power outage.

In addition, the network connections to the vacuum racks in the LVEA have been reconfigured such that all components are on UPS.  This means we should be able to monitor these systems from the control room as long as UPS power lasts.  The outbuilding switches are in varying states of UPS readiness, and their operation through a power outage is not guaranteed - the local workstations are available for monitoring in these locations (again, for as long as the UPS batteries last).

I've attached a flowmeter to the vent from the CP1 dewar. I'll be using this to monitor the exhaust flow for the next couple of weeks. One concern is this may ice up and block the flow, I'll be keeping an eye on it to make sure this doesn't happen. There is an overpressure vent upstream of the flowmeter as well.