I used Tony and I's statecounter.py to take a look at the past year of data using minute trends. Minute and second trends do some rounding that I had to consider in my search, ETMX goes from -8.9 to +6.05425 during a lock aquisition, these average to ~ -1.42. I ended up searching the chan H1:SUS-ETMX_L3_LOCK_BIAS_OFFSET for above and below -2.0 doing the following calls:

This gave me an outfile.txt file full of results in the format "idx (data_idx_start, data_idx_stop) gpsstart gpsstop duration" which I did some brief analysis on yielding:

Percentage of the time the Bias offset was [+]: 43.70 %  *Past locked LOWNOISE_ESD_ETMY
Percentage of the time the Bias offset was [-]: 56.25 %   *Between PREP_FOR_LOCKING and LOWNOISE_ESD_ETMY
Total duration in [+]: 205.3 [Days], [-]: 159.5 [Days] over 365.0 [Days]
Data timespan missing due to minute rounding and FW restarts 0.1965 [Days] (0.054 % of total time).

Where the BIAS_OFFSET is changed from negative to positive has changed a little over the past year but it's always during one of the final states of ISC_LOCK and it's always reset to negative PREP_FOR_LOCKING.

Jenne sent me a Mattermost message this afternoon pointing out an odd "oscillation" in Amp2 output power, so I took a look.  Sure enough, it was doing something weird.  Ever since the power outage, on a roughly 2-hour period, the output power would drop slightly and come back up.  I looked at items that directly impact the amplifier: Water/amp/pump diode temperatures, pump diode operating currents, and pump diode output.  Everything looked good with the temperatures and operating currents, but the pump diode output for 3 of the 4 pump diodes (1, 2, and 4) showed the same periodic behavior as the amp output; every ~2 hours, the pump diode output would spike and come back down, causing the "oscillation" in Amp2 output power.  See first attachment.  But what was causing this behavior?

At first, I couldn't think of anything beyond, "Maybe the pump diodes are finally starting to fail..."  Gerardo, who was nearby at the time, reminded me that during the last power outage the H2 enclosure had an odd sound coming from its environmental control system, and that it was not showing up on the control panel for the system; the control panel showed everything as OFF, but when Randy climbed on top of the enclosure he found one of the anteroom fans moving very slowly and haltingly, and making a noise as it did so.  Turning the fans off at the control panel fixed the issue at the time.  So, I took a look at the signal for the PEM microphone in the PSL enclosure, which Gerardo also reminded me of, and sure enough, the mic was picking up more noise than it was before the power outage (see 2nd attachment).  Around the same time, from the front of the Control Room Sheila noted that Diag_Main was throwing an alarm about the PSL air conditioner being ON.  It had been doing this throughout the day, but everytime I checked the AC temperature it was reading 70 °F, which was a little lower than normal but not as low as it reads when the AC is actually on (which is down around 67 °F or so).  This time, however, when I checked the AC temperature it was reading 68 °F.  Huh.  So I pulled up a trend of the PSL enclosure temps and sure enough, every 2 hours it looks like the AC comes on, drops the temperature a little, then turns off, and this behavior lines up with the "oscillation" in Amp2 output (see 3rd attachment; not much data for the enclosure temps since those come in through Beckhoff, which was recovered earlier this afternoon, but enough).

I went out and turned every PSL environmental item (HEPA fans, ACs, and make-up air) ON then OFF again and placed the enclosure back in Science Mode (HEPA fans and AC off, make-up air at 20%).  Won't know for sure if this cured the issue, as it's been happening on a 2-hour period, but looking at the PEM microphone in the enclosure shows promise.  The PEM mic is not picking up the extra noise, it's back to where it was before the power outage (see final 2 attachments).  Also encouraging, at the time of writing the AC temperatures are above the temperature where the ACs would kick on before I cycled the environmental controls.  I'll continue to monitor this over the weekend.

Dave, Jonathan, Tony, operators, ...

This is a compilation of recovery actions based off of a set of notes that Tony took while helping with recovery.  This is to augment the already existing log entries 88381 and 88376.  Times are localtime.

Thurs 4 Dec

At 12:25 PST power went out.  Tony and Jonathan had been working to shut down some of the systems so that they could have a graceful power off.  The UPS ran out around 1:17 PST.  At 2:02 the power came back.

Tony checked the status of the network switch, making sure they all powered on and we could see traffic flowing.

We started up the DNS/DHCP servers, as well as made sure the FMCS system was coming up.

Then we got access to the management box.  We did this with a local console setup.

The first steps were to get file servers up, we needed to get /ligo and /opt/rtcds up.  We started on /opt/rtcds as that is what FE computers need.  We turned on h1fs0 and made sure it was exporting file systems.  H1fs0 was problematic for us.  The opt/rtcds file system is a ZFS file system.  We think that the box came up, exported the /opt/rtcds path, and then got the zpool ready to use.  In the mean time another server came up and wrote to /opt/rtcds.  This appears to have happened before the zfs filesystem could be mounted, so it created directories in /opt/rtcds and kept the zfs filesystem (which had the full contents of /opt/rtcds) from mounting.  When we noticed this we deleted the /opt/rtcds contents on h1fs0, made sure the zfs file system mounted, and then re-exporting things.  This gave all the systems a populated /opt/rtcs.  We had to reboot everything that had started as they ended up now having stale file handles. There were still problems with the mount.  The file system performance was very slow over nfs.  Direct disk access was fast when testing on the file server.  We fixed this the next day after rulling out network conjestion and errors.

We then turned on the h1daq*0 machines to make it possible to start recording data.  However they would need a reboot to clear issues with /opt/rtcds, and would need front end systems up in order to have data to collect.

Then we went to get /ligo started.  We logged onto cdsfs2.  As a reminder cdsfs2,3,4,5 are a distributed system with the files.  We don't start this much so we had forgotten how.  Our notes hinted at it.  Dan Moraru helped here.  What we had to do was to tell pacemaker (pcs) to leave maintenance mode, then it started the cdsfsligo ip address.  Dan did a zpool reopen to fix zfs errors.  Then we restarted the nfs-server service.  At this point we had a /ligo file system.  We updated the notes on cdswiki as well so that we a reminder for next time.  The system was placed back into maintenance mode (the failover is problematic).

The next step was to get the boot server running.  This was h1vmboot5-5, that lived on h1hypervisor.  This is a kvm based system, that does not use proxmox, like our vm cluster.  So it took us a moment to get on, we ended up going in via the console and doing a virsh start on h1vmboot5-5.

Dave started front-ends at this point.  Operators were checking the IOC for power.

We started the 0 leg of the daq.

We started the procmox cluster up and began starting ldap and other core services.  To get the VMs to start we had to do some work on cdsfs0 as the VM images are stored on a share there.  This was an unmount of the share, starting zfs on cdsfs0 and a remount.

Ldap came up around 4:31.  This allowed users to start logging into systems as themselves.

Turned on the following vms

• ldapmgmt
• ldaprep1
• ldaprep2
• extalert
• cdsioc0
• beckhoffimages
• cdsioc1
• condamirror
• h0epics
• epicsburt
• seismon2
  • Note we should get rid of seismon and seismon1 vms

We powered on the epics gateway machine.

We needed to reboot h1daqscript0 to get mounts right and to start the daqstat ioc.  This was around 5pm.  The overview showed that TW1 was down.  We needed to bring and interface up on h1daqdc1 and start a cds_xmit process so that data would flow to TW1.  We got TW1 working around 5:15pm.

Powered on h1xegw0, and fmcs-epics.  Note with fmcs-epics the button doesn't work (it is a mac mini with a rack mount kit), you need to go the back and find the power button there.

Reviewing systems, we turned off the old h1boot1 (there are no network connections, so it doesn't break anything to power on, but it should be cleaned up).  Powered on the ldasgw0,1 machines so that h1daqnds could mount the raw trend archive.

The epics gateways did not start automatically, so we went onto cdsegw0 around 5:45 and ran the startup procedure.

The wap controller did not come up.  Something is electrically wrong (maybe a failed power supply).

At 6:01 Dave powered down susey for Fil.  It was brought back around 6:11.

Through this Dave was working on starting models.  The /opt/rtcds was very slow and models would time out while reading the safe.snap files.  Eventually Dave got things going.

Patrick, Tony, and Jonathan did some work on the cameras while Dave was restarting systems.  We picked a camera, cycled the network switch port to power cycle the camera.  Then restarted the camera service.  However this did not result in a video stream.

Friday

Jonathan found a few strange traffic flows while looking for slow downs.  These were not enough to cause the slow downs we had.  h1daqgds1 did not have it's broadcast interface come up and was transmitting the 1s frames out over its main interface and through to the router.  So this was disabled until it could be looked at.  The dolphin manager was sending more traffic to all daqd boxes trying to establish the health of the dolphin fabric.  It was not a complete fabric as dc0 was not brought back up as it isn't doing anything at this point.  In response we started dc0 to remove another source of abnormal traffic from the network.

These were not enought to explain the slow downs.  Further inspection showed that there was not abnormal traffic going to/from h1fs0 and that other than the above noted traffic there was no unexpected traffic on the switch.

It was determined the slow downs were strictly nfs related.  We changed cdsws33 to get /opt/rtcds from h1fs1.  This was done to test the impact of restarting the file server.  After testing restarts with h1fs1 and cdsws33 we stopped the zfs-share service on h1fs0 (so there would not be competing nfs servers) and restarted nfs-kernel-server.  In general no restarts where required, access just returned to normal speed for anything touching /opt/rtcds.

After this, TJ restarted the guardian machine to try and put it into a better state.  All nodes came up except one, which he then restarted.

Dave restarted h1pemmx, which had been problematic.

We restarted h1digivideo2,4,5,6.  This brought all the cameras back.

Looking at the h1daqgds machines the broadcast interface had not come back.  So starting the enp7s0f1 interfaces and restarting the daqd fixed its transmission problems.  The 1s frames are flowing to DMT.  At Dan's request Jonathan took a look at a DMT issue after starting h1dmt1,2,3.  the /gds-h1 share was not mounted on h1dmtlogin, so nagios checks were failing due to not have that data available.

The wap control was momentarily brought to life.  It's cmos battery was dead so it needed intervention to boot and needed a new date/time.  However it froze while saving settings.

The external medm was not working.  Looking at the console for lhoepics, its cmos battery had failed as well and needed intervention to boot.

After starting ldasgw0,1 yesterday we were able to mount the raw minute trend archive to the nds servers.

Fw2 needed a reboot to reset it's /opt/rtcds.

We also brought up more of the test stand today to allow Ryan Crouch and Rahul to work on the sustriple in the staging building.

A note, things using kerberos for authentication are going slower.  We are not sure why.  We have reached out to the LIAM group for help.

TITLE: 12/05 Day Shift: 1530-2300 UTC (0730-1500 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
INCOMING OPERATOR: Oli
SHIFT SUMMARY: We've continued to recover after the power outage yesterday. The major milestones were getting the /opt/rtcds/ file server reset and running back up to normal speed. This allowed us to get all guardian nodes up and running and start other auxillary scripts and IOCs. The Beckhoff work wrapped up around noon and we were able to get the IMC locked and ready to start initial alignment just before 1300PT. We then ran into some IMC whitening issues, which have since been solved. The main issue at this point is the COMM and DIFF beatnotes are very low. Investigation ongoing. See many other alogs for specific information on the power outage recovery.

• 1828 h1guardian1 reboot - all but one node came up successfully.  Restarted the failed node (ISI_ETMY_ST2_BLND) and it started up without issue.

While TJ was running initial alignment for the green arms, I noticed that the ALS X beam on the camera appeared to be too far to the right side of the screen. The COMM beatnote was at -20 dBm, when it is normally between -5 and -10 dBm. I checked both the PR3 top mass osems and the PR3 oplevs. The top mass osems did not indicate any significant change in position, but the oplev seems to indicate a significant change in the yaw position. PR3 yaw was around -11.8 but then changed to around -11.2 after the power outage. It also appears that the ALS X beam is closer to its usual position on the camera.

I decided to try moving PR3 yaw. I stepped 2 urad and which brought the oplev back to -11.8 in yaw and the COMM beatnote to -5 dBm. Previous slider value: -230.1, new slider value: -232.1.

The DIFF beatnote may not be great yet, but we should wait for beamsplitter alignment before making any other changes.

Closes FAMIS38811, last checked in alog87797.

We can see yesterdays outage on every plot as expected and the BRS issues from October 21st. There's doesn't look to be any trend in the driftmon, and ETMY looks to be still slowly increasing in temperature before the outage. The aux plot looks about the same as during the last check except for ETMY DAMP CTL looks to have come back at a different lower spot but the EY SEI configuration looks like is not quite fully recovered so that may be why.

All systems have been recovered enough to start relocking at this point. We will not be relocking the IFO fully at this time though since the ring heaters have been off and their time to thermalize would take too long. We will instead be running other measurements that will not need the ring heaters on.

There is a good chance that other issues will pop up, but we are starting initial alignment now.

[Oli, Jenne]

Oli brought all of the IMC optics back to where they had been yesterday before the power outage.  We squiggled MC2 and the IMC PZT around until we could lock on the TEM00 mode, and let the WFS (with 10x usual gain) finish the alignment.  We offloaded the IMC WFS using the IMC guardian.  We then took the IMC to OFFLINE, and moved MC1 and the PZT such that the DC position on the IMC WFS matched a time from yesterday when the IMC was offline before the power outage. We relocked, let the alignment run again, and again offloaded using the guardian.   Now the IMC is fine, and ready for initial alignement soon.

I restarted all the dust monitor IOCs, they all came back nicely. I then reset the alarm levels using the 'check_dust_monitors_are_working' script.

Marc Daniel

We finished the software chanegs for EtherCAT Corner Station Chassis 4.

We found 2 issues related to the power outage:

• Corner chassis 3 terminal R10 EL9410 needed replacement.
• EndX chassis 3 coupler EK1101 (left rail) needed replacement.

Baffle PD chassis in EX has a bench suppy that needs to be turned on by hand.

The system is back up and running.

Jonathan, EJ, Richard, TJ, Dave:

Slow /opt/rtcds file access has been fixed.

All Guardian nodes running again.

The long-range-dolphin issue was due to the end station Adnaco chassis being down, Richard powered these up and all IPCs are working correctly now.

Cameras are now working again.

Currently working on:

 . GDS broacaster and DAQ min trend archived data

 . Alarms and Alerts

Closes FAMIS#27621, last checked 88194

Everything looking okay except PMC REFL being high. It does look like it's been higher since they recovered the PSL after yesterday's power outage, but we are also still recovering and more will still need to be done for the PSL anyway, so maybe this is expected/will be adjusted anyway.

Laser Status:
    NPRO output power is 1.83W
    AMP1 output power is 70.67W
    AMP2 output power is 139.9W
    NPRO watchdog is GREEN
    AMP1 watchdog is GREEN
    AMP2 watchdog is GREEN
    PDWD watchdog is GREEN

PMC:
    It has been locked 0 days, 17 hr 9 minutes
    Reflected power = 25.84W
    Transmitted power = 105.6W
    PowerSum = 131.5W

FSS:
    It has been locked for 0 days 16 hr and 19 min
    TPD[V] = 0.5027V

ISS:
    The diffracted power is around 4.0%
    Last saturation event was 0 days 0 hours and 0 minutes ago

Possible Issues:
    PMC reflected power is high

Following yesterdays power outage, primary FAC systems (HVAC, domestic water supply, fire alarm/suppression) all seem to be operating normally. Fire panels required acknowledgement of power supply failures, and battery backups did their job. There is an issue plaguing the supply fan drive at the FCES which revisited us this morning, but that arising from the outage is highly unlikely. Temperatures are within normal ranges in all areas. Eric continues to physically walk systems down to ensure normal operation of points in systems not visible via FMCS.

C. Soike E. Otterman R. McCarthy T. Guidry

TITLE: 12/0 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Planned Engineering
OUTGOING OPERATOR: None

QUICK SUMMARY: Still in the midst of recovery from the power outage yesterday. Looks like all suspensions stayed damped overnight except the BS which tripped at 0422UTC. SEI is still in a non-nominal state as Jim left it yesterday (alog88373). Many medm screens, scripts, other processes are running *very* slow. 

Recovery will continue, one forward step at a time.

Related: https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=87729

We disconnected everything from the ISS array installation spare unit S1202965 and stored it in the ISS array cabinet in the vac prep area next to the OSB optics lab. See the first 8 pictures.

The  incomplete spare ISS array assy originally removed from LLO HAM2 (S1202966) was moved to a shelf under the work table right next to the clean loom in the optics lab (see the 9th picture). Note that one PD was pulled from that and was transplanted to our installation spare S1202965.

Metadata for both 2965 and 2966 were updated.

ISS second array parts inventory https://dcc.ligo.org/E2500191 is being updated.

Jennie W, Keita,

Since we don't have an easy way of scanning the input beam in the vertical direction, Keita used the pitch of the PZT steering mirror to do the scan and we read out the DC voltages for each PD.

The beam position can be inferred from the pictures setup - see photo. As the pitch actuator on the steering mirror is rotated the allen key which is in the hole in the pitch actuator moves up and down relative to the ruler.

height on ruler above table = height of centre of actuator wheel above table + sqrt((allen key thickness/2)^2 + (allen key length)^2) *np.sin(ang - delta_theta)

where ang is the angle the actuator wheel is at and delta_theta is the angle from the centre line of the allen key to its corner which is used to point at the gradations on the ruler.

The first measurement from our alignment that Keita found yesterday that minimised the vertical dither coupling is shown. It shows voltage on each PD vs. height on the ruler.

From this and from the low DC voltages we saw on the QPD and some PDs yesterday Keita and realised we had gone too far to the edge of the QPD and some PDs.

So in the afternoon Keita realigned onto all the of PDs.

Today as we were doing measurements on it Keita realised we still had the small aperture piece in place on the array so we moved that for our second set of measurements.

The plot of voltage with ruler position and voltage with pitch wheel angle are attached.