SUS - power outage recovery

Oli, Ryan S, Rahul

Except ETMX and ETMY (timing error, work currently ongoing), we have recovered all other suspensions by un-tripping the WD and setting it to SAFE for tonight. The inmons looks fine - eyeballed them all (bosems and aosem), nothing out of the order.

For ETMX and ETMY, Dave is currently performing a computer restart, following which they will set to safe as well.

 

Thurs DAY Ops Summary

TITLE: 12/04 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Planned Engineering
INCOMING OPERATOR: None
SHIFT SUMMARY:

After CDS & H1-Locking work of yesterday, today we transitioned to starting prep work for the upcoming vent of HAM7.

However, there was more CDS work today which was related to ISC Slow Controls....but in the middle of this...

There was a 90min power outage on site!!
 

LOG:

• early-1615 Randy was at the FCES putting stickers on items
• 1539-1900 pre-vent technical cleaning begins (kim.nellie)
  • 1716 SR3 + HAM5 tripped; gave it some time, untripped/redamped @1805
• 1545 SEI_ENV transitioned to MAINTENANCE for vent
• 1609 TCSx Laser off (TJ contacted; prob due to cleaning near the rack)
• 1624-1745 BSC2 platform & then sorting parts (randy)
  • Tripped BS, ISI, HEPI---leaving tripped until randy finishes in about an hr
  • Once he was done, BS was taken to ALIGNED & SEI_BS taken to DAMPED_HEPI_OFFLINE
  • 1745-1845 Sorting through rail parts in North Bay
• 1626 quick assistance for cleaning (chris)
• 1628-1635 lvea walkthru (richard)
• 1630 cer to pull slow control chassis for ISC (fil)
  • This affects a PZT for the IMC, so it lost lock.  Keeping IMC_LOCK at OFFLINE during this beckhoff work.
• 1718-1810 Unboxing the JAC in the optics lab (jennie, jason)
• 1728 Taking Integrating sphere (had been at KAGRA) into PCal Lab (tony)
• 1730-1855 Grabbing parts at offsite vendor (mitch)
• 1751-1900 Building 2nd CHETA enclsure in the CHETA/JAC Lab (matt)   
• 1803-2012 OPO Testing cont (KarMeng.Eric)
  • 1812 Shiela joining but will be in/out with them + looking for parts
• 1813-1820 chamber measurements for JAC (jason)
• 1814 Kobelco (purge air compressor) being turned on (travis,jordan)
• 1848-2000 Inventory work at EX + EY (randy)
• 2005 Grabbing wipes in Cleaning Room (ibrahim)
• 2023 CHETA Viewport part search (gerardo)
• 2025 Power Outage ON Site!
  • Ended up down for over 90min
• See oli's alog for a summary of recovery.

PSL Recovered After Power Outage

R. Short, P. Thomas, J. Oberling

We have recovered the PSL after today's power outage.  Some notes for the future:

• When recovering the PSL Beckhoff PC and control software, make sure both Control Boxes are ON before trying to start the software
• The PSL's calibration data was lost, again (this seems to happen after every software restart, we're still not sure why)

I've attached a picture of the Settings table for PSL sensor calibration and operating hours for future reference.  Again, our persistent operating hours (that track total uptime of PSL laser components; OPHRS A in the attached picture) will continue to be wrong as we cannot update this value.  The current operating hours, which tracks operating hours of currently operating components (i.e. we've been running this specific NPRO for X hours; OPHRS in the attached picture) are correct.

We have the PMC and FSS RefCav locked, but have left the ISS OFF overnight while the PMC settles.  The PMC requires a beam alignment tweak (normal after an extended time off, like a 90 minute power outage) but we don't yet have Beckhoff so we don't have access to our picomotor mounts.  I'll tweak the beam alignment tomorrow once Beckhoff has been recovered.

VOPO crystal mount stage test

[Sheila, Eric, Karmeng]

We checked the NLG of the OPO without cavity lock, threshold power is roughly at 3.7mW, at the same order as the test performed at MIT (E2500270)

Checked the crystal alignment, found 9 good dual resonant position, from -2689um to 1223um, with 490um separation between each point. Everything seems to be in a good position. 

Plus a clipped dual resonance position at -3190um. Crystal edge is roughly at 1500um and -3200um.

Red alignment does not shift between the various crystal position.

POWER OUTAGE At LHO At 1225pm PDT!

It's been a few minutes so far.  There is emergency lighting.  Luckily since it was lunchtime there were no people out on the floor.  Recovery Begins!

Ground Seismometer Mass Position Check - Monthly

FAMIS 38886, last checked in alog88025

Both HAM5 and the BS chambers were in 'ISI_DAMPED_HEPI_OFFLINE' at the time this was run, but all other chambers were nominal.

There are 15 T240 proof masses out of range ( > 0.3 [V] )!
ETMX T240 2 DOF X/U = -1.565 [V]
ETMX T240 2 DOF Y/V = -1.633 [V]
ETMX T240 2 DOF Z/W = -0.886 [V]
ITMX T240 1 DOF X/U = -2.256 [V]
ITMX T240 1 DOF Z/W = 0.467 [V]
ITMX T240 3 DOF X/U = -2.386 [V]
ITMY T240 3 DOF X/U = -1.073 [V]
ITMY T240 3 DOF Z/W = -2.895 [V]
BS T240 1 DOF Y/V = -0.323 [V]
BS T240 2 DOF Z/W = 0.327 [V]
BS T240 3 DOF X/U = -0.547 [V]
BS T240 3 DOF Z/W = -0.401 [V]
HAM8 1 DOF X/U = -0.638 [V]
HAM8 1 DOF Y/V = -0.729 [V]
HAM8 1 DOF Z/W = -1.11 [V]

All other proof masses are within range ( < 0.3 [V] ):
ETMX T240 1 DOF X/U = 0.007 [V]
ETMX T240 1 DOF Y/V = -0.049 [V]
ETMX T240 1 DOF Z/W = -0.017 [V]
ETMX T240 3 DOF X/U = -0.005 [V]
ETMX T240 3 DOF Y/V = -0.056 [V]
ETMX T240 3 DOF Z/W = -0.051 [V]
ETMY T240 1 DOF X/U = 0.045 [V]
ETMY T240 1 DOF Y/V = 0.198 [V]
ETMY T240 1 DOF Z/W = 0.249 [V]
ETMY T240 2 DOF X/U = -0.073 [V]
ETMY T240 2 DOF Y/V = 0.218 [V]
ETMY T240 2 DOF Z/W = 0.037 [V]
ETMY T240 3 DOF X/U = 0.268 [V]
ETMY T240 3 DOF Y/V = 0.076 [V]
ETMY T240 3 DOF Z/W = 0.147 [V]
ITMX T240 1 DOF Y/V = 0.254 [V]
ITMX T240 2 DOF X/U = 0.151 [V]
ITMX T240 2 DOF Y/V = 0.258 [V]
ITMX T240 2 DOF Z/W = 0.223 [V]
ITMX T240 3 DOF Y/V = 0.104 [V]
ITMX T240 3 DOF Z/W = 0.094 [V]
ITMY T240 1 DOF X/U = 0.074 [V]
ITMY T240 1 DOF Y/V = 0.123 [V]
ITMY T240 1 DOF Z/W = -0.013 [V]
ITMY T240 2 DOF X/U = 0.017 [V]
ITMY T240 2 DOF Y/V = 0.22 [V]
ITMY T240 2 DOF Z/W = 0.148 [V]
ITMY T240 3 DOF Y/V = 0.091 [V]
BS T240 1 DOF X/U = 0.184 [V]
BS T240 1 DOF Z/W = -0.223 [V]
BS T240 2 DOF X/U = 0.094 [V]
BS T240 2 DOF Y/V = -0.224 [V]
BS T240 3 DOF Y/V = 0.004 [V]

There are 2 STS proof masses out of range ( > 2.0 [V] )!
STS EY DOF X/U = -4.625 [V]
STS EY DOF Z/W = 2.321 [V]

All other proof masses are within range ( < 2.0 [V] ):
STS A DOF X/U = -0.498 [V]
STS A DOF Y/V = -0.915 [V]
STS A DOF Z/W = -0.485 [V]
STS B DOF X/U = 0.154 [V]
STS B DOF Y/V = 0.939 [V]
STS B DOF Z/W = -0.337 [V]
STS C DOF X/U = -0.685 [V]
STS C DOF Y/V = 0.77 [V]
STS C DOF Z/W = 0.505 [V]
STS EX DOF X/U = -0.195 [V]
STS EX DOF Y/V = -0.111 [V]
STS EX DOF Z/W = 0.112 [V]
STS EY DOF Y/V = 1.271 [V]
STS FC DOF X/U = 0.071 [V]
STS FC DOF Y/V = -1.254 [V]
STS FC DOF Z/W = 0.567 [V]
 

Calibration measurement post DAC swap

Last night I took a calibration measurement (broadband and simulines) after the IFO was locked for about 3.5 hours. This generated calibration report 20251204T035347Z.

The results from broadband and simulines on the front page look reasonable. The plot label indicates that these are from PCAL X and Y but I think the broadband and simulines are run on PCAL Y only, so I don't know if PCAL X agrees with Y. I believe as of last night, the DAC for PCAL X had been swapped.

I made no changes to the pydarm ini file to account for the DAC change before running the report.

Thu CP1 Fill

Thu Dec 04 10:15:45 2025 INFO: Fill completed in 15min 42secs

Plot has zoomed y-scale to reduce number of divisions in order to show detail.

PSL Cooling Water pH Test - Monthly

FAMIS 27645

pH of PSL chiller water was measured to be between 10.0 and 10.5 according to the color of the test strip.

CDS Frontend Upgrade Summary: Thursday 4th December 2025

Summary of where we are in the upgrade:

h1susey

All Gen Std DACs replaced with 2 LIGO-DACs. All AI-Chassis upgraded to SCSI daisy-chain.

Timing card replaced with one having latest firmware

New models [h1iopsusey, h1susetmy, h1sustmsy, h1susetmypi]

All models running RCG-5.5.2

h1susex

All Gen Std DACs and first-gen LIGO-DAC replaced with 2 LIGO-DACs. All AI-Chassis upgraded to SCSI daily-chain.

Timing card firmware upgraded in place

New models [h1iopsusex, h1susetmx, h1sustmsx, h1susetmxpi]

All models running RCG-5.5.2

h1iscex

18bit-DAC replaced with 20bit-DAC.

Timing card firmware upgraded in place

New models [h1iopiscex, h1calex, h1pemex]

All models running RCG-5.5.2

h1iscey

Timing card firmware upgraded in place

h1seiex, h1seiey

Timing card firmware upgraded in place

h1susauxex, h1susauxey

Timing card firmware upgraded in place.

All models running RCG-5.5.2 (first test of new RCG)

h1pemmx, h1pemmy

Timing card firmware upgraded in place.

 

Mega Cleanroom Powered On

At approx 8am the cleanroom above HAM 5/6/7 was powered on in prep for vent activities and initial cleanings. As to be expected, zone 4g temp has spiked. No action on behalf of FAC is required at this time.

T. Guidry K. Stewart N. Sanchez

IO Chassis Timing Card Firmware Versions

On Tuesday 02dec2025 Marc upgraded the firmware version of the IO Chassis timing cards at both end stations and mid stations. This upgrade was done "in place" using a laptop connected to the JTAG port on the timing card.

The current timing card firmware versions for H1 are:

All are running 0xfe0 except the following which are running 0x635

h1omc0: upgraded some time ago during O4 to move its DUOTONE frequencies from [960/961Hz] to [1920/1921Hz]

h1susey: upgraded Mon 01dec2025 when its timing card was replaced as part of the LIGO-DAC upgrade

Upgraded by Marc Tue 02dec2025:

h1iscex 
h1iscey
h1pemmx
h1pemmy
h1seiex
h1seiey
h1susauxex
h1susauxey
h1susey

h1iscex 20bit-DAC replacement

Jonathan, Jeff, Tony, Dave:

Following the upgrade of h1iscex 18bit-DAC upgrade to a 20bit-DAC Tuesday 02dec2025 we discovered the 20bit-DAC was not outputting any voltage.

Wed 03dec2025 Tony and I went to h1iscex to replace the suspect 20bit-DAC with the second DAC which was removed from the h1susex upgrade Tue02dec2025.

All while H1 was locked, we powered down h1iscex and its IO Chassis, pulled the chassis from the rack, took its lid off, replaced the 20bit-DAC and powered the chassis up while it was open.

I powered up the computer and auto started the models. H1 lost lock soon after the IOP model started running (possible Dolphin IPC minor break in transmission?).

We verified that the replacement 20bit-DAC was working, looking at both the DAC_chan7->ADC_chan30 DUOTONE inteface card loop back and h1calex drive of DAC_chan6.

We closed the chassis and pushed it back into place.

Suspect broken 20bit-DAC (removed)	190219-10
Replacement 20bit-DAC (installed)	150311-01 (S1700286)

Note that both of these 20bit-DACs came out of h1susex during its upgrade to LIGO-DACs Tue02dec2025. I'm not sure which one was driving the PI signals, and initially I chose the 2019 card for h1iscex over the older 2015 card.

Also note that the 2015 card has a LIGO S-number tag, the first I've seen on a GenStd IO card.

Thurs DAY Ops Transition

TITLE: 12/04 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Planned Engineering
OUTGOING OPERATOR: None
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 3mph Gusts, 1mph 3min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.33 μm/s 
QUICK SUMMARY:

H1 has been locked almost 15hrs....but H1 is going DOWN in prep for the HAM7 vent and will be set to IDLE. 

There will be more CDS (have heard Beckhoff) work, but no "proof of life" for H1 after that work will be done. 

Now the focus is on prep to vent with Kim & Nellie commencing shortly with thorough technical cleaning of HAM chambers.  Other possible items on the list for today:  

• Turn on Kobelco purge
• Forklifting furniture
• HAM7 vent prep
• Prep for HAM7 -Y door removal

instructions for recovering alignment after gate valves are excercised

It seems like it would be useful to have this recipie written in one place.  After the gate valves are opened and closed, we often have a hard time relocking because of alignment.  

Here's what we think we should do to recover after gate valves open.  

1. Do initial alignment without using green cameras in the loop, as the cameras are right next to the gate valves. 
   1. Start by using the baffle PD dither alignment script to set the TMS alignment: use the suspension guardians to misalign both ITMs and ETMs, run dither align script for both TMSs found on the INIT ALIGN MEDM, (dither align scripts, TMSX and TMSY, can run at the same time).  Open the ITM baffle PD scope and watch the script find the pointing of TMS that sets the beam to PD1, PD4, and finally centered on the optic.  When this finishes sucsesfully for both TMSs, there should be a dim green beam visible on both ALS cameras, if there is not check the alignment of PR3 (affects both arms) and the BS (only moves the Y arm beam).  This TMS alignment has to be set before you can move onto the next step, otherwise the ITM alignment won't be right. 
   2. Use the suspension guardians to align both ITMs, but leave the ETMs misaligned.  Run the dither alignment script for ITMX and ITMY, this time watching the ETM baffle PD scope to watch the script's progress.  Once this is finished, the ITMs should be set, and we should not move them based on the cameras. 
   3. Use the suspension guardians to set the ETMs to aligned, and you will hopefully see mulitple beams or cavity flashes on the ALS cameras.  You can manually align the ETMs to increase the flashes until they are good enough to lock the arm.  Use the ALS guardians (in auto mode might be helpful) to set them to ETM_TMS_WFS_OFFLOADED, this will not run the green camera servos, it runs green WFS to the ETM and TMS alignment only to increase build up. Don't run INITIAL_ALIGNMENT or INITIAL_ALIGNMENT offloaded. 
   4. Align the vertex to these ITMs: set the ALIGN_IFO guardian to manual, init it to control the suspension guardians.  Choose INPUT_ALIGN_OFFLOADED, once that finishes PRC_ALIGN_OFFLOADED, then MICH_BRIGHT_OFFLOADED, then SRC_ALIGN_OFFLOADED.  
2. Make two guardian changes before you start locking the full IFO
   1. Don't use the SRC1 + SRC2 ASC loops in DRMI, these don't seem to work until the alignment is set well for full lock.  You can turn these off by going to lscparams, and searching for use_DRMI_ASC, setting SRC1+ SRC2 to false.  Load ISC_DRMI for this change to take effect. 
   2. Don't shutter the ALS beams once they are no longer used for locking.  In ISC_LOCK, find the edge     ('IDLE_ALS',  'CARM_OFFSET_REDUCTION', 5), and reduce the weight which is normally 5 to 1, so that this path which avoids shuttering the ALS becomes the default path.  Load ISC_LOCK for this change to take effect.
3. Start to relock as usual, going through PRMI.  Stop at CHECK_DRMI_ASC, where all the DRMI ASC loops will be running other than SRC1+SRC2, and manually adjust SRM sliders to reduce POP90 and increase POP18. 
4. You can try to lock all the way to ENGAGE_ASC for full IFO, but this may not work.  If it does not, you may need to manually align DHARD, CHARD, and SRM. 
   1. If you are adjsuting alignments by hand, you want to keep the green beams locked to the arms as the alignment changes.  To get the green alignment to follow any changes you make, stop at IDLE_ALS and turn on the scripts that servo the green TMS QPD offsets based on the green WFS.  You can access this script by going to either ALS end station overview and clicking the button that says QPD OFFSETs, then using the pink drop down button to start the servo for each arm, and open the ndscope that lets you watch the progress of these servos (screenshot)
   2. If the power in the two arms is mismatched (visible on the PRMI SB4 min ndscope on the front wall), this could be a sign that DHARD is misaligned. 
      1. If you notice this in a state before DHARD WFS are closed, RF_DARM or CARM_OFFSET reduction , you can use the move_arm_dev script to step DHARD pitch and yaw to bring the arms closer to each other.  userapps/asc/h1/scripts/move_arm_dev.py DH P 0.01, or similar.  
      2. If you notice the arm powers diverging from each other after DHARD WFS are closed, this might be fixed by moving SRM while watching to make sure POP 90 stays low.  
   3. If the power recycling gain is low, you may need to adjust CHARD.  You can step through CARM_OFFSET reduction or stop after it finishes and use 62110 to check if the refl power is too low for the arm transmission that you are at, and try to move CHARD while trying to increase the refl power (watch build_up_slow ndscope linked from the ISC guardians overview and other places).  
5. Once you are all the way locked (congratulations), stop at PREP_ASC_FOR_FULL_IFO before you try turning on ASC.  It may be a good idea to note the green QPD offsets with a screenshot, if you loose lock they wil be reverted with SDF revert but you would need to reset them to these values to relock.   Also look at the ITM green camera offsets, from each ALS end station overview in the green WFS section there is a button that says CAM ITM YAW or PIT, you can take a screenshot of this screen for both ITM cameras and compare the current location of the beam centroid to the setting (screenshot)
6. Use a guardian shell (guardian -i ISC_LOCK) to copy and paste lines from ENGAGE_ASC_FOR_FULL_IFO manually, skipping SRC1 +SRC2.  (you might grab another screenshot of the green QPDs and cameras at this point)
7. Then you can manual the guardian to TMS_SERVO, engage_soft_loops, and DARM offset, and take another green QPD offset + camera screenshot.  
8. Having the DARM offset on as well as the soft loops seems to help the SRC ASC work, so try engaging those once those steps are done.  (Congratulations on getting all the ASC controlled)
9. Important step: reset the green references.  In the past we have reset the green QPD offsets by accepting them in SDF as well as the camera offsets, but it doesn't make sense to be changing the QPD offsets as they shouldn't be impacted by the gate valve moving the camera.  Today once we had all the ASC engaged, we stepped the QPD offsets back to their original settings from SDF, watching the arm transmissions in the ndscope.  The arm transmissions seemed fine with the original QPD offsets, so we left those there and only updated the green camera offsets.  I think this should be the way we do things in the future, not reseting the QPD offsets by default.  
10. undo the gaurdian changes from step 2 above.  After the next lockloss, run initial alignment as usual, letting the green cameras be used to point the ITMs.  DRMI ASC should work now that the alignment has been set.  

 

Touched SQZR Settings & Sliders

TITLE: 12/04 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Planned Engineering & LOCKED
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 10mph Gusts, 6mph 3min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.28 μm/s 
QUICK SUMMARY:

SQZ_OPO_LR was giving us this error: 
"CLF frequency out of range, check CLF CMB".
I do not recall how this was resolved by Daniel while he was checking out the CMB. 

Once that was resolved we got the classic: "pump fiber rej power in ham7 high, nominal 35e-3, align fiber pol on sqzt0."  issue.
The Quarter & Half wave plates on SQZT0 were touched up to minimize H1:SQZ-SHG-FIBR_REJECTED_DC_POWER_MON. 
OPO was locked. Great!

Then the FC wasn't locking: 
So FC2 was moved from its nominal location. only after I accidentally moved ZM1 and reverted it.
Fumbled with the FC2 sliders for a while, then used the clear history button.
Fumbled around some more with the FC2 sliders. 
Locked the the FC with FC2 P @ 246.1 Y @ 51.1

And now the SQZ has been SQuoZe.

Forcast for tonight's wind looks good for a stable lock.