PSL Recovered After Power Outage

R. Short, P. Thomas, J. Oberling

We have recovered the PSL after today's power outage.  Some notes for the future:

• When recovering the PSL Beckhoff PC and control software, make sure both Control Boxes are ON before trying to start the software
• The PSL's calibration data was lost, again (this seems to happen after every software restart, we're still not sure why)

I've attached a picture of the Settings table for PSL sensor calibration and operating hours for future reference.  Again, our persistent operating hours (that track total uptime of PSL laser components; OPHRS A in the attached picture) will continue to be wrong as we cannot update this value.  The current operating hours, which tracks operating hours of currently operating components (i.e. we've been running this specific NPRO for X hours; OPHRS in the attached picture) are correct.

We have the PMC and FSS RefCav locked, but have left the ISS OFF overnight while the PMC settles.  The PMC requires a beam alignment tweak (normal after an extended time off, like a 90 minute power outage) but we don't yet have Beckhoff so we don't have access to our picomotor mounts.  I'll tweak the beam alignment tomorrow once Beckhoff has been recovered.

VOPO crystal mount stage test

[Sheila, Eric, Karmeng]

We checked the NLG of the OPO without cavity lock, threshold power is roughly at 3.7mW, at the same order as the test performed at MIT (E2500270)

Checked the crystal alignment, found 9 good dual resonant position, from -2689um to 1223um, with 490um separation between each point. Everything seems to be in a good position. 

Plus a clipped dual resonance position at -3190um. Crystal edge is roughly at 1500um and -3200um.

Red alignment does not shift between the various crystal position.

POWER OUTAGE At LHO At 1225pm PDT!

It's been a few minutes so far.  There is emergency lighting.  Luckily since it was lunchtime there were no people out on the floor.  Recovery Begins!

The power was out from 2025-12-04 or Thursday Dec 04 2025 from 20:25 UTC until 22:02 UTC (12:25 PST to 14:02 PST)

Ground Seismometer Mass Position Check - Monthly

FAMIS 38886, last checked in alog88025

Both HAM5 and the BS chambers were in 'ISI_DAMPED_HEPI_OFFLINE' at the time this was run, but all other chambers were nominal.

There are 15 T240 proof masses out of range ( > 0.3 [V] )!
ETMX T240 2 DOF X/U = -1.565 [V]
ETMX T240 2 DOF Y/V = -1.633 [V]
ETMX T240 2 DOF Z/W = -0.886 [V]
ITMX T240 1 DOF X/U = -2.256 [V]
ITMX T240 1 DOF Z/W = 0.467 [V]
ITMX T240 3 DOF X/U = -2.386 [V]
ITMY T240 3 DOF X/U = -1.073 [V]
ITMY T240 3 DOF Z/W = -2.895 [V]
BS T240 1 DOF Y/V = -0.323 [V]
BS T240 2 DOF Z/W = 0.327 [V]
BS T240 3 DOF X/U = -0.547 [V]
BS T240 3 DOF Z/W = -0.401 [V]
HAM8 1 DOF X/U = -0.638 [V]
HAM8 1 DOF Y/V = -0.729 [V]
HAM8 1 DOF Z/W = -1.11 [V]

All other proof masses are within range ( < 0.3 [V] ):
ETMX T240 1 DOF X/U = 0.007 [V]
ETMX T240 1 DOF Y/V = -0.049 [V]
ETMX T240 1 DOF Z/W = -0.017 [V]
ETMX T240 3 DOF X/U = -0.005 [V]
ETMX T240 3 DOF Y/V = -0.056 [V]
ETMX T240 3 DOF Z/W = -0.051 [V]
ETMY T240 1 DOF X/U = 0.045 [V]
ETMY T240 1 DOF Y/V = 0.198 [V]
ETMY T240 1 DOF Z/W = 0.249 [V]
ETMY T240 2 DOF X/U = -0.073 [V]
ETMY T240 2 DOF Y/V = 0.218 [V]
ETMY T240 2 DOF Z/W = 0.037 [V]
ETMY T240 3 DOF X/U = 0.268 [V]
ETMY T240 3 DOF Y/V = 0.076 [V]
ETMY T240 3 DOF Z/W = 0.147 [V]
ITMX T240 1 DOF Y/V = 0.254 [V]
ITMX T240 2 DOF X/U = 0.151 [V]
ITMX T240 2 DOF Y/V = 0.258 [V]
ITMX T240 2 DOF Z/W = 0.223 [V]
ITMX T240 3 DOF Y/V = 0.104 [V]
ITMX T240 3 DOF Z/W = 0.094 [V]
ITMY T240 1 DOF X/U = 0.074 [V]
ITMY T240 1 DOF Y/V = 0.123 [V]
ITMY T240 1 DOF Z/W = -0.013 [V]
ITMY T240 2 DOF X/U = 0.017 [V]
ITMY T240 2 DOF Y/V = 0.22 [V]
ITMY T240 2 DOF Z/W = 0.148 [V]
ITMY T240 3 DOF Y/V = 0.091 [V]
BS T240 1 DOF X/U = 0.184 [V]
BS T240 1 DOF Z/W = -0.223 [V]
BS T240 2 DOF X/U = 0.094 [V]
BS T240 2 DOF Y/V = -0.224 [V]
BS T240 3 DOF Y/V = 0.004 [V]

There are 2 STS proof masses out of range ( > 2.0 [V] )!
STS EY DOF X/U = -4.625 [V]
STS EY DOF Z/W = 2.321 [V]

All other proof masses are within range ( < 2.0 [V] ):
STS A DOF X/U = -0.498 [V]
STS A DOF Y/V = -0.915 [V]
STS A DOF Z/W = -0.485 [V]
STS B DOF X/U = 0.154 [V]
STS B DOF Y/V = 0.939 [V]
STS B DOF Z/W = -0.337 [V]
STS C DOF X/U = -0.685 [V]
STS C DOF Y/V = 0.77 [V]
STS C DOF Z/W = 0.505 [V]
STS EX DOF X/U = -0.195 [V]
STS EX DOF Y/V = -0.111 [V]
STS EX DOF Z/W = 0.112 [V]
STS EY DOF Y/V = 1.271 [V]
STS FC DOF X/U = 0.071 [V]
STS FC DOF Y/V = -1.254 [V]
STS FC DOF Z/W = 0.567 [V]
 

Calibration measurement post DAC swap

Last night I took a calibration measurement (broadband and simulines) after the IFO was locked for about 3.5 hours. This generated calibration report 20251204T035347Z.

The results from broadband and simulines on the front page look reasonable. The plot label indicates that these are from PCAL X and Y but I think the broadband and simulines are run on PCAL Y only, so I don't know if PCAL X agrees with Y. I believe as of last night, the DAC for PCAL X had been swapped.

I made no changes to the pydarm ini file to account for the DAC change before running the report.

Thu CP1 Fill

Thu Dec 04 10:15:45 2025 INFO: Fill completed in 15min 42secs

Plot has zoomed y-scale to reduce number of divisions in order to show detail.

PSL Cooling Water pH Test - Monthly

FAMIS 27645

pH of PSL chiller water was measured to be between 10.0 and 10.5 according to the color of the test strip.

CDS Frontend Upgrade Summary: Thursday 4th December 2025

Summary of where we are in the upgrade:

h1susey

All Gen Std DACs replaced with 2 LIGO-DACs. All AI-Chassis upgraded to SCSI daisy-chain.

Timing card replaced with one having latest firmware

New models [h1iopsusey, h1susetmy, h1sustmsy, h1susetmypi]

All models running RCG-5.5.2

h1susex

All Gen Std DACs and first-gen LIGO-DAC replaced with 2 LIGO-DACs. All AI-Chassis upgraded to SCSI daily-chain.

Timing card firmware upgraded in place

New models [h1iopsusex, h1susetmx, h1sustmsx, h1susetmxpi]

All models running RCG-5.5.2

h1iscex

18bit-DAC replaced with 20bit-DAC.

Timing card firmware upgraded in place

New models [h1iopiscex, h1calex, h1pemex]

All models running RCG-5.5.2

h1iscey

Timing card firmware upgraded in place

h1seiex, h1seiey

Timing card firmware upgraded in place

h1susauxex, h1susauxey

Timing card firmware upgraded in place.

All models running RCG-5.5.2 (first test of new RCG)

h1pemmx, h1pemmy

Timing card firmware upgraded in place.

 

Mega Cleanroom Powered On

At approx 8am the cleanroom above HAM 5/6/7 was powered on in prep for vent activities and initial cleanings. As to be expected, zone 4g temp has spiked. No action on behalf of FAC is required at this time.

T. Guidry K. Stewart N. Sanchez

IO Chassis Timing Card Firmware Versions

On Tuesday 02dec2025 Marc upgraded the firmware version of the IO Chassis timing cards at both end stations and mid stations. This upgrade was done "in place" using a laptop connected to the JTAG port on the timing card.

The current timing card firmware versions for H1 are:

All are running 0xfe0 except the following which are running 0x635

h1omc0: upgraded some time ago during O4 to move its DUOTONE frequencies from [960/961Hz] to [1920/1921Hz]

h1susey: upgraded Mon 01dec2025 when its timing card was replaced as part of the LIGO-DAC upgrade

Upgraded by Marc Tue 02dec2025:

h1iscex 
h1iscey
h1pemmx
h1pemmy
h1seiex
h1seiey
h1susauxex
h1susauxey
h1susey

h1iscex 20bit-DAC replacement

Jonathan, Jeff, Tony, Dave:

Following the upgrade of h1iscex 18bit-DAC upgrade to a 20bit-DAC Tuesday 02dec2025 we discovered the 20bit-DAC was not outputting any voltage.

Wed 03dec2025 Tony and I went to h1iscex to replace the suspect 20bit-DAC with the second DAC which was removed from the h1susex upgrade Tue02dec2025.

All while H1 was locked, we powered down h1iscex and its IO Chassis, pulled the chassis from the rack, took its lid off, replaced the 20bit-DAC and powered the chassis up while it was open.

I powered up the computer and auto started the models. H1 lost lock soon after the IOP model started running (possible Dolphin IPC minor break in transmission?).

We verified that the replacement 20bit-DAC was working, looking at both the DAC_chan7->ADC_chan30 DUOTONE inteface card loop back and h1calex drive of DAC_chan6.

We closed the chassis and pushed it back into place.

Suspect broken 20bit-DAC (removed)	190219-10
Replacement 20bit-DAC (installed)	150311-01 (S1700286)

Note that both of these 20bit-DACs came out of h1susex during its upgrade to LIGO-DACs Tue02dec2025. I'm not sure which one was driving the PI signals, and initially I chose the 2019 card for h1iscex over the older 2015 card.

Also note that the 2015 card has a LIGO S-number tag, the first I've seen on a GenStd IO card.

Thurs DAY Ops Transition

TITLE: 12/04 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Planned Engineering
OUTGOING OPERATOR: None
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 3mph Gusts, 1mph 3min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.33 μm/s 
QUICK SUMMARY:

H1 has been locked almost 15hrs....but H1 is going DOWN in prep for the HAM7 vent and will be set to IDLE. 

There will be more CDS (have heard Beckhoff) work, but no "proof of life" for H1 after that work will be done. 

Now the focus is on prep to vent with Kim & Nellie commencing shortly with thorough technical cleaning of HAM chambers.  Other possible items on the list for today:  

• Turn on Kobelco purge
• Forklifting furniture
• HAM7 vent prep
• Prep for HAM7 -Y door removal

instructions for recovering alignment after gate valves are excercised

It seems like it would be useful to have this recipie written in one place.  After the gate valves are opened and closed, we often have a hard time relocking because of alignment.  

Here's what we think we should do to recover after gate valves open.  

1. Do initial alignment without using green cameras in the loop, as the cameras are right next to the gate valves. 
   1. Start by using the baffle PD dither alignment script to set the TMS alignment: use the suspension guardians to misalign both ITMs and ETMs, run dither align script for both TMSs found on the INIT ALIGN MEDM, (dither align scripts, TMSX and TMSY, can run at the same time).  Open the ITM baffle PD scope and watch the script find the pointing of TMS that sets the beam to PD1, PD4, and finally centered on the optic.  When this finishes sucsesfully for both TMSs, there should be a dim green beam visible on both ALS cameras, if there is not check the alignment of PR3 (affects both arms) and the BS (only moves the Y arm beam).  This TMS alignment has to be set before you can move onto the next step, otherwise the ITM alignment won't be right. 
   2. Use the suspension guardians to align both ITMs, but leave the ETMs misaligned.  Run the dither alignment script for ITMX and ITMY, this time watching the ETM baffle PD scope to watch the script's progress.  Once this is finished, the ITMs should be set, and we should not move them based on the cameras. 
   3. Use the suspension guardians to set the ETMs to aligned, and you will hopefully see mulitple beams or cavity flashes on the ALS cameras.  You can manually align the ETMs to increase the flashes until they are good enough to lock the arm.  Use the ALS guardians (in auto mode might be helpful) to set them to ETM_TMS_WFS_OFFLOADED, this will not run the green camera servos, it runs green WFS to the ETM and TMS alignment only to increase build up. Don't run INITIAL_ALIGNMENT or INITIAL_ALIGNMENT offloaded. 
   4. Align the vertex to these ITMs: set the ALIGN_IFO guardian to manual, init it to control the suspension guardians.  Choose INPUT_ALIGN_OFFLOADED, once that finishes PRC_ALIGN_OFFLOADED, then MICH_BRIGHT_OFFLOADED, then SRC_ALIGN_OFFLOADED.  
2. Make two guardian changes before you start locking the full IFO
   1. Don't use the SRC1 + SRC2 ASC loops in DRMI, these don't seem to work until the alignment is set well for full lock.  You can turn these off by going to lscparams, and searching for use_DRMI_ASC, setting SRC1+ SRC2 to false.  Load ISC_DRMI for this change to take effect. 
   2. Don't shutter the ALS beams once they are no longer used for locking.  In ISC_LOCK, find the edge     ('IDLE_ALS',  'CARM_OFFSET_REDUCTION', 5), and reduce the weight which is normally 5 to 1, so that this path which avoids shuttering the ALS becomes the default path.  Load ISC_LOCK for this change to take effect.
3. Start to relock as usual, going through PRMI.  Stop at CHECK_DRMI_ASC, where all the DRMI ASC loops will be running other than SRC1+SRC2, and manually adjust SRM sliders to reduce POP90 and increase POP18. 
4. You can try to lock all the way to ENGAGE_ASC for full IFO, but this may not work.  If it does not, you may need to manually align DHARD, CHARD, and SRM. 
   1. If you are adjsuting alignments by hand, you want to keep the green beams locked to the arms as the alignment changes.  To get the green alignment to follow any changes you make, stop at IDLE_ALS and turn on the scripts that servo the green TMS QPD offsets based on the green WFS.  You can access this script by going to either ALS end station overview and clicking the button that says QPD OFFSETs, then using the pink drop down button to start the servo for each arm, and open the ndscope that lets you watch the progress of these servos (screenshot)
   2. If the power in the two arms is mismatched (visible on the PRMI SB4 min ndscope on the front wall), this could be a sign that DHARD is misaligned. 
      1. If you notice this in a state before DHARD WFS are closed, RF_DARM or CARM_OFFSET reduction , you can use the move_arm_dev script to step DHARD pitch and yaw to bring the arms closer to each other.  userapps/asc/h1/scripts/move_arm_dev.py DH P 0.01, or similar.  
      2. If you notice the arm powers diverging from each other after DHARD WFS are closed, this might be fixed by moving SRM while watching to make sure POP 90 stays low.  
   3. If the power recycling gain is low, you may need to adjust CHARD.  You can step through CARM_OFFSET reduction or stop after it finishes and use 62110 to check if the refl power is too low for the arm transmission that you are at, and try to move CHARD while trying to increase the refl power (watch build_up_slow ndscope linked from the ISC guardians overview and other places).  
5. Once you are all the way locked (congratulations), stop at PREP_ASC_FOR_FULL_IFO before you try turning on ASC.  It may be a good idea to note the green QPD offsets with a screenshot, if you loose lock they wil be reverted with SDF revert but you would need to reset them to these values to relock.   Also look at the ITM green camera offsets, from each ALS end station overview in the green WFS section there is a button that says CAM ITM YAW or PIT, you can take a screenshot of this screen for both ITM cameras and compare the current location of the beam centroid to the setting (screenshot)
6. Use a guardian shell (guardian -i ISC_LOCK) to copy and paste lines from ENGAGE_ASC_FOR_FULL_IFO manually, skipping SRC1 +SRC2.  (you might grab another screenshot of the green QPDs and cameras at this point)
7. Then you can manual the guardian to TMS_SERVO, engage_soft_loops, and DARM offset, and take another green QPD offset + camera screenshot.  
8. Having the DARM offset on as well as the soft loops seems to help the SRC ASC work, so try engaging those once those steps are done.  (Congratulations on getting all the ASC controlled)
9. Important step: reset the green references.  In the past we have reset the green QPD offsets by accepting them in SDF as well as the camera offsets, but it doesn't make sense to be changing the QPD offsets as they shouldn't be impacted by the gate valve moving the camera.  Today once we had all the ASC engaged, we stepped the QPD offsets back to their original settings from SDF, watching the arm transmissions in the ndscope.  The arm transmissions seemed fine with the original QPD offsets, so we left those there and only updated the green camera offsets.  I think this should be the way we do things in the future, not reseting the QPD offsets by default.  
10. undo the gaurdian changes from step 2 above.  After the next lockloss, run initial alignment as usual, letting the green cameras be used to point the ITMs.  DRMI ASC should work now that the alignment has been set.  

 

Touched SQZR Settings & Sliders

TITLE: 12/04 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Planned Engineering & LOCKED
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 10mph Gusts, 6mph 3min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.28 μm/s 
QUICK SUMMARY:

SQZ_OPO_LR was giving us this error: 
"CLF frequency out of range, check CLF CMB".
I do not recall how this was resolved by Daniel while he was checking out the CMB. 

Once that was resolved we got the classic: "pump fiber rej power in ham7 high, nominal 35e-3, align fiber pol on sqzt0."  issue.
The Quarter & Half wave plates on SQZT0 were touched up to minimize H1:SQZ-SHG-FIBR_REJECTED_DC_POWER_MON. 
OPO was locked. Great!

Then the FC wasn't locking: 
So FC2 was moved from its nominal location. only after I accidentally moved ZM1 and reverted it.
Fumbled with the FC2 sliders for a while, then used the clear history button.
Fumbled around some more with the FC2 sliders. 
Locked the the FC with FC2 P @ 246.1 Y @ 51.1

And now the SQZ has been SQuoZe.

Forcast for tonight's wind looks good for a stable lock.

Locking Summary

[Sheila, Jeff, Elenna, Corey]

Today we relocked with some issues. We think that the soft close of the gate valves shook the ITM green cameras enough that the green references were no longer good, which caused many alignment problems through carm offset reduction.

The SRC ASC is OFF in DRMI ASC at this time, by having the use_DRMI_ASC['SRC1'/'SRC2'] flag set to False. When locking, SRM and SR2 might need to be moved by hand before offloading DRMI ASC.

Sheila will write a more detailed alog about the process of updating the green references, because we think we have a better method now. For the summary, we have updated the green camera references, SDFed them in safe, and run an initial alignment after the lockloss.

Other guardian changes:

• DARM OFFSET now occurs after DRMI TO POP and befor PREP ASC FOR FULL IFO. This is how it used to be before the OFI problems. We had some problems with SRC1 P during full IFO engagement, but the problems seemed to go away as soon as we went through DARM OFFSET
• CLOSE BEAM DIVERTERS is now out of the ladder. While we troubleshoot, we want these open, and to avoid lockloss they are closed at 2 W. I removed them from the ladder completely for now.

Tony, Oli and I have been watching violin mode damping. So far, there doesn't seem to be any problems.

By eye, Jeff and I think the PCAL X crosses do not match the line height in CAL DELTA L, so there may be some calibration mismatch. I will run a cal measurement when we are thermalized.

Squeezer is not working, Daniel and others are trying to troubleshoot now.

Wed DAY Ops Summary

TITLE: 12/03 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Planned Engineering
INCOMING OPERATOR: None
SHIFT SUMMARY:

Main goal for today was to continue to see troubleshoot H1 to see if it survived the CDS work from earlier this week (by trying to get H1 back to NLN)---Elenna and Sheila worked on this.  There continued to be issues with ASC SRC---Elenna said this is common for H1 recoveries.  Sheila also mentioned that the alignment issue most likely is a result of Gate Valve closures affecting the Green Camera alignment.

Recovery work above had to wait until about 11amPDT due to remaining LVEA crane inspections.  

During Recovery work 20-bit DAC card was swapped out for one which was installed yesterday that had issues.
LOG:

• 1532-1605 Initial Alignment
• 1541 Crane inspection begins in South Bay [randy & CraneTech contractor]----> done in LVEA
  • After lunch will do Staging and then FCES
• 1606-1613 Restart all of the models on h1iscex (this is for the the issue with the PCal for EX).  [dave]
• 1640-1730 EY, EX trips to check on Dustmon Pumps (tj)
• 1659-1706 lvea walkthru (richard)
• 1707-1714 meet w/ crane inspector (tyler)
• 1729-1901 Moving magnetometer around HAM1 (rene, alicia)
• 1740-1755 Swapping in new Corner Station Dust Monitor pump & then checking flow  (tj)
• 1756-1808 Testing out new dust monitors in the diode room (ryanC)
• 1803-1854 Power supply measurements at EX related to PCal & new DAC (tony)
• 1819-1840 Looking for picomotor mount assys in optics lab (jennie)
• 1839-1852 Parts hunt in West Bay (mitch)
• 1852-1857 LVEA walkthru (sheila)
• 1850 Restart locking H1 
  • darm to rf -> idle als -> dhard wfs
  • Thought is GV soft closes affect green camera alignment so this causes recovery grief.
• 1941-2111 LVEA Tour for Michael Turner (cosmologist) & Dennis Overbye (NYT reporter) which was in CR and now moves to LVEA (mike, jenne)
  • 2127-2156 Driving to EY
  • 2206-2218 Drive to Overpass & then Back to LExC
• 2040-2139 Gantry crane inspections at Mechanical Room & FCES.  This COMPLETES crane inspection. (randy, cranetech)
• 2140-2348 OPO testing + laser Hazard in the optics lab (KarMeng)
• 2145-2157 magnetometer tests in CER (alicia,rene)
• 2158 CHETA table work in the CHETA/JAC Lab (matt)
• 2200-2207 Hi-Bay trip (ibrahim)
• 2230-2308 h1iscex being shut down for the to investigate issues with 20-bit DAC (dave, tony)
• 2250 JAC table work in CHETA/JAC Lab (jennie)
  • 2342 Looking for JAC parts in the LVEA (while we are locking H1). [jennie]
• 2310 Initial Alignment #2 of the day (post EX 20bit DAC swap)
•