We have had multiple locklosses right near the end of the locking sequence around CLOSE BEAM DIVERTERS and OMC WHITENING. Today, we held at different states while locking. First, we pasued at MAX POWER while we waited for the R waves of a few different earthquakes to pass. Then, we proceeded successfully through to LASER NOISE SUPPRESSION. There are large glitches in this state, probably because of the CARM loop, so we stayed in the state to wait for the glitches to die down. Then, Corey selected ADS TO CAMERAS. This state proceeded fine, but we saw the 1 Hz starting to ring up. Corey selected the ASC Hi Gain script button on the seismic page to damp the 1 Hz back down. It damped down. We next went to INJECT SQUEEZING, which was fine. We waiting again about a minute after the state finished. Finally, Corey selected CLOSE BEAM DIVERTERS and we lost lock. Here is the guardian log so we can see the timestamps of each step.
We had the same lockloss earlier, it's logged at being from "omc whitening" but usually the beam diverters state is very fast so we are in omc whitening by the time the lockloss happens. Attached is the suspension signals from both locklosses. It looks like something rings up in DARM.
2025-11-05_00:36:48.787114Z ISC_LOCK [LASER_NOISE_SUPPRESSION.run] ezca: H1:IMC-REFL_SERVO_IN2GAIN => -29
2025-11-05_00:36:48.787719Z ISC_LOCK [LASER_NOISE_SUPPRESSION.run] ezca: H1:IMC-REFL_SERVO_FASTGAIN => -16
2025-11-05_00:37:24.445085Z ISC_LOCK REQUEST: ADS_TO_CAMERAS
2025-11-05_00:37:24.445085Z ISC_LOCK calculating path: LASER_NOISE_SUPPRESSION->ADS_TO_CAMERAS
2025-11-05_00:37:24.446434Z ISC_LOCK new target: ADS_TO_CAMERAS
2025-11-05_00:37:24.501243Z ISC_LOCK EDGE: LASER_NOISE_SUPPRESSION->ADS_TO_CAMERAS
2025-11-05_00:37:24.501532Z ISC_LOCK calculating path: ADS_TO_CAMERAS->ADS_TO_CAMERAS
2025-11-05_00:37:24.505287Z ISC_LOCK executing state: ADS_TO_CAMERAS (578)
2025-11-05_00:37:24.506308Z ISC_LOCK [ADS_TO_CAMERAS.enter]
2025-11-05_00:37:24.532550Z ISC_LOCK [ADS_TO_CAMERAS.main] ezca: H1:GRD-CAMERA_SERVO_REQUEST => CAMERA_SERVO_ON
2025-11-05_00:37:24.533290Z ISC_LOCK [ADS_TO_CAMERAS.main] camera servo guardian state
2025-11-05_00:37:24.533992Z ISC_LOCK [ADS_TO_CAMERAS.main] ezca: H1:ASC-AS_A_RF36_Q_YAW_SW1 => 8
2025-11-05_00:37:24.785056Z ISC_LOCK [ADS_TO_CAMERAS.main] ezca: H1:ASC-AS_A_RF36_Q_YAW => ON: OFFSET
2025-11-05_00:38:24.227696Z ISC_LOCK [ADS_TO_CAMERAS.run] ezca: H1:SUS-ETMX_L2_DRIVEALIGN_P2L_SPOT_GAIN => 3.13
2025-11-05_00:38:24.228345Z ISC_LOCK [ADS_TO_CAMERAS.run] ezca: H1:SUS-ETMY_L2_DRIVEALIGN_P2L_SPOT_GAIN => 4.89
2025-11-05_00:38:24.230162Z ISC_LOCK [ADS_TO_CAMERAS.run] ezca: H1:SUS-ETMX_L2_DRIVEALIGN_Y2L_SPOT_GAIN => 4.85
2025-11-05_00:38:24.230577Z ISC_LOCK [ADS_TO_CAMERAS.run] ezca: H1:SUS-ETMY_L2_DRIVEALIGN_Y2L_SPOT_GAIN => 0.78
2025-11-05_00:39:40.380293Z ISC_LOCK REQUEST: INJECT_SQUEEZING
2025-11-05_00:39:40.383415Z ISC_LOCK calculating path: ADS_TO_CAMERAS->INJECT_SQUEEZING
2025-11-05_00:39:40.386240Z ISC_LOCK new target: INJECT_SQUEEZING
2025-11-05_00:39:40.448133Z ISC_LOCK EDGE: ADS_TO_CAMERAS->INJECT_SQUEEZING
2025-11-05_00:39:40.448133Z ISC_LOCK calculating path: INJECT_SQUEEZING->INJECT_SQUEEZING
2025-11-05_00:39:40.453186Z ISC_LOCK executing state: INJECT_SQUEEZING (580)
2025-11-05_00:39:40.453467Z ISC_LOCK [INJECT_SQUEEZING.enter]
2025-11-05_00:39:40.471153Z ISC_LOCK [INJECT_SQUEEZING.main] ezca: H1:GRD-SQZ_MANAGER_REQUEST => FREQ_DEP_SQZ
2025-11-05_00:39:40.472228Z ISC_LOCK [INJECT_SQUEEZING.main] timer['CheckSqz'] = 90
2025-11-05_00:39:40.473316Z ISC_LOCK [INJECT_SQUEEZING.main] ezca: H1:GRD-SUS_PI_REQUEST => PI_DAMPING
2025-11-05_00:39:40.720211Z ISC_LOCK [INJECT_SQUEEZING.run] USERMSG 0: SQZ_MANAGER: has notification
2025-11-05_00:39:45.281174Z ISC_LOCK [INJECT_SQUEEZING.run] USERMSG 0: SQZ_MANAGER: has notification
2025-11-05_00:40:51.377382Z ISC_LOCK REQUEST: CLOSE_BEAM_DIVERTERS
2025-11-05_00:40:51.378087Z ISC_LOCK calculating path: INJECT_SQUEEZING->CLOSE_BEAM_DIVERTERS
2025-11-05_00:40:51.378087Z ISC_LOCK new target: CLOSE_BEAM_DIVERTERS
2025-11-05_00:40:51.459548Z ISC_LOCK EDGE: INJECT_SQUEEZING->CLOSE_BEAM_DIVERTERS
2025-11-05_00:40:51.459721Z ISC_LOCK calculating path: CLOSE_BEAM_DIVERTERS->CLOSE_BEAM_DIVERTERS
2025-11-05_00:40:51.464167Z ISC_LOCK executing state: CLOSE_BEAM_DIVERTERS (590)
2025-11-05_00:40:51.465760Z ISC_LOCK [CLOSE_BEAM_DIVERTERS.enter]
2025-11-05_00:40:51.475467Z ISC_LOCK [CLOSE_BEAM_DIVERTERS.main] Closing REFL beam diverter
2025-11-05_00:40:51.476845Z ISC_LOCK [CLOSE_BEAM_DIVERTERS.main] ezca: H1:SYS-MOTION_C_BDIV_A_CLOSE => 1
2025-11-05_00:40:51.476939Z ISC_LOCK [CLOSE_BEAM_DIVERTERS.main] Closing AS beam diverter
2025-11-05_00:40:51.478119Z ISC_LOCK [CLOSE_BEAM_DIVERTERS.main] ezca: H1:SYS-MOTION_C_BDIV_D_CLOSE => 1
2025-11-05_00:40:51.478494Z ISC_LOCK [CLOSE_BEAM_DIVERTERS.main] ezca: H1:VID-CAM16_EXP_REQ => 36000
2025-11-05_00:40:51.579758Z ISC_LOCK [CLOSE_BEAM_DIVERTERS.main] ezca: H1:VID-CAM16_EXP_SET => 1
2025-11-05_00:40:51.579758Z ISC_LOCK [CLOSE_BEAM_DIVERTERS.main] Closing POP beam diverter
2025-11-05_00:40:51.580545Z ISC_LOCK [CLOSE_BEAM_DIVERTERS.main] ezca: H1:SYS-MOTION_C_BDIV_B_CLOSE => 1
2025-11-05_00:40:53.156310Z ISC_LOCK [CLOSE_BEAM_DIVERTERS.run] Unstalling IMC_LOCK
2025-11-05_00:40:53.347649Z ISC_LOCK [CLOSE_BEAM_DIVERTERS.run] Unstalling VIOLIN_DAMPING
2025-11-05_00:40:53.521338Z ISC_LOCK JUMP target: LOCKLOSS
2025-11-05_00:40:53.521845Z ISC_LOCK [CLOSE_BEAM_DIVERTERS.exit]
2025-11-05_00:40:53.582367Z ISC_LOCK JUMP: CLOSE_BEAM_DIVERTERS->LOCKLOSS
2025-11-05_00:40:53.582367Z ISC_LOCK calculating path: LOCKLOSS->CLOSE_BEAM_DIVERTERS
2025-11-05_00:40:53.585523Z ISC_LOCK new target: DOWN
2025-11-05_00:40:53.590140Z ISC_LOCK executing state: LOCKLOSS (2)
I double checked that the squeezer wasn't doing anything (just in case waiting longer than 1 min might help disentangle things). The SQZ_MANAGER was hanging out in state 69, where it asks the SQZ_ANG_ADJUST guardian to go to 'ADJUST_SQZ_ANG_ADF'. On the way to that state, the last thing the sqz manager had done was tell the SQZ_ANG_ADJUST guardian to go to WAIT_SQZ_AND_ADF, which just sits and waits for ISC_LOCK to get to NomLowNoise. So, I think I've convinced myself the lockloss is nothing to do with the squeezer.
I plotted several channels (starting from the lockloss 'scope). I'm still not really sure why the test mass L2 channels get a bit fuzzy about one second before the beam diverters close, but I think it's a red herring anyway. The real thing that seems to be interesting is that the HAM1 GS13s see a kick (due to the beam diverter moving) right when DARM sees a kick (the kick that Elenna pointed out).
So, I agree with Elenna and Corey's plan for now, to do the last few locking states slowly, then doing CLOSE_BEAM_DIVERTERS by hand to figure out which beam diverter is causing the problem. I also think it could be interesting to trend the GS13s for HAMs 1 and 6 for other locks (the other one tonight that we lost lock, as well as a few that were successful), to see if the kicks have gotten bigger or if they are the same size as always.
I also think that if closing the beam diverters causes another lockloss, we should just manual past the CLOSE_BEAM_DIVERTERS state, run for the night with them open, and tag DetChar. I'm pretty sure we've shown that they no longer really matter if they are open or closed, so I think we'd rather try to have some data with the beam diverters open than none at all.
