Sat Oct 25 10:08:58 2025 INFO: Fill completed in 8min 55secs

TITLE: 10/25 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
OUTGOING OPERATOR: Tony
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 9mph Gusts, 6mph 3min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.44 μm/s 
QUICK SUMMARY: H1 was down overnight due to high winds and microseism coupled wiht ongoing locking troubles. Will start with a fresh initial alignment then see how far in the locking sequence H1 can get this morning.

• Wind low, microseism slowly dropping and just below 90th percentile
• All other systems nominal

TITLE: 10/25 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Corrective Maintenance // Environment
INCOMING OPERATOR: Tony
SHIFT SUMMARY: The high winds and microseism made locking pretty much impossible after the 03:01 UTC lockloss.

• The next time we make it through the ESD transitions we want to run this measurement: "diaggui sheila.dwyer/LSC/DARM/DARM_model/PUM_crossover_2024.xml" (OPSINFO)

LOG: No log.

• 01:28 UTC Lockloss from Bounce mode EX/EY (Maybe, the monitors aren't super trustworthy) 12 seconds after we reached NLN, They started to ringup when we passed through INCREASE_DARM_OFFSET. I tried to damp one of the modes, EY using a gain of 1 which I saw in an old comment in ISC_LOCK, but it did not help and in fact might have made them increase faster and angered ITMY so now 3 of them were ringing up by the monitors. The Roll modes were fine during this. Screenshot of the B&R modes during the ringup, a look at the quads M2 WIT chans during this
• 02:06 UTC I put the gain of the Roll mode back to zero in ISC_LOCK and reloaded it so it won't go in automatically
• 02:40 UTC The winds picked up drastically, from around 10 mph to over 40mph, windy.com predicts they'll calm down around 11/12pm PST 06/07 UTC
  • The wind is increasing ground motion in the 0.03-0.1 band on all axes but in the 0.3 - 3.0 its only really increased in the EY Yaxis
• 03:01 UTC  lockloss during LOWNOISE_ESD_ETMX, likely from 30-40mph gusts while we were in this sensitive state
• 04:57 UTC Holding in IDLE

Summary:

We aligned everything such that none of 8 PDs was excellent but all were OK (we were also able to set up such that 4 pds were excellent but a few were terrible but decided not to take that), we were preparing for putting the array in storage until the installation, only to find that something is wrong with the design of the asymmetric QPD clamp D1300963-V2. It's unusable as is.

QPD clamp doesn't constrain the position of the QPD laterally, and there's a gross mismatch between the position of properly aligned QPD and that of the center hole of the QPD clamp. Because of that, when QPD is properly positioned, one of the QPD pins will touch the QPD clamp and be grounded unless the QPD connector is fixed such a way to pull the QPD pins sideways. Fortunately but sadly, the old non-tilt QPD clamp D1300963-V1 works better, so we'll use that.

Another minor issue, is that there seems to be a confusion as to the direction of the QPD tilt in terms of the word "pitch" and "yaw". The way the QPD is tilted in D1101059-v5 (this is how things are set up in the lab as of now) doesn't seem to follow the design intent of ECR E1400231 though it follows the word of it. After confirming that this is the case with systems, we'll change the QPD tilt direction (or not). This means that we're not ready to put everything in storage quite yet.  

None of these affect the PD array alignment we've done, this is just a problem of the QPD.

Pin grounding issue due to the QPD clamp design.

I loosened the screws for the QPD connector clamps (circled in blue in the first attachment) and the output of the QPD preamp got crazy with super large 60Hz noise and large DC SUM even though there was no laser light.

I disconnected the QPD connector, removed the connector clamps too, and found that one pin of the QPD was short circuited to the ground via the QPD clamp (not to be confused with the QPC connector clamps, see 2nd attachment).

Turns out, the offending pin was isolated during our adjustments all the time because the QPD connector clamps were putting enough lateral pressure as well as down such that the pins were slightly bent from the offending side. I was able to reattach the connector, push it laterally while tightening the clamp screws, and confirm that the QPD functioned fine. But this is not really where we wanted to be.

I rotated the QPD clamp 180 degrees (which turns out to make more sense judging from the drawings in the first attachment), which moved the QPD. Since the beam radius is about 0.2mm, if the QPD moves by 0.2mm it's not useful as a reference of the in-lab beam position. I turned the laser on, repositioned the QPD back to where it should be, but the pin on the opposite side started touching. (Sorry no picture.)

I put the old non-tilt version clamp and it was much, much better (attachment 3). It's annoying because the screw holes don't have an angled recess. The screw head is tilted relative to the mating surface on the clamp, contacting at a single point, and tightening/loosening the screw tend to move the QPD. But it's possible to carefully tighten one screw a bit, then the other one a bit, repeat that dozen times or so until nothing moves even when pushed firmly by finger. After that, you can still move the QPD by tiny amounts by tapping the QPD assy by bigger Allen key. Then tighten again.

What's going on here?

In the 4th attachment, you can see that the "center" hole of the QPD clamp is offset by 0.55" (1.4mm) in the direction orthogonal to A-A, and about 0.07" (even though this number is not specified anywhere in the drawing) or 1.8mm in A-A direction. So the total lateral offset is sqrt(1.4^2+1.8^2)~2.3mm. OTOH, the QPD assy is only 0.5" thick, so the lateral shift arising from the 1.41deg tilt at the back of the QPD assy is just 1.41/180*pi*0.5=0.0123" or 0.3mm.

Given that the beam position relative to the array structure is determined by the array itself and not by how the QPD is mounted, 2.3mm lateral shift is impossibly large, something must be wrong in the design. The 5th attachment is a visual aid for you.

Anyway, we'll use the old clamp, it's not worth designing and manufacturing new ones at this point.

QPD tilt direction.

If you go back to the first attachment, the QPD is tilted in a direction indicated by a red "tilt" arrow in the lab as we just followed the drawing.

The ECR E1400231 says "We have to tilt the QPD 1 deg in tip (pitch) and 1 deg in tilt (yaw)" and it sounds as if it colloborates with the drawing.

However, I suspect that "pitch" and "yaw" in the above sentence might have been misused. In the right figure of the 6th attachment (screeshot of ECR unedited), it seems that the QPD reflection hits the elevator (the red 45 degree thing in the figure) at around 6 O'clock position around the eliptic exit hole, which means that the QPD is tilted in its optical PIT. If it's really tilted 1 degree in optical PIT and 1 degree in optical YAW, the reflection will hit something like 7:30 position instead of 6:00.

That makes sense as the design intent of the ECR is to make sure that the QPD reflection will not go back into the exit hole. The 7th attachment is a side view I made, red lines represent the IR beams, yellow lines the internal hole(s) in the elevator, and green lines the aperture of the two eliptical exit holes. Nothing is to scale, but hopefully you agree that, in order to steer the QPD reflection outside of the exit hole aperture, PIT UP requires the largest tilt and PIT DOWN requires the least tilt. We have a fixed tilt of QPD, so it's best to PIT DOWN, that's what I read from the ECR. If you don't know which angle is bigger or smaller, see attachment 8.

Anyway, I'll ask Callum if my interpretation is correct, and will act accordingly.

Using this Summary website I was able to grab all times that the ISC_LOCK Guardian was in Acquire_DRMI_1F [101] since the beginning of the  O4 run.
Along with it comes what ISC_LOCK State it came From and what State it went To. 
Pecilar Issues with the website data: The Local and UTC time is the same for some reason which i thought was strange, but the GPS times are fine.

I was able to use the data to make a spreadsheet which I color coded for your TechnoColor viewing pleasure! 
Green is a successful LOCK of DRMI that took us to the next locking state [102].
Red..... is a lockloss, boo!
Orange indicates that ISC_LOCK went from trying to DRMI to trying to PRMI.
 
I then took the duration that ISC_Lock took in Acquire_DRMI_1F and pasted the duration into a different cell depending on if the lock was successful or went to PRMI or Lockloss or another state.

That allowed me to make more Histograms of DRMI data!  
AcqDRMI1f-DRMILocked_Hist.png is a histogram of all the durations that DRMI has successfully locked since the beggining of O4. 
DRMI1F-PRMI.png depicts the durations of all the times DRMI failed to lock and got kicked down to PRMI.
DRMI-Lockloss.png shows us all the times we locklossed from Acquire_DRMI_1F.

Also I used Statecounter to try and break up the O4 run into ~month long sections that are 30 days long starting on the first day of the run.
Month with the lowest Average time spent Locking DRMI: October 20th - Nov 19th 2023 @ 1.439 minutes  
Month with the Highest Average time spent locking DRMI : July 2024  at 13.750 minutes (context we only locked twice in this time frame)
Month with the 2nd Highest Average time spent locking DRMI : October 2025 with 8.351 minutes. 

If the metric we want to use is what percent of the DRMI Locking states lasted longer than 5 minutes to determine the WORST DRMI locking era. That award would go to August 10th -Sep 9th 2025 with a stunning 61% of attempts to lock DRMI were above 5 minutes!
While at the beggining of the run we spent 4 months in a row never exceeding 5 minutes in DRMI. It does seem like it got progressively worse. 
 

TITLE: 10/24 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
INCOMING OPERATOR: Ryan C
SHIFT SUMMARY: Busy day of continuous troubleshooting of H1. Things have been learned and progress has been made, but currently still waiting to relock after discovering an incorrect whitening setting and swapping a sat amp to see if those help with some of our instabilities. We've had some locklosses at DRMI_3F, but waiting for DRMI ASC to fully converge on this most recent try seemed to work. Currently relocking up to MOVE_SPOTS.

• 14:35 - Starting initial alignment
• 15:06 - Starting lock acquisition
  • Applied roll mode damping (G = -200) during LOWNOISE_LENGTH_CONTROL
  • Waited for 30 minutes to watch for roll mode ringups, saw none
  • Went through LOWNOISE_ASC once DHARD_P lownoise steps commented out (alog87712)
  • Saw 1 HZ ASC oscillation once at NLN, transitioned to high gain ASC configuration
• 17:03 - Lockloss after ~1:15 at max power with 5.8 Hz DARM "pulse" behavior (alog87717)
• Holding down for sat amp swap (alog87722)
• 17:38 - Starting lock acquisition
  • Applied roll mode damping (G = -200) during LOWNOISE_LENGTH_CONTROL
  • Skipped over LOWNOISE_ASC entirely
• 20:59 - Lockloss similar to before with ~1:15 at max power and 5.8 Hz DARM "pulse"
• Discovered incorrect POP A RF45 whitening setting, fixed (alog87728)
• 21:30 - Starting initial alignment
• 21:55 - Starting lock acquisition

LOG:

My first SDF post today, which was based on getting archive files from subversion commits, doesn't appear to be complete and in addition may have some false positives.

I have generated new lists, this time getting the archived OBSERVE.snap files from the CDS backups of userapps by the file server.

In the directory /ligo/home/david.barker/tuesdaymaintenance/24oct2025/sdf/observe there are:

mon20oct2025/ Directory of backups as of 00:39 early monday morning

wed22oct2025/ Directory of backups as of 00:39 early wednesday morning

thu23oct2025/ Directory of snaps as of 00:39 ealy thursday morning

mon-wed_diffs.txt Text file of settings changed between the monday and wednesday backups

mon-thu_diffs.txt Text file of settings changed between the monday and thursday backups

snap_diffs.bsh Shell script to generate the diffs files

snapfiles Text file of the userapps "OBSERVE.snap" files as symbolically linked from the target directories

TITLE: 10/24 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
OUTGOING OPERATOR: Ryan S
CURRENT ENVIRONMENT:
    SEI_ENV state: USEISM
    Wind: 9mph Gusts, 6mph 3min avg
    Primary useism: 0.04 μm/s
    Secondary useism: 0.53 μm/s 
QUICK SUMMARY:

• We've had 2 ~1 hour locks today, 2 locklosses so far
• Locking itself has been straightfoward, for the 3rd lock attempt we had to run the 2nd IA of the day
• Microseism seems steady around the 90th percentile
• Wind is low

We (Sheila, Jennie, Ryan, Oli) have found that the POP A RF45 antiwhitening gain was set incorrectly. We tracked this problem down to the SDF being saved incorrectly Monday night along with with various dark offsets (87598). The antiwhitening gain should be -15 dB but was set to -21 dB. Additionally, the new dark offsets were doubled. We think this problem explains why the MICH gain needed to be doubled this week (87645).

I reset the safe SDF to have the correct LSC POP antiwhitening filters, and reverted the dark offsets as well.

Extra kudos to Jennie for scrolling through Dave's SDF difference file (400 settings!) to find this mismatch, 87718.

Sheila edited the guardian to remove the MICH gain doubling.

Marc, Oli

In 87713 it's noted that some of the strange behavior we've been seeing with the locks seem to be related to the ETMY L2 stage. The ETMY L2 satamp was swapped out on Oct 14th from S1100137 to S1100127 as part of the ECR E2400330 upgrade (87469). So, even though we were seemingly doing fine with this ETMY L2 satamp, we decided to swap it out this morning to see if that is the cause of our issues. Coincidentally, the spare satamp that we have available that was modified for ECR E2400330 and that Jeff had characterized was the satamp that had come from ETMY L2, S1100137. So we decided to put that one back in - it was fine before, although it obviously has now been modified for the satamp upgrade. So it's not been swapped back to exactly what it was before, but this is the closest we can get!

Marc and I took S1100137 down to EY and swapped out S1100127 with S1100137. I edited the fitresults file for S1100137 to work for ETMY L2 and replaced the previous OSEMINF 5.3:0.1 compensation filters to be the best possible compensation filters for this new satamp. 

Updating compensation filters
$ py satampswap_bestpossible_filterupdate_ECR_E2400330.py -o ETMY_L2
All updated filters grabbed for ETMY
ETMY L2 UL compensation filter updated to zpk([5.25],[0.0959],1,"n")
ETMY L2 LL compensation filter updated to zpk([5.21],[0.0953],1,"n")
ETMY L2 UR compensation filter updated to zpk([5.18],[0.0949],1,"n")
ETMY L2 LR compensation filter updated to zpk([5.21],[0.0955],1,"n")
write /opt/rtcds/userapps/release/sus/h1/filterfiles/H1SUSETMY.txt
Done writing updated filters for ETMY

Swap timeline

Satamp characterization data

Here's the characterization data and fit results for S1100137, assigned to ETMY L2's ULLLURLR OSEMs.

This sat amp is a UK 4CH sat amp, D0900900 / D0901284. The data was taken per methods described in T080062-v3, using the diagrammatic setup shown on PAGE 1 of the Measurement Diagrams from LHO:86807.

The data was processed and fit using ${SusSVN}/trunk/electronicstesting/lho_electronics_testing/satamp/ECR_E2400330/Scripts/
plotresponse_S1100137_ETMY_L2_20251020.m

Explicitly, the fit to the whitening stage zero and pole, the transimpedance feedback resistor, and foton design string are:

The attached plot and machine readable .txt file version of the above table are also found in ${SusSVN}/trunk/electronicstesting/lho_electronics_testing/satamp/ECR_E2400330/Results/
2025-10-20_UKSatAmp_S1100137_D0901284-v5_fitresults.txt

Per usual, R_TIA_kOhm is not used in the compensation filter -- but after ruling out an adjustment in the zero frequency (by zeroing the phase residual at the lowest few frequency points), Jeff nudged the transimpedance a bit to get the magnitude scale within the ~0.25%, shown in the attached results. Any scaling like this will be accounted for instead with the absolute calibration step, i.e. Side Quest 4 from G2501621, a la what was done for PR3 and SR3 top masses in LHO:86222 and LHO:84531 respectively.

Fri Oct 24 10:10:02 2025 INFO: Fill completed in 9min 58secs

To continue with the "what has changed since Monday" I have created the SDF reports. For now I've just concentrated on the OBSERVE.snap settings changes.

Models with changed settings over this time period: h1alsex  h1alsey  h1asc  h1ascimc  h1iscex  h1iscey  h1isietmy  h1lsc  h1omc  h1suspr3  h1sussr3

There are approximately 400 settings changes.

Full details available in /ligo/home/david.barker/tuesdaymaintenance/23oct2025/sdf/observe

We had a lockloss at 17:03 UTC that has the same "pulse" behavior noted by Jennie and Oli yesterday. This pulses can be seen in the DARM loop, on the L3 master out, and in the EX and EY L2 master outs. These pulses are about 5.8 Hz in frequency.

I'm going to use this post to keep running notes of the 1 Hz investigations:

• After being at high power for more than 30 minutes, we went to low noise and left DHARD P in its high bandwidth configuration.
• After some minutes, the 1 Hz oscillation appeared in the ASC
• Indication that DHARD P is NOT the only culprit here
• I averted the lockloss by running the High Gain ASC script
• after the script completed I reengaed the LSC FF since that is not the issue here
• we had been at high power long enough that the CSOFT P gain was dropped to 20, and I can't change it because the thermalization guardian won't let me
• I measured DHARD P around 1 Hz with all ASC in high bandwidth
• DHARD P looks very stable at 1Hz, the phase is fine and there is no UGF at this frequency
• at 9:54 local I transitioned all the YAW arm loops to their low bandwidth state
• LOCKLOSS at 10:03- no oscillations, no saturation warnings, just a "fast" lockloss

DHARD P at 1 Hz attached.

We have been getting saturation warnings on EY after going through the lownoise coil drivers state. We finally tracked it down to the R0 F2 and F3 drivers. The reaction chain length drive (aka reaction chain tracking) is causing these saturations. This servo is run using the L2 OSEMs as a witness to then drive the R0 chain to follow the main chain. I turned off the length drive and the saturations stopped. When I turn it back on it continues to saturate.

The only recent change I can think of is the L2 satamp swap on all the test masses.

TITLE: 10/24 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
OUTGOING OPERATOR: None
CURRENT ENVIRONMENT:
    SEI_ENV state: USEISM
    Wind: 8mph Gusts, 6mph 3min avg
    Primary useism: 0.04 μm/s
    Secondary useism: 0.62 μm/s 
QUICK SUMMARY: H1 was down overnight due to ongoing locking issues. I'll start by running through an initial alignment while I get caught up on events of the past couple days.

• Wind low, microseism steady around 90th percentile
• DR1 dust monitor reports no change in counts
• All other systems nominal

WP 12836
ECR E2400330
Modified List T2500232

The following SUS SAT Amps were upgraded per ECR E2400330. Modification improves the whitening stage to reduce ADC noise from 0.05 to 10 Hz.

F. Clara, J. Kissel, O. Patane