Got called at 10:57PM (PT) and IFO was actually locked in NLN with one SDF Diff, which I accepted and is screenshot is below (IMC_DOF_1_P TRAMP from 0 to 3 and IMC_DOF_2_Y TRAMP from 0 to 3).

Good timing for this since the 1Hz ringup was happening. I saw earlier today that turning on the ASC High Gain might help with this so I turned it on and it seems to be working.

SQZ also looked terrible inthe high-frequency domain. I ran the OPO temp scan and the SQZ Angle, but this did not seem to improve anything (plot attached).

Additionally, the EY Roll Ringup that plagued Ryan C's EVE shift is indeed happening again. I'm unsure what to do about this since it seems he and Rahul tried a multitude of damping techniques without avail.

ASC High Gain was turned off ~5 minutes after it seemed to be working. A minute later, we it rang up again and we lost lock.

So there's two issues (or maybe they're related I don't know):

1. 1Hz Ringup that can be survived while locked but only in High Gain settings

2. ETMY Roll mode which just rings up consistently

Out of ideas - will continue to investigate but nothing seems to be helping.

TITLE: 10/23 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
INCOMING OPERATOR: Ibrahim
SHIFT SUMMARY: We made it to NLN but the ETMY roll mode took us down, it appears to have rung up each time we get into OMC_WHITENING since maintenance.

I found new settings for ITMX mode13, FM1+FM2+FM10 G= -0.5. 

The first 2 aquisitions we saw the Roll mode ringup but this 3rd time we did not, we slowly brought the IFO up, pausing to look at the mode in CHECK_VIOLINS, 10Ws, 25Ws, MAX_POWER,  TRANSITION_FROM_ETMX, and LOWNOISE_ESD_ETMX.
LOG: No log.

• 00:43 UTC We reached OMC_WHITENING
  • ETMY roll mode starts to ringup
  • 01:35 UTC lockloss from ETM glitch?
• 02:21 UTC OMC_WHITENING again, we're close to the limit so it shouldn't take very long
  • The ETMY roll mode looks to slowly increasing again, I'm going to try to damp it
  • 02:39 UTC NLN
    • SQZ looked bad, so I ran SCAN_OPO_TEMP then SCAN_SQZ_ANG, still looked bad after
    • I had some SDFs to accept, mostly offsets (ALSEY & LSC, ISCEY & ASC & ASCIMC, ALSEX & ISCEX & OMC)
    • I never made into Observing as I was fighting the ever ringing up ETMY roll mode until it made us lose lock
      • The 1Hz oscillation also showed up, as soon as I turned FM4 back on the Oscillation returned
• 03:13 UTC lockloss from ETMY roll mode
  • Rahul logged on to take a look with me, we paused at various states
  • 04:40 UTC NLN
    • 1Hz showed up, I turned off FM8 waited until it was fully off then FM4, as soon as FM4 is turned off and on the 1Hz seems to appear on DHARD
    • To turn them back on go in reverse, slowly
    • Something is growing, as seen in ASC-INP1 and LSC-PR_GAIN, Its at 1Hz even though I turned off the DHARD_P filters?
    • 04:52 UTC lockloss from the oscillation I'm assuming

I had been in NLN for 35 minutes fighting the ETMY roll mode, I also ran some SQZ stuff since it looked terrible (OPO_TEMP_SCAN, then ANG_SCAN) while in NLN. I was unsuccessful in damping the mode and it eventually grew saturated the DCPDs and killed the lock, there was also increased ground motion from a 5.6 from eastern Russia during this. In the middle of fighting the roll mode I saw the 1Hz oscillation start in DHARD_P so I turned off FM8 then FM4 on DHARD_P which fixed it.

Turns out that I had forgotten to turn the input on for the PR3 P estimator contribution from L (H1:SUS-PR3_M1_EST_P_FUSION_MODL_DRV_L_2GAP) (thanks to Edgard and Brian for having us even consider the estimators!), and that was causing the 0.6 Hz ASC ringups (87658). Even though we're pretty sure we found the cause, we've decided to temporarily turn off the PR3 and SR3 L and P estimators until we're back up and can verify that they are now working correctly.

TITLE: 10/23 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
INCOMING OPERATOR: Ryan C
SHIFT SUMMARY: Currently relocking and at LOWNOISE_COIL_DRIVERS. Still struggling with a 1 Hz oscillation(87658), but we've solved issues with MICH phase margin and with a 0.6 Hz oscillation caused by incorrect estimator settings, so 2/3 of the issues solved?
LOG:

14:30UTC Started relocking
    Initial alignment
    Lockloss from OMC_WHITENING due to 1 Hz ringup
    Multiple locklosses from DRMI and under due to ALSX getting noisy and ALSY having some glitchy drops in power
    Locklosses from LOWNOISE_LENGTH_CONTROL, MOVE_SPOTS, etc.                                                                                                                                                                

We managed to track down the source of the 0.6 Hz oscillation that we see at move spots and full power. It seems to be the result of the new pitch and length estimators on PR3 and SR3 that were installed on Tuesday. We tested this by sitting at move spots (after the spots converged), and turning on and off the estimators for a few minutes at a time. When turning ON the estimators, the 0.6 Hz ringing is not immediately apparent- it will slowly grow. However, once the 0.6 Hz oscillation is visible in the controls, turning OFF the estimator immediately stops the ringing. Therefore, we think this is the culprit for this instability. Oli will be turning off the length and pitch estimators for PR3 and SR3 for now. The yaw estimators are still on because they have been operating stably for a long time and were not changed this Tuesday.

We have also put the factor of 2 increase of MICH2 gain into the guardian, since we kept forgetting to add it in. Several locklosses that we thought were due to various other things were actually due to the MICH loop ringing up.

Finally, we just had a lockloss after moving to camera servos because the 1 Hz oscillation began to grow. Here is everything I tried to stop the ringing (none of it worked):

• CSOFT P gain was already at 30. I dropped it to 25 and re-increased to 30 several times, no change either way
• INP1 P gain was at -1, I doubled it and then tripled it, no change
• INP1 Y gain was at -0.1, doubled it, no change
• DSOFT P gain was at 5, I increased to 10, then 15, no change
• DSOFT Y gain was at 5, increased to 10, no change

I then had an idea to go to high bandwidth ASC using the seismic script, but we lost lock as soon as I thought of that.

Next plan: go to lownoise length control, skip lownoise asc, go to the final spots and camera servos, stay in HBW ASC and wait for the 1 Hz ring up and see if we can ride it out this way.

Attached scope screenshot shows the 0.6 Hz oscillation disappearing as soon as Oli flipped the length and pitch estimator switches OFF.

TITLE: 10/22 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
OUTGOING OPERATOR: Oli
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 21mph Gusts, 17mph 3min avg
    Primary useism: 0.05 μm/s
    Secondary useism: 0.30 μm/s 
QUICK SUMMARY:

• Microseism is decreasing over the past ~14 hours
• Wind has been rising the past 2.5 hours
• We're at MAX_POWER currently

Sheila, TJ, Oli, Matt

We reverted the ASC gain changes (87582) setting INP1 P to 1 and CSOFT P to 25 at MAX power (rather than 30).  Most of our locklosses overnight were from states before Max power, so the CSOFT P gain can't have been the problem, but we are undoing it for now just for the sake of reverting recent changes. 

We looked at one of the overnight locklosses, it seems that MICH is ringing up at 4.7 Hz.  Since Oli is sitting at 2W DCreadout to damp violin modes, we measured the MICH OLG there, and we see it is pretty close to unstable at 5Hz, with low coherence there.  We added a factor of 2 gain in LSC MICH2, to bring the UGF to 10 Hz similar to the references, and give us 35 degrees phase margin. 

Next we checked the SRCL UGF, which is also locked on POP45, and it's gain looks low by a factor of 5 compared to the reference.  This loop has a large phase bubble so it's still stable. POP 45 whitening gain and filters seem to be the same as a few days ago, and the POP LF power is the same as it was a few days ago at 2W.  PRCL gain also looks low compared to the reference, but is stable.  

We were able to get to OMC_WHITENING (almost to low noise), but while we were sitting there we had the old ring up.  We tried manually increasing the CSOFT P gain to 30, but the osciallation had already grown large and we lost lock anyway.  I've now put the csoft P gain back to 30 in max power in the guardian.  

Something happened with BRSY this morning during maintenance that caused it to ring up more than normal and now the damping is not behaving quite as expected. For now, I have paused the ETMY sensor correction guardian with the BRSY out of loop and turned off the output of the BRS so it won't be used for eq mode, should that transition happen.

So far today, I did a bunch of "recapturing frames" in the BRS C# code, which has usually fixed this issue in the past. We also restarted the beckhoff computer, then the plc, C# and epics ioc. This did not recover the BRS either. Marc, Fil and I went to EY and looked at the damping drive in the enclosure and I think it was behaving okay. When the drive came on, the output would reach ~1.8V, then go to 0V when it turned off.

I've contacted UW and we will take a look at this again tomorrow. 

[Joan-Rene Merou, Alicia Calafat, Anamaria Effler, Sheila Dwyer, Robert Schofield, Jenne Driggers]

We have looked at the near-30 Hz and near-100 Hz combs (Detchar issue 340) in all of LHO Fscan channels (Full 148 channels list can be found in O4_H1_Fscan_ch_info.yml) to find witnesses and also channels where the amplitude and coherence change at the same dates as DARM.

The list of combs is the following one:


  

    

      
spacing (Hz)
      
offset (Hz)
    
  
  

    
29.9695138888
0
    
99.99845486125
70.02888889
    
99.99845679
0
    
99.99846
29.9694
    
99.9984722225
89.90847222
    
99.99865
0
    
99.99845
10.08992
    
29.96952
520.17208
    
29.9695211
589.9007589
    
29.96951374
760.22840625
  


As reported by Xuanye Fan in Detchar issue 296, "it can be determined that the high-order harmonics disappeared on April 8, 2024, and reappeared on July 7, 2024". Hence, we have compared coherence with DARM in all the channels and the difference between July 1st and July 7th 2024.

Following the findings by Xuanye, we have redone the comb time-evolution plots for all of these combs in order to check for dates where their amplitude changed. As can be seen in the plots below, there are two clear dates where changes in the behavior of the combs took place, April ~4th 2024 and July ~7th 2024. The attached plots to this post are similar to the ones generated by Fscan using FscanFineToothPlot plots, however, they have two differences: They use data from the non-normalized spectras instead of from the time average plots. Additionally, they compute the relative harmonic amplitude with respect to the median of the surrounding 1 Hz spectrogram. The attached plots confirm that there is a significant change in comb amplitude between April 4th and July 7th 2024. One clear example is shown here:



We list here the channels that do show most of these Combs. These same channels do show changes in coherence between July 1st and July 7th 2024, but do not show changes in the amplitude of the combs.

- H1_IMC-F_OUT_DQ
- H1_LSC-MCL_IN1_DQ
- H1_LSC-MICH_IN1_DQ
- H1_LSC-SRCL_IN1_DQ
- H1_PEM-CS_MAG_EBAY_LSCRACK_X_DQ
- H1_PEM-CS_MAG_EBAY_LSCRACK_Y_DQ
- H1_PEM-CS_MAG_EBAY_LSCRACK_Z_DQ
- H1_PEM-CS_MAG_LVEA_INPUTOPTICS_X_DQ
- H1_PEM-CS_MAG_LVEA_INPUTOPTICS_Y_DQ
- H1_PEM-CS_MAG_LVEA_INPUTOPTICS_Z_DQ

In most channels, the comb amplitude tends to get quite low after ~1500 Hz. The following sets of channels show differences between X, Y and Z:

- H1_PEM-CS_MAG_EBAY_SUSRACK_X_DQ (Higher amplitudes and towards higher frequencies)
- H1_PEM-CS_MAG_EBAY_SUSRACK_Y_DQ (Lower comb amplitudes)
- H1_PEM-CS_MAG_EBAY_SUSRACK_Y_DQ (Lower comb amplitudes)

Regarding CS_MAG_LVEA_OUTPUTOPTICS, these combs can be seen best in X, weaker in Y and almost non-existent in Z. (In CS_MAG_LVEA_INPUTOPTICS they look roughly the same height)

- H1_PEM-CS_MAG_LVEA_OUTPUTOPTICS_X_DQ (Strongest)
- H1_PEM-CS_MAG_LVEA_OUTPUTOPTICS_Y_DQ (Weaker lines)
- H1_PEM-CS_MAG_LVEA_OUTPUTOPTICS_Z_DQ (Almost no lines)

Same behavior at:
- H1_PEM-CS_MAG_LVEA_VERTEX_X_DQ (Strongest lines)
- H1_PEM-CS_MAG_LVEA_VERTEX_Y_DQ (Weaker lines)
- H1_PEM-CS_MAG_LVEA_VERTEX_Z_DQ (Almost no lines)

We can see that these combs mostly appear in the corner station. The combs do not appear in neither EX nor EY channels. However, comb 99.99865 Hz offset 0.000 appears in many EX, EY channels and does become more coherent after July 7th. However, it is very close to 100 Hz so it may be influenced by other round-number combs (?)