Lockloss @ 06/17 05:28 UTC from unknown causes. Definitely not from wind or an earthquake

Closes FAMIS#26310, last checked 78362

Corner Station Fans (attachment1)
- All  fans are looking normal and within range.

Outbuilding Fans (attachment2)
- All fans are looking normal and within range.

Lockloss at 06/17 01:37 UTC - looks like it may have been due to a jump in the wind speed?

Link to report.

• The duty factor for the week was 15.43%, much lower than previous weeks. Locking problems persisted for the second half of the week (see, e.g., alog 78308). It was better in the first couple of days, at 51.91% and 38.43%, but dropped substantially thereafter. No observing data was taken from Friday through Sunday.
• The week started with a wind advisory in the region with wind gusts in excess of 25mph some times. Winds were partially responsible for lockloss and caused difficulties in maintaining lock. The wind speed was around 10 m/s or less on June 2 (UTC) but increased to as much as 17.5 m/s around 15:45hrs UTC on June 3. June 4 also remained windy; while it ebbed in the night hours, it climbed back up in the day hours to as high as 18.5 m/s by 18:00 UTC. The wind finally slowed down to below 10m/s around 4:00 UTC on June 5 and did not rise above that for the rest of the week.
• June 5 onward the duty factor remained below 10%, while several commissioning / troubleshooting exercises attempted to get the detector to maintain lock / relock.
• "Whistle" glitches in SQG FC WFC were associated with range drops in a couple of instances, as has been observed in May too -- in this alog 78263. A drop in the relative vacuum in H1:SQZ DC PhotoDiode sum/null ratio BLRMS was also found to have some correlation with range drops.
• Low-frequency (10-50Hz) noise in the IMC spectrogram appears to be correlated with high SNR glitches in Omicron (especially, on June 3).
• Hveto: 12 channels appeared in Hveto this week of which 8 were not seen in the previous week.
• FScan: Plots from June 3 seem to be missing. All plots have appeared for June 4 – 6, for which the line counts are 656, 267 and 291, respectively.

TITLE: 06/16 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
INCOMING OPERATOR: Oli
SHIFT SUMMARY:

IFO was in NLN and OBSERVING as of 06:05 UTC (21hr 37 min lock) but is NOW in CORRECTIVE_MAINTENANCE while we briefly restart the ADS Camera.

Like 5 minutes before my shift ended, the ADS Pitch1 Inmon and Yaw1 Inmon are stuck. It seems that they keep trying to turn on, but can't get past TURN_ON_CAMERA_FIXED_OFFSET. This has happened before alog 77499 and it is likely that the cameras just need to be restarted. We did not lose lock. Oli (incoming Op) has called Dave and they are working on it.
 

LOG:

None

TITLE: 06/16 Eve Shift: 2300-0800 UTC (1600-0100 PST), all times posted in UTC
STATE of H1: Observing at 152Mpc
OUTGOING OPERATOR: Ibrahim
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 13mph Gusts, 11mph 5min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.07 μm/s
QUICK SUMMARY:

Observing and locked for 21.5 hours.

IFO is in NLN and OBSERVING (Now 18hr 35 min lock!)

• LDAS seems to have come back online in the last few hours.
• We survived a 6.0 from Peru

Nothing else of note.

Sun Jun 16 10:10:34 2024 INFO: Fill completed in 10min 30secs

Note TCs did not reach -200C because lower outside temps this morning (15C, 59F).

TITLE: 06/16 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 156Mpc
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 9mph Gusts, 4mph 5min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.09 μm/s
QUICK SUMMARY:

IFO is in NLN and OBSERVING since 06:06 UTC (12hr 45 min lock)

NUC27 Glitch screen is giving a warning: "Cluster is down, glitchgram is not updated", which I haven't seen before.

TITLE: 06/16 Eve Shift: 2300-0800 UTC (1600-0100 PST), all times posted in UTC
STATE of H1: Observing at 158Mpc
INCOMING OPERATOR: Corey
SHIFT SUMMARY:  We are Observing at 158 Mpc and have been locked for 6 hours now. The only issues I had today was the OPO ISS maxing out at 54uW and so not being able to catch at 80uW and so needing to adjust the setpoint, and then relocking after the 06/16 00:01 UTC lockloss , I had some issues with getting PRMI and DRMI to lock, but it makes sense since the wind was still a bit high at the time. After that the night has been quiet.
LOG:

23:00 Detector relocking and at DARM_TO_RF

23:39 NOMINAL_LOW_NOISE
- SQZ OPO ISS pump having trouble locking. The OPO transmission couldn't go higher than 54.6uW
    - Adjusted OPO temp, but the current temp was the best so I put it back. Reloaded the OPO guardian (So I had changed nothing!). OPO was able to get up to 72 after this, but still not to 80.
    - Naoki came on teamspeak and lowered the threshold to 70, and it caught very soon after that.
00:01 While dealing with the OPO, we lost lock. We had been locked for 22 minutes

00:29 Lockloss from ACQUIRE_DRMI
00:30 Started an initial alignment
00:51 Initial alignment done, relocking
01:04 Lockloss from ACQUIRE_DRMI
01:52 NOMINAL_LOW_NOISE
01:56 Observing
02:01 Our range was low so I took us out of Observing and ran the sqz tuning guardian states
02:10 Back to Observing, with a 7Mpc increase in range

05:25 Left Observing and started calibration measurements
05:53 Calibration measurements done, running sqz alignment (new optic offsets accepted)

06:05 Back into Observing

Calibration was run between 06/16 05:25 and 05:53 UTC

Calibration monitor screenshot

Broadband (2024/06/16 05:25 - 05:30 UTC)

File: /ligo/groups/cal/H1/measurements/PCALY2DARM_BB/PCALY2DARM_BB_20240616T052532Z.xml

Simulines (2024/06/16 05:32 - 05:53 UTC)

Files:

/ligo/groups/cal/H1/measurements/DARMOLG_SS/DARMOLG_SS_20240616T053211Z.hdf5
/ligo/groups/cal/H1/measurements/PCALY2DARM_SS/PCALY2DARM_SS_20240616T053211Z.hdf5
/ligo/groups/cal/H1/measurements/SUSETMX_L1_SS/SUSETMX_L1_SS_20240616T053211Z.hdf5
/ligo/groups/cal/H1/measurements/SUSETMX_L2_SS/SUSETMX_L2_SS_20240616T053211Z.hdf5
/ligo/groups/cal/H1/measurements/SUSETMX_L3_SS/SUSETMX_L3_SS_20240616T053211Z.hdf5

 

06/16 05:24 UTC I took us out of Observing to run a calibration sweep that we weren't able to run earlier.

Currently Observing at 152 Mpc and have been Locked for 55 mins. We had a lockloss at 06/16 00:01 UTC, 22 minutes into NOMINAL_LOW_NOISE, and relocking required me helping with PRMI and DRMI, but we were able to get up eventually. The wind is slowly going down and is around 25mph now.

Lockloss @ 06/16 00:01UTC (LDAS currently down so no link). We had only been locked for 20 minutes and I was trying to get the SQZ OPO to lock, so we had not been in Observing. There was a small jump up in wind at that time, but not sure if that would have caused the LL

TITLE: 06/15 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Wind
INCOMING OPERATOR: Oli
SHIFT SUMMARY:

IFO is in and is locking at LOWNOISE_COIL_DRIVERS

Wind gusts have been high since the last lockloss and per usual, I would put H1 in down when the wind would go higher than 30mph and wait for oppurtunistic low-wind periods to lock ALS - here's a timeline since the last LL. The only sdf revert I had to accept was in my midshift alog 78457

All times are UTC

19:31 - NLN/Observing Lockloss

19:38 - ALS Lockloss

19:39 - DOWN due to 40-50mph winds

21:43 - Attempt to re-lock due to wind speeds below 30mph

22:08 - Two failures at MICH_FRINGES and no ability to lock PRMI let alone DRMI - decied to run initial alignment

22:25 - Initial Alignment done fully auto, attempt to relock - winds picking up about to surpass 30mph

22:41 - DOWN again after 2 ALS locklosses and 3 IR Locklosses, with winds around 35mph.

22:54 - Saw small oppurtunistic sub 30mph calmness (that only lasted 10 mins) and decided to try locking again - it worked!

23:02 - DRMI locked!

23:30 - Shift ended while in LOWNOISE_COIL_DRIVERS

LOG:

None