Sun Dec 22 10:09:57 2024 INFO: Fill completed in 9min 54secs

TCmins = [-110C, -50C]

TITLE: 12/22 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: Oli
CURRENT ENVIRONMENT:
    SEI_ENV state: USEISM
    Wind: 6mph Gusts, 4mph 3min avg
    Primary useism: 0.08 μm/s
    Secondary useism: 0.76 μm/s
QUICK SUMMARY:

• Relocking at engage_asc from a 15:00 UTC lockloss
• Thick fog all over site and town, ~30ft of visibility?
• Microseism is just above the 90th percentile but looks to have flattened, low wind

I was called to intervene with the detector and found that we were in NLN but couldn't get into Observing because of an SDF diff on the ramp time for the ETMY Lock ESD bias. When TJ and I were testing out the code for the function that generates the INCREASE/DECREASE BIAS states for SUS_PI, we had decided on leaving the line in that sets the bias ramp time to 30 for going from a bias offset of -9 back to the nominal bias of -4.9. We thought it would be fine because the ramp time would get reverted when we lost lock, but we forgot about the part where we have to go back into Observing after the PI gets damped at the higher bias (oops). At least it was me who had to intervene so I can suffer the consequences of my mistakes! I changed that line in the code(attachment) so that when we are going from the higher bias to the nominal lower bias, the ramp time is the nominal 120 seconds. So it'll take longer to ramp back to nominal bias, but at least we'll be able to go back to Observing!

TITLE: 12/22 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Oli
SHIFT SUMMARY: Quiet shift until an EQ caused a lockloss. Relocking has been challenging due to high microseism, but H1 has been able to make progress and is currently relocking up to ENGAGE_SOFT_LOOPS. Fortunately, it looks like the microseism has turned around and started to trend down.

• 04:08 - EQ lockloss - alog81941
• 04:55 - DRMI alignment looks poor, starting initial alignment

Lockloss @ 04:08 UTC - link to lockloss tool

Cause likely due to M5.6 EQ out of Alaska and aftershocks; H1 had been locked for just over 7 hours.

[OpsInfo] I noticed that verbal alarms didn't announce the transition to earthquake mode in this instance. Found out that this is because EQ transitions were only announced for the regular EARTHQUAKE state in SEI_ENV, but we were in USEISM_EARTHQUAKE due to the high secondary microseism. I've updated the verbal alarms code to now also announce the transition to earthquake mode when going to the USEISM_EARTHQUAKE state.

TITLE: 12/21 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 153Mpc
OUTGOING OPERATOR: Ryan C
CURRENT ENVIRONMENT:
    SEI_ENV state: USEISM
    Wind: 34mph Gusts, 26mph 3min avg
    Primary useism: 0.08 μm/s
    Secondary useism: 0.66 μm/s
QUICK SUMMARY: H1 has been locked for 3 hours.

• Wind speeds and secondary microseism are on the rise
• All other systems nominal

TITLE: 12/21 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 154Mpc
INCOMING OPERATOR: Ryan S
SHIFT SUMMARY: One lockloss from an earthquake, continuously increasing microseism made relocking more difficult. The wind started to pick up around 22:00UTC. We've been locked for just 3 hours, I was not able to get the calibration measurement in.
LOG: No log

• 16:21 UTC lockloss
  • 17:39 UTC IA, SR2 was really off in pitch
  • 18:54 UTC LASER_NOISE_SUPR lockloss, now DRMI can't lock
  • 19:31 UTC 2nd IA
  • DRMI still can't lock, I ran though PRMI to confirm it was SRM that was misaligned, I trended SRM and moved it to where it was during the last successful lock acquisition (DRMI took 35 minutes). I had to move SRM 60 mrad in Pitch! This was right after finishing an IA as well
  • 21:00 Observing
• 21:54 PSL laser room yellow dust alarm, red at 23:33 (only 100 cts)

Sat Dec 21 10:05:49 2024 INFO: Fill completed in 5min 46secs

Running on TC-A only. TC-B (min=-52C) has been removed from plot to reduce y-axis log decade span.

Ryan S, Jason, Dave:

In the past few days cds_report was flagging a large log file for h1daqnds1. On investigation I found it was due to the DIAG_MAIN guardian node asking for 10 second trends of H1:PSL-ISS_DIFFRACTION_AVG eight times every second. This was generating +630k logs daily over the past few days.

Working with Ryan S, the problem was found to be due to the ISS diffraction avg signal trending downwards and going below DIAG_MAIN's  2.5 low limit, which was then requesting the 10 second averages of this channel from the DAQ.

In consultation with Jason, Ryan made two changes:

1. Boosted the ISS diffraction avg back up above 2.5 [see attached trend]

2. Removed the code which requests the 10 second average of the average channel, the 1 second average this channel represents is sufficient lookback [DIAG_MAIN.py code change shown below]

  665 @SYSDIAG.register_test
 666 def PSL_ISS():
 667     """ISS diffracted power
 668 
 669     """
 670     diff_high = 5.5
 671     diff_low = 2.5
 672     if PSL_on():
 673     ¦   diff_now = ezca['PSL-ISS_DIFFRACTION_AVG']
 674     ¦   # if diff_now > diff_high:
 675     ¦   #     diff_pwr = try_avg(-10, 'PSL-ISS_DIFFRACTION_AVG')
 676     ¦   #     if diff_pwr >= diff_high:
 677     ¦   #         yield "Diffracted power is high"
 678     ¦   # elif diff_now < diff_low:
 679     ¦   #     diff_pwr = try_avg(-10, 'PSL-ISS_DIFFRACTION_AVG')
 680     ¦   #     if diff_pwr <= diff_high:
 681     ¦   #         yield "Diffracted power is low"
 682     ¦   if diff_now > diff_high:
 683     ¦   ¦   yield "Diffracted power high"
 684     ¦   elif diff_now < diff_low:
 685     ¦   ¦   yield "Diffracted power low"
 686 
 

We were getting hit by a 6.1 from Vanuatu at the time, https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/index.cgi?event=1418832746.

SEI went to earthquake mode 1 minute before the LL.

TITLE: 12/21 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 158Mpc
OUTGOING OPERATOR: Oli
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 14mph Gusts, 10mph 3min avg
    Primary useism: 0.05 μm/s
    Secondary useism: 0.49 μm/s
QUICK SUMMARY:

We've been locked for 6 hours

2ndary microseism is increasing, wind is low/moderate

TITLE: 12/21 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 159Mpc
INCOMING OPERATOR: Oli
SHIFT SUMMARY: H1 has been locked and observing for about 4.5 hours. Been a quiet evening since H1 relocked early this shift.

• 00:11 - Lockloss - alog81930
• 01:38 - Observing

FAMIS 26348, last checked in alog81819

All fans look well within acceptable noise levels and largely unchanged from last check.

Lockloss @ 00:11 UTC - link to lockloss tool

Locked for about 1.5 hours. No obvious cause, but interestingly the first saturating suspension here was ETMY L2, which I don't think is typical.

TITLE: 12/21 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Ryan S
SHIFT SUMMARY: 2 locklosses, recovering from the last one currently. New o4b record of 47 hours?
LOG:                                                                                                                             

20:59 UTC lockloss from a 47 hour lock which I think is the new O4b record. Easy relock with an IA, which I had to help out inbetween SR2 and SRY align as it got couldn't get SRY (lots of SRM sats), I reran SR2 and them SRY was fine.

00:11 UTC lockloss

TITLE: 12/21 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 158Mpc
OUTGOING OPERATOR: Ryan C
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 5mph Gusts, 3mph 3min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.41 μm/s
QUICK SUMMARY: H1 has been locked for just over an hour. If there's a lockloss this evening, I plan to adjust the ISS RefSignal to bring the diffracted power back up to around 4% (currently down to around 2.5%).

• Wind is low; secondary microseism moderate
• LAB2 dust monitor error (known)

New calibration lines turned on 7/25/2023 resulted in unexpected additional inter-modulation products. These new calibration lines were turned off on 8/9/2023.

When looking at coherence data between the main detector and safe channels, we expect no change in the before-during-after periods of the new calibration lines and additional inter-modulation products. However, in “safe” channel CS MAG EBAY LSCRACK Y, we see coherence jump at the specific frequencies of the calibration lines and inter-modulation products. This indicates that the channel may actually be unsafe.

Plots for H1_PEM-CS_MAG_EBAY_LSCRACK_Y_DQ  before / during / after

Research Presentation: https://dcc.ligo.org/G2402592

ETMX glitch lockloss tool