Sun Dec 29 10:16:20 2024 INFO: Fill completed in 16min 16secs

TITLE: 12/29 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 160Mpc
OUTGOING OPERATOR: Ryan C
CURRENT ENVIRONMENT:
    SEI_ENV state: USEISM
    Wind: 13mph Gusts, 7mph 3min avg
    Primary useism: 0.04 μm/s
    Secondary useism: 0.42 μm/s
QUICK SUMMARY:

H1 has been locked for 10+hrs!  Secondary microseism is mostly under the 95th percentile line, and winds are only around 10mph (with light rain on the drive in).

TITLE: 12/29 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 158Mpc
INCOMING OPERATOR: Ryan C
SHIFT SUMMARY: Currently Observing and have been Locked for almost 2 hours. Just spent an hour working with Vicky to get sqz back up and running because it had unlocked and I couldn't get it to relock. SQZ_LO guardian had the message "Beam diverter is open but no 3MHz on OMC, could be an alignment problem". 

With Vicky, we first tried resetting the SQZ angle by manualing to the RESEST_SQZ_ASC_FDS guardian state in SQZ_Manager, but that didn't work. The problem was that the FC couldn't even lock because the LO couldn't lock. What had happened is that the "squeezer angle servo ran away, and because squeezer ASC alignment setpoints are coupled to the SQZ angle, so this time when SQZ ANGLE runs away it also drags the SQZ ASC away. This time it dragged the SQZ ASC away enough that the LO loop could not lock (because nearly no signal on SQZ-OMC_RF3 beatnote (RF3 was around -22, normally around -9)" (Vicky, 2024).

What we did that ended up working (and the ideal way to do this in the future):

     - Reset SQZ angle (CLF_REFL_RF6 to 185)

     - Go to NO_SQZ

     - Trend ZM4 and ZM6 driftmon and adjust sliders to bring SUS back to last time in FDS (last lock)

     - Go back to FDS

Vicky's notes on sqz todo's regarding this problem:

     - "Improve the notification given (currently "Beam diverter is open but no 3MHz on OMC, could be an alignment problem") when SQZ_LO can't lock to alert ops to check SQZ ASC or check ZM4/6 LOCK filters, etc, to make this problem more diagnosable"

     - "SQZ MANAGER needs a more robust way to clear SQZ ASC now that integrators are in the ZM4/6 LOCK filter banks (see https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=80534). Oli tried to RESET_SQZ_ASC_FDS but that didn't clear the suspension lock filter banks."

     - "We should get at more robust guardian state to clear SQZ ASC in the lock filter integerators, and corresponding OPS instructions for when OMC RF3 LO locking fails + SQZ ANGLE is railed (I found it at 235 ish, it should be like 185-195). "

* All quotes by Vicky

On a different note, I figured out why the LOCKLOSS_SHUTTER_CHECK guardian had been stuck stalled and had been requiring us to change the states between high and low arm power - it's because it is NOT normally managed by ISC_LOCK like I had thought. Yesterday after I had MANUAL'd it, thinking that it had previously been managed by ISC_LOCK, I set it to be managed by ISC_LOCK. It actually is normally just on AUTO. I set it to AUTO and had no issues with it during the locking sequence.

LOG:

00:05 Started trying to relock
    - Sitting in OMC_WHITENING for an hour damping violins
02:45 NOMINAL_LOW_NOISE
02:45 Observing

03:51 Lockloss https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=82037
05:23 NOMINAL_LOW_NOISE
05:25 Observing

05:46 Out of Observing due to squeezer unlock
    - SQZ_LO gives message "Beam diverter is open but no 3MHz on OMC, could be an alignment problem"
    - With help from Vicky, we figured it out
07:02 Back to Observing

Lockloss @ 12/29 03:51UTC

(Lockloss tool still hasn't been restarted since the cluster issues on Thursday)

TITLE: 12/28 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Wind
INCOMING OPERATOR: Oli
SHIFT SUMMARY:

Winds put us over the tipping point for not making it possible to Lock H1 (with the continued horrendous µseism); so marking Observatory Mode at WIND state.  For the latter part of the shift, took H1 to IDLE (to help prevent ringing up violins).

With H1 down, attempt a full Site Walkthru to the End Stations (only made it to EY).
LOG:

• 1527 Started to look into why ISCLOCK was stuck at CHECK AS SHUTTERS
  • 1535 MANUAL to PREP TR CARM & then return to AUTO to NLN
  • 1538 LOCKLOSS (start of many locklosses assume due to very high winds and very high microseism)
• 1545 Sent Dave text regarding Guardian Log "leap second" issue
  • 1600 chatted with Dave and he started work
• 1906-2017 H1 to IDLE due to winds
  • tried locking at 2017, but still lots of ALS locklosses, back to IDLE at 2030
  • 2247 Tried locking again, but only for a few minutes due to locklosses
• 2126-2240 Site WalkThroughs VEA Edition: This was done since H1 was down due to high winds + high microseism (Corey)
  • 2126-2154 LVEA + FCES 
    • Hard hat caution sign was on the floor so I removed that.
  • 2155-2240 EY & attempt of EX (too many tumbleweeds after MX)
    • At EY the Wind Fence leftmost wall panel has come undun (see attached).
    • Also noticed some sort of plumbing outside near the entrance to EY which looks like something has been trying to dig a hole around it (see attached).
    • Could not drive beyond MX due to impassable "X2" road (see attached).
    • While driving around the site, also knocked some ice off the CP1 exhaust (but this required pitchforking some tumbleweeds out of the way) (see attached).
• 2248-2340 Site Walk Through Common Spaces Edition (Oli)

TITLE: 12/29 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Wind
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
    SEI_ENV state: USEISM
    Wind: 20mph Gusts, 16mph 3min avg
    Primary useism: 0.08 μm/s
    Secondary useism: 0.72 μm/s
QUICK SUMMARY:

Currently trying to relock and at FIND_IR. We were able to catch DRMI for a few seconds before losing it. It's still very windy and the microseism is still super high but I wanted to try relocking for a bit while the wind is slightly lower at 25 (still gusting up to 30+)

Closes FAMIS#26450, last checked 81585

BRS Driftmon

Auxiliary BRS Channels

At the beginning of December, Ryan noted (81585) that the BRSY temperature had been drifting up a bit, but it looks like it has now settled at a temperature just below 23°C. Everything else is looking normal and within range.

Closes FAMIS#26350, last checked 81932

Corner Station Fans (attachment1)
- All fans are looking normal and within range.

Outbuilding Fans (attachment2)
- All fans are looking normal and are within range.

Shift Update: 

Lots of tumbleweeds rolling by outside & Tons of ALS locklosses as the winds are back at 30+mph (this morning there were only two short lulls in the winds...where H1 was able to try MAX POWER on one of these windows).  Because of the double whammy of high microseism AND high winds (which can be heard in the Control Room, will take H1 to IDLE to help prevent ringing up violins.  Will check back hourly (time for lunch).

Laser Status:
    NPRO output power is 1.849W
    AMP1 output power is 69.87W
    AMP2 output power is 137.0W
    NPRO watchdog is GREEN
    AMP1 watchdog is GREEN
    AMP2 watchdog is GREEN
    PDWD watchdog is GREEN

PMC:
    It has been locked 11 days, 0 hr 28 minutes
    Reflected power = 24.6W
    Transmitted power = 104.2W
    PowerSum = 128.8W

FSS:
    It has been locked for 0 days 0 hr and 5 min
    TPD[V] = 0.7813V

ISS:
    The diffracted power is around 3.5%
    Last saturation event was 0 days 0 hours and 6 minutes ago
Possible Issues:
        PMC reflected power is hig

Sat Dec 28 10:15:41 2024 INFO: Fill completed in 15min 38secs

h1guardian1's /usr/share/zoneinfo/leap-seconds.list file expired 28 Dec 2024 resulting in guardlog errors.

I updated the tzdata package on h1guardian1 which resolved the problem. I verified the CDS workstations already have updated leap-second files (they expire 28 Jun 2025).

TITLE: 12/28 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: Ryan C
CURRENT ENVIRONMENT:
    SEI_ENV state: USEISM
    Wind: 34mph Gusts, 24mph 3min avg
    Primary useism: 0.09 μm/s
    Secondary useism: 0.56 μm/s
QUICK SUMMARY:

Still some logic issues with H1 after the HAM6 Vacuum Gauge Failure / Interlock change where ISC_LOCK is stuck at CHECK AS SHUTTERS.  I "manually" ran a test of the Fast Shutter via the AS Port Protection medm and pushing the TEST button (it completed and was OK, also get a "HAM6 Verbal alarm which I assume is the Fast Shutter being cycled open/closed and seen by the ISI).  I went to the FAST SHUTTER Guardian, but it was MANAGED by ISC_LOCK, so I took it to AUTO and ran an INIT to get it to READY (green state), but now this node is "stolen" by USER (me).......Aaaaand, we are still stuck at CHECK AS SHUTTERS.......Aaaand....

We continue to fly blind because Guardian Logs have not been available since yesterday due to "leap second" issue.

BUT---we are still stuck at CHECK AS SHUTTERS, so I did a "MANUAL" on ISC_LOCK from CHECK AS SHUTTERS and went to PREP TR CARM; then returned to AUTO for ISC LOCK......

....Then there was a  LOCKLOSS at CARM TO ANALOG at 1538. 

Lockloss is not surprising since we are still at high microseism (above 95th prcentile) AND for the last 6hrs winds have been between 20-40+mph.

13:00 UTC lockloss, the lockloss tool is not updating. Appears to be from an ITMX CPS glitch, the suspension tripped.

(Dave Barker, Fil Clara, Jenne Driggers, Corey Gray, Gerardo Moreno, plus those who signed work permit and whom I phoned last night)

Last Night Recap:
Last night at 959pmPT, there was an audible alarm for the HAM6 vacuum gauge (PT110).  Gerardo went out to check to see if it was dead.  He phoned Fil and they did a power cycle of the gauge to see if this would improve the situation, but it did not.  After this, H1 was taken to IDLE for the night (OWL shift cancelled).  (Should have taken Observatory Mode to CORRECTIVE MAINTENANCE at this point, but only remembered to do this this morning.)

Work Permit 12260 generated.  Fil arrived on-site around 9amPT (17utc).

This Morning's Summary of Fil's Work:

The HAM6 Interlock (for operations of the Fast Shutter + PZT) is now Using the BSC3 Interlock (or VAC gauge from BSC3 since it was decided to BYPASS the HAM6 VAC gauge which failed last night [alog 82005]).  This means that the BSC3 Interlock is no longer being used for the ITM ESDs (which sounds like it's fine because Jenne/Fil said we don't use the ESDs for the ITMs).

Fil noted that he confirmed that the Fast Shutter Chassis in the LVEA is OPERATIONAL.  Fil then left the site after giving me this summary.

As far as a test of this new HAM6 Interlock configuration, I intentionally ran a test of the AS Port Protection system (sitemap/SYS/ AS Port Protection....this was my first time doing this).  After clicking the RUN (test) button, after a few seconds received an "OK" for the (1) Test & (2) Fault.  After this test and the signed-off Work Permit, we are OK to GO for High Power locking.

Currently running an Initial Alignment (note:  ISC_LOCK initially went into ERROR [red box] when I selected Initial Alignment (screenshot of error message attached).  LOAD-ing ISC_LOCK (suggested in the Log) cleared this up.

ALSO: 

• Attaching a screenshot of the AS Port Protection medm
• CDS Alarm box on the CDS Overview remains (related to the lingering issue of the failed PT110 gauge).