Tue Nov 19 10:12:46 2024 INFO: Fill completed in 12min 42secs

WP12201

Marc, Fil, Dave, Ryan

h1susex is powered down to restore cabling from the new 28bit LIGO-DAC to the original 20bit General Standards DACs.

Procedure:

Put h1iopseiex SWWD into long bypass

Safe the h1susetmx, h1sustmsx and h1susetmxpi models

Stop all models on h1susex

Fence h1susex from the Dolphin fabric

Power down h1susex.

15:22 UTC lockloss - 1416065051

FSS_FAST_MON grew just below the LL, IMC lost it at the same time as ASC. FSS, and ISS lost it at the same time.

TITLE: 11/19 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Preventative Maintenance
OUTGOING OPERATOR: Oli
CURRENT ENVIRONMENT:
    SEI_ENV state: USEISM
    Wind: 7mph Gusts, 5mph 3min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.54 μm/s
QUICK SUMMARY:

• We lost lock at 15:22, 2 minutes into the magnetic injections (we had been locked for 9:56!)
  • We're not going to get the inlock charge measurements this morning
• 15:30 UTC brought ISC-_LOCK to IDLE

TITLE: 11/19 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 161Mpc
INCOMING OPERATOR: Oli
SHIFT SUMMARY:
Shift consisted of 3 locks and 2 IMC locklosses
Locking has been fairly straight forward once the IMC decides to get and stay locked.
Survived a 5.6M Tonga Quake, and a few PI ring ups.

Tagging SUS because ITMY Mode 5 is ringing up.  I have turned OFF the Gain to it as the current Nominal state is making IYM5 worse and has been for the last few locks.

LOG:

4:20 UTC Lockloss https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/index.cgi?event=1416025235

This Alog was brought to you by your favorite IMC Lockloss tag. 

TITLE: 11/19 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Lock Acquisition
CURRENT ENVIRONMENT:
    SEI_ENV state: USEISM
    Wind: 16mph Gusts, 9mph 3min avg
    Primary useism: 0.04 μm/s
    Secondary useism: 0.67 μm/s
QUICK SUMMARY:
1:57:53 UTC  IMC Lockloss
https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/?event=1416016690

Fought with IMC a little to get the IMC to lock, by requesting Down, and Init, and offline a few times. But once it finally locked, relocking went fast.
2:55:01 UTC  Nominal LowNoise

Incoming 5.6M EQ.

3:04 UTC Observing Reached

TITLE: 11/19 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 152Mpc
OUTGOING OPERATOR: Ibrahim
CURRENT ENVIRONMENT:
    SEI_ENV state: USEISM
    Wind: 9mph Gusts, 6mph 3min avg
    Primary useism: 0.04 μm/s
    Secondary useism: 0.70 μm/s
QUICK SUMMARY:
Despite the Usiesm being elevated, H1 has been Locked for 20 minutes.
The Plan is to continue Observing all night.

 
 

TITLE: 11/19 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing
INCOMING OPERATOR: Tony
SHIFT SUMMARY:

IFO is in NLN and OBSERVING as 00:17 UTC

A rough day for locking.

First, we had planned comissioning from 8:30AM to 11:30AM, during which comissioners continued investigating the IMC.

Sheila went into the LVEA to realign onto the POP and ALS beatnotes following a PR3 move that got higher buildups, which was successful. For this, we stayed at CHECK_VIOLINS_BEFROE_POWERUP (losing lock twice from there due to IMC losing lock).

We then got to locking and were able to acquire NLN and go into OBSERVING for 30 minutes (starting 21:44 UTC). The IMC glitch then caused a Lockloss (alog 81338).

After this LL, we were unable to lock DRMI despite just having done an initial alignment. Tried touching up PRM and SRM during PRMI and DRMI respectively, succeeding in the former and failing in the latter.

I decided to do an initial alignment, which happened fully automatically. We were able to lock DRMI but a lockloss at CARM_5_PICOMETERS brought us down due to the IMC glitch. We were then able to come back up to NLN fully automatically, finally getting to OBSERVING.

Overall, high microseism and IMC glitches made locking difficult. We still got there in the end though.
LOG:

Dave, Fernando, Patrick

Fernando and I powered off the computer, unplugged the power and other cables from the back of it to slide it forward in the rack, took the top off the chassis, disconnected the blue timing fiber from the PCI board, capped the fiber ends, pulled the fiber out of the chassis, slid the computer back into the rack, and plugged the cables back in, except for the kvm switch cable that is not used anyway. We left the computer on.

This was to stop the timing overview from turning red when the computer is powered on.

We did not put in a work permit, but cleared it with Daniel and informed the operator.

IMC caused Lockloss 30 mins into observing.

We had another instance of the -800nS timing glitch of the EY CNS-II GPS receiver this morning from 05:47:50 to 06:00:04 PDT (duration 12min 11 sec).

A reminder of this receiver's history:

At EX and EY we had no CNS-II issues for the whole of O4 until the past few months.

The EY 1PPS, nominally with a -250 +/- 50 nS difference to the timing system, went into a -800nS diff state on 30th Sept, then again 1st October and again 18th October. 

We replaced its power supply Tue 22nd October 2024, after which there have been no glitches for 27 days until this morning.

Closes FAMIS26017, last checked in alog81183.

HAMs:

HAM sensor noise at 7-9Hz seems to be reduced for most chambers (HAMs and BSCs both stages of BSCs).

BSCs:

ITMY_ST1_CPSINF_V2 looks reduced above 30 Hz.

All the BSC ST2 specifically, sensor noise at 6-9Hz is reduced.

Mon Nov 18 10:13:07 2024 INFO: Fill completed in 13min 4secs

Sheila, Ibrahim,  Jenne, Vicky, Camilla

After Ibrahim told Sheila that DRMI has been struggling to lock, she checked the POP signals and found that somethings been drifting and POP now appears to be clipping, see POP_A_LF trending down. We use this PD in full lock so don''t want any clipping.

We has similar issues last year that gave us stability issues. These issues last year didn't have "IMC" lock-losses so we think this isn't the main issue we've having now but maybe effecting stability.

Trends showing the POP clipping last year, and now. Last year we moved PR3 to removing this slipping while looking at the coherence between ASC-POP_A_NSUM (one of the QPD on the sled in HAM3) and LSC-POP_A (LSC sensor in HAM1): 74578, 74580, 74581.

Similar coherence plots to 74580, for now are show the coherence is bad:

• It's got worse since July, ca see as step down once we moved PR2 in 78878. Compare Green (July 1st) then Red (July 10th), then Blue (now), plot here.
• It's been bad since the emergency vent recovery in August plot here. 
• The increased coherence seen now <1Hz is the micoseism being elevated.

Sheila is moving the PRC cavity now which is improving POPAIR signals, plot attached is the ASC-POP_A_NSUM to LSC-POP_A coherence with DRMI only locked before and during the move. See improvements. Sheila is checking she's in the best position now.

H1 was dropped out of OBSERVING due to the TCS ITMy CO2 laser unlocking at 0118utc.  The TCS_ITMY_CO2 guardian relocked the 2min.

It was hard to see the reason why at first (there were no SDF Diffs, but eventually saw a User Message via GRD IFO (on Ops Overview) pointing to something wrong with TCS_ITMY_CO2.  Oli was also here and they mentioned seeing this, along with Camilla, on Oct9th (alog)---this was the known issue of the TCSy laser nearing the end of its life.  It was replaced a few weeks later on  Oct22nd (alog).

Here are some of the lines from the LOG:

2024-11-13_19:43:31.583249Z TCS_ITMY_CO2 executing state: LASER_UP (10)
2024-11-14_01:18:56.880404Z TCS_ITMY_CO2 [LASER_UP.run] laser unlocked. jumping to find new locking point

.

.
2024-11-14_01:20:12.130794Z TCS_ITMY_CO2 [RESET_PZT_VOLTAGE.run] ezca: H1:TCS-ITMY_CO2_PZT_SET_POINT_OFFSET => 35.109375
2024-11-14_01:20:12.196990Z TCS_ITMY_CO2 [RESET_PZT_VOLTAGE.run] ezca: H1:TCS-ITMY_CO2_PZT_SET_POINT_OFFSET => 35.0
2024-11-14_01:20:12.297890Z TCS_ITMY_CO2 [RESET_PZT_VOLTAGE.run] timer['wait'] done
2024-11-14_01:20:12.379861Z TCS_ITMY_CO2 EDGE: RESET_PZT_VOLTAGE->ENGAGE_CHILLER_SERVO
2024-11-14_01:20:12.379861Z TCS_ITMY_CO2 calculating path: ENGAGE_CHILLER_SERVO->LASER_UP
2024-11-14_01:20:12.379861Z TCS_ITMY_CO2 new target: LASER_UP

While looking at locklosses today, Vicky and I noticed that after we reach NLN, the MC2/IM4 TRANS power increases 0.1 to 0.5%.  

Daniel helped look at this and we expect the ISS to change to keep the power out of the IMC constant, but the power after the IMC on IM4 TRANS (not centered) changes ~1% too. Everything downstream of the ISS AOM sees this change plot, something is seeing a slow ~1hour thermalization.

• Plot Nov 13th, MC2 TRANS increases 0.5%
• Plot Nov 9th, MC2 TRANS increases 0.35%
• Plot September 2nd, MC2 TRANS increases 0.1%
• Plot September 11th, MC2 TRANS increases 0.2% (IFO DOWN for 6 hours before)

The same signals at LLO are show a similar amount of noise (slightly more in the MC2_TRANS) but no thermalization drift, but LLO has a lot less IFO thermalization.

After discussions with control room team: Jason, Ryan S, Sheila, Tony, Vicky, Elenna, Camilla 

• Prior to September 13th:
  • No IMC locklosses 80951
  • NPRO/FSS glitches present but not bothering us (June 80520, last year 67718).
• September 12th: IMC gain redistribution 79998
• September 13th: First IMC lockloss
• September to October:
  • We start noticing FSS glitches and do of tests to understand these.
  • They were present in NPRO only with everything else unlocked (part of reason for swapping NPRO)
  • Found that these glitches are not new, present in June 80520 and even last year 67718. Maybe they got worse, or maybe we just stopped surviving them as well.
  • IMC has periods of being unstable (have to wait in DOWN)
• October 9th: IMC gain redistribution reverted 80566
  • We have 6 days with only one IMC lockloss 
• October 15th: NPRO current increased 80687
  • IMC lockloses started again 
• Ocotber 18th: NPRO current reverted 80746
  • IMC lockloses continued
• October 23rd to 29th: NPRO swapped 80929
  • Swapped to new faraday and EOM, both found to be dodgy
• Ocotber 30th: Back to NLN
  • Had issues with NPRO 81073: found to be NPRO mode hopping 
  • IMC locklosses continued 
• October 30th to November 1st: Tests.
  • CARM OLG 80956
  • IMC OLG 80979
  • FSS OLG 80993 
  • If the PSL ISS is unlocked the IMC can stay locked: 81056
  • Attempt to swap TTFSS (reverted) 80983
  • Hardboot of PSL chassis h1psl0: 81048 
• November 6th: PSL mode hopping was fixed by changing NPRO temperature, 81107. 
  • PSL now stable
  • IMC has periods of being unstable
  • IMC locklosses continue 

Conclusions: The NPRO glitches aren't new. Something changed to make us not be able to survive them as well in lock. The NRPO was swapped so isn't the issue. Looking the timing of these "IMC" locklosses, they are caused by something in the IMC or upstream 81155.

Tagging OpsInfo: Premade templates for looking at locklosses are in /opt/rtcds/userapps/release/sys/h1/templates/locklossplots/PSL_lockloss_search_fast_channels.yaml and will come up with command 'lockloss select' or 'lockloss show 1415370858'.

Camilla, Oli

Recently, because of the PSL/IMC issues, we've been having a lot of times where the IFO (according to verbals) seemingly goes into READY and then immediately goes to DOWN because the IMC is not locked. Camilla and I checked this out today and it turns out that these locklosses are actually from LOCKING_ARMS_GREEN - the checker in READY that is supposed to make sure the IMC is locked was actually written as nodes['IMC_LOCK'] == 'LOCKED' (line 947), which just checks that the requested state for IMC_LOCK is LOCKED, and it doesn't actually make sure the IMC is locked. So READY will return True, it will continue to LOCKING_ARMS_GREEN, and immediately lose lock because LOCKING_ARMS_GREEN actually makes sure the IMC is locked. This all happens so fast that verbals doesn't have time to announce LOCKING_ARMS_GREEN before we are taken to DOWN.

To (hopefully) solve this problem, we changed nodes['IMC_LOCK'] == 'LOCKED' to be nodes['IMC_LOCK'].arrived and nodes['IMC_LOCK'].done, and this should make sure that we stay in READY until the IMC is fully locked. ISC_LOCK has been reloaded with these changes.