TITLE: 10/05 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 152Mpc
INCOMING OPERATOR: Ryan S
SHIFT SUMMARY: 1 lockloss, automated relock, the wind is decreasing and the micoseism also seems to be (slowly...). We've been locked for just over an hour.
LOG: No log

• 23:17 OMC_WHITENING to damp violins
• 10/04 23:27 UTC Observing
  • I ran a coherence check right before we lost lock
• 02:50 UTC lockloss alog80472
  • IMC_LOCK is struggling as the FSS keeps unlocking
  • 03:59 UTC Observing
  • 04:10 UTC, I adjusted the OPO temperature and we gained ~10 Mpc (~145 -> ~155)
  • 05:00 UTC, I again adjusted the OPO temp

More investigation similar to alog80358

NLN 1412064049 lockloss tool, no tag, FSS chans looks stable.

AS_A_DC and IMC-TRANS loose lock within ~250 ms

PREP_DC_READOUT_TRANSITION 1412066248 lockloss tool, FSS tag

AS_A_DC and IMC-TRANS loose lock within 1 ms of each other, but the FSS chans look stable before hand

NLN 1412077192 lockloss tool, tool failed

AS_A_DC and IMC-TRANS loose lock within ~5 ms, small FSS glitching starts almost 2 mins before LL, large glitch 1/2 a sec before

NLN 1412084422 lockloss tool, No FSSery

AS_A_DC and IMC-TRANS loose lock within ~250 ms

NLN 1412092628 lockloss tool, FSS tag

AS_A_DC and IMC-TRANS loose lock within ~5 ms, FSS glitching starts 5 secs before LL

NLN 1412101888 lockloss tool, FSS tag

AS_A_DC and IMC-TRANS loose lock within ~250 ms

TRANSITION_FROM_ETMX 1412105312 lockloss tool, FSS tag

AS_A_DC and IMC-TRANS loose lock within ~ 13 ms, the FSS starts glitching 40 seconds before LL, FSS_PC_MON_OUTPUT starts rising ~40 seconds before LL

LOW_NOISE_ESD_ETMX 1412113952 lockloss tool, no FSSery

AS_A_DC and IMC-TRANS loose lock within ~250 ms

NLN 1412131865  lockloss tool, FSS tag

AS_A_DC and IMC-TRANS loose lock within ~1 ms, there's a FSS glitch 1 second before the LL

02:50 UTC lockloss

TITLE: 10/04 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 147Mpc
INCOMING OPERATOR: Ryan C
SHIFT SUMMARY:

A bit of a rough day with H1.  Had two short locks and then it was rough going with locking, and then it got worse with winds in the afternoon.  BUT, at the end of the shift H1 was able to get to OMC WHITENING---the violin modes are elevated.
LOG:

• 1445 H1 back to OBSERVING
• 1454-1508 Optics Lab check & clean (kim)
• 1517 Phoned non-emergency Hanford Patrol dispatch to let them know that Robert & colleagues will be making magnetic field measurements from the road near the power lines this morning.
• 1556 LOCKLOSS
  • 1621 DRMI lockloss---I had touched SRM in yaw to assist locking; DRMI locked (then lost lock); manually selected CHECK MICH FRINGES and eventually made it to past DRMI
  • For this lock, I followed Sheila's instructions (alog 80461) to make an OPO temperature adjustment before ISC_LOCK injected squeezing (this adjustment only took a few minutes).  Apparent results are a slightly better range around 160Mpc.
• 1606-1743 Magnetic field measurements along Y-arm with 2 vans and a couple rental cars (Anamaria, Robert, Cosmic Explorer group)
  • 1743-1913 Then they moved offsite around the intersection of Rt10 & Alabama Rd to take more of these measurements until about noon (local time)
• 1617-1632 Pcal lab photos (rick)
• 1831 LOCKLOSS
  • Locklosses at TRANSITION TO ETMX, SHUTTER ALS, FIND IR, soooo, went for an....
  • 1956-2016 Initial Alignment
  • Locklosses at ALS, MAX POWER, ALS, FIND IR, LOWNOISE ESD...
  • useism is above the 50th percentile and the winds have picked up this afternoon.
  • 2204 Marking Observatory Mode as WINDY (should have done this earlier)
• 2015 Picking up parts at MX (mitch, ibrahim)

TITLE: 10/04 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Wind
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 29mph Gusts, 16mph 3min avg
    Primary useism: 0.05 μm/s
    Secondary useism: 0.22 μm/s
QUICK SUMMARY:

• Between the high winds (gust of >30mph & 3min avg ~20mpg), and high microseism (fully above the 50th percentile) holding ALS has been quite tough
• We've had 3 high state locklosses so far on the way to recover NLN

H1 continues to have a rough day with short locks, and it's gotten worse with H1 dropping out at various states while trying to get to NLN.  Although H1, had been locking DRMI sort of decently, ran an Initial Alignment to see if this can help (it had been over 24hrs since the last IA was run).

For the last lock about 3hrs ago, OPO temperature was adjusted and this yielded a better range, but it was only locked 47min.

In the process of investigating the locklosses due to FSS glitching and working on spare chassis for the FSS in the PSL, we compared the power spectrum of the PZT monitor between H1 and L1.  We found some difference in the power spectrum, plot attached.

Because we moved the OPO crystal position yesterday, 8045, the crystal absorption will be changing for a few weeks, and we will need to adjust the temperature setting for the OPO until it settles, as Tony and I did last night. 80455.  Here are two sets of instructions, one that can be used while in observing to adjust this, and one that can be done when we loose lock or are out of observing for some other reason. 

For these first few days, please follow the out of observing instructions when relocking, if this hasn't been done in the last few hours, and please try to do it in the last few hours of the evening shift so that the temperature is close to well tuned at the start of the owl shift.

Out of observing instructions (to be done while recovering from a lockloss, can be done while ISC_LOCK is locking the IFO) (screenshot) :

• Make sure that ISC_LOCK will not set SQZ_MANAGER request to INJECT_SQZ, by setting the request to ADS_TO_CAMERAs or any lower state.
• From sitemap, SQZ_OVERVIEW, set SQZ MANGER request to LOCKED_SEED_NLG.
• from the purple !SQZ scopes button, open the NLG scope
• Once the SQZ manger is in LOCKED_SEED_NLG, you should see that the red OPO_IR_TRANS trace on the scope is oscillating.  Adjust the OPO temperature using the slider which is right under the purple SQZ scopes button, to maximize the peak of the oscillations. 
• SET_SQZ MANGER request back to FDS_READY_IFO, and once that's ready you can set the ISC_LOCK request to nominal low noise.
• Before going to observing, you can adjust the squeezing angle by requesting SQZ_MANAGER to SCAN_SQZANG.  It is best practice to do this after the IFO has thermalized, but it can be done early in the lock.  If lock stretches are long and this has been done only early in the lock, it may need adjusting for a thermalized IFO.  Right now the squeezing angle has a strong frequency dependence early in the lock stretch, so let's not adjust it until the IFO has been locked for at least an hour.

In observing instructions:

Both the OPO temperature and the SQZ phase are ignored in SDF and can be adjusted while we are in observing, but it's important to log the times of any adjustments made. 

• To adjust the temperature in full lock, go to the purple SQZ_SCOPE button and choose the OPO temp scope, adjust the OPO temperature to maximize CLF_REFL_RF6_ABS (the green trace). 
• After doing this it might be necessary to adjust the SQZ PHASE, which can be adjusted in steps of 5 degrees or so.  Set this to maximize the squeezing level, which you can do by watching the DARM FDS dtt template.  It is available from the SQZ DTT purple button, or on NUC33.  It is easiest to adjust this for high frequency squeezing (around 1kHz), this should get you close to good squeezing at 100 Hz.

Changes made to make this easier (Thanks Corey for beta testing the instructions):

•  I've commented out the request to SQZ_MANGER that ISC_LOCK makes in low noise length control, because that took control of the squeezer from Corey when he was giving these instructions a trial.
• The OPO temp is not monitored in SDF
• I removed the goto= True from the LOCK_FSS state.  This will force SQZ_MANAGER to go through down after the NLG measurement (and turn off the seed PZT ramp).

Corey has had a series of locklosses from DRMI/PRMI and ALS, I've looked into one of these: 1412094082

In this case it looks like there is a large glitch seen in all DRMI LSC signals (and ASC, but it seems to start in LSC), with a small glitch about 4 seconds before lockloss, which DRMI recovers from.  Then there is a larger glitch that happened about 1 second before lockloss, where POP18 drops below the trigger threshold, and the DRMI guardian goes to DOWN, at the same time the ALS Y arm looses green lock, which then unlocks the ALS X arm. 

This glitch doesn't seem to be cause by the DRMI ASC, although we were in the process of engaging that ASC when it happened.  It also isn't due to the BS ISI isolation coming on, which hadn't happened yet. 

Edit:  I looked into a second lockloss, from PRMI: 1412095134, this one is different in that it happens very quickly and PRMI and ALS signals drop out at close to the same time.  This happened while the DRMI guardian was waiting for the refl WFS centering to converge.

For FAMIS 26303:
Laser Status:
    NPRO output power is 1.818W (nominal ~2W)
    AMP1 output power is 64.55W (nominal ~70W)
    AMP2 output power is 137.7W (nominal 135-140W)
    NPRO watchdog is GREEN
    AMP1 watchdog is GREEN
    AMP2 watchdog is GREEN
    PDWD watchdog is GREEN

PMC:
    It has been locked 0 days, 22 hr 2 minutes
    Reflected power = 22.1W
    Transmitted power = 104.0W
    PowerSum = 126.1W

FSS:
    It has been locked for 0 days 0 hr and 7 min
    TPD[V] = 0.8089V

ISS:
    The diffracted power is around 2.4%
    Last saturation event was 0 days 0 hours and 7 minutes ago

Possible Issues:
    AMP1 power is low
    PMC reflected power is high

Monthly FAMIS Check (#26494)

T240 Centering Script Output:

Averaging Mass Centering channels for 10 [sec] ...
2024-10-04 09:08:54.264102
There are 14 T240 proof masses out of range ( > 0.3 [V] )!
ETMX T240 2 DOF X/U = -0.745 [V]
ETMX T240 2 DOF Y/V = -0.637 [V]
ETMX T240 2 DOF Z/W = -0.529 [V]
ITMX T240 1 DOF X/U = -1.51 [V]
ITMX T240 1 DOF Y/V = 0.325 [V]
ITMX T240 1 DOF Z/W = 0.447 [V]
ITMX T240 3 DOF X/U = -1.563 [V]
ITMY T240 3 DOF X/U = -0.773 [V]
ITMY T240 3 DOF Z/W = -1.934 [V]
BS T240 1 DOF Y/V = -0.394 [V]
BS T240 3 DOF Y/V = -0.307 [V]
BS T240 3 DOF Z/W = -0.49 [V]
HAM8 1 DOF Y/V = -0.475 [V]
HAM8 1 DOF Z/W = -0.786 [V]
All other proof masses are within range ( < 0.3 [V] ):
ETMX T240 1 DOF X/U = -0.083 [V]
ETMX T240 1 DOF Y/V = -0.1 [V]
ETMX T240 1 DOF Z/W = -0.159 [V]
ETMX T240 3 DOF X/U = -0.106 [V]
ETMX T240 3 DOF Y/V = -0.165 [V]
ETMX T240 3 DOF Z/W = -0.093 [V]
ETMY T240 1 DOF X/U = 0.009 [V]
ETMY T240 1 DOF Y/V = 0.116 [V]
ETMY T240 1 DOF Z/W = 0.173 [V]
ETMY T240 2 DOF X/U = -0.083 [V]
ETMY T240 2 DOF Y/V = 0.171 [V]
ETMY T240 2 DOF Z/W = 0.03 [V]
ETMY T240 3 DOF X/U = 0.182 [V]
ETMY T240 3 DOF Y/V = 0.029 [V]
ETMY T240 3 DOF Z/W = 0.109 [V]
ITMX T240 2 DOF X/U = 0.127 [V]
ITMX T240 2 DOF Y/V = 0.258 [V]
ITMX T240 2 DOF Z/W = 0.22 [V]
ITMX T240 3 DOF Y/V = 0.117 [V]
ITMX T240 3 DOF Z/W = 0.104 [V]
ITMY T240 1 DOF X/U = 0.078 [V]
ITMY T240 1 DOF Y/V = 0.083 [V]
ITMY T240 1 DOF Z/W = -0.061 [V]
ITMY T240 2 DOF X/U = 0.054 [V]
ITMY T240 2 DOF Y/V = 0.185 [V]
ITMY T240 2 DOF Z/W = 0.092 [V]
ITMY T240 3 DOF Y/V = 0.066 [V]
BS T240 1 DOF X/U = -0.162 [V]
BS T240 1 DOF Z/W = 0.087 [V]
BS T240 2 DOF X/U = -0.09 [V]
BS T240 2 DOF Y/V = 0.007 [V]
BS T240 2 DOF Z/W = -0.113 [V]
BS T240 3 DOF X/U = -0.206 [V]
HAM8 1 DOF X/U = -0.28 [V]

Assessment complete.

STS Centering Script Output:

Averaging Mass Centering channels for 10 [sec] ...
2024-10-04 09:22:39.265893
There are 1 STS proof masses out of range ( > 2.0 [V] )!
STS EY DOF X/U = -2.389 [V]
All other proof masses are within range ( < 2.0 [V] ):
STS A DOF X/U = -0.514 [V]
STS A DOF Y/V = -0.927 [V]
STS A DOF Z/W = -0.438 [V]
STS B DOF X/U = 0.323 [V]
STS B DOF Y/V = 0.945 [V]
STS B DOF Z/W = -0.432 [V]
STS C DOF X/U = -0.723 [V]
STS C DOF Y/V = 0.743 [V]
STS C DOF Z/W = 0.539 [V]
STS EX DOF X/U = -0.224 [V]
STS EX DOF Y/V = -0.05 [V]
STS EX DOF Z/W = 0.084 [V]
STS EY DOF Y/V = 0.007 [V]
STS EY DOF Z/W = 1.209 [V]
STS FC DOF X/U = 0.24 [V]
STS FC DOF Y/V = -1.04 [V]
STS FC DOF Z/W = 0.636 [V]

Assessment complete.

Fri Oct 04 08:09:58 2024 INFO: Fill completed in 9min 55secs

TITLE: 10/04 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: Ryan S
CURRENT ENVIRONMENT:
    SEI_ENV state: EARTHQUAKE
    Wind: 6mph Gusts, 3mph 3min avg
    Primary useism: 0.06 μm/s
    Secondary useism: 0.22 μm/s
QUICK SUMMARY:

Walked in to see H1 at MAX POWER after a 59min lock.  (There have been EQ spikes, in last few hours, but they do NOT line up with atleast the last 2 locklosses---for this current lock there was a M5.2 Iceland EQ which occured as H1 was stepping though ISC LOCK).  This last 1340utc lockloss also does not have an FSS tag; 1139utc lockloss had a FAIL; 0845, 840, 837, utc locklosses all have an FSS Oscillation tag.

1445 Back to Observing.

Lockloss @ 22:19 UTC - link to lockloss tool

No obvious cause, but it looks like the IMC lost lock at the same time as the arms. However, I don't suspect the FSS in this case.

TITLE: 10/02 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 148Mpc
CURRENT ENVIRONMENT:
    SEI_ENV state: USEISM
    Wind: 21mph Gusts, 13mph 3min avg
    Primary useism: 0.08 μm/s
    Secondary useism: 0.45 μm/s
QUICK SUMMARY:

Sudden Lockloss @ 21:55 UTC very likley cause by a PSL FSS issue.

IMC Had a hard time relocking locking after the lockloss.

NLN Reached @ 22:54 UTC
Observing reached @ 22:56 UTC