TITLE: 10/05 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 152Mpc
INCOMING OPERATOR: Ryan S
SHIFT SUMMARY: 1 lockloss, automated relock, the wind is decreasing and the micoseism also seems to be (slowly...). We've been locked for just over an hour.
LOG: No log
More investigation similar to alog80358
NLN 1412064049 lockloss tool, no tag, FSS chans looks stable.
AS_A_DC and IMC-TRANS loose lock within ~250 ms
PREP_DC_READOUT_TRANSITION 1412066248 lockloss tool, FSS tag
AS_A_DC and IMC-TRANS loose lock within 1 ms of each other, but the FSS chans look stable before hand
NLN 1412077192 lockloss tool, tool failed
AS_A_DC and IMC-TRANS loose lock within ~5 ms, small FSS glitching starts almost 2 mins before LL, large glitch 1/2 a sec before
NLN 1412084422 lockloss tool, No FSSery
AS_A_DC and IMC-TRANS loose lock within ~250 ms
NLN 1412092628 lockloss tool, FSS tag
AS_A_DC and IMC-TRANS loose lock within ~5 ms, FSS glitching starts 5 secs before LL
NLN 1412101888 lockloss tool, FSS tag
AS_A_DC and IMC-TRANS loose lock within ~250 ms
TRANSITION_FROM_ETMX 1412105312 lockloss tool, FSS tag
AS_A_DC and IMC-TRANS loose lock within ~ 13 ms, the FSS starts glitching 40 seconds before LL, FSS_PC_MON_OUTPUT starts rising ~40 seconds before LL
LOW_NOISE_ESD_ETMX 1412113952 lockloss tool, no FSSery
AS_A_DC and IMC-TRANS loose lock within ~250 ms
NLN 1412131865 lockloss tool, FSS tag
AS_A_DC and IMC-TRANS loose lock within ~1 ms, there's a FSS glitch 1 second before the LL
To summarize, Ryan looked at 8 locklosses from today, 6 of them had the FSS tag, 5 of them had the IMC loose lock withing 15ms of the IFO losing lock, and only 3 of them had the pattern that Ian and Camilla found was normal in O4a, where the IMC looses lock ~250 ms after the IFO losses lock. This FSS issue seems to be having a serious impact on our duty cycle.
I had a helpful conversation with Rick this afternoon about this issue, he sugested that we trying looking for the glitches in other places in the PSL, so that we could think about tests where we don't run some of the feedback servos and see if we still see the issue. I made an ndscope template with all the PSL fast channels I could find, it is saved in sheila.dwyer/ndscope/PSL/PSL_fast_channels.yaml
Looking at 1412111658 with this scope, it looks like several of these channels that we are saving at a fast rate might not be connected, for example the OSC and HPL channels are around 0. The only other channel that clearly shows these glitches is H1:PSL-PWR_NPRO_OUT_DQ.
Request for operators: If you are having locking difficulties over the weekend, it may be due to the PSL/FSS problems. To help narrow down the cause of the problem one could sit for an hour or two with the FSS unlocked, and the ISS unlocked. If the glitches show up in the NPRO channel even without these servos locked, then we will know that the problem is not from the FSS, but from the NPRO.
02:50 UTC lockloss
TITLE: 10/04 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 147Mpc
INCOMING OPERATOR: Ryan C
SHIFT SUMMARY:
A bit of a rough day with H1. Had two short locks and then it was rough going with locking, and then it got worse with winds in the afternoon. BUT, at the end of the shift H1 was able to get to OMC WHITENING---the violin modes are elevated.
LOG:
TITLE: 10/04 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Wind
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 29mph Gusts, 16mph 3min avg
Primary useism: 0.05 μm/s
Secondary useism: 0.22 μm/s
QUICK SUMMARY:
23:27 UTC Observing
H1 continues to have a rough day with short locks, and it's gotten worse with H1 dropping out at various states while trying to get to NLN. Although H1, had been locking DRMI sort of decently, ran an Initial Alignment to see if this can help (it had been over 24hrs since the last IA was run).
For the last lock about 3hrs ago, OPO temperature was adjusted and this yielded a better range, but it was only locked 47min.
Locking woes continue. Have had a couple of locks make it all the way past MAX POWER only to have locklosses, but most of our locklosses of late are early ISC LOCK stages. This is some weather. Winds are now in the 30mph range and microseism is above 50th percentile.
In the process of investigating the locklosses due to FSS glitching and working on spare chassis for the FSS in the PSL, we compared the power spectrum of the PZT monitor between H1 and L1. We found some difference in the power spectrum, plot attached.
Because we moved the OPO crystal position yesterday, 8045, the crystal absorption will be changing for a few weeks, and we will need to adjust the temperature setting for the OPO until it settles, as Tony and I did last night. 80455. Here are two sets of instructions, one that can be used while in observing to adjust this, and one that can be done when we loose lock or are out of observing for some other reason.
For these first few days, please follow the out of observing instructions when relocking, if this hasn't been done in the last few hours, and please try to do it in the last few hours of the evening shift so that the temperature is close to well tuned at the start of the owl shift.
Out of observing instructions (to be done while recovering from a lockloss, can be done while ISC_LOCK is locking the IFO) (screenshot) :
In observing instructions:
Both the OPO temperature and the SQZ phase are ignored in SDF and can be adjusted while we are in observing, but it's important to log the times of any adjustments made.
Changes made to make this easier (Thanks Corey for beta testing the instructions):
Corey has had a series of locklosses from DRMI/PRMI and ALS, I've looked into one of these: 1412094082
In this case it looks like there is a large glitch seen in all DRMI LSC signals (and ASC, but it seems to start in LSC), with a small glitch about 4 seconds before lockloss, which DRMI recovers from. Then there is a larger glitch that happened about 1 second before lockloss, where POP18 drops below the trigger threshold, and the DRMI guardian goes to DOWN, at the same time the ALS Y arm looses green lock, which then unlocks the ALS X arm.
This glitch doesn't seem to be cause by the DRMI ASC, although we were in the process of engaging that ASC when it happened. It also isn't due to the BS ISI isolation coming on, which hadn't happened yet.
Edit: I looked into a second lockloss, from PRMI: 1412095134, this one is different in that it happens very quickly and PRMI and ALS signals drop out at close to the same time. This happened while the DRMI guardian was waiting for the refl WFS centering to converge.
For FAMIS 26303:
Laser Status:
NPRO output power is 1.818W (nominal ~2W)
AMP1 output power is 64.55W (nominal ~70W)
AMP2 output power is 137.7W (nominal 135-140W)
NPRO watchdog is GREEN
AMP1 watchdog is GREEN
AMP2 watchdog is GREEN
PDWD watchdog is GREEN
PMC:
It has been locked 0 days, 22 hr 2 minutes
Reflected power = 22.1W
Transmitted power = 104.0W
PowerSum = 126.1W
FSS:
It has been locked for 0 days 0 hr and 7 min
TPD[V] = 0.8089V
ISS:
The diffracted power is around 2.4%
Last saturation event was 0 days 0 hours and 7 minutes ago
Possible Issues:
AMP1 power is low
PMC reflected power is high
Monthly FAMIS Check (#26494)
T240 Centering Script Output:
Averaging Mass Centering channels for 10 [sec] ...
2024-10-04 09:08:54.264102
There are 14 T240 proof masses out of range ( > 0.3 [V] )!
ETMX T240 2 DOF X/U = -0.745 [V]
ETMX T240 2 DOF Y/V = -0.637 [V]
ETMX T240 2 DOF Z/W = -0.529 [V]
ITMX T240 1 DOF X/U = -1.51 [V]
ITMX T240 1 DOF Y/V = 0.325 [V]
ITMX T240 1 DOF Z/W = 0.447 [V]
ITMX T240 3 DOF X/U = -1.563 [V]
ITMY T240 3 DOF X/U = -0.773 [V]
ITMY T240 3 DOF Z/W = -1.934 [V]
BS T240 1 DOF Y/V = -0.394 [V]
BS T240 3 DOF Y/V = -0.307 [V]
BS T240 3 DOF Z/W = -0.49 [V]
HAM8 1 DOF Y/V = -0.475 [V]
HAM8 1 DOF Z/W = -0.786 [V]
All other proof masses are within range ( < 0.3 [V] ):
ETMX T240 1 DOF X/U = -0.083 [V]
ETMX T240 1 DOF Y/V = -0.1 [V]
ETMX T240 1 DOF Z/W = -0.159 [V]
ETMX T240 3 DOF X/U = -0.106 [V]
ETMX T240 3 DOF Y/V = -0.165 [V]
ETMX T240 3 DOF Z/W = -0.093 [V]
ETMY T240 1 DOF X/U = 0.009 [V]
ETMY T240 1 DOF Y/V = 0.116 [V]
ETMY T240 1 DOF Z/W = 0.173 [V]
ETMY T240 2 DOF X/U = -0.083 [V]
ETMY T240 2 DOF Y/V = 0.171 [V]
ETMY T240 2 DOF Z/W = 0.03 [V]
ETMY T240 3 DOF X/U = 0.182 [V]
ETMY T240 3 DOF Y/V = 0.029 [V]
ETMY T240 3 DOF Z/W = 0.109 [V]
ITMX T240 2 DOF X/U = 0.127 [V]
ITMX T240 2 DOF Y/V = 0.258 [V]
ITMX T240 2 DOF Z/W = 0.22 [V]
ITMX T240 3 DOF Y/V = 0.117 [V]
ITMX T240 3 DOF Z/W = 0.104 [V]
ITMY T240 1 DOF X/U = 0.078 [V]
ITMY T240 1 DOF Y/V = 0.083 [V]
ITMY T240 1 DOF Z/W = -0.061 [V]
ITMY T240 2 DOF X/U = 0.054 [V]
ITMY T240 2 DOF Y/V = 0.185 [V]
ITMY T240 2 DOF Z/W = 0.092 [V]
ITMY T240 3 DOF Y/V = 0.066 [V]
BS T240 1 DOF X/U = -0.162 [V]
BS T240 1 DOF Z/W = 0.087 [V]
BS T240 2 DOF X/U = -0.09 [V]
BS T240 2 DOF Y/V = 0.007 [V]
BS T240 2 DOF Z/W = -0.113 [V]
BS T240 3 DOF X/U = -0.206 [V]
HAM8 1 DOF X/U = -0.28 [V]
Assessment complete.
STS Centering Script Output:
Averaging Mass Centering channels for 10 [sec] ...
2024-10-04 09:22:39.265893
There are 1 STS proof masses out of range ( > 2.0 [V] )!
STS EY DOF X/U = -2.389 [V]
All other proof masses are within range ( < 2.0 [V] ):
STS A DOF X/U = -0.514 [V]
STS A DOF Y/V = -0.927 [V]
STS A DOF Z/W = -0.438 [V]
STS B DOF X/U = 0.323 [V]
STS B DOF Y/V = 0.945 [V]
STS B DOF Z/W = -0.432 [V]
STS C DOF X/U = -0.723 [V]
STS C DOF Y/V = 0.743 [V]
STS C DOF Z/W = 0.539 [V]
STS EX DOF X/U = -0.224 [V]
STS EX DOF Y/V = -0.05 [V]
STS EX DOF Z/W = 0.084 [V]
STS EY DOF Y/V = 0.007 [V]
STS EY DOF Z/W = 1.209 [V]
STS FC DOF X/U = 0.24 [V]
STS FC DOF Y/V = -1.04 [V]
STS FC DOF Z/W = 0.636 [V]
Assessment complete.
Fri Oct 04 08:09:58 2024 INFO: Fill completed in 9min 55secs
TITLE: 10/04 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: Ryan S
CURRENT ENVIRONMENT:
SEI_ENV state: EARTHQUAKE
Wind: 6mph Gusts, 3mph 3min avg
Primary useism: 0.06 μm/s
Secondary useism: 0.22 μm/s
QUICK SUMMARY:
Walked in to see H1 at MAX POWER after a 59min lock. (There have been EQ spikes, in last few hours, but they do NOT line up with atleast the last 2 locklosses---for this current lock there was a M5.2 Iceland EQ which occured as H1 was stepping though ISC LOCK). This last 1340utc lockloss also does not have an FSS tag; 1139utc lockloss had a FAIL; 0845, 840, 837, utc locklosses all have an FSS Oscillation tag.
1445 Back to Observing.
Lockloss @ 22:19 UTC - link to lockloss tool
No obvious cause, but it looks like the IMC lost lock at the same time as the arms. However, I don't suspect the FSS in this case.
TITLE: 10/02 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 148Mpc
CURRENT ENVIRONMENT:
SEI_ENV state: USEISM
Wind: 21mph Gusts, 13mph 3min avg
Primary useism: 0.08 μm/s
Secondary useism: 0.45 μm/s
QUICK SUMMARY:
Sudden Lockloss @ 21:55 UTC very likley cause by a PSL FSS issue.
IMC Had a hard time relocking locking after the lockloss.
NLN Reached @ 22:54 UTC
Observing reached @ 22:56 UTC
Definitely looks like the FSS had a large glitch and lost lock before DARM saw the lockloss. This lockloss didn't have the FSS glitches happening before though.