Yesterday we went back to the higher range CARM configuration, in which the fast/IMC locklosses first appeared  80938.  Overnight we had two observing locklosses tagged with the IMC, neither of which had glitches visible in the FSS fast mon channel, 1414411965 (1st attached screenshot), and 1414381791. 

This morning I changed the gains by 6 dB in the other direction, so we further limit the range available to the frequency actuator. (CARM sliders and ndscope) Our microseism is around 5um/second right now, slightly above the 90th percentile, but there was no wind and no earthquakes this morning.  At times the IMC splitmon was reaching -8V, it's range is +/- 10V so this is about as far as we can go in this direction, and might be kind of marginal.

We've since had one lockloss 1414426971, this is tagged as commissioning because we were in commissioning time at the moment, but no one was actively doing anything at the time so this wasn't caused by commissioners.  There was a glitch in the FSS fast mon for this lockloss, although it was below the threshold for the tag to be applied, and around the time of that glitch the drive to the ISS AOM becomes noisy.  We were not saturating the IMC splitmon at the time of the lockloss so that doesn't seem to be the cause.

Lockloss @ 10/31 16:01 UTC after almost an hour locked

All models on h1seih45 glitched at 07:09:46 PDT (14:09:46 UTC). This appears to have broken the lock and caused guardian to start an initial alignment.

Symptoms are:

• All IPC receivers, Dolphin and SHMEM, on all h1seih45 models reported a single receive error
• DAQ (both DC0 and DC1) reported a single CRC error for all h1seih45 models
• dmesg, DAQ and central loggers show no logs at this time
• No evidence of a timing glitch at this time

The models appeared to have resumed normal operation following the glitch, no restarts were needed. At the time of writing H1 is locked with a range of 160MPc.

TITLE: 10/31 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: Ryan S
CURRENT ENVIRONMENT:
    SEI_ENV state: USEISM
    Wind: 5mph Gusts, 3mph 3min avg
    Primary useism: 0.04 μm/s
    Secondary useism: 0.47 μm/s
QUICK SUMMARY:

Looks like we just finished an initial alignment and are starting to relock

H1 called for assistance at 13:13 UTC when SRM watchdogs tripped during the SRC alignment steps of initial alignment. After resetting the WDs, I just let Guardian try SRC align again, but SRM tripped during ACQUIRE_SRY. I reset the WDs again and took ALIGN_IFO to DOWN to manually align SRM a bit (didn't get too much improvement on the AS_A fringing, but some), and had Guardian try again. This time, there were no issues, and initial alignment finished without issue.

H1 is currently relocking and just got through DRMI.

TITLE: 10/31 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Ryan S
SHIFT SUMMARY:  Currently relocking, but we've had a few lock losses after DRMI and one at engage_asc_for_full_ifo. We've just made it past that point again.

• Gave up trying to bring PR2 back to its old position, it was close but as I get closer it seemed to only make things worse. I then ran an initial alignment sans green because it I had moved the PRs a fair amount and DRMI was definitely not going to catch. During the IA, it seemed the PR2 preferred to be at it's place post earthquake.
• DRMI post 2nd initial alignment was a great sucess!
• Lost lock at LOWNOISE_ESD_ETMX - saw a pitch ringup in what looks like CHARD.
• Next time around started to see a ringup again, but it calmed itself down after a minute. Made it to low noise at 0201UTC

Lockloss 0349

• It has been able to reaquire DRMI, but there have been a many lock losses, seeming to be from the FSS glitching, or something else that I don't know of.

Averaging Mass Centering channels for 10 [sec] ...
2024-10-30 14:34:04.266740


There are 14 T240 proof masses out of range ( > 0.3 [V] )!
ETMX T240 2 DOF X/U = -0.755 [V]
ETMX T240 2 DOF Y/V = -0.707 [V]
ETMX T240 2 DOF Z/W = -0.473 [V]
ITMX T240 1 DOF X/U = -1.549 [V]
ITMX T240 1 DOF Y/V = 0.328 [V]
ITMX T240 1 DOF Z/W = 0.452 [V]
ITMX T240 3 DOF X/U = -1.611 [V]
ITMY T240 3 DOF X/U = -0.797 [V]
ITMY T240 3 DOF Z/W = -1.997 [V]
BS T240 1 DOF Y/V = -0.404 [V]
BS T240 3 DOF Y/V = -0.317 [V]
BS T240 3 DOF Z/W = -0.502 [V]
HAM8 1 DOF Y/V = -0.436 [V]
HAM8 1 DOF Z/W = -0.758 [V]


All other proof masses are within range ( < 0.3 [V] ):
ETMX T240 1 DOF X/U = -0.04 [V]
ETMX T240 1 DOF Y/V = -0.066 [V]
ETMX T240 1 DOF Z/W = -0.098 [V]
ETMX T240 3 DOF X/U = -0.055 [V]
ETMX T240 3 DOF Y/V = -0.111 [V]
ETMX T240 3 DOF Z/W = -0.058 [V]
ETMY T240 1 DOF X/U = 0.015 [V]
ETMY T240 1 DOF Y/V = 0.136 [V]
ETMY T240 1 DOF Z/W = 0.192 [V]
ETMY T240 2 DOF X/U = -0.069 [V]
ETMY T240 2 DOF Y/V = 0.187 [V]
ETMY T240 2 DOF Z/W = 0.029 [V]
ETMY T240 3 DOF X/U = 0.206 [V]
ETMY T240 3 DOF Y/V = 0.04 [V]
ETMY T240 3 DOF Z/W = 0.128 [V]
ITMX T240 2 DOF X/U = 0.129 [V]
ITMX T240 2 DOF Y/V = 0.265 [V]
ITMX T240 2 DOF Z/W = 0.229 [V]
ITMX T240 3 DOF Y/V = 0.124 [V]
ITMX T240 3 DOF Z/W = 0.102 [V]
ITMY T240 1 DOF X/U = 0.083 [V]
ITMY T240 1 DOF Y/V = 0.084 [V]
ITMY T240 1 DOF Z/W = -0.063 [V]
ITMY T240 2 DOF X/U = 0.05 [V]
ITMY T240 2 DOF Y/V = 0.182 [V]
ITMY T240 2 DOF Z/W = 0.1 [V]
ITMY T240 3 DOF Y/V = 0.072 [V]
BS T240 1 DOF X/U = -0.174 [V]
BS T240 1 DOF Z/W = 0.082 [V]
BS T240 2 DOF X/U = -0.099 [V]
BS T240 2 DOF Y/V = 0.002 [V]
BS T240 2 DOF Z/W = -0.126 [V]
BS T240 3 DOF X/U = -0.213 [V]
HAM8 1 DOF X/U = -0.286 [V]
 

1414381791

IMC tag on this one, ending a 1 hr 48min lock.

Following instructions from alog76751 and from Sheila herself, I ran the CARM olg python script located in /ligo/gitcommon/psl_measurements/. Sheila had already hooked up the SR785 at the PSL racks earlier in the day while we were relocking, so it was easy enough for me to check that we could connect to it through GPIB (checked with ipython like Craig did in alog64204), then run the commands listed in alog76751. After I ran it, I then went out into the LVEA and unplugged and turned off the SR785, as well as the LVEA lights.

Since this plot is saved as a pdf, its hard to say exactly what the UGF is, but it looks around 17kHz just as it has been, consistent with 76751 & 76448  but a little higher than 70920 and 65676 and 67584.

• Gave up trying to bring PR2 back to its old position, it was close but as I get closer it seemed to only make things worse. I then ran an initial alignment sans green because it I had moved the PRs a fair amount and DRMI was definitely not going to catch. During the IA, it seemed the PR2 preferred to be at it's place post earthquake.
• DRMI post 2nd initial alignment was a great sucess!
• Lost lock at LOWNOISE_ESD_ETMX - saw a pitch ringup in what looks like CHARD.
• Next time around started to see a ringup again, but it calmed itself down after a minute. Made it to low noise at 0201UTC

TITLE: 10/31 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Earthquake
OUTGOING OPERATOR: Oli
CURRENT ENVIRONMENT:
    SEI_ENV state: USEISM
    Wind: 14mph Gusts, 9mph 3min avg
    Primary useism: 0.04 μm/s
    Secondary useism: 0.49 μm/s
QUICK SUMMARY: After the earthquake we started an initial alignment, in which Oli only had to adjust IM4. Now while locking PRMI, the POP90 signal isn't as high has before the earthquake when locked. Trending the PRs, PR2 is a little bit off in Y from before, so I've tried to keep PRC1 on and move PR2 slowly toward its old position while PRM followed. I'm not too sure this is helping at all though. Once this is done I'll try more DRMI.

TITLE: 10/30 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Earthquake
INCOMING OPERATOR: TJ
SHIFT SUMMARY: Trying to relock after the huge earthquake off the coast of Oregon knocked us down. Currently at PRMI_ASC and TJ is trying to touch up PRMI.
LOG:

14:30 In tail end of an initial alignment
14:33 Initial alignment done, relocking
15:26 NOMINAL_LOW_NOISE
15:26 Observing

15:49 Out of Observing to adjust CARM gain
    - Trying to recreate FSS glitches
15:50 Back into Observing
16:09 Lockloss

16:10 Initial alignment
    - SRC wouldn't lock and SRM tripped twice
16:37 Took us to DOWN and just started relocking
    - Lockloss from LOCKING_ARMS_GREEN
    - Lockloss from ACQUIRE_DRMI_1F
    - IMC took 15 minutes to relock. Relocked after I took it to OFFLINE for a few minutes
17:55 NOMINAL_LOW_NOISE
17:58 Observing

18:00 Out of Observing for PSL makeup air check
18:32 Back to Observing

20:17 Lockloss due to earthquake
    Tripped:
    - ISI HAM{2,3,4,5,7,8}, BS, {E,I}TM{X,Y}
    - SUS PR2, PR3, SR2, SRM, IM4, OFI, TMSY
21:45 Starting initial alignment
    - Could not get XARM IR due to the IMs having drifted during the earthquake. I trended them back with driftmon and adjusted sliders until they were back to where they were before the earthquake.
22:36 Initial alignment finished, relocking
    - Lockloss from PRMI_ASC x2
    - Lockloss from LOCKING_ARMS_GREEN
    - Lockloss from FIND_IR

Reran all O4b locklosses on the locklost tool with the most up to date code, which includes the "IMC" tag.

I used the lockloss tool data since the emergency OFI vent (back online ~23rd August) until today and did some excel wizardry (attached) to make the two attached plots, showing the number of locklosses per day tagged with IMC and without tag IMC. I made this plots for just locklosses from Observing and all NLN locklosses (can include commissioning/maintenance times), including:

• No IMC locklosses August to September 13th (Iain checked prior to August G2401576)
• Setpeber 12th: IMC gain redistribution 79998
• October 9th: IMC gain redistribution reverted 80566
• October 15th: NPRO current increased 80687
• Ocotber 18th: NPRO current reverted 80746

Our frequency noise and contrast defect (measured at the OMC) are lower than they ever have been in O4 (contrast defect, frequency noise). As a part of trying to track down what may have caused this reduction, I tracked the coherence of DARM with the LSC REFL A RIN, using that as a witness to frequency noise, as it was used in OAF for online frequency noise cleaning in O4a (source: 72276 and poking around the OAF screens).

I tried to pick times that were a few hours into the lock to avoid thermalization confusion. I also used only observing data.

There are a couple ideas about what could have caused this improvement. A short list:

• ring heater/power changes: we have stepped up and down the ring heaters by 0.1 W to avoid PIs, and briefly stepped up the input power by 1 W for the same reason
• AS WFS yaw offset: this was turned on due to an investigation in the O4 break, and then turned off after the OFI vent recovery
• OFI change: this doesn't help match the arms, but changes the mode matching to the OMC which could effect the contrast defect

These are the times I used. All of these times were at the same 60 W input power from the PSL:

Dec 10 2023: O4a time, no WFS offset in use, EY ring heater set to 1 W (1386244818)

Apr 11 2024: O4b time, before the OFI problems, WFS offset in use, EY ring heater set to 1 W (1396950886)

Jul 10 2024: O4b time, just before OFI repair vent, WFS offset in use, EY ring heater set to 1 W (1404656647)

Sept 14 2024: O4b, after OFI repair, WFS offset in use, EY ring heater set to 1 W (1410432000)

Sept 20 2024: O4b, after OFI repair, WFS offset set OFF, EY ring heater set to 1 W (1410855190)

Oct 22 2024: O4b, after OFI repair, WFS offset set OFF, EY ring heater set to 1.1 W (1413635762)

Here are some notes for the comparison of these times:

• the frequency noise coherence dropped significantly from O4a to O4b (red trace to blue trace)
• the frequency noise coherence increased again after the problems with the OFI (blue trace to green trace)
• the frequency noise coherence improved again after the OFI vent recovery (green trace to brown trace)
• the frequency noise coherence improved further after we turned off the WFS offset (brown trace to pink trace)
• the frequency noise coherence stayed the same when we changed the EY ring heater to 1.1 W (pink trace to light blue trace)

All of these O4 times show less coherence than the O4a time. Based on this data, it seems like the WFS offset did have an impact on the frequency noise. It also seems like the various vents with output port changes could affect the frequency noise, but the overall beam alignment in the arms could have changed during the vent. For example, we did adjust the camera offsets/ADS gains during the vent commissioning times. The change in frequency noise during the OFI problems (between April and July) could have a similar source, since we had to change a lot of alignment during that time. I'm not sure if any arm alignment was significantly changed though. Finally, it seems like this small ring heater change had no effect on the frequency noise.