TITLE: 09/14 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
INCOMING OPERATOR: None
SHIFT SUMMARY:
IFO is in IDLE and CORRECTIVE MAINTENANCE
The "good" news:
DHARD P loop has been successfully automatic since closing it this afternoon. We can lock all the way to CARM_5_PICOMETERS quite quickly. DRMI has also been good to me this evening.
The bad news:
We thought we were losing lock at RESONANCE, but we were really losing lock at CARM_5_PICOMETERS.
Stepping through this state has showed that its only at the end of this, where the matrix involving REFL_BIAS, TR_REFL9 and TR_CARM is loaded, we lose lock roughly 10s later due to SRM becoming unstable quickly.
I investigated this via alogs 86909, 86910, 86911, 86912, which are comments to my OPS shift start (alog 86908). After being led down a rabbit hole of ramp times, from 3 other times where that was the problem, I can confirm that this isn't it. Curiously, lock was lost faster with a longer ramp time.
With confirmation from TJ and Jenne, I'm leaving IFO in IDLE with the plan being to solve this problem in tomorrow's shift.
I do feel like we're close to solving this problem or at least figuring out where the issue lies. Specific details are in the alogs listed below.
LOG:
None
Writing this before I attempt to look at ISC_library but:
Our last stable state took us to CARM_5_PICOMETERS. I stayed there just to be sure it was stable.
I then followed the procedure and stepped through the next state, CARM_TO_REFL.
I will begin investigating what I can but I think this an issue with loading the matrix from ISC_Library, implying something is up with REFLBIAS or TR_REFL9? Somewhat out of my depth but will investigate alogs, wiki, ISC_library and anything else I can.
Either way, it seems that we thought we were losing at RESONANCE but it was the CARM_TO_REFL's last line, unless I got something wrong in the if-loop I didn't run through, which is also something I will investigate.
In looking at the problem lines loaded by the matrix, I see REFL_BIAS, TR_REFL9 and TR_CARM.
According to ISC_library.py, these are REFL_BIAS=9, TR_REFL9 = 27 and TR_CARM = 26. I don't know what this means yet.
Then, I looked at alogs hoping to find the last time something like this happened, which from vague memory was during post-vent recovery this year (with Craig and Georgia):
Looking for REFL_BIAS, I found Sheila's 2018 alog 43623 talking about the TR_CARM transition and how SRM would saturate first, which was what I was experiencing. Specifically the line is: TR_CARM transition: This has been unreliable for the last several days, our most frequent reason for failure in locking. We found that the problem was a large transient when the REFL_BIAS (TR CARM) path is first engaged would saturate SRM M1. We looked at the old version of the guardian and found that this used to be ramped on over 2 seconds, but has been ramped over 0.1 seconds recently. This transition has been sucsesfull both of the times we tried it since changing the ramp time, in one try there was no transient and in the other there was a transient but we stayed locked.
Looking for TR_REFL9, I found Craig's alog 84679 from the aforementioned post-vent recovery. Same issue with ramp time too, specifically referencing the few lines of ISC_LOCK that I posted above. He moved the ramp time to 2s, which I confirmed is still there. He has a picture of the lockloss (attached below) and it looks very similar to the ones we have been having. I trended the same channels he did and found that after this 2s ramp time, PR_GAIN would become unstable after 10s (attached below). Also found Elenna's alog (where I was also on-shift) dealing with the same issue, but before Craig had increased the ramp time - alog 84569. The deja vu was real. Buut I'm unsure this is the same issue because the ramp time is indeed 2s.
Looking for TR_CARM, I found this recent alog from Elenna discussing CARM, REFL and DHARD WFs - unknown whether it is relevant. Alog 86469. I will read this more closely.
While the I'm being led to just increase the ramp time further, I truly doubt this will change much since 2s is quite large of a ramp time and doesn't really explain much. Given all that, I'm going to try it before looking further since we've gotten back here in the time it took to write and investigate this.
As expected, that did not fix it, rather it seemed as though the faster ramp time caused a weird offset in REFLAIR9, which didn't happen last time. I checked the log and confirmed that there was a 4s instead of 2s ramp time but I'm kind of struggling to see it (attached).
There seems to be a 25hz signal on POP_A as soon as I increased the ramp time. Additionally, there wasn't really an instability upon turning on CARM_TO_REFL. It was stable for 9s until LL, showing a kick 3s before rather than a degradation. Additionally, the lockloss wasn't as violent.
With the tiniest font, maybe an even longer ramp time would work? Very unconvincing.
I don't have many ideas right now, but I'm reading the alog, the wiki and then devising some plan based off an idea of how this part of lock works.
I found an alog from Craig about REFLAIR9 (alog 43784). I'm somewhat more convinced that something involving the REFLAIR9_OFFSET and POP_LF?
The line from the alog: "we have made it through this transition every time now, but the PR_GAIN number on the striptool still spikes, meaning the POP_LF power changes rapidly through this transition, which is probably bad." sounds relevant?
I'm still confused why the REFLAIR9_OFFSET turning on causes an instability in PR_GAIN, which causes an instability in SRM which causes a lockloss.
I'm sufficiently confused at this stage so will attempt to go through the usual lockloss again to see if it's the same. Then I'll try stepping through again as a sanity check.
So far, I've only changed ramp times, immediately changing them back.
One thing that's confusing me is what the "REFL_BIAS" means. The REFL_BIAS Gain is 86 but the REFL_BIAS value in ISC_Library is 9.Nevermind, I think one refers to the location within a matrix in terms of the element number whereas the other is the value of that number within the matrix.