TITLE: 09/13 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
OUTGOING OPERATOR: Ryan S
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 17mph Gusts, 11mph 3min avg
Primary useism: 0.02 μm/s
Secondary useism: 0.15 μm/s
QUICK SUMMARY:
IFO is DOWN in CORRECTIVE MAINTENANCE
There was a lot of good work done today. Primarily rephasing REFLAIR (alog 86903) and closing DHARD_P loop. This has allowed us to easily get past DRMI. It seems that we are still losing lock around resonance due to a 35hz fast ringup.
The Procedure for the EVE:
- Try DHARD_WFS and step to resonance (state before lockloss). If DHARD_WFS doesn't woprk,
- comment out line 2430, 2404 of ISC_LOCK.py
- Then go to CARM_5_PICOMETERS and turn DHARD_P Gain to -1, then -10. This is slow
- If DHARD_WFS works, keep going and step through states by:
- once at previous state of the one you want to step
- "guardian -i ISC_LOCK"
- Copy the ezca lines
- copt intrix load lines
- do NOT copy "self.lines" or the library - that's done by the guardian -i
- Copy line by line and see it work. If state is done, MANUAL to state AFTER desired state
What has happened so far:
Same PR_G ringup caused lockloss at RESONANCE and CARM_TO_REFL so I'm going to the state before, which is CARM_5_PICOMETERS. Will follow the blueprint above. Thankfully DHARD_P worked 2x now.
Writing this before I attempt to look at ISC_library but:
Our last stable state took us to CARM_5_PICOMETERS. I stayed there just to be sure it was stable.
I then followed the procedure and stepped through the next state, CARM_TO_REFL.
I will begin investigating what I can but I think this an issue with loading the matrix from ISC_Library, implying something is up with REFLBIAS or TR_REFL9? Somewhat out of my depth but will investigate alogs, wiki, ISC_library and anything else I can.
Either way, it seems that we thought we were losing at RESONANCE but it was the CARM_TO_REFL's last line, unless I got something wrong in the if-loop I didn't run through, which is also something I will investigate.
In looking at the problem lines loaded by the matrix, I see REFL_BIAS, TR_REFL9 and TR_CARM.
According to ISC_library.py, these are REFL_BIAS=9, TR_REFL9 = 27 and TR_CARM = 26. I don't know what this means yet.
Then, I looked at alogs hoping to find the last time something like this happened, which from vague memory was during post-vent recovery this year (with Craig and Georgia):
Looking for REFL_BIAS, I found Sheila's 2018 alog 43623 talking about the TR_CARM transition and how SRM would saturate first, which was what I was experiencing. Specifically the line is: TR_CARM transition: This has been unreliable for the last several days, our most frequent reason for failure in locking. We found that the problem was a large transient when the REFL_BIAS (TR CARM) path is first engaged would saturate SRM M1. We looked at the old version of the guardian and found that this used to be ramped on over 2 seconds, but has been ramped over 0.1 seconds recently. This transition has been sucsesfull both of the times we tried it since changing the ramp time, in one try there was no transient and in the other there was a transient but we stayed locked.
Looking for TR_REFL9, I found Craig's alog 84679 from the aforementioned post-vent recovery. Same issue with ramp time too, specifically referencing the few lines of ISC_LOCK that I posted above. He moved the ramp time to 2s, which I confirmed is still there. He has a picture of the lockloss (attached below) and it looks very similar to the ones we have been having. I trended the same channels he did and found that after this 2s ramp time, PR_GAIN would become unstable after 10s (attached below). Also found Elenna's alog (where I was also on-shift) dealing with the same issue, but before Craig had increased the ramp time - alog 84569. The deja vu was real. Buut I'm unsure this is the same issue because the ramp time is indeed 2s.
Looking for TR_CARM, I found this recent alog from Elenna discussing CARM, REFL and DHARD WFs - unknown whether it is relevant. Alog 86469. I will read this more closely.
While the I'm being led to just increase the ramp time further, I truly doubt this will change much since 2s is quite large of a ramp time and doesn't really explain much. Given all that, I'm going to try it before looking further since we've gotten back here in the time it took to write and investigate this.
As expected, that did not fix it, rather it seemed as though the faster ramp time caused a weird offset in REFLAIR9, which didn't happen last time. I checked the log and confirmed that there was a 4s instead of 2s ramp time but I'm kind of struggling to see it (attached).
There seems to be a 25hz signal on POP_A as soon as I increased the ramp time. Additionally, there wasn't really an instability upon turning on CARM_TO_REFL. It was stable for 9s until LL, showing a kick 3s before rather than a degradation. Additionally, the lockloss wasn't as violent.
With the tiniest font, maybe an even longer ramp time would work? Very unconvincing.
I don't have many ideas right now, but I'm reading the alog, the wiki and then devising some plan based off an idea of how this part of lock works.
I found an alog from Craig about REFLAIR9 (alog 43784). I'm somewhat more convinced that something involving the REFLAIR9_OFFSET and POP_LF?
The line from the alog: "we have made it through this transition every time now, but the PR_GAIN number on the striptool still spikes, meaning the POP_LF power changes rapidly through this transition, which is probably bad." sounds relevant?
I'm still confused why the REFLAIR9_OFFSET turning on causes an instability in PR_GAIN, which causes an instability in SRM which causes a lockloss.
I'm sufficiently confused at this stage so will attempt to go through the usual lockloss again to see if it's the same. Then I'll try stepping through again as a sanity check.
So far, I've only changed ramp times, immediately changing them back.
One thing that's confusing me is what the "REFL_BIAS" means. The REFL_BIAS Gain is 86 but the REFL_BIAS value in ISC_Library is 9.Nevermind, I think one refers to the location within a matrix in terms of the element number whereas the other is the value of that number within the matrix.