After some digging, I found that this is not a new issue and was commented on briefly earlier today.
Alog 87625 shows a near-identical screenshot to the attached MICH LSC ringup, which Oli was experiencing during their DAY shift and no doubt Ryan C also experienced during EVE.
In this alog Sheila noted: "This change should have made a 5% change in the loop gain, so if this was the reason for the lockloss one of our loops is very close to unstable."
I believe since we've lost lock around here for the fifth time now, with all of them showing this ringup, that one of our loops is probably very close to unstable.
What's weird is that from the wording of that alog - the power scaling reversion should have fixed this issue - but it didn't?
So either the power scaling change reversion wasn't loaded/done OR this is an entirely seperate issue.
Well, I checked and it seems this was indeed reverted and we're scaling based on IMC_PWR_IN_OUTMON.
Maybe this is a seperate loop instability issue or a microseism issue or a combination then..
H1 Lockloss Tool is Tagging ADS Excursion on all of these, if it's relevant.
I reloaded the guardian at the next Lockloss (which just happened in the same place). It was indeed reloaded prior though at 15:21 (with the first ringup happening at 15:18, and the alog reporting reversion at 15:53).
Thus, this seems to be an unrelated LSC Loop instability.
I will keep investigating.
Well, it's been two locklosses now and the same thing is happening - The only changes I can think of are the gain changes made to INP P (-1 to -2) and CSOFT P (25 to 30) gains (alog 87582) that were put there to avoid the 1Hz ringup after NLN. I frankly have no idea how these would affect powerup. I trended the gains with the ringup but did not find that they happened during a gain change or anything of the sort so I'm doubting this. Trended the values of CSOFT_P and INP_P with the ring-up and they don't seem to be affecting it other than CSOFT_P sees it later on (as expected).
Just checked to confirm the first time this issue has happened was after maintenance and indeed it was. The lock acquisition before the CSOFT P and INP 1 gains were changed did not see even a small ringup in the MICH.
What I can also confirm is that it's a 3.5Hz ringup and it seems to happen during powerup to 25W and not always in the same state. Sometimes it's at MOVE_SPOTS and sometimes it's after. I then checked whether it happens a set time after powerup or sometime spontaneously after 25W node. Sometimes it happens 8 minutes after POWER_25W; sometimes 10 minutes and other times 13 minutes after 25W.
Currently out of ideas. Letting H1 go automatic. Either way microseism is quite high. If I get called back, the next step is to step through POWER_25W or wait at POWER_10W and see if the ringup eventually happens.
Attached trends of the ringup at different times in last 24 hrs as well as the last time it didn't happen.
