Sheila, TJ, Oli, Matt
We reverted the ASC gain changes (87582) setting INP1 P to 1 and CSOFT P to 25 at MAX power (rather than 30). Most of our locklosses overnight were from states before Max power, so the CSOFT P gain can't have been the problem, but we are undoing it for now just for the sake of reverting recent changes.
We looked at one of the overnight locklosses, it seems that MICH is ringing up at 4.7 Hz. Since Oli is sitting at 2W DCreadout to damp violin modes, we measured the MICH OLG there, and we see it is pretty close to unstable at 5Hz, with low coherence there. We added a factor of 2 gain in LSC MICH2, to bring the UGF to 10 Hz similar to the references, and give us 35 degrees phase margin.
Next we checked the SRCL UGF, which is also locked on POP45, and it's gain looks low by a factor of 5 compared to the reference. This loop has a large phase bubble so it's still stable. POP 45 whitening gain and filters seem to be the same as a few days ago, and the POP LF power is the same as it was a few days ago at 2W. PRCL gain also looks low compared to the reference, but is stable.
We were able to get to OMC_WHITENING (almost to low noise), but while we were sitting there we had the old ring up. We tried manually increasing the CSOFT P gain to 30, but the osciallation had already grown large and we lost lock anyway. I've now put the csoft P gain back to 30 in max power in the guardian.
Following the reversion of gains in lscparams.py and the reload of ISC_LOCK, I also reloaded the nodes which import this file but do not use the modified lines in order to green-up GRD-CFC. Nodes loaded are:
ALIGN_IFO,ALS_COMM,ALS_DIFF,ALS_XARM,ALS_YARM,CAMERA_SERVO,H1_MANAGER,IMC_LOCK,INIT_ALIGN,ISC_DRMI,LASER_PWR,LOCKLOSS_SHUTTER_CHECK,OMC_LOCK,SEI_CONF,SEI_ENV,SQZ_FC,SQZ_MANAGER,SUS_CHARGE,TCS_ITMX_CO2_PWR,TCS_ITMY_CO2_PWR,TEST,THERMALIZATION,TMS_SERVO,VIOLIN_DAMPING
The ringup and lockloss in OMC_WHITENING was indeed the usual 1 Hz ringup
We've had a lot of 0.6Hz wobbling in the move spots state, so Elenna made the move spots a2l gains different from final (undoing a change from this spring), and made the move spots ones the same a2l gains as they were before the camera bump (87473)
Attached is a comparison of the build ups in the move spots states before and after this change, things look about the same, so this hasn't fixed the wobbles.
This isn't our main locking problem today, it doesn't seem to be causing our locklosses and gets better as we power up.