We managed to track down the source of the 0.6 Hz oscillation that we see at move spots and full power. It seems to be the result of the new pitch and length estimators on PR3 and SR3 that were installed on Tuesday. We tested this by sitting at move spots (after the spots converged), and turning on and off the estimators for a few minutes at a time. When turning ON the estimators, the 0.6 Hz ringing is not immediately apparent- it will slowly grow. However, once the 0.6 Hz oscillation is visible in the controls, turning OFF the estimator immediately stops the ringing. Therefore, we think this is the culprit for this instability. Oli will be turning off the length and pitch estimators for PR3 and SR3 for now. The yaw estimators are still on because they have been operating stably for a long time and were not changed this Tuesday.
We have also put the factor of 2 increase of MICH2 gain into the guardian, since we kept forgetting to add it in. Several locklosses that we thought were due to various other things were actually due to the MICH loop ringing up.
Finally, we just had a lockloss after moving to camera servos because the 1 Hz oscillation began to grow. Here is everything I tried to stop the ringing (none of it worked):
- CSOFT P gain was already at 30. I dropped it to 25 and re-increased to 30 several times, no change either way
- INP1 P gain was at -1, I doubled it and then tripled it, no change
- INP1 Y gain was at -0.1, doubled it, no change
- DSOFT P gain was at 5, I increased to 10, then 15, no change
- DSOFT Y gain was at 5, increased to 10, no change
I then had an idea to go to high bandwidth ASC using the seismic script, but we lost lock as soon as I thought of that.
Next plan: go to lownoise length control, skip lownoise asc, go to the final spots and camera servos, stay in HBW ASC and wait for the 1 Hz ring up and see if we can ride it out this way.
Attached scope screenshot shows the 0.6 Hz oscillation disappearing as soon as Oli flipped the length and pitch estimator switches OFF.
Here's the measurements taken for MICH that show the phase margins at different points in the locking sequence
We have been thinking that maybe the MICH UGF is too low (I had a memory that it was supposed to be 8 Hz). Then I found this alog from Evan where he designed it to be 5.5 Hz and the phase margin looks about the same 76631. Note that Oli's measurement was taken after we DOUBLED the gain, so somewhere along the way we lost 6 dB of gain.
I held us at high bandwidth ASC for 9 minutes and there was no 1 Hz oscillation. A few minutes after running lownoise ASC the 1 Hz oscillation came back. It begins first in DHARD P. I stopped the oscillation by running the ASC HI GAIN script that we set up for earthquakes. It looks like the oscillation stopped as soon as DHARD P was put into HBW state. After waiting a few minutes, I put us back into the lownoise state. I have previously been suspicious of DHARD P for being the culprit causing the 1 Hz ring up. My recommendation is that if we see this again, try turning OFF FM8 in the DHARD P filter module (labelled GVBoost). If that doesn't work, just run the HI GAIN ASC script like you would for an earthquake, on the IS sensor config screen.
EDIT: I posted this comment too soon- the 1 Hz ringing came back. I tried just turning off DHARD P FM8, but that did nothing. However, going to Hi Gain ASC worked again, so that seems to be a good way to get around this (for now).
I've been flipping back and forth between high and low bandwidth DHARD P. The 1 Hz definitely begins when we are in LBW, and it seems to actually be the lowpass/reshaping in FM4 that is the issue.
I also noticed that there is a glitch in DARM whenever FM4 is turned on or off, so I increased the filter ramp time from 15 to 20 seconds.
To take DHARD P to HBW, disengage FM8, wait for the filter to ramp, then disengage FM4.
To take DHARD P to LBW, engage FM4, wait for the filter to ramp, then engage FM8.
I tried to measure DHARD P to understand what the instability is but we had a lockloss. Looks like it's an ETM glitch?
My recommendation is to put DHARD P in HBW if the 1 Hz starts oscillating again.