Jeff, Ryan S, Oli
During last week's set of issues, something that we saw happen a few times during our lock reacquisition attempts were lots of EY saturations while going through LOWNOISE_COIL_DRIVERS/TRANSITION_FROM_ETMX. The saturations would stop once L2 to R0 damping was turned off (87713), so it seemed like the issue was with the ETMY L2 satamp, so we swapped it out with a different one (87722). We didn't see any of these repeating saturations after that, but also we were changing a lot of things at the time trying to figure out the problem, plus we hadn't been seeing these saturations during every single relock attempt.
The DAC channels that were showing saturations were from DAC1 channels 1 and 2. Checking the model, those channels line up with R0 F2 and F3, which are the channels that control Length on R0. We plotted R0's MASTER OUTs during times where we saw lots of the EY saturations, and at times where the saturations heard on verbals were normal, including a time before we swapped the ETMY L2 satamp on October 14th.
Here's a breakdown of the different examples we looked at:
| Normal amount of saturations during LOWNOISE_COIL_DRIVERS / TRANSITION_FROM_ETMX | |||
| Date | SatAmp | verbals | ndscope |
| Oct 1 | unmodified | oct1_verbals | oct1_ndscope |
| Oct 20 | modified | oct20_verbals | oct20_ndscope |
| Oct 22 | modified | oct22_verbals | oct22_ndscope |
| Lots of EY saturations during LOWNOISE_COIL_DRIVERS / TRANSITION_FROM_ETMX | |||
| Oct 23 | modified | oct23_verbals | oct23_ndscope oct23_ndscope_zoomout |
| Oct 24 | modified | oct24_verbals | oct24_ndscope |
We saw that for the times where the amount of saturations were what we consider 'normal', the LOWNOISE_COIL_DRIVERS/TRANSITION_FROM_ETMX states behave similarly in the R0 MASTER OUT channels, including after swapping the satamp. The OSEMs will see some movement, but it's not too far outside of where they usually sit. However, for the two times that we checked where we had the excessive EY saturations, we saw that right before they started, there was a high frequency glitch seen in the ETMY L2 Length witness channel. This glitch only moved L2 a small amount, about 0.5 um, but it was causing R0 to move a lot in Length, saturating or nearly saturating for a long time.
Plotting the impulse response of the SUS-ETMY_L2_R0DAMP_L filter bank, we see that these filters have an impulse response time of ~16 seconds, and breaking down the impulse response by each filter's contribution, we see that the FM8 (module 7) filter module, invPsmoo, has a wild impulse response. Because of this filter module, the impulse response of the entire R0_L2DAMP_L filter bank is extremely long, and the signal is very large. The frequency response plot for module 7 shows us that it approaches the 10^15 gain at higher frequencies. Additionally, these new satamps have about double the gain at high frequencies as compared to the old satamps, so that would also be exacerbating any issues at higher frequencies.
With all that said, it looks like the conclusion is that the EY saturation issues from last week were not caused by a faulty satamp, but instead by something else that caused L2 to glitch, and the long impulse response and high gain causing R0 to take forever to calm down.
A temporary solution would be to keep the L2 to R0 damping off during locking until after LOWNOISE_LENGTH_CONTROL has finished, to make sure that we are avoiding having it on during all the sudden movements that could upset R0.
I am confused about the conclusions of this alog. Attached is a screenshot of the last time we had these saturations before the satamp was replaced. I did a test where I turned off the R0 tracking loop (ETMY L2->R0 length) by ramping the gain of the loop to zero on a 15 second ramp. The saturations stopped. I then ramped back on and saw the saturations return. I waited seven minutes and tried ramping on again and got the same saturation warnings. This test was done while we sat in state 560, which is lownoise_length_control. The state had completed and we held there in order to track down the saturations.
I can see the glitch that Jeff and Oli found in this alog, but I don't see any other glitches that caused the subsequent saturations when I was turning the gain on and off. The ramp time should be long enough to avoid any sort of issues with the impulse response, and the on/off test happened many minutes after the noted glitch, so I don't think they can be explained by this impulse response issue.
I don't necessarily think this indicates the satamp is the problem, except that we haven't had these saturations since the replacement, and this loop has been running for a long time without issue (my understanding is since O3b, but I don't know for certain).
I agree that a good way to avoid this issue is to engage the R0 tracking later on in the guardian.
