[Jenne, TJ, Elenna]
The last three locks saw a violin mode ring up (see 87933), which managed to saturate the DCPDs and cause a large amount of excess noise, see this screenshot of DARM near the time of lockloss. I am assuming that all of the extra noise is a result of some crazy nonlinearity due to the saturation DCPDs.
Jenne and I have concluded that the mode that rung up must be ETMX mode 1, which the violin mode monitor screen reports is at f = 511.2993 Hz. However, we have looked at GDS calib strain when the mode has rung up using both diaggui_test and with exported data in python at double precision with fft length of 4096 and 1024 seconds respectively and find that the mode is at 511.269 Hz, DTT screenshot, matplotlib screenshot.
Jenne has plotted both the damping filter and the monitor filters here. We can see that the damping filter is broad enough to cover this frequency, but the monitor filter is not. The violin mode guardian is supposed to track the modes via the monitor filters and disengage damping if the mode is ringing up. However, we think the monitor filter might be missing this frequency and this is why the damping did not turn off.
The attached timeseries also shows that the DCPDs were railed for some time, the suspension drive on ETMX L2 was very nearly saturated, and the monitor filter level was low overall, near 3.5. Jenne has included channel H1:FEC-88_ACCUM_OVERFLOW, maybe this is a useful channel for us to monitor.
I just want to add that I am being specific about how we measured the mode frequency above because we have been confused by insufficient precision before. I think we have sufficient precision to measure this frequency accurately, but the results show a shift of 0.02 Hz from the previously reported value.
Here is a follow up on the mode frequency that is very confusing! The ETMX mode 1 and mode 6 damping was disengaged about 20 minutes before our last lockloss. If I plot the strain spectrum in the 20 minutes before the damping was off and then the 20 minutes after, I see the expected ETMX mode 1 and 6 peaks at 511.18 Hz and 511.2993 Hz Hz. Only in the spectrum with the damping on do I see a large peak at 511.27 Hz. Jenne and I don't understand that yet, but are thinking about it. Plotted in diaggui_test with fft length 512 seconds, 2 averages with 50% overlap in each measurement.
The attached plot shows ETMX mode 1 and 6 damping ON in red and ETMX mode 1 and 6 damping OFF in blue.
However, the mode 1 filter is more likely to be the culprit, since it actually covers 511.27 Hz whereas the mode 6 filter does not, filter screenshot.
And, here is a version of the spectrum over time. Probably the 'true' ETMX mode 1 is the one closer to 511.3 Hz, which is consistent with the filters as well as Rahul's violin mode identification spreadsheet, and the 511.269 Hz that we've had trouble with today is something else that is outside the width of the monitor filter, but inside the width of the damping filter. As Elenna's alog shows, it's not really clear what the filters are picking up to drive at 511.269 Hz, but probably the damping filter was set up for 511.3 Hz and so when being applied to something at 511.269 Hz it just feeds nonsense.
In the attachment, I show strain spectra from 3 different times when the alog has mentioned violin modes at around 511 Hz. Green is today (after we were saturating, hence the elevated background), brown is this Oct 2025, and pink is Aug 2023. All of the times have a small peak at 511.302 Hz (which is not so far from the ETMX mode1 monitor and damping filter central frequecy of 511.299 Hz), so that's probably the true violin mode. Oct and Nov have also this much larger peak at 511.269 Hz, which is this new confusing peak. I haven't looked at any times other than these 3, so it's possible that the 511.269 Hz has come and gone before Oct 2025.
Also shown is the ETMX mode 1 monitor and damping filters, showing that as Elenna said the monitor filter seems to only catch the 'true' mode, while the damping filter is broad enough for both of these frequencies.
