I can't find the right alog for directions, so here is an easily searchable set of directions for HAM1 FF.

There is a master switch for the HAM1 feedforward, but it can sometimes cause problems to slam it on and off that way. Instead, you can ramp the input to the feedforward down to zero. From sitemap:

SEI > ISI Sensor Config > [middle of the screen, see attachment] HAM1 ASC FF > L4CINF

This opens a filter bank page with four filter banks. They each have a gain of 1 and a ramp time of 20 seconds. Set all of these gains to zero to turn off the input to the feedforward. Ramp them back to 1 to engage.

As several of us just talked about in the control room, if The Noise is happening when folks are on site and there's a plan of a thing to try, please feel free to drop Observing to check.  I think we'll keep the list of things to try elsewhere more dynamic than the alog (probably the LHO commissioning google doc).  Some examples of things that we're thinking of right now are (a) turning off the HAM1 FF, or (b) walking (gently) around electronics racks (CER, EX ESD driver area) to see if we can hear any electronics 'whining' or otherwise going bad.

Please do send me a mattermost message, which should audibly ping my phone, but this is causing such problems for our data quality that there is no need to wait for a response from me before trying something.

TITLE: 12/12 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 155Mpc
OUTGOING OPERATOR: Ryan C
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 4mph Gusts, 2mph 3min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.28 μm/s
QUICK SUMMARY: H1 has been observing for 23 hours. The vac prep lab dust monitor is reporting as disconnected.

This alog presents the first steps I am taking into answering the question: "what is the calibrated residual test mass motion from the ASC?"

As a reminder, the arm alignment control is run in the common/differential, hard/soft basis, so we have eight total loops governing the angular motion of the test masses: pitch and yaw for differential hard/soft and common hard/soft. These degrees of freedom are diagonalized in actuation via the ASC drive matrix. The signals from each of these ASC degrees of freedom are then sent to each of the four test masses, where the signal is fed "up" from the input of the TST ISC filter banks through the PUM/UIM/TOP locking filters banks (I annotated this screenshot of the ITM suspension medm for visualization). No pitch or yaw actuation is sent to the TST or UIM stages at Hanford. The ASC drive to the pum is filtered through some notches/bandstops for various suspension modes. The ASC drive to the TOP acquires all of these notches and bandstops and an additional integrator and low pass filter, meaning that the top mass actuates in angle at very low frequency only (sub 0.5 Hz).

Taking this all into account involves a lot of work, so to just get something off the ground, I am only thinking about ASC drive to the PUM in this post. With a little more time, I can incorporate the drive to the top mass stage as well. Thinking only about the PUM makes this a "simple" problem:

• we know exactly what the ASC drive is in counts- we just measure the "H1:ASC-[DOF]_[P/Y]_OUT_DQ" channel
• We propagate drive that to each test mass knowing the drive matrix we have set
  • screenshots of the pitch matrix and yaw matrix
• We calibrate this from drive counts to Newton*meters knowing the drive calibration from G1100968
  • for PUM: 40 V /2**20 ct [DAC] * 0.268 mA / V [drive strength] * 0.0309 N / A [force coeff] * 70.7 mm [lever arm], pit/yaw lever arm is the same for the PUM
  • I conveniently chose a time from before the ETMX DAC change to avoid thinking too hard about the DAC counts
• We transform this N*m PUM drive to radians of test mass motion using the PUM-TST state space suspension model transfer function, which is calibrated into real units
  • I did all of this in python, so I used Kevin's exported quad state space model, found here: T2300299
  • I used the damped model which applies the ETMX top mass damping filters

I have done just this to achieve the four plots I have attached to this alog. These plots show the ITM and ETM test mass motion in rad/rtHz from each degree of freedom and the overall radian RMS value. That is, each trace is showing you exactly how much radians of motion each ASC degree of freedom is sending to the test mass through the PUM drive. The drive matrix value is the same in magnitude for each ITM and each ETM, meaning that the "ITM" plot is true for both ITMX and ITMY (the drives might differ by an overall sign though).

Since I am just looking at the PUM, I also didn't include the drive notches. Once I add in the top mass drive, I will make sure I capture the various drive filters properly.

Some commentary: These plots make it very evident how different the drive is from each ASC degree of freedom. This is confusing because in principle we know that the "HARD" and "SOFT" plants are the same for common and differential, and could use the same control design. However, we know that the sensor noise at the REFL WFS, which controls CHARD, is different than the sensor noise at the AS WFS that control DHARD, so even with exact same controller, we would see different overall drives. We also know that we don't use the same control design for each DOF, due to the sensor noise limitations and also the randomness of commissioning that has us updating each ASC controller at different times for different reasons. For example, the soft loops both run on the TMS QPDs, but still have different drive levels.

Some action items: besides continuing the process of getting all the drives from all stages properly calibrated, we can start thinking again about our ASC design and how to improve it. I think two standout items are the SOFT P and CHARD Y noise above 10 Hz on these plots. Also, the fact that the overall RMS from each loop varies is something that warrants more investigation. I think this is probably related to the differing control designs, sensor noise, and noise from things like HAM1 motion or PR3 bosems. So, one thing I can do is project the PR3 damping noise that we think dominates the REFL WFS RMS into test mass motion.

As part of a different investigation (still trying to understand why our range goes bad sometimes), I incidentally may have found a source for some glitches / range drops. 

I'm not going to look further into this, but instead tag detchar-request in hopes that someone else has some time to think about it.  I'll also directly send messages to Jim and Elenna, who manage this feedforward system.

In the attached screenshot there are a few channels that include 'TTL4C' - those are the tabletop L4C seismic sensors on HAM1.  There are also 'FFHAM1' channels that take a channel derived from those L4Cs, and feed it forward to the error signal of the ASC loop that is referred to in the name.  A moment or so after there is a glitch in those channels, there is a range drop in DARM. 

I will say that when I zoom in, it looks like the glitch appears in the FFHAM1 channel before it appears in the TTL4C channel, but based on my understanding of the signal flow, I'm not entirely sure how that's possible.  I'm hoping that Elenna (who knows much more than I do) can help think this through.

The broadband glitching that was present in the early hours of Dec 11 (UTC) appears to have suddenly and entirely stopped at 10:36:30 UTC - this sharp feature can be seen in the daily range plot. I completed a series of LASSO investigations around this time in the hopes that such a sharp feature would make it easier for LASSO to identify correlations. I find a number of trend channels that have drastic changes at the same time as this turn-off point related to TCS-ITMY_CO2, ALS-Y_WFS, and SUS-MC3_M1.

The runs I completed are linked here: 

1. LASSO run of the first half of Dec 11 with the sensemon range as the primary channel 
2. LASSO run of times near the turn-off point with TCS-ITMY_CO2 as the primary channel 
3. LASSO run of times near the turn-off point with the sensemon range as the primary channel

Run #1 was a generic run of LASSO in the hopes of identifying a correlation. While no channel was highlighted as strongly correlated to the entire time period, this run does identify  H1:TCS-ITMY_CO2_QPD_B_SEG2_OUTPUT (rank 11) and H1:TCS-ITMY_CO2_QPD_B_SEG2_INMON (rank 15) as having a drastic feature at the turn-off point (example figure). Based on this information, I launched targeted runs #2 and #3. 

Run #2 is a run of LASSO using H1:TCS-ITMY_CO2_QPD_B_SEG2_OUTPUT as the primary channel to correlate against. This was designed to identify any additional channels that may show a drastic change in behavior at the same time. Channels of interest from this run include H1:ALS-Y_WFS_B_DC_SEG3_OUT16 (example figure) and H1:ALS-Y_WFS_B_DC_MTRX_Y_OUTMON (example figure). SEISMON channels were also found to be correlated, but this is likely a coincidence. 

Run #3 targets the same turn-on point, but with the standard sensemon range as the primary channel. This run revealed an additional channel with a change in behavior at the time of interest, H1:SUS-MC3_M1_DAMP_P_INMON (example figure). 

Based on these runs, the TCS-IMTY_CO2 and ALS-Y_WFS channels are the best leads for additional investigations into the source of this glitching. 

TITLE: 12/11 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 156Mpc
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 5mph Gusts, 3mph 3min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.32 μm/s
QUICK SUMMARY:

• We've been locked for 14:20, in observing since 03:33
• 2ndary microseism is decreasing, its just below the 90th percentile
• Low wind

TITLE: 12/11 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 102Mpc
INCOMING OPERATOR: TJ
SHIFT SUMMARY: Currently Observing and have been Locked for almost 5 hours.  Our range is still all over the place unfortunately. I jumped in and out of Observing a few times by turning squeezing on and off to check for differences with the range (81753), but didn't find anything.
LOG:

00:30 Relocking
01:08 NOMINAL_LOW_NOISE
01:14 Observing

    02:57 Went out of Observing and turned off SQZ to see if that fixes the mystery noise 
    03:09 Back to FDS
    03:19 Turned off SQZ
    03:33 Back to FDS and back to Observing

[Jason, Masayuki]

Summary

The PMC heater calibration was performed last week (Tuesday). The calibration involved adjusting the temperature loop set point and monitoring the corresponding changes in PMC temperature and length. The results were validated using previous measurements and compared to similar evaluations at LLO. Additionally, the necessity of the heater for JAC operations was assessed, highlighting it would be needed for long term operation.

Details

PMC Heater Calibration

• The calibration of the PMC heater was conducted. The PMC remained locked while the temperature loop set point was adjusted from the usual 3.7V to 1.7V. This adjustment corresponds to a PMC length change of 0.30 μm based on a prior measurement (alog entry 81385).
• A PMC temperature change of 0.023 K was recorded using the thermometer. From this result, the thermal expansion coefficient was calculated as:

  0.30 μm / 0.023 K = 13 μm/K

  This result is consistent with the thermal expansion coefficient of aluminum, 22e-6/K, and the approximate PMC length of 0.5 m.
• No significant changes in heater output were detected before and after the set point adjustment.
• First Plot Description:
  • Top panel: PZT input calibrated for PMC length.
  • Middle panel: PMC temperature measured with the thermometer.
  • Bottom panel: Heater output, with histograms shown for the middle and bottom panels.

Reference to LLO Calibration

• To further calibrate the PMC heater, the LLO heater calibration (alog entry 51467, attached plot) was used as a reference.
• At LLO, heater input changes from 2.5V to 4V resulted in PMC temperature changes from 292.2 K to 291.125 K, indicating a heater efficiency of \(0.67 \, \text{K}/\text{V}\). Using this efficiency, the PMC expansion rate per heater voltage was calculated as:

  13 μm/K × 0.67 K/V = 8.7 μm/V

Monthly Drift Analysis

• With this calibration, the monthly heater output was evaluated.
• Second Plot Description:
  • Top panel: Heater-induced temperature actuation (left axis) and the corresponding PMC expansion (right axis).
  • Bottom panel: Required PZT input for equivalent actuation without the heater.
• Periodic large spikes were observed daily. However, since each spike occurred within approximately 10 seconds, as shown in the attached screenshot, these spikes are not attributed to actual temperature changes. A moving median (red line) was applied to highlight the spikes. Note that this moving median is not an ideal representation, as the extracted data is already averaged over 1-minute intervals. However, it is still useful for illustrating the spikes.
• Despite the spikes, the heater compensated for a length change equivalent to approximately 500 V in PZT input is typical day-time drift, but sometime, it can drift more than 1000V in a day.
• A 10-day analysis of LLO PMC showed no spikes, and the results were consistent with LHO data. Result is shown in the third plot as same structure as the second plot.

Implications for JAC Operations

The PZT can be railed in a day from this measurement, so we would need the heater for JAC operation.

Additional Observations

Spikes observed at LHO caused glitches in transmitted power, as shown in the attached plot, and may require resolution. Redesigning the filters could potentially mitigate these anomalies.

Just went out of Observing and took SQZ manager to no squeezing to check if the noise issues are related to squeezing. We'll be doing no squeezing for 10mins, SQZ for 10, no sqz for 10, and then back to sqzing and Observing

As part of WP 12239 we moved h0epics, cdsvmscript1, epics-burt, autoburt off of the cds0proxmox hypervisor.  This resulted in a few minutes of down time for the dust monitor IOC around 8:19am localtime.

After this we done we were able to look at cdsproxmox.  Its boot drive had failed.  After some help from Fil we got the drives replaced and reinstalled proxmox ve on cdsproxmox.  We adjusted DNS and renamed the box cdsproxmox0.

Some notes:

 * This is now in a temporary state.  We aim to retire this hardware by or at the end of O4.  New hypervisor computers are being procured.  As such we did not provision much storage on this.  Just enough to run the hypervisor, relying on the shared storage layer to handle the VM.

 * As per the proxmox administrators guild we removed cdsproxmox from the cluster prior to attaching it back as cdsproxmox0.