After the EtherCAT reboot we had a few SDF Diffs that needed to be accepted.
Screenshot.

This morning Tony and TJ had a hard time locking the OMC.

We've found that OMC DCPD A and B output are very assymmetric only when there was a fast transient (1st attachment) but not when the OMC length was slowly brought close to the resonance (2nd attachment), which suggested whitening problem.

The transfer function from OMCA to B suggested that the switchable hardware whitening was ON for DCPD_A and OFF for B when it was supposed to be OFF for both. 3rd attachment shows the transfer function from DCPD_A to B, and 4th attachment shows the anti-whitening filter shape.

Switching ON the anti-whitening only for DCPD_A made the frequency response flat. Trying to switch the analog whitening ON and OFF by toggling H1:OMC-DCPD_A_GAINTOGGLE didn't change the hardware whitening status, it's totally stuck.

We tried to lock the IFO by only using DCPD_B, but IFO unlocked for some reason.

After IFO lost lock, people found on the floor that it's the problem of the whitening chassis, not the BIO. It's not clear if we can fix the board in the chassis (which is preferrable) or have to swap the whitening chassis (less preferable as calibration group needs to measure the analog TF and generate a compensation filter).

We'll update as we make progress.

TITLE: 09/26 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 12mph Gusts, 6mph 3min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.22 μm/s
QUICK SUMMARY:

Locking notes:
Ran an initial Alignment.
After Initial Alignment I started the Locking process, and bounced to PRMI 3 times even though I had good DRMI flashes.
On the 3rd time of jumpin to PRMI,  I manualed back to Align_recycling mirrors, then requested NLN. that took us to DRMI , then I touched up SRM in Yaw.
 
OMC issue in Prep_DC_READOUT_TRANSITION.
Tridents looking good, but the TUNE_OFFSETS OMC_LOCK Guard state tunes Away from the desired value.
TJ Tried locking by hand...
OMC DCPDS are not balanced? (I'm not sure what they mean by this.
Sheila suggests that it might be a Whitening issue and Keita confirms this.
We are trying to switch the OMC DCPD Whitening that we are using. We are now using DCPD B. and Plan to switch back to DCPD A later on. An effort wasmade to try to keep the configuration the same for the CAL team, so Swapping Whitening Chassis was Not the best option. We are stuck in the "High State", which makes the output of the OMC's DCPD's high. But the High State is where we need it to be for NLN.
Sheila and Keita, did the DCPD Switching to get around the Whitening issues, just to have a LockLoss in the next ISC_Lock state DARM_TODC_READOUT. :(

Relocking again got to PRMI....

17:43 UTC EtherCat Failure, Fernando, Fil, and Sigg are working on resolving the issue.

Current H1 status is Down for Corrective maintenance.


 

In the past few weeks have seen rocky performance out of the Calibration pipeline and its IFO-tracking capabilities. Much, but not all, of this is due to [my] user error.

Tuesday's bad calibration state is a result of my mishandling of the recent drivealign L2L gain changes for the ETMX TST stage (LHO:78403,
LHO:78425, LHO:78555, LHO:79841).

The current practice adopted by LHO with respect to these gain changes is the following:

1. Identify that KAPPA_TST has drifted from 1 by some appreciable amount (1.5-3%), presumably due to ESD charging effects.
2. Calculate the necessary DRIVEALIGN gain adjustment to cancel out the change in ESD actuation strength. This is done in the DRIVEALIGN bank so that it's downstream enough to only affect the control signal being sent to the ESD. It's also placed downstream of the calibration TST excitation point.
3. Adjust the DRIVEALIGN gain by the calculated amount (if kappaTST has drifted +1% then this would correspond to a -1% change in the DRIVEALIGN gain).
3a. Do not propagate the new drivealign gain to CAL-CS.
3b. Do not propagate the new drivealign gain to the pyDARM ini model.

After step 3 above it should be as if the IFO is back to the state it was in when the last calibration update took place. I.e. no ESD charging has taken place (since it's being canceled out by the DRIVEALIGN gain adjustments). 
It's also worth noting that after these adjustments the SUS-ETMX drivealign gain and the CAL-CS ETMX drivealign will no longer be the copies of each other (see image below). 

The reasoning behind 3a and 3b above is that by using these adjustments to counteract IFO changes (in this case ESD drift) from when it was last calibrated, operators and commissioners in the control room could comfortably take care of performing these changes without having to invoke the entire calibration pipeline. The other approach, adopted by LLO, is to propagate the gain changes to both CAL-CS and pyDARM each time it is done and follow up with a fresh calibration push. This approach leaves less to 'be remembered' as CAL-CS, SUS, and pyDARM will always be in sync but comes at the cost of having to turn a larger crank each time there is a change.


 
Somewhere along the way I updated the TST drivealign gain parameter in the pyDARM model even though I shouldn't have. At this point, I don't recall if I was confused because the two sites operate differently or if I was just running a test and left this parameter changed in the model template file by accident and subsequently forgot about it. In any case, the drivealign gain parameter change made its way through along with the actuation delay adjustments I made to compensate for both the new ETMX DACs and for residual phase delays that haven't been properly compensated for recently (LHO:80270). This happened in commit 0e8fad of the H1 ifo repo. I should have caught this when inspecting the diff before pushing the commit but I didn't. I have since reverted this change (H1 ifo commit 41c516).

During the maintenance period on Tuesday, I took advantage of the fact that the IFO was down to update the calibration pipeline to account for all of the residual delays in the actuation path we hadn't been properly compensating for (LHO:80270). This is something that I've done several times before; a combination of the fact that the calibration pipeline has been working so well in O4 and that the phase delay changes I was instituting were minor contributed to my expectation that we would come back online to a better calibrated instrument. This was wrong. What I'd actually done was install a calibration configuration in which the CAL-CS drivealign gain and the pyDARM model's drivealign gain parameter were different. This is bad because pyDARM generates FIR filters that are used by the downstream GDS pipeline; those filters are embedded with knowledge of what's in CAL-CS by way of the parameters in the model file. In short, CAL-CS was doing one thing and GDS was correcting for another. 

-- 

Where do we stand?

At the next available opportunity, we will be taking another calibration measurement suite and using it reset the calibration one more time now that we know what went wrong and how to fix it. I've uploaded a comparison of a few broadband pcal measurements (image link). The blue curve is the current state of the calibration error. The red curve was the calibration state during the high profile event earlier this week. The brown curve is from last week's Thursday calibration measurement suite, taken as part of the regularly scheduled measurements.

--
Moving forward, I and others in the Cal group will need to adhere more strictly to the procedures we've already had in place: 
1. double check that any changes include only what we intend at each step
2. commit all changes to any report in place immediately and include a useful log message (we also need to fix our internal tools to handle the report git repos properly)
3. only update calibration while there is a thermalized ifo that can be used to confirm that things will back properly, or if done while IFO is down, require Cal group sign-off before going to observing

Daniel has the Beckhoff slow controls system offline for investigation.

I've bypassed the following alarms:

Bypass will expire:
Thu Sep 26 10:54:08 PM PDT 2024
For channel(s):
    H1:PEM-C_CER_RACK1_TEMPERATURE
    H1:PEM-C_MSR_RACK1_TEMPERATURE
    H1:PEM-C_MSR_RACK2_TEMPERATURE
    H1:PEM-C_SUP_RACK1_TEMPERATURE
 

I loaded the new H1LSC.txt filter file into h1lsc. This has added Elenna's "new0926" filter to PRCLFF. This filter is currently turned off and has not been switch on recently.

Thu Sep 26 08:14:57 2024 INFO: Fill completed in 14min 52secs

Jordan confirmed a good fill curbside.

TITLE: 09/26 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 3mph Gusts, 1mph 3min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.23 μm/s
QUICK SUMMARY:

When I arrived the IFO was trying to lock its self after a Lockloss from NLN. .
Unknown Lockloss

While getting the screenshots and typign this up the IFO went to PRMI twice.so I decided to get it an Initial Alignment after that big gusty wind storm yesterday.
 

A follow up to alog78711 with all of the 11/12 completed assemblies.

I made 3 comparison plots; all suspensions(contains the legend on the 1st page, measurement date to corresponding sus s/n), both of the suspended versions, all 9 of the freestanding versions (there is 1 left to be finished that we're waiting on a part rework/repair).

Continuing in the investigation of the PRCL coupling and the attempts at applying a PRCL feedforward...

Since we have performed several injections into PRCL over the last few weeks, we can track the PRCL coupling. I made this plot comparing the DARM/PRCL transfer function from three different times: an injection performed before testing a new feedforward on Sept 16, an injection from the noise budget, and an injection done while updating the sensing matrix for SRCL and PRCL after rephasing POP9, Sept 23. The PRCL coupling is relatively stable from all of these tests.

However, the feedforward we have tried has not been working, which has been very confusing. I think I have discovered the reason why: in addition to the measure of the DARM/PRCL transfer function, there is a measure of the "preshaping", which includes the high pass filter we apply to the feedforward. Looking at my code, I found that I had been using the SRCL preshaping to calculate the PRCL coupling, which uses a different high pass filter (comparison of the three high pass filters). I should have been using the MICH preshaping measurement instead, since MICH and PRCL use the same high pass, and everything downstream from that point is the same. Sorry everyone for that mistake.

So, when using the proper preshaping, a PRCL feedforward fit should work. Since we already have this data taken, I ran a quick fit of the feedforward, and compared it to previous fits. Camilla's fit from July 11 was successful, and I can see that the low frequency gain and phase is very similar to the new fit. I believe one reason her fit no longer works is the coupling shape from up to about 70 Hz has changed. Meanwhile, the incorrect fits I tried recently are very different in magnitude and phase (shown in green as "old incorrect fit"), explaining why they failed.

This new fit requires about a factor of 5 more gain than Camilla's fit above 200 Hz, which I think should be ok. The PRCL injections see a large bandstop around 250 Hz, so we don't measure the coupling there. If we can turn off these bandstops we can probably get a much better measurement up to 1000 Hz, which will help this shaping.

Overall, the fit should subtract up to 10 times the noise from 10-30 Hz and on average 3 times the noise up to 100 Hz. Based on our recent noise budget projections, which show that PRCL is directly limiting DARM noise up to 30 Hz, this will have a positive effect on the low frequency sensitivity.

The new filter is placed in the PRCL FF bank in FM6, labeled as "new0926". Since we are in observing, I didn't load the model, just saved the filter. To test, load the filter bank, and engage with a gain of 1, along with the  FM10 highpass.

I recommend that this filter is tested along with the commissioning work for Thursday. It would be useful to have an injection before the filter is applied and after the filter is applied, each with enough averages to provide a good fit. If this feedforward filter doesn't work, we can refit. If it does, we can look at fitting iteratively for further subtraction. Please try to get about 30 averages for both measurements, and if possible, boost the injection strength above 50 Hz to get a bit better coherence.

If the feedforward doesn't work, please also additionally run a PRCL preshaping measurement. This means running with the PRCL FF input set OFF, with the high pass filter ON and a gain of 1. I think we have a template for this in the LSC feedforward folder. If not, the preshaping template of the MICH preshaping ("MICHFF") can be repurposed with the appropriate channels. This will help avoid the confusing mistake I previously made.

TITLE: 09/26 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 147Mpc
INCOMING OPERATOR: Corey
SHIFT SUMMARY:

IFO is in NLN and OBSERVING since 03:09 UTC (2hr 10 min lock)

Wind was actively inhibiting locking for the first few hrs of shift. Gusts were ranging from 35-48mph. There were a few times where it dipped below 35mph at which point I began to lock, to no avail, losing lock usually at CHECK_IR or DRMI.

There was a lull with <20mph winds ~1:40 UTC. We got all the way to MICH_FRINGES bu IFO was not able to lock PRMI or DRMI so I began INITIAL_ALIGNMENT. From this point on, locking was fully automatic and we reached NLN approximately 1.5hrs after the initial alignment started.

Other:

• There was an error in the TEST guardian so I reloaded it
• There was an issue with LASER_PWR guardian due to the new 61W laser power not being reloaded. Before I knew this was the issue, I thought it was some recent Test Point thing, so I reset Test Points.
  • After contacting TJ, we checked the LASER_PWR guardian to find out it had to be reloaded and upon reloading, we were able to go into OBSERVING, where we've been since.
• Squeezing still seems sub-optimal (known recent issue).
• I experienced no issues locking ALS (woohoo!)
• I experienced no PI Ringups worthy of verbal alarm (woohoo part twoo)

LOG:

None

Since we've gotten back from the OFI vent, we've had chunks of time where we struggle with PI's ringing up (attachment1). During that time, we've made changes to filter bank settings and adjusted the max power during our efforts to try and tame them.

Timeline of PI ringups and associated changes (all times in UTC)
August 22-23rd (alog79673)
    - Multiple locklosses from PI24 (lockloss1,lockloss2,lockloss3)
    - Changes made to PI damping (alog79665)
August 24
    - nominal max power setting changed to 61W so as to bypass it for the weekend
August 27 (alog79753)
    - PI31 ringup (no lockloss)
August 31 (alog79836)
    - PI28 & PI29 ring up and cause lockloss
September 02 (alog79860)
    - PI28/29 ring up and caused lockloss
    - nominal max power changed back to 60W
    - PI28/29 ringup causing lockloss
    - PI28/29 ringup causing lockloss
    - PI28/29 ringup causing lockloss
        - These last three lockloss were with 60W
        - These four locklosses all happened one after the other.
September 17
    - PI24 ringup and lockloss
September 21
    - PI24 ringup and lockloss
September 23
    - PI24 ringup and lockloss
September 25
    - PI24 ringup and lockloss
    - PI24 ringup and lockloss
    - PI24 ringup and lockloss

Something weird first noticed by TJ during the 17:52UTC ring up today was that PI24 looked a lot noisier than usual. I looked back at the previous PI24 ringups(attachment2) and noticed that only the ringups from today (September 25th) are this noisy, compared to the other PI24 locklosses we've seen since the vent.
After this latest lockloss, the nominal max power has been increased back up to 61W (alog80295). We are discussing changes to ring heater power since it looks like the max power changes are enough to stop one mode from ringing up, but might be causing the others to start ringing up.

J. Oberling, O. Patane

Today we started to re-center the SR3 optical lever after SR3 alignment was reverted to its pre-April alignment.  That's not quite how it went down, however...

We started by hooking up the motor driver and moving the QPD around (via the crossed translation stages it is attached to), and could not see any improvement to the OpLev signal.  While moving the horizontal translation stage it suddenly stopped and started making a loud grinding noise, like it had hit its, or a, limit.  Not liking the sound of that, we launched on figuring out fall protection to climb on top of HAM4 to investigate.  While the fall protection was getting figured out we took a look at the laser and found it dead.  No light, no life, all dead.  So we grabbed a spare laser from the Optics Lab and installed it (did not turn it on yet).

Once the fall protection was figured out I climbed on top of HAM4 and opened the OpLev receiver.  I couldn't visually see anything wrong with the stage.  It was near the center of its travel range, and nothing else looked like it was hung up.  I removed the QPD plate and the vertically mounted translation stage to get a better view of the stuck stage, and could still see nothing wrong.  Oli tried moving the stage with the driver and it was still making the loud noise, and the stage was not moving.  So it was well and truly stuck.  We grabbed one of the two spare translation stages from the EE shop (where Fernando was testing the remote OpLev recentering setup), tested it to make sure it worked (it did!), and installed it in the SR3 OpLev receiver.  The whole receiver was reassembled and the laser was turned on.  Oli slowly turned up the laser power while I watched for the beam, and once it was bright enough Oli then moved the translation stages to roughly center it on the QPD.

Something interesting, as Oli was turning up the laser power it would occasionally flash bright and then return to the brightness it was at before the flash.  They got it bright enough to see a SUM count of ~3k, and then re-centered the OpLev.  At this point I closed up the receiver and came down from the chamber.  I turned the laser power up to return the SUM counts to the ~20k it was at before the SR3 alignment shift and saw the SUM counts jump just like the beam would flash.  This happened early in the power adjustment (for example: started at ~3k SUM, adjusted up and saw a flash to ~15k, then back down to ~6k) but leveled off once the power was higher (I saw no jumps once the SUM counts were above 15k or so).  Maybe some oddness with a low injection current for the laser diode?  Not sure.  The OpLev is currently reading ~20k SUM counts and looks OK, but we'll keep an eye out to see if it remains stable starts behaving oddly.

The SR3 optical lever is now fixed and working again.

New laser SN is 197-3, old laser SN is 104-1.  SN of the new translation stage is 10371

Looking at data from a couiple of days ago, there is evidence of some transient bumps at multiples of 11.6 Hz. Those are visible in the summary pages too around hour 12 of this plot.

Taking a spectrogram of data starting at GPS 1410607604, one can see at least two times where there is excess noise at low frequency. This is easier to see in a spectrogram whitened to the median. Comparing the DARM spectra in a period with and without this noise, one can identify the bumps at roughly multiples of 11.6 Hz.

Maybe somebody from DetChar can run LASSO on the BLRMS between 20 and 30 Hz to find if this noise is correlated to some environmental of other changes.

-Brice, Sheila, Camilla

We are looking to see if there are any aux channels that are affected by certain types of locklosses. Understanding if a threshold is reached in the last few seconds prior to a lockloss can help determine the type of lockloss, which channels are affected more than others, as well as

We have gathered a list of lockloss times (using https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/index.cgi) with:

1. only Observe and Refined tags (plots, histogram)
2. only Observe, Refined, and Windy tags (plots, histogram)
3. only Observe, Refined, and Earthquake tags (plots, histogram)
4. Observe, Refined, and Microseism tags (note: all of these also have an EQ tag, and all but the last 2 have an anthropogenic tag) (plots, histogram)

(issue: the plots for the first 3 lockloss types wouldn't upload to this aLog. Created a dcc for them: G2401806)

We wrote a python code to pull the data of various auxilliary channels 15 seconds before a lockloss. Graphs for each channel are created, a plot for each lockloss time are stacked on each of the graphs, and the graphs are saved to a png file. All the graphs have been shifted so that the time of lockloss is at t=0.

Histograms for each channel are created that compare the maximum displacement from zero for each lockloss time. There are also a stacked histogram based on 12 quiet microseism times (all taken from between 4.12.24 0900-0930 UTC). The histrograms are created using only the last second of data before lockloss, are normalized by dividing by the numbe rof lockloss times, and saved to a seperate pnd file from the plots.

These channels are provided via a list inside the python file and can be easily adjusted to fit a user's needs. We used the following channels:

anthony.sanchez@cdsws29: python3 /ligo/home/francisco.llamas/COMMISSIONING/commissioning/k2d/KappaToDrivealign.py

Fetching from 1409164474 to 1409177074

Opening new connection to h1daqnds1... connected
    [h1daqnds1] set ALLOW_DATA_ON_TAPE='False'
Checking channels list against NDS2 database... done
Downloading data: |█████████████████████████████████████████████████████████████████████████████████████| 12601.0/12601.0 (100%) ETA 00:00

Warning: H1:SUS-ETMX_L3_DRIVEALIGN_L2L_GAIN changed.

Average H1:CAL-CS_TDEP_KAPPA_TST_OUTPUT is -2.3121% from 1.
Accept changes of    
H1:SUS-ETMX_L3_DRIVEALIGN_L2L_GAIN from 187.379211 to 191.711514 and
H1:CAL-CS_DARM_FE_ETMX_L3_DRIVEALIGN_L2L_GAIN from 184.649994 to 188.919195
Proceed? [yes/no]
yes
Changing
H1:SUS-ETMX_L3_DRIVEALIGN_L2L_GAIN and
H1:CAL-CS_DARM_FE_ETMX_L3_DRIVEALIGN_L2L_GAIN
H1:SUS-ETMX_L3_DRIVEALIGN_L2L_GAIN => 191.7115136134197
anthony.sanchez@cdsws29: