h1cam26 (BS) stopped updating around 22:22 PDT Wed 21aug2024.

I power cycled the camera via sw-lvea-aux POE and restarted the process on h1digivideo2. 

h1cam26 is on switch port 0/35

The offload of the past 6 month's of raw minute trend files from h1daqtw1's SSD-RAID to permanent archival on spinning media is complete. TW1's SSD-RAID disk usage has been reduced from 92% to 2%.

First NDS1 restart: 09:56 Tue 20aug2024

Start file copy: 10:53 Tue 20aug2024

File copy completed: 12:07 Wed 21aug2024 (25hrs 14mins)

Second NDS1 restart: 15:47 Wed 21aug2024

Start file deletion on TW1: 09:32 Thu 22aug2024

File deletion complete: 11:40 Thu 22aug2024 (2hrs 8mins)

I wanted to check the last lock loss concerning the power upstream/downstream of the fast shutter, but quickly remembered that the upstream signal, i.e. the non-whitened analog version of the ASC-ASC_SUM, was disconnected from the fast PEM ADC (H1:PEM-CS_ADC_5_19_2K_OUT_DQ) before the last vent to fix OFI, and it was not reconnected since.

I went to the floor and restored the connection.

Original alogs about this connection: 78968 and 79077.

After HAM7/FC GVs were open this morning, I aligned FC1/2 and FC GR is locked. FC GR trans is ~110, which is reasonable. 

I measured NLG following 76542. The measured NLG is 16 (0.247/0.0154=16), which is a bit lower than the last measurement in 78000. The OPO temperature is very different from before as shown in the attached SDF.

Famis 26321 HVAC Vibrometer Check

HVAC fan channel  H0:VAC-EY_FAN1_ACC_INCHSEC saw a decrease in signal but an increase in noise on the signal starting on Tuesday.
H0:VAC-MR_FAN2_170_2_ACC_INCHSEC saw an increase in the counts signal starting on Tuesday.

Thu Aug 22 08:15:16 2024 INFO: Fill completed in 15min 12secs

TITLE: 08/22 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 8mph Gusts, 5mph 5min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.07 μm/s
QUICK SUMMARY: H1 is holding in PREP_FOR_LOCKING after losing lock last night (and was locked at NLN for almost 4 hours!). I'm starting to go through notes from locking attempts last night, and once I have, I'll start trying to bring up H1.

TITLE: 08/21 Eve Shift: 2300-0800 UTC (1600-0100 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
INCOMING OPERATOR: Ryan C
SHIFT SUMMARY:

LOG:

• 2320-2338 Going out to center OMC trans camera (tj, camilla)
• 1650 Started Initial Alignment & then went into locking
• 0407 PI Mode 24 alerts came up.  I went to the PI Overview screen, found the SUS PI guardian, and saw it was in the IFO DOWN state!  So I transitioned it to its nominal state of PI Damping.  But looks like PI24 is railing around 2000 on nuc25.

[Ryan S, Corey, Camilla, TJ, Elenna]

Quick, must be fixed ASAP summary: something is wrong with ALS Y. The WFS are not engaging when they should. DOF2 seems to be bad right now, and the camera offsets are also not good. Guardian does not notice the WFS are not engaging. Do not trust the green camera offsets on the Y arm.

Ryan S brought the IFO to DC readout successfully, and began damping violin modes. Some noisy peaks are evident, probably due to pumps that are still running and/or the HAM5/7 relay tube gate valve is still closed.

I set SRC1 P and Y offsets and engaged the loops. SRC1 P offset = -0.538 and SRC1 Y offset = 1.327

These offsets keep us in a good place for the SRM, and then I can walk SR2 to converge the SRC2 loop.

This worked for pitch, and I was just about to engage SRC2 pitch when there was a lockloss due to HAM6 motion. Both the vacuum team and Camilla and TJ were around that area for various activities, potentially causing the lockloss.

After TJ and Camilla aligned the OMC trans camera, we ran initial alignment (trying to lock DRMI failed because the alignment was very poor).

Initial alignment failure:

• we tried to run a full initial alignment with the new green camera offsets, but it failed
• ALS locks successfully for both arms, and the X arm completes "initial alignment offloaded"
• However, the Y arm gets stuck in the "initial alignment" state and the WFS do not turn on
• We found a button that can "force" turn them on, but the alignment pulls the Y arm in a bad direction
• We tried this a few times, turning off various alignment loops to pinpoint which ones were bad
• it appears that dof2 and the camera loops are not good
• I don't know if there is some other issue I am missing, but it doesn't seem like the camera offsets are in the right place (even though they should be from earlier!)
• We couldn't figure out why the ALS guardian would just hang in "initial alignment" either, waiting for the WFS to converge when it wasn't even engaging the WFS. Ryan hypothesized it was a "beckhoff issue"

I had us give up and run a manual initial alignment without using the green cameras.

It appears that even in "Locked slow wfs" the ALS Y WFS do not turn on, but it's somehow not a problem in the guardian because the error signals are pretty well converged already? Something is going wrong with the ALS Y WFS. Trending back it seems that the ALS Y WFS did not run at all today, and we didn't seem to notice!

In order to properly run the WFS, I have to take the ALS Y guardian to "locked slow ETM TMS WFS" and then "force" the autocentering trigger in order to turn on the WFS. Then I can let the guardian move on to offloading.

Additionally, ALS Y DOF 2 is taking the Y arm in a bad direction again after it converges the alignment in green locking. Once I force on the Y arm WFS, I turn off the DOF2 signals and that keeps the Y arm stable through the next locking steps.

To get the SRC ASC on:

• Corey and I went to "prep asc for full ifo"
• I stepped through by hand, skipping the SRC ASC steps
• I set offsets in SRC1 at about -0.5 for pitch and 1.4 for yaw.
• I engaged the SRC1 loops successfully with the offsets on
• I stepped SR2 in pitch and yaw to converge the SRC2 error signal and keep the buildups good
• I engaged SRC2
• I turned on soft loops, corner alignment got worse (POP18 dropped, POP90 increased)
• to improve the POP buildups, I then slowly stepped the SRC1 offsets to zero, which improved the alignment (and SR2 and other optics could follow!)
• These offsets are SDFed for now, but should be turned off once we know the alignment is "saved" properly otherwise

Indeed now that we are in full good alignment with ASC engaged, the ALS Y green is no longer locked on a 00 mode, which means that the camera offsets from earlier are maybe incorrect. However, I could not get the ALS Y to lock on the 00 beam in order to reset the offsets. First, I tried moving the green QPD offset sliders around, then I tried very slowly moving the TMS in 0.1 urad steps. All I can get on the camera is this.

However, I did another set of the ALS X green references (very minor change), with the SDF screenshot attached. The ITMX CAM values were still good, so I didn't change them.

Everything else executed well: power up, low noise transitions. I skipped over "inject squeezing" since we don't have the valves open for the FC yet. We have arrived in OMC whitening, where the guardian is holding us to damp the violins.

DARM is very noisy, probably due to whatever vacuum pumps are still running and any extra noise due to the relay tube being closed. Most calibration lines have no SNR due to the high noise.

Major problem to solve is the ALS Y alignment and camera offsets, but locking has succeeded and we are almost back to nominal!

Today's activities:

- The large LVEA turbos (OMC, X-manifold, Y-manifold) have been valved out
- The large Gate Valves (GV5 and GV7) have been opened
- The large turbos have been switched off - see their statistics in the comments
- The HAM6 RGA bakeout is being ramped down. Tomorrow it should be ready to be valved in, and make the 1st scan.

Tomorrow the whole FCT will be valved together with the corner, so everything will go back to normal. The reason why it will only be done tomorrow is that the pressures of the corner need some time to catch up with HAM7

The pressures:
Corner (PT-120 and PT131): 7.5E-8 Torr
HAM6: 2.5E-7 Torr
HAM7: 7.45E-8 Torr
CP1: ~3E-8 Torr
CP2: ~5E-8 Torr

TITLE: 08/21 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
INCOMING OPERATOR: Corey
SHIFT SUMMARY:

Arm gate valves (GV5 and GV7) were opened this morning, which allowed us to start recovering the IFO. See Elenna's alog for details in the process. Currently running through a full initial alignment now that the green camera references have been set.

Other items to note from the day are that the LVEA remains laser HAZARD following the camera adjustments on HAM6.

LOG:

WP 12056.  TJ and I realigned the AS AIR camera (using the steering mirror) and OMC TRANS camera (sliding and adjusting the camera height after Ryan S and Elenna locked OMC in single bounce) cameras that were removed and replaced when HAM6 was vented 79556. They are now centered.  

TITLE: 08/21 Eve Shift: 2300-0800 UTC (1600-0100 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
OUTGOING OPERATOR: Ryan S
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 15mph Gusts, 11mph 5min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.07 μm/s
QUICK SUMMARY:

Walked in to H1 being at DARM to DC.  TJ and Camilla went out to center the OMC Trans camera, but then H1 lost lock.  Ryan has been working on getting them an OMC single-bounce so they can align the camera.

[Sheila, TJ, Ryan S, Elenna]

Here we go!

Ryan S began by running the dither scripts. The ITM and TMS dither scripts worked well, the ETM dither scripts did NOT. Note for future: do not run ETM dither scripts.

ALS X and Y locked well, X was faster than Y as usual. Beatnotes look good!

Input Align successful.

Sheila and I checked SR3 oplevs, and noticed while pitch was very similar to July 12th, pre-vent locking, the yaw oplev had moved by about 20 microrad. We checked that the sliders are about the same. Note: SR2 had to be moved significantly in yaw to find the beam at AS. We spoke to Jim and TJ; they pointed out that HAM5 had been moved (see 79183). Jim moved the HAM5 ISI back by 65 microradians, and this recovered the SR3 yaw oplev alignment to match pre-vent. Note for everyone: 65 micrad of HAM5 ISI motion compared to about 20 microrad of SR3 oplev motion. We should figure out this discrepancy!

Since this step fixed the SR3 yaw alignment, Sheila reverted SR2 back to the July alignment using the sliders.

Ryan began moving PRM by hand to get AS_A alignment better- after this PRC align was successful. However, as PRC1 converged, the AS_A_DC_NSUM decreased. The sensor for PRC1 in this step is REFL B. We did not offload this alignment and instead went to DOWN, and then reran the alignment of PRM by hand. Good alignment onto AS_A for PRM does not correspond to a converged PRC1 alignment in the PRC align state right now.

While struggling with MICH Bright align, Sheila realized that the AS centering was likely the problem. We went to down, turned on AS WFS centering, and saw a huge improvement in the signals, indicating that our issue was that the AS signals were no good. MICH Bright align occured quickly after this change. This may also explain the PRC align issue.

Ryan took us back to PRC align. Indeed, the PRC1 alignment signal converged while maximizing the AS A NSUM. We ran PRC align offloaded this time.

Ryan ran SR2 align successfully.

Sheila had to do a big move of SRM to find the fringing, otherwise, no issues with SRC align. (-577 urad yaw, +300 urad in pitch compared to early July slider positions)

We then paused so Camilla and TJ could realign the AS air camera on the floor.

***********

Lock failure at DRMI ASC- SRC ASC is probably no good, the PRMI ASC ran well. Sheila changed the flag so only MICH ASC runs for DRMI.

Next lock attempt failed at "resonance". Sheila checked the PRG based on the REFL dip and arm transmission, and it is between 48 and 54 just before lockloss (34% off resonance and 1230 arm transmission, see 62110)

Next lockloss towards the end of "engage asc for full ifo". Hard to tell the cause, but the corner alignment decayed as the various loops engaged.

At the next lock attempt, we stepped engage ASC by hand. All ASC is fine to engage except SRC ASC. Both the SRC1 and SRC2 error signals are far off, and it appears that they are not converging properly to maximize the buildups. This is extra confusing given that SRC2 actuates both SRM and SR2 (using AS_C), however even SRC1 is not very good (SRM actuation with AS RF72). We left the SRC ASC off and engaged the soft loops. By moving SRM and SR2 by hand we can keep the buildups good, but the ASC error signals are still bad.

After the soft loops converged, Sheila reset the green references so our alignment is saved. SDF screenshot attached.

We still don't know what to do about SRC ASC, but we have decided to go to DC readout and try damping some violins while we think. DARM offset is on, and we have made it to prep DC readout transition.

Naoki, Camilla.

After SQZ laser was turned on laser week, today we locked PMC and SHG with no issues. We struggled with OPO locking (CLF dual). In the end Naoki could lock by hand but the trigger wasn't triggering. We will continue to troubleshoot tomorrow.  The OPO dither lock  worked well but only after we swapped H1:SQZ-OPO_IR_LSC_SERVO_GAIN sign from -10 to 10: strange!

I put ZM4/5 PSAMS and ZM4/5/6, ZM2/3 and FC1 alignments back to the same values as in alog79193. Todays plot and July 17th plot attached. There was only around 0.3mW on the IR trans, when we expect 1mW. So may need some more alignment moved. We have the SQZT7 iris to help if needed.

I turned the CO2s back on today and CO2X came back to its usual 53W, but CO2Y came back at 24W. We've seen in the past that it will jump up a handful of watts overnight after a break, so maybe we will see something similar here. If not, we will have to investigate this loss. Trending the output of this laser, it has definitely been dropping in the last year, but we should be higher than 24W.