We had a failure of h1omc0 at 16:37:24 PDT which precipitated a Dolphin crash of the usual corner station system.

TITLE: 08/15 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
SHIFT SUMMARY:

After the HAM6 cameras were reinstalled and the high voltage was turned back on then we were ready for light in the vacuum. The mode cleaner locked within about 30 seconds of requesting IMC_LOCK to Locked! Tomorrow we will bring back our seismic systems as best we can and then try some single bounce to ensure that we can get some type of light into HAM6.

Ion pumps 1,2,3,14 are in a known error state. Gerardo warned this was intentional.

LOG:                                                                                                                                                                                                      

We were exploring any weird behavior during the locklosses preceding the OFI burns to try and narrow down possible causes, and we recently learned from a scientist experienced with KTP optics that the movement of a high powered beam passing through the KTP could cause damage to the optic, so that created the theory that this could have happened due to earthquakes.

There were a couple of decently-sized earthquakes before the incidents:

April Incident (seismic/LL summary alog79132)

- April 18th - nearby earthquake from Canada - we lost lock from this (lockloss ndscope)

- April 20th - nearby earthquake from Richland - stayed locked

- April 23rd - drop in output power noticed - the lockloss right before this had NOT been caused by an earthquake (previous lockloss ndscope)

July Incident

- July 11th - nearby earthquake from Canada (alog79023) (lockloss ndscope, zoomed out lockloss ndscope)

- July 12th - noticed similarities to the April incident

We used ndscope to compare ground motion to DARM, AS_A, AS_C, and the IFO state. In looking over the ndscopes, we don't see anything that would make us think that these earthquakes changed anything in the output arm.

So yes, we did have two decently sized earthquakes (+a local one) before the IFO burns took place, but we also have earthquakes hitting us all the time, many with higher ground velocities. Overall, we did not see anything strange during these earthquake locklosses in the AS_A and AS_C channels that would lead us to think that the earthquakes played a part in the OFI issues.

I turned the CO2s back on today and CO2X came back to its usual 53W, but CO2Y came back at 24W. We've seen in the past that it will jump up a handful of watts overnight after a break, so maybe we will see something similar here. If not, we will have to investigate this loss. Trending the output of this laser, it has definitely been dropping in the last year, but we should be higher than 24W.

Per WP12042 high voltage was reactivated at the corner station via M1300464v13.

1 - ESD ITM's supply enabled
2 - HAM6 Fast Shutter and OMC PZT supply enabled
3 - Ring Heaters ITMs & SR3 enabled in TCS racks
4 - HAM6 Fast Shutter at Rack verified HV enabled
5 - MER SQZ PZT & Piezo driven PSAMs supply enabled

M. Pirello, F. Mera

WP12040. Oli and I reinstalled the HAM6 cameras/housings we revmoved beore the vent in 79213. Daniel plugged them in. 

The new corner station scroll pump air compressor was valved in at 09:39 PDT this morning. I has gone through 3 compression cycles since then. Here are the stats

• Compression starts when pressure has dropped to 87PSI
• Compression stops when pressure reaches 116PSI
• Compression takes about 7 minutes
• Pressure drop takes about 14 minutes
• Compression cycle time about 21.5 minutes.

I have reset the cell phone alarms with the levels: LOW = 80PSI, HIGH = 120PSI

The new compressor runs at a higher pressure so the alarms needed to move with that. I changed the minor and major high alarms for H0:VAC-MR_INSTAIR_PT199_PRESS_PSIG for 127 and 130 respectively in epics via the vacuum computer in the back of the control room. These alarm levels cannot be changed outside of the vacuum network. I did not see this channel in the VAC SDF.

Patrick will check if these values are hard coded into Beckhoff and will adjust accordingly.

Conda packages updated.  Controls rooom tools are updated to version 4.1.4.  The only change is a fix to the Python foton module. 

FilterDesign::get_zroots() would throw an exception when called.  Thanks to Chris Wipf for finding and fixing the problem.

Thu Aug 15 08:08:59 2024 INFO: Fill completed in 8min 55secs

TITLE: 08/15 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
CURRENT ENVIRONMENT:
    SEI_ENV state: MAINTENANCE
    Wind: 10mph Gusts, 7mph 5min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.06 μm/s
QUICK SUMMARY: Pumpdown continues.

TITLE: 08/14 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
SHIFT SUMMARY:  The pump down continues (5e-7).
LOG:                              

Yesterday Sam and I went around the LVEA with a portable accelerometer (H1:PEM-CS_ADC_5_28_2K_OUT_DQ) to investigate a ~35.4 Hz peak. This peak dates back to at least June 2023, possibly longer. We placed the portable accelerometer in seven different spots, each time against a wall except for one spot near BS1. See the attachment 1 for the reference map. The peak appears strongest at REF0 and REF5 as seen in attachment 2. We attempted to test outside near the Mitsubishis (outside the OSB kitchen) and the southeast-facing mechanical room wall as well but had data retrieval issues, so we'll go back to that soon.

We are also investigating the ~36 Hz peak seen in attachment 2, strongest near the X OL. It seems to have been around since at least 2023-05-08, but more investigation is needed to find out when it started and if it shows up consistently every day. The frequency of this peak oscillates on a range of about 1 Hz. Attachment 3 shows the peak in H1:PEM-CS_LVEAFLOOR_XCRYO_Z_DQ and H1:GDS-CALIB_STRAIN_CLEAN. This peak also travels between 30-40 Hz, generally near 40 Hz at the beginning and end of the UTC day and dropping towards 30 Hz in the middle of the day, as seen in attachment 4 from the summary pages.

WP12035

Yesterday thermocouple B (TC-B) failed during the fill. Its wire broke and its temperature railed at +1372C. The fill proceded with TC-A to completion.

If TC-A had similarly failed, the fill would have continued until either the discharge line pressure exceeded its trip pressure; or the fill timed out after 30 minutes.

Today's code modification adds a check when reading the TCs that they are not greater than the nominal_max temps (+40C).

Providing at least one TC is in operational range the fill will continue, but now if no TCs are operational the fill will abort with an error.

The new code was installed at 9am today, it will run for the first time during Thursday's fill.

During this investigation I noted that the discharge_line_pressure trip is 2.0PSI. Operationally this pressure has never exceeded 0.6PSI so I lowered the trip down to 1.0PSI.

Arianna, Derek

We noticed that the calibration line subtraction pipeline is often not correctly subtracting the calibration lines at the start of lock segments.

A particularly egregious case is on July 13, where the calibration lines are still present in the NOLINES data for ~4 minutes into the observing mode segment but then quickly switch to being subtracted. This on-to-off behavior can be seen in this spectrogram of H1 NOLINES data from July 13, where red lines at the calibration frequencies disappear at 1:10 UTC. Comparing the H1 spectrum at 1:06 UTC and 1:16 UTC shows that the H1:GDS-CALIB_STRAIN spectrum is not changed, but the H1:GDS-CALIB_STRAIN_NOLINES spectrum has calibration lines in the 1:06 UTC spectrum and no lines in the 1:16 UTC spectrum. This demonstrates that the problem is with the subtraction rather than the actual calibration lines. 

This problem is still ongoing for recent locks. The most recent lock on August 15 has the same problem. The calibration lines "turn off" at 4:46 UTC as seen in the attached spectrogram. 

The on-to-off behavior of the subtraction is particularly problematic for data analysis pipelines as the quickly changing line amplitude can result in the data being over- or under-whitened.