TITLE: 12/11 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Preventive Maintenance
INCOMING OPERATOR: Oli
SHIFT SUMMARY: Long maintenance day with a great many activities wrapped up late afternoon and initial alignment started around 23:40. The biggest issue we've encountered is that the FSS autolocker is having trouble locking the RefCav; it can grab the resonance but loses it after about a second, so it's been taking quite a while to lock (I manually did it twice this afternoon to speed things along). However, if the FSS is locked, it seems happy, the only issue appears to be with relocking. Otherwise, initial alignment ran smoothly and main locking started at 00:24. Currently up to PREP_ASC_FOR_FULL_IFO.
LOG:

After our CO2 tests this morning 81723 we've reverted the PEM_MAG_INJ and SUS_CHARGE back to their nominal Tuesday start times of 7:20 and 7:45am. This was changed in 81474 but today was the first chance we got to turn off the CO2s before Tuesday maintenance started.

TITLE: 12/11 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Preventive Maintenance
OUTGOING OPERATOR: Ryan C / Ryan S
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 8mph Gusts, 4mph 3min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.43 μm/s
QUICK SUMMARY:

Just started relocking after finishing an initial alignment. Little bit of a spike in the secondary microseism but nothing too bad

Sheila D, TJ S

There has been a bout of SRM M3 watch dog trips around the SRY locking step of initial alignment over the last few weeks (alog81670 for example). Today Sheila and I discussed locking without the FE triggering at all, and just having the ALIGN_IFO node see good flashes and turn on the necessary filters and banks. I tried this out today but I wasn't able to get it to lock any more reliably than what we have now. No matter the method, it would seem to catch with low AS_A values, ~2500 on NSUM vs the normal 5000. From here, SRM would start to be driven before it would realize that it wasn't quite locked. I tried adding some code in ACQUIRE_SRY to try to turn on and off the SRM M1 LOCK L input and the SRCL FM4 filters and even clear the history if necessary, but this didn't work without long settling periods between attempts.

I ended up keeping the ISC_library.is_locked('SRY') looking as AS_A_DC_NSUM_OUTPUT > 4000. This value is a bit higher but safer. It will let the node run through Down and reset drives, integrators, and let SRM settle a bit. I don't think this is a fix, barely even a band aid. It will need some more thought on how to only catch on the correct mode.

Oli, Ibrahim

IFO has been swept.

Of note:

• Crane had to be re-parked to right spot.
• Scissor lift on pallet jack (pictured below) was found plugged in and unplugged. Tagging FMCS.
• Unplugged extension cord (was plugged in with nothing plugged into it).

Per the WP12246:

Visual inspections were performed in the LVEA to track the wires and verify the picomotor controllers existence, connections and spares (physically only). The information gattered was updated in the document E1200072. Important findings were found and the document is pretty much near to the real installation. Electrical part investigations will follow to determine the spares. The names of the rack as well as the physical location for the controllers were verified using the O5 ISC wiring diagram D1900511.

Marc, Fernando

WP12239 New VM server

Erik, Jonathan:

The old proxmox machine was taken out of service and repaired. In production it was replaced with a W2275 (aka oaf FE). Please see Jonathan's alog for details. After h0epics was moved, its iocs were started by hand as per wiki instructions:

dust monitors (lvea, lab, ey, ex, dr), weather stations (ex, ey, mx, my) and 3ifo dewpoint sensor.

WP12236 New RGC and Timing Card h1omc0

Jonathan, Erik, EJ, Marc, Daniel, Dave:

h1omc0 was upgraded to a custom RCG5.30 and a new LIGO Timing Card (ver1589). This will permit changing the Duotone frequency from 960,961Hz to 1920,1921Hz in conjection with LLO next week. Erik verified the duotone is currently unchanged at 960,961Hz.

Please see Erik's alog for details.

The RCG upgrade of h1iopomc0 added one slow channel to the DAQ INI (H1:FEC-179_TIMING_CARD_TEMP_DEG_C). A DAQ restart was required, which permitted an EDC change for several pending WPs (marked as *).

WP12243 Slow Controls Timing Card Version Check

Daniel

Daniel changed the slow-controls Beckhoff code to permit the following versions of the LIGO timing card in the FE IO-Chassis

The code was restarted at 10:19. This fixed the persistant EX timing error due to h1susex TCver=1000 and preempted a corner station h1omc0 error with TCver=1589.

WP12164 Add new Guardian PCALX_STAT channels to DAQ*

Tony, TJ, Dave:

The new PCALX_STAT guardian node's DAQ channels were added to the EDC as part of the DAQ restart.

WP12195 Add missing CDSRFM slow channel to DAQ*

Dave:

A new H1EPICS_CDSRFM.ini was loaded, adding the missing channel H1:CDS-RFM_LRS_EX2CS_CHCNT

WP12185 add VACSTAT slow channels to DAQ*

Dave:

A new H1EPICS_VACSTAT.ini added the new state channels to the DAQ.

CDS Hardware Status IOC

Erik, Dave:

Following the upgrade of h1omc0's timing card the CDS HW stat reported problems with this card. We found that one unused bit in the LPTC_STATUS PV was unset in h1omc0 and set in all the other front ends, including h1susex.

Now that Daniel has fixed the TCver error in EX, the temporary code in cds_status_ioc.py to verify the only EX error is the TCver error was removed.

DAQ Restart

Jonathan, Erik, EJ, Dave:

The DAQ was restarted primarily for the h1iopomc0 INI change due to its RCG upgrade. Several EDC changes which have been waiting for target-of-opportunity were also done.

This was a messy DAQ restart. Adding h1omc0 to the custom RCG boot server h1vmboot0 inadvertently changed the DAQ FE list to only include two frontends; h1susex and h1omc0. When the 0-leg was started initially DC0 looked good, but then we noticed that on the CDS overview only the DC0 EPICS channels for omc0 and susex were connecting, and Jonathan found GDS0 had lost a lot of channels. Around this time I also restarted the EDC on h1susauxb123 and also found that the expected number of channels was down by about 1000. At this time FW0 was writing tiny frames (only two frontends).

DC0s DAQ configuration was fixed and the 0-leg was restarted. At this point the EDC was out of sync, so it was restarted. This then generated a new H1EDC.ini which necessitated a third and final 0-leg restart (plus another GDS0).

The 1-leg restart was a single restart, with a GDS1 restart also needed.

Change of RCG reporting color on CDS Overview

Dave:

We now have two front end systems in production which are running with a custom version of the RCG. To highlight this, I have changed the non-standard-RCG indicator color from purple to black on the CDS Overview. Please see attached.

[Erik v.R., Dave, Jonathan]

h1omc0 was upgraded to support alternative duotone frequencies.

A new parameter was added to the iop model, h1iopomc0, 'duotone_frequency', currently set to '960', which configures the duotone frequency to 960/961 Hz, the standard frequencies.

The h1omc0 front end was moved from a the primary bootserver, h1vmboot1, to the secondary bootserver, h1vmboot0 using changes to the puppet configuration.

All models for h1omc0 were built and installed from RCG source rather than packages.  The git commit of the RCG source is tagged as 'h1_2024_12_10_variable_duotone'.

The omc0 front end and io chassis were shut down.  The timing card SN S2101117 was replaced with SN S2101084, which had newer firmware version 1589 that supports alternative duotone frequencies.

The IO chassis and front-end were restarted.

A few problems were encountered:

Running puppet on the secondary bootserver overwrote the DAQ master template and some EDC INI files.   The master file problem was revealed in the DAQ overview where most of the front ends were reporting no channels.  The EDC overwrite showed up as too few total EDC channels.

Running the puppet configuration on the production server re-enabled DHCP for hosts that were supposed to boot from the secondary bootserver.  These had to be manually commented out from the DHCPD configuration, then the service restarted.

To change the duotone to 1920 Hz, edit the h1ipomc0 model, set 'duotone_frequency=1920' in the parameter block, save the model.  Build and install the model then restart all models.

As part of WP 12239 we moved h0epics, cdsvmscript1, epics-burt, autoburt off of the cds0proxmox hypervisor.  This resulted in a few minutes of down time for the dust monitor IOC around 8:19am localtime.

After this we done we were able to look at cdsproxmox.  Its boot drive had failed.  After some help from Fil we got the drives replaced and reinstalled proxmox ve on cdsproxmox.  We adjusted DNS and renamed the box cdsproxmox0.

Some notes:

 * This is now in a temporary state.  We aim to retire this hardware by or at the end of O4.  New hypervisor computers are being procured.  As such we did not provision much storage on this.  Just enough to run the hypervisor, relying on the shared storage layer to handle the VM.

 * As per the proxmox administrators guild we removed cdsproxmox from the cluster prior to attaching it back as cdsproxmox0.

Closes FAMIS#28383, last checked 81688

Once again, the coherence for ITMX bias drive bias off is below the coherence threshold - this time the coherence is 0.01, much lower than the threshold of 0.1, so there are once again no new analyzed measurements for ITMX.

ETMX, ETMY, ITMX, ITMY

I've used the pico to center IM4 trans QPD.   See alogs about this history: 80604 78962 78943 78856

 After difficulty with input alignment shifts in the spring, we centered IM4 with the hopes that we could use that as a reference for any future input alignment shifts to reset the IMs.  However this didn't work well when the PMC was swapped.  Now we are recentering IM4 trans to get this new reference since we've been operating here for a while.

TJ, Camilla. WP12162. Continuing work in 76030

We installed a CFCS11-A adjustable SMA fiber collimator on the 50um fiber for the ETMX HWS fiber coupled LED source M530F2. This improved the size of the beam but it was still too large at the edge of the collimators range.

We removed HWS-L3 (D1800270) which made the beam a better size but still a little large ~ 10-15mm diameter at the periscope. We couldn't see a change in the beam by adjusting HWS-L2 (on translation stage). We aligned the system using the retrofection of the ALS beam: aligned ALS beam to output coup;er with HWS-MS1 and then adjusted the fiber collimator (mounted in mirror mount) to aligned the HWS output to the final iris. Confirmed we were getting a reflection of ETMX by mis-aligning it ~20urad. Note that the beam looks cleaner than usual photo, probably because the GV is closed so we are getting no retroflection of ITMX. Set frequecy to 1Hz. Replaced the mask photo and started the HWS code with new references.

The ETMX HWS 520nm beam in now injected into the vacuum where it hasn't been for ~months. Tagging DetChar. 

To do: measure the beam profile of beam out of fiber and re- calculate an imaging solution of the ETMX HR surface. 

GV5 and GV7 were soft closed at ~10am local time today to facilitate equipment craning for crane inspections.  They were opened at ~11am local time at the completion of the vertex crane inspection.

This morning, Ryan let me know that he got a "Check PSL chiller" verbal alarm.  Upon checking the chiller I saw the water level was a little low but not at minimum, but the usual oscillations in the water level as the chiller does its thing were getting a little close; this is likely what caused the alarm.  I added 175 mL of water to the chiller to bring the water level back up to the MAX line.

Ansel reported that a peak in DARM that interfered with the sensitivity of the Crab pulsar followed a similar time frequency path as a peak in the beam splitter microphone signal. I found that this was also the case on a shorter time scale and took advantage of the long down times last weekend to use  a movable microphone to find the source of the peak. Microphone signals don’t usually show coherence with DARM even when they are causing noise, probably because the coherence length of the sound is smaller than the spacing between the coupling sites and the microphones, hence the importance of precise time-frequency paths.

Figure 1 shows DARM and the problematic peak in microphone signals. The second page of Figure 1 shows the portable microphone signal at a location by the staging building and a location near the TCS chillers. I used accelerometers to confirm the microphone identification of the TCS chillers, and to distinguish between the two chillers (Figure 2).

I was surprised that the acoustic signal was so strong that I could see it at the staging building - when I found the signal outside, I assumed it was coming from some external HVAC component and spent quite a bit of time searching outside. I think that this may be because the suspended mezzanine (see photos on second page of Figure 2) acts as a sort of soundboard, helping couple the chiller vibrations to the air. 

Any direct vibrational coupling can be solved by vibrationally isolating the chillers. This may even help with acoustic coupling if the soundboard theory is correct. We might try this first. However, the safest solution is to either try to change the load to move the peaks to a different frequency, or put the chillers on vibration isolation in the hallway of the cinder-block HVAC housing so that the stiff room blocks the low-frequency sound. 

Reducing the coupling is another mitigation route. Vibrational coupling has apparently increased, so I think we should check jitter coupling at the DCPDs in case recent damage has made them more sensitive to beam spot position.

For next generation detectors, it might be a good idea to make the mechanical room of cinder blocks or equivalent to reduce acoustic coupling of the low frequency sources.