I startedtesting the PCALX_STAT Guardian node today.

/opt/rtcds/userapps/release/cal/h1/guardian/PCALX_STAT.py

It created a new ini file, But the DAQ was not restarted after this new ini file creation.
As it currently stands this is a draft of the final product that will be tested for a week and further refined.
This Guardian node, does not make any changes to the IFO, it's only job is to determine if PCALX arm is broken or not. TJ has already added it to the Guardian Ignore list. 
 

TITLE: 11/19 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Preventive Maintenance
INCOMING OPERATOR: Ryan S
SHIFT SUMMARY: 2ndary microseism has been consistently high (above the 90th percentile) for the whole day and has risen even more in the last few hours of the shift and has reached the top of the plot. It will likely will get even worse tomorrow as the storm off the pacific coast closes in. Wind has been pretty low today. ITMY5/6 is still kind of high.

As of 23:30 UTC we're back to sitting in idle, we unlocked the FSS, ISS, and IMC just leaving the PMC locked.
LOG:

IFO/Locking:

• 16:35 UTC While we were down Sheila set the IMC to offline and turned off the ISS first loop off (PMC and FSS locked).
• 16:53 UTC HAM7 ISI tripped, from a CPS glitch, possibly from the fire crew checking out extinguishers. This is a sensitive area, people walking around can cause CPS glitches.
• 17:22 UTC LVEA back to LASER SAFE (Transition)
• 21:23 UTC I started relocking, ALS_X had to be recovered due to the restart. I just trended top mass OSEMs and set the sliders accordingly
• 21:55 - 22:02 UTC I ran a manual IA to align the SRC as DRMI didn't look good but PRMI did. DRMI looked great afterwords
• 22:00 UTC Oli swept the LVEA alog81363
• 22:15 UTC "FAST_SHUTTER test failed", Keita's AS_B FM4 gain filters were still on when they shouldn't have been
• 23:12 UTC Lockloss at MOVE_SPOTS, lockloss tool. ASC loops were working pretty hard
• 23:30 UTC ISC_LOCK to IDLE

Some of the maintenance work completed includes:

• Glycol piping checks
• Fire alarm testing & inspection around site (confirmed PSL encl strobes are working)
• CO2Y table loose cables investigated and beam mask adjustment alog81362
• New VAC scroll pumps at LVEA and Ends
• Replaced bad pressure gauge at PT154 alog81361
• EPO pictures and videos of site
• ALS beam profiling alog81358
• ETMX DAC swap, back to the old DAC alog81350
• LVEA oscilloscope installed alog81359
• IMC testing alog81356
  • IMC offline ISS off, PMC and FSS locked
  • Unplugging and plugging in ISS FF cable
  • IMC offline, ISS off, FSS off, only PMC

We did not do a DAQ restart

Oli, Camilla WP12203. Repeat of some of the work done in 2019: EX: 52608, EY: 52636, older: part 1, part 2, part 3.

We misaligned ITMY and turned off the ALS-Y QPD servo with H1:ALS-Y_PZT_SWITCH and placed the Ophir Si scanning slit beam profiler to measure both the 532nm ALSY outgoing beam and the ALSY return beam in the HWS path.

The outgoing beam was a little oblong in the measurements but looked pretty clean and round by eye, the return beam did not! Photos of outgoing and return beam attached.  Outgoing beam was 30mW, return beam 0.75mW. 

Attached is the 13.5% and D4sigma measurements, I also have photos of the 50% measurements if needed. Distances are measured from the optic where HWS and ALS beams combine, ALS-M11 in D1400241. 

We had previously removed HWS-M1B and HWS-M1C and translated HWS-M1A from whats shown in D1400241-v8 to remove clipping. 

TITLE: 11/19 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Preventive Maintenance
OUTGOING OPERATOR: Ryan C
CURRENT ENVIRONMENT:
    SEI_ENV state: USEISM
    Wind: 10mph Gusts, 8mph 3min avg
    Primary useism: 0.04 μm/s
    Secondary useism: 1.47 μm/s
QUICK SUMMARY: Due to very rapidly rising microseism, H1 will not be locking this evening or overnight, which allows us to run a long test for PSL glitch hunting by leaving just the PMC locked (no FSS, ISS, IMC). The DB37 cable was unlplugged from the FSS fieldbox at 23:45 UTC.

Camilla C., TJ

Recently, the CO2Y laser that we replaced on Oct 22 has been struggling to stay locked for long periods of time (alog81271 and trend from today). We've found some loose or bad cables in the past that have caused us issues, so we went out on table today to double check they are all ok.

The RF cables that lead into the side of the laser can slightly impact the output power when wiggled, in particular the ones with a BNC connector, but not to the point that we think it would be causing issues. The only cable that we found loose was for the PZT that goes to the head of the laser. The male portion of the SMA that comes out of the laser head was loose, and cannot be tightened from outside of the laser. We verified that the connection from this to the cable were good, but wiggling it did still introduce glitched in the PZT channel. I don't think that we've conviced ourselves that this is a problem though, because the PZT doesn't seem to glitch when the laser loses lock and instead it will run away.

An unfortunate consequence of the cable wiggling was that one of the Beckhoff plugs at the feedthrough must have been unseated slightly and caused our mask flipper read backs to read incorrectly. The screws for this plug were not working so we just pushed the plug back in to fully seat it and all seemed to work again.

We still are not sure why we've been having these lock losses lately, the 2nd and 3rd attachments show a few of them from the last day or so. They remind me of back in 2019 when we saw this - example1, example2. The fix ultimately a chiller swap (alog54980), but the flow and water temp seem more stable this time around. Not completely ruling it out yet though.

LVEA has been swept. Just had to unplug a(n already off) PEM function generator and turn off the lights for the HAM5/6 giant cleanroom.

Today we swapped the PT154 gauge on the FC-A section that was reported faulty in alog 81078 with a brand new gauge, same make/model.

FCV1 & 2 were closed to isolate the FCA section, and the angle valve on the A2 cross closed to isolate the gauge. Volume was vented with dry nitrogen and gauges swapped. CF connection was helium leak checked, no He signal above the HLD background ~2e-10 Torr-l/s.

Closing WP 12200

Setup a scope near the PSL rack. The channels are FSS test2, PMC mixer out, ISS PDB, and IMC servo test 1. The trigger has been connected to the IMC REFL shutter. The shutter usually triggers upon a lock loss, or when more than ~4W are detected in reflection of the IMC.

Sheila, writing for a large crew (Jason, Vicky, Daniel, Keita, Marc, Rick)

This morning we spent about 3 hours with the IMC offline and the ISS autolocker requested to off, (from 16:30 UTC) then unplugged the IMC feedback to the IMC VCO (from 17:08 UTC to 11:54 UTC) . 

In the attached screenshot you can see that there are a few large disturbances during this time, which show up with an amplitude of 0.1 on the PMC mixer channel, and large oscillations in the FSS fast monitor (30 counts), dips in the reference cavity transmission, and glitches in the PMC high voltage. The second screenshot shows a zoomed in version of the problem times.

We also noticed that unplugging the feedback from the IMC to the VCO changed the reference cavity transmitted power, by about 1%, Daniel suggests that this might be OK because the change in laser frequency causes a small change in alignment out of the AOM.

We set the FSS autolocker to off at 19:54 UTC.  This doesn't acutaly disable the FSS, but we can track times when the reference cavity would go through resonance by watching in the trans PD.  At 21:19 UTC, Ryan Crouch started locking the IFO as the winds are supposed to be calm this afternoon (we did not see any glitches in this short test but need a longer one). 

We will plan to continue this PMC only test overnight when the wind is supposed to come back up.

Related alog: https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=81320

Sometime ago (July/09/2024) the analog DC gain of H1:ASC-AS_B_DC was set from "High" to "Low" when I was looking to extract more information from WFSs about the power coming past the Fast Shutter when the shutter bounces down after high power lock losses. Since this is not necessary any more (see plots in alog 81130) and since keeping this means that we have to either adjust dark offset once in a while and/or making the "shutter closed" threshold for Fast Shutter test (alog 81320), I set the gain switch back to "HIGH", and disabled the +20dB digital gain in H1:ASC-AS_B_DC_SEG[1234] FM4.

Interestingly, when I flipped the gain switch, SEG3 output didn't change (1st attachment). It could be that the specific DC channel is stuck to HIGH or LOW, but it could be that the analog offset happens to be really small for that channel. I cannot test it right now as we're leaving IMC unlocked. As soon as the light comes back I will go to the floor and test.

The second attachment is the relevant MEDM screen (that shows the state it should be in now) and the third one is the picture of the interface chassis (the switch I flipped is circled in red, and it's supposed to be HIGH now).

It looks like usually the PSL PMC + FSS can stay locked for long periods of time, for example >1 month over the emergency vent in August. See the attached trends of the # of locked days and hours in the last few months (ndscope saved as /ligo/home/victoriaa.xu/ndscope/PSL/pmc_fss_can_stay_locked_over_emergency_o4b_vent.yaml).

The last time the PMC was locked for almost a month ended on Sept 23, 2024. Since then the PSL PMC has not stayed locked for over 5 days, but this is most likely due to commissioning tests and debugging which started around then.

Bypass will expire:
Tue Nov 19 10:45:37 PM PST 2024
For channel(s):
    H0:FMC-CS_FIRE_PUMP_1
    H0:FMC-CS_FIRE_PUMP_2
 

Tue Nov 19 10:12:46 2024 INFO: Fill completed in 12min 42secs

WP12201

Marc, Fil, Dave, Ryan

h1susex is powered down to restore cabling from the new 28bit LIGO-DAC to the original 20bit General Standards DACs.

Procedure:

Put h1iopseiex SWWD into long bypass

Safe the h1susetmx, h1sustmsx and h1susetmxpi models

Stop all models on h1susex

Fence h1susex from the Dolphin fabric

Power down h1susex.

2024 Nov 12

Neil, Fil, and Jim installed an HS-1 geophone in the biergarten (image attached). HS-1 is threaded to plate and plate is double-sided taped to the floor. Signal given was non-existent. Must install pre-amplifier to boost signal.

2024 Nov 13

Neil and Jim installed an amplifier (SR560) to boost HS-1 signal (images attached). Circuitry checked to ensure signal makes it to the racks. However, when left alone there is no signal coming through (image attached, see blue line labelled ADC_5_29). We suspect the HS-1 is dead. HS-1 and amplifier are now out of LVEA, HS-1's baseplate is still installed. We can check one or two more things, or wait for more HS-1s to compare.

After discussions with control room team: Jason, Ryan S, Sheila, Tony, Vicky, Elenna, Camilla 

• Prior to September 13th:
  • No IMC locklosses 80951
  • NPRO/FSS glitches present but not bothering us (June 80520, last year 67718).
• September 12th: IMC gain redistribution 79998
• September 13th: First IMC lockloss
• September to October:
  • We start noticing FSS glitches and do of tests to understand these.
  • They were present in NPRO only with everything else unlocked (part of reason for swapping NPRO)
  • Found that these glitches are not new, present in June 80520 and even last year 67718. Maybe they got worse, or maybe we just stopped surviving them as well.
  • IMC has periods of being unstable (have to wait in DOWN)
• October 9th: IMC gain redistribution reverted 80566
  • We have 6 days with only one IMC lockloss 
• October 15th: NPRO current increased 80687
  • IMC lockloses started again 
• Ocotber 18th: NPRO current reverted 80746
  • IMC lockloses continued
• October 23rd to 29th: NPRO swapped 80929
  • Swapped to new faraday and EOM, both found to be dodgy
• Ocotber 30th: Back to NLN
  • Had issues with NPRO 81073: found to be NPRO mode hopping 
  • IMC locklosses continued 
• October 30th to November 1st: Tests.
  • CARM OLG 80956
  • IMC OLG 80979
  • FSS OLG 80993 
  • If the PSL ISS is unlocked the IMC can stay locked: 81056
  • Attempt to swap TTFSS (reverted) 80983
  • Hardboot of PSL chassis h1psl0: 81048 
• November 6th: PSL mode hopping was fixed by changing NPRO temperature, 81107. 
  • PSL now stable
  • IMC has periods of being unstable
  • IMC locklosses continue 

Conclusions: The NPRO glitches aren't new. Something changed to make us not be able to survive them as well in lock. The NRPO was swapped so isn't the issue. Looking the timing of these "IMC" locklosses, they are caused by something in the IMC or upstream 81155.

Tagging OpsInfo: Premade templates for looking at locklosses are in /opt/rtcds/userapps/release/sys/h1/templates/locklossplots/PSL_lockloss_search_fast_channels.yaml and will come up with command 'lockloss select' or 'lockloss show 1415370858'.