Ryan had two locklosses from an oscillation just below 1 Hz that showed up in the SRCL error signal and CSOFT P (and a little less in yaw) this evening during the state TRANSITION_FROM_ETMX. 

I had a look at the state and found that there was a step where we were clearing histories repeatedly while waiting for the bias to reach 0, this probably wasn't the problem but I added and increment to the counter to stop doing this. 

Since the oscillation rings up very slowly, I lowered the bias ramp times from 60 seconds to 30 seconds.  This worked once, although I'm not sure it always will. 

We've now passed through that state and the guardian is waiting in OMC_WHITENING.  I will set it to automatically go into observing when the violins are low enough.

TITLE: 11/23 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: None
SHIFT SUMMARY: Once Jason, Vicky, and I finished recovering the PSL and taking closeout transfer functions, initial alignment began right away with the intent to get H1 back to observing tonight. These are just some quick notes about the locking progress this evening, which at this point have not been successful due to locklosses at TRANSITION_FROM_ETMX and from ALS-X randomly dropping out.

• 02:46 - Starting initial alignment
  • Both arms needed help locking ALS, BS needed quite a bit of adjustment to align MICH, but otherwise straightforward
• 03:34 - Starting lock acquisition
  • Despite just doing an alignment and buildups looking good, DRMI and PRMI wouldn't lock, so went to CHECK_MICH_FRINGES
  • 04:02 - Noticed while SEI_CONF was set to 'WINDY' (and showing it on the wall Ops Overview), SEI_ENV was still in 'MAINTENANCE' so sensor correction had not been fully engaged until this point; set SEI_ENV to 'AUTOMATIC'
  • PRMI locked immediately after turning sensor correction on
  • 04:30 - Straightforward locking until lockloss @ TRANSITION_FROM_ETMX; looked similar to this known issue (alog81195)
  • 04:42 - Lockloss @ ACQUIRE_PRMI due to ALS-X transmission looking noisy then dropping out completely (example of this behavior attached; this does not always unlock the arm)
  • 04:48 - Lockloss @ CHECK_MICH_FRINGES due to same ALS-X issue as before
  • 05:30 - Another lockloss @ TRANSITION_FROM_ETMX; notable ringup in DHARD_P leading up to this'
  • 05:54 - Another lockloss @ ACQUIRE_PRMI from ALS-X

H1 is currently relocking up to ENGAGE_ASC_FOR_FULL_IFO, but if TRANSITION_FROM_ETMX continues to be unsuccessful, H1 will be left down overnight.

V. Xu, R. Short, J. Oberling

Short short version: The PSL NPRO swap is done and IFO recovery has begun.  I'll get a more detailed alog in tomorrow, right now I'm exhausted.

After the very successful PSL work this week, we're electing to not offload the temp changes for the ALS or SQZ lasers to their physical knobs tonight.  This means that they all need Crystal Freqs of something near 1.6 GHz (1600 in the channel H1:ALS-X_LASER_HEAD_CRYSTALFREQUENCY).  The new-ish ALS state CHECK_CRYSTAL_FREQ is written assuming that all the lasers have the changes offloaded to their knobs, so that the value in that channel is close to zero.  After we went through SDF revert, we lost those values (and the search values, which Daniel has updated in SDF for the weekend), so we we lost the PLL lock of the aux lasers.  The CHECK_CRYSTAL_FREQ state was 'fighting' us, by putting in candidate values closer to zero.  I've updated its candidate values to be closer to 1600.  Once we offload their temps to the knobs on their front panels (early next week), then we'll want to undo this change.

While the PSL team finishes up their work, I wanted to get a jump start on alignment, so I moved the ITMs, ETMs, and TMSs back to where their top mass OSEMs said that they were, the last time that we had good transmission for both ALSX and ALSY while trying to lock.  This indeed got somewhat okay flashes on both ALSs, so will be a fine place to start from, once the reference cavity is locked again.

However, I found that TMSY's pitch slider has a minus sign with respect to the OSEM readbacks for pitch.  This is the only slider (out of pit or yaw, for all 6 optics that affect ALS alignment for either arm) that seems to have this issue. 

In the attached screenshot, when I try to move the TMS up the OSEM readbacks say it went down, and vice versa. 

Not any kind of urgent matter, but may make it more challenging for ALS auto-alignment scripts to do their work.

TITLE: 11/22 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
INCOMING OPERATOR: None
SHIFT SUMMARY:

NPRO Swap work continued today by Jason, Vicky, and Ryan, with quite a bit of progress this morning, but the PSL crew is still on the floor  (currently seeing flashes of RefCav Trans) and so the status for the return to observing is not clear for tonight.  (I will be in for my DAY shift tomorrow.)

Trend of the new PSL HV monitor H1:PEM-CS_ADC_4_28_2K_OUT_DQ is attached.  It has been flat/constant for most of the last 24+hrs, but in the last couple hours it has moved a bit (correlated with PMC locking or ISS?).
LOG:

• 1553-1627 LVEA clean-up/restocking mainly around the h1 psl & then Optics lab (karen)
• 1645-2045 NPRO swap (jason,vicky)
  • 2150-NPRO swap continues (jason, vicky, ryan)
• 1705 alsy work from CR (tj)
  • 2200 Continuing after lunch
• 1711 EY Wind Fence inspection (mitch, jim)
• 1736-1752 laser hunting (camilla)
• SEI:  itmx
• 1752-1759 h1 msl tv monitor powered on (tony)
• 1800 replacing ceiling light fixtures in PCal lab which also required clean room move (Eric w/ help from tony, gerardo, fil)
• 2030-2100 Photos of temporary seismometer (lance)
• 2115-2145 Mounting seismoeter in Beer Garden area (jim, neil)
• 2120-2145 UW Physics tour (camilla)
• 2155-2240 EX for taking photos, Jim thing, "microphone", magnetometer (lance)
  • 2241-2303 EY to take similar photos (lance)
• 2155-2208 PSL racks (richard)

Adam Mullavey from LLO made some code to move the ETM and TMS in a spiral to scan for flashes in the arm. LLO has been using this for O4, and in terms of an automation step, this would be comparable to our Increase_Flashes state. Increase_Flashes is very simple in how it works - move one direction, one degree of freedom at a time, look for better cavity flashes, move the other way if they get worse. While this is reliable, it is very slow since we have to wait for one period of the quad between each step (20seconds) to ensure we dont miss a flash.

The last two days I spent some time converting Adam's state, Scan_Alignment, for use at LHO, adjusting thresholds and other parameters, and trying to improve on parts of it so it might work a bit more reliably. The most notable change I've made is to get data from the fast channel for the arm transmission, rather than collecting slow channel data. This seemed to help a bit, but this completely relies on nds calls that we've historically found not 100% reliable. I've also lowered minimum thresholds, thanks to the previous change. This allows it to start off from basically no light in the cavity and bring it up to a decent alignment.

After these changes it seems to really improve the flashes from a very misaligned starting point, but I'm not sure it's any faster than the Increase_Flashes state. In the attached example it took around 20-30 min to go from little to no light to a decent amount. I'm testing this without the PLL and PDH locked, so it's hard to say exactly how well aligned it is and how much better it can get. Next I'd like to take some time on a Tuesday to test with a PDH&PLL locked ALS, and compare it to the time it takes compared to Increase_Flashes for a very misaligned cavity and a barely misaligned cavity.

I've committed the changes I've made to ALS_ARM.py, ALS_YARM.py (both in common) to the SVN. This created a new state - SCAN_ALIGNMENT - that I'll keep there, but isn't in the state graph so it cannot be reached.

Closes FAMIS 26342. Last checked in alog 81307.

All trends look well. No fans above or near the (general) 0.7 ct threshold. Screenshots attached.

Here are some plots relevant to understanding our uptime and down time from the start of O4 until Nov 13th, with some comparisions to Livingston.  I'm looking at times when the interferometer is in the low noise state (for H1 ISC_LOCK >=600, for L1 ISC_LOCK >=2000). 

The first piechart shows what guardian states we spend the most time in, this is pretty similar to what Mattia reported in 79623

The histograms of lock segement lengths and relocking times show that L1's lock stretches are longer than H1s, and that we've had 34 individual instances where we were down for longer than 12 hours.  (And that H1 has a lot of short locks)

The rolling and cumulative avergage plot shows how the drop in duty cycle in O4b compare to O4a is due to individual problems, including the OFI break, pressure spikes, and laser issues.

Lastly, the final plot shows how we accumulate uptime and down time bined by length of the segments.  This shows that L1 accumulates more uptime than Hanford by having more locks in the 30-50 hour range.  The downtime accumulation shows that just under half of our downtime is from times when we were down for more than 16 hours (serious problems), and about 1/4 of it is due to routine relocking that takes less than 2.5 hours.

The script and data used to make these plots can be found in DutyCycleO4a.py and H1(L1)ISCLockState_04.txt in this git repo.

Closes FAMIS#26018, last checked 81331

Nothing looks out of the norm

Yesterday Gerardo noticed that the south most section of the EY wind fence had some broken cables on the lower half of the fence on the panel we did NOT replace last summer. I think we have a couple of ways we could go about repairing this. We will discuss options, but the weather is not great for this kind of work.

Fri Nov 22 10:10:42 2024 INFO: Fill completed in 10min 39secs

Gerardo confirmed a good fill curbside through the tumbleweeds. Minimum TC temps are getting close to their trip temps (trip=-100C, TCmins=-117C,-115C). I have increased the trip temps to -90C for tomorrow's fill. Looking at a yearly trend, we had to do this on 25 Nov last year, so we are on schedule.

On Tuesday NOV 19th Eric started the replacement of the ceilign light fixtures in the PCAL Lab.
Francisco and I grabbed 3 HAM Door covers to stretch out over the PCAL table to minimize dust particles on the PCAL optical bench.
I also put all the Spheres away in the cabinet and made sure that they all had their apature covers on.
I went in to shutter the laser using the internal PCAL TX module shutter, and the shutter stopped working.
I then just powered off the laser, removed the shutter, and repaired the shutter in the EE lab.
Put in an FRS ticket: https://services1.ligo-la.caltech.edu/FRS/show_bug.cgi?id=31730
The Shutter is repaired and just waiting for the FAC team to finish the Light fixture replacement to reinstall.

Update: Friday NOV 22nd, there are 2 more light fixtures in the PCAL LAB that need to be replaced. One directly above the PCAL Optics table.
PCAL Laser Remains turned off with the key out. 
 

Obviously, lots of reds and zeros due to ongoing NPRO work.
Laser Status:
    NPRO output power is 1.839W
    AMP1 output power is -0.6214W
    AMP2 output power is 0.07093W
    NPRO watchdog is GREEN
    AMP1 watchdog is RED
    AMP2 watchdog is RED
    PDWD watchdog is RED

PMC:
    It has been locked 0 days, 0 hr 0 minutes
    Reflected power = -0.4062W
    Transmitted power = -0.02552W
    PowerSum = -0.4317W

FSS:
    It has been locked for 0 days 0 hr and 0 min
    TPD[V] = -0.01703V

ISS:
    The diffracted power is around 4.0%
    Last saturation event was 0 days 22 hours and 52 minutes ago

Possible Issues:
    AMP1 power is low
    AMP2 power is low
    AMP1 watchdog is inactive
    AMP2 watchdog is inactive
    PDWD watchdog is inactive
    FSS TPD is low
    Service mode error, see SYSSTAT.adl

Using the lockloss page, I picked out some times when the tag "IMC" or "FSS_OSSCILATION" was triggered. Then, I just made some simple time series comparison plots between several PSL channels and PSL PEM channels. In particular, the PSL channels I used were:

H1:PSL-FSS_FAST_MON_OUT_DQ, H1:PSL-FSS_TPD_DC_OUT_DQ, H1:IMC-MC2_TRANS_SUM_IN1_DQ, H1:PSL-PWR_NPRO_OUT_DQ, H1:PSL-PMC_MIXER_OUT_DQ, H1:PSL-PMC_HV_MON_OUT_DQ. I compared those to the following PSL PEM channels: 

H1:PEM-CS_ACC_PSL_PERISCOPE_X_DQ, H1:PEM-CS_ACC_PSL_PERISCOPE_Y_DQ, H1:PEM-CS_ACC_PSL_TABLE1_X_DQ, H1:PEM-CS_ACC_PSL_TABLE1_Y_DQ, H1:PEM-CS_ACC_PSL_TABLE1_Z_DQ, H1:PEM-CS_ACC_PSL_TABLE2_Z_DQ, H1:PEM-CS_MIC_PSL_CENTER_DQ. For this analysis, I've only looked at 3 time periods but will do more. Below are some things I've seen so far:

1. I definitely do see correlations between the IMC/FSS (IMC_MC2_TRANS & FSS_FAST_MON)  channels and the PEM PSL sensors. In particular, it seems like the most consistent correlations are with the PSL_PERISCOPE_X/Y and PSL_TABLE1_X/Y/Z channels. In some cases, there was a visible glitch in PSL_TABLE2_Z, but the amplitude is much less than in the other PEM channels.
2. Some PEM glitches correlate with the glitching in PMC_HV_MON, but this was only the case for 2 of the 3 times I looked at.
3. Some PEM glitches correlate with PMC_MIXER, but I wonder if that is just a coincidence (see slide 9 for an example)
4. I have seen no obvious correlation with the PEM PSL sensor glitches and the NPRO_PWR channel.

From my very limited dataset so far, it seems like the most interesting area is where the PSL_PERISCOPE_X/Y and PSL_TABLE1_X/Y/Z channels are located.

WP12214

Marc, Ryan C, Dave:

h1susex has been fenced from Dolphin and powered down in preparation to re-establishing the LIGO DAC 28ao32 as the driver of the ETMX L1/L2 and ESD channels, essentially undoing what was done on Tuesday.

Setup a scope near the PSL rack. The channels are FSS test2, PMC mixer out, ISS PDB, and IMC servo test 1. The trigger has been connected to the IMC REFL shutter. The shutter usually triggers upon a lock loss, or when more than ~4W are detected in reflection of the IMC.

2024 Nov 12

Neil, Fil, and Jim installed an HS-1 geophone in the biergarten (image attached). HS-1 is threaded to plate and plate is double-sided taped to the floor. Signal given was non-existent. Must install pre-amplifier to boost signal.

2024 Nov 13

Neil and Jim installed an amplifier (SR560) to boost HS-1 signal (images attached). Circuitry checked to ensure signal makes it to the racks. However, when left alone there is no signal coming through (image attached, see blue line labelled ADC_5_29). We suspect the HS-1 is dead. HS-1 and amplifier are now out of LVEA, HS-1's baseplate is still installed. We can check one or two more things, or wait for more HS-1s to compare.