TITLE: 12/31 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 4mph Gusts, 3mph 3min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.18 μm/s
QUICK SUMMARY: H1 lost lock at 14:43 UTC, ending this lock stretch at just over 29 hours. The online lockloss tool has not been updating since 12/26 at 20:14 UTC (5 days ago), so I don't know of a cause yet, but I plan to look into it. H1 is currently automatically relocking and up to RESONANCE.

• Wind low, secondary microseism trending down to around the 50th percentile
• LAB2 DM still in error
• All other systems nominal

TITLE: 12/31 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 159Mpc
INCOMING OPERATOR: TJ
SHIFT SUMMARY: Currently Observing at 160 Mpc and have been Locked for over 20.5 hours. A few minutes ago we got another superevent candidate! Looking nice in H1! The evening has been quiet with nothing needing to be touched by me!

I did notice a notification on SQZ_OPO_LR that says "CMB EXC is on", and looking at the channel it checks, it looks like SQZ-OPO_SERVO_COMEXCEN went from 0 to 1 around 09:57 UTC on 12/30, which is a bit after we got locked last night and while we were having the SQZ issues. There are no excitations listed on the test point or CDS Overview screens, so I am wondering if this is some error? Tagging SQZ
LOG:

05:41 Superevent S241231bg

TITLE: 12/31 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 159Mpc
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 6mph Gusts, 5mph 3min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.22 μm/s
QUICK SUMMARY: Forgot to put in my start of shift alog, so here it is as a midshift update. Currently Observing and have been locked for 17.5 hours. Quiet evening with nothing to report.

TITLE: 12/30 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 132Mpc
INCOMING OPERATOR: Oli
SHIFT SUMMARY:

Locked entire shift (of a 15+hr lock) with (2) drops from Observing mostly for Squeezer (and also due to PI24).  H1 was also mostly with No Squeezing since last night, and then Sheila and Daniel were able to get squeezing back around lunchtime (alog)---just in time for a superevent in the afternoon!

Environmentally we are looking good with lower microseism and low winds!
LOG:

• 1556 H1 Knocked out of OBSERVING due to rung up PI24 (it would redamp its self within a minute)
  • While out of observing looked at Squeezer issues for about 20min, but then returned to Observing with NO Squeezing, to return to this when Sheila would be available.
• 1805 Tyler phoned about heads up of King Soft arriving today (est 1930utc) for taking a water sample at woodshop.  They have a card to get through gate and will be non-invasive for observing
• 1900 Noticed that I couldn't hear Verbal.  Restarted verbal to no avail and then noticed the speaker icon on the desktop was muted.  Not sure why/how that happened, so it was unmuted.
• 1955 Out of Observing for Sheila to look at the Squeezer.  She noted the issue with the ISS (as RyanC mentioned last night) and also mentioned the SQZT0 AOM.  She made adjustments with a SQZT0 half waveplate which is just downstream of that AOM (same one Daniel adjusted to address high rejected power issue from yesterday).  She could only look at this briefly and then messaged Daniel about what she saw.  She said if her change did not work for getting the MANAGER to FDS_READY_IFO, then we can return to Observing with NO SQUEEZING.  After a few minutes, Daniel hopped online after getting Sheila's message.  
  • After a while, Daniel was able to get the SQZ to a good spot
• 2043 Back to Observing
• 2336 Superevent S241230ev
• 2230--2350 Site WalkThrough:  Corner Area + Road-To-MX (corey---with a pitchfork for tumbleweeds!)

We seem to be running into repeated problems where the green power is not sufficient to get to the nominal OPO green transmitted power.

Today the SHG output was particular low, so I adjusted the input pointing as well as lowered the temperature to 35.0C (from 35.5C). This gave us about 10% more green power, but still not enough! I then further increased the green power by adjusting the waveplate in front of the cube.

The rejected power (H1:SQZ-SHG_REJECTED_DC_POWERMON) is currently showing 1.8mW, down from around 4.7mW when I started. There is a little bit further to go, but we may have to realigning the green fiber coupler on the table. And if this doesn't work, we probably need to coonsider moving the OPO crystal.

Link to Shift Summary 

• The duty cycle for this week was 30.4%
• H1 was out of observing for almost the whole day of 12/24 and 12/27. See aLOG(81960), aLOG(81964), aLOG(82012) and aLOG(82016)
• Some days, the lock losses were due to earthquakes, wind and ground motion, some due to HAM issues and some have unknown causes.
• Average range was between 155-165 Mpc for the days that were majorily in locked mode.
• The week had pretty high wind and ground motion for the majority part of multiple days. Some days seismic activity was also high contributing to mutiple locklosses. 
• The Corner BSC 10 has constant high amplitude low frequency noise multiple days.
• H1 EX and H1 EY Acoustic PEM somedays has long periods of low frequency noise in the spectrograms. 
• Some HVETO round winner channels were new while some repeated for multiple days. See the shift summary for more details. 
• FScan plots were not available for 3 days of the week. While, the lines remained under 600 for the rest of the days. 

Mon Dec 30 10:16:30 2024 INFO: Fill completed in 16min 27secs

TCmins [-91C, -45C] OAT [+6C, 42F]. TC-A started positive, hence multiple decades in log plot.

1556utc H1 was taken out of observing due to PI24 ringing up (it was damped out within a minute), BUT....

Since we were out of Observing (and I had already gotten a workspace ready to work on the Squeezer), I wanted to try a "simple" Squeezer check (basically what Oli did with taking the Manager to DOWN and going back to FDS.

I thought I was close to FDS, because all of the sub-nodes went to NOMINAL states and then I believe the Manager would get to FDS, but then go to NO SQUEEZING.....Then I realized all their NOMINAL states were set for NO SQUEEZING! 

So I worked on transitioning all 8-nodes Nominal states for SQUEEZING, to work on squeezing, but the issue I ran into was that the OPO_LR node had trouble getting to LOCKED_CLF_DUAL due to "USERMSG 0: No pump ISS means injecting bad squeezing. Going DOWN."

--->This sounds like yet another different SQZ issue.  I did not have issues with the SQZ_FC as far as I could tell (also did not get to see the "leap second" issue RyanC noted last night).

At this point, I was stumped, but got a mattermost from Sheila saying she could help in a bit. So I switched all SQZ node Nominal states for NO Squeezing to get more Observing in the mean time.

SIDE NOTE:  The buisiness for transitioning SQZ Nominal States is not quick or trivial (atleast for my slow self).  While  transitioning nodes for Observing WITH Squeezing, I ran into a LOADING error for the OPO_LR node.  Eventually found that it was a typo in the wiki (for editing all the nodes I copied the state from the wiki into the node python files.  I discovered the wiki had a typo [or the state had a slightly different name int he past].  For OPO_LR, the wiki said to make the Nominal -> "LOCK_CLF_DUAL".  Eventually figured out from the log, that this name was wrong it is "LOCKED_CLS_DUAL".  At any rate, I updated wiki.  And now have more Nominal node experience...I think this is only my 2nd or 3rd time going through the buisiness of "editing nominal states" for observing.

TITLE: 12/30 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 132Mpc
OUTGOING OPERATOR: Ryan C
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 6mph Gusts, 5mph 3min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.20 μm/s
QUICK SUMMARY:

H1's been locked for 6.25hrs (with no squeezing); we have triple coincidence now, but debating whether to drop out of Observing for Squeezer investigations---It feels like we've had a variety of squeezer issues for the last 36 hours through multiple differnt locks.  Microseism continues it's downward trend with it down to touching the 50% percentile line.  Winds have been at or under 10mph for most of the last 24hrs.

(Although Oli spotted the male elk while they left the site, he was not spotted this morning.)

SQZer not locked, SQZ_FC gave a notification about Leap data being expired in its log. I tried bringing FC1 and FC2 back to where they were the last time we locked the SQZer to to avail, following the wiki I tried to walk through the locking process but the ISS keeps losing lock and couldn't stay locked, I keep seeing the 10 locklosses notification. I wasn't able to get the ISS to lock, I ended up taking us to obs without SQZing, I had to accept a few SDFS.

10:31 UTC Observing

TITLE: 12/30 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 152Mpc
INCOMING OPERATOR: Ryan C
SHIFT SUMMARY: Currently Observing at 155Mpc and have been Locked for 2.5 hours. One lockloss during my shift and relocking was entirely hands-off, but we had trouble getting into Observing because the squeezer wouldn't lock.

The FC would go to FC_ASC_ON, then less than a second later the OPO would drop from LOCKED_CLF_DUAL to LOCKED_CLF_DUAL_NO_ISS and then ENGAGE_PUMP_ISS, and either at the same time or immediately following this, the LO guardian would unlock from the OMC, and the filter cavity would go to DOWN a few seconds later. This happened over and over until I took SQZ_MANAGER to DOWN and then back to FDS, at which point we were able to get squeezing and go into Observing. I originally thought this issue was related to the problem Corey and Daniel worked on earlier, but the values for SQZ-SHG_FIBR_REJECTED_DC_POWERMON were still nice and low to where Daniel had adjusted them to, and I don't think there was an issue with the OPO temperature since we were able to lock with the ISS with the setpoint at 80uW?? Tagging SQZ
 

LOG:

00:30 Relocking and at MOVE_SPOTS
00:46 NOMINAL_LOW_NOISE
00:48 Observing

02:19 Lockloss
03:34 NOMINAL_LOW_NOISE
    - SQZ was having issues locking
    - Got it figured out by going to NO_SQUEEZING and then back to FDS
03:49 Observing

Lockloss @ 12/30 02:19 UTC

TITLE: 12/29 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 161Mpc
INCOMING OPERATOR: Oli
SHIFT SUMMARY:

H1 was locked over 17hrs until it had a LockLoss (whose cause is unclear--LockLoss tool is down).  Before the lockloss there was a drop from Observing due to the Squeezer.

Microseism continues it's downward trend of the last 24hrs (and winds were not an issue today).
LOG:

• 1557 Seismic to CALM (from USEISM since 12/25 ~18utc)
• 1620-1715 Bull elk sighting on-site with LExC camera. 
• 1730--1808 tumbleweed clearing w/ pitchfork on X-arm for LN2 delivery on NYE (corey)
• 1820 Chatted w/ Dave about his work to free space for nds1 & to also make the CDS Alarm box on CDS Overview GREEN (it had been RED due to the HAM6 PT110 failure a few days ago).
• 1925-2056 Out of Observing Due to SQZ_OPO_LR / halfwave plate adjustment (phoned Daniel to help)
• 2107 Saw the PCal trace on DARM FOM have a big glitch!
• 2229 LOCKLOSS (after 17hrs!)
  • ALS Scan Alignment completed, but DRMI/PRMI didn't look good.
  • 2249-2305 Manual Initial Alignment (since green arms went through Scan Alignment
  • ALSx had 5 locklosses (looked as if ASC drove it off)
  • Eventhough ran a Manual Alignment, PRMI & DRMI were a bit tough to acquire (even with lower microseism and low winds), but currently on our way to NLN
• 2250-2300 EQ Mode activated  (M5.1 Ethiopia EQ)  L1 loses lock around this time.
• 2323 Site Walkthru: Corner Area (Oli)

TITLE: 12/30 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 4mph Gusts, 2mph 3min avg
    Primary useism: 0.05 μm/s
    Secondary useism: 0.27 μm/s
QUICK SUMMARY:

Currently relocking and at ENGAGE_SOFT_LOOPS. Secondary microseism has gone way down over the last half a day and wind is low.

(CoreyG, DanielS)

At 1925utc, H1 dropped out of Observing due to the Squeezer.  After a couple minutes, could see that the MANAGER & LO_LR nodes were cycling and not getting anywhere. 

Wiki First

At this point, I opened the Troubleshooting SQZ wiki's "Can't Get To FREQ_DEP_SQZ" section.  From this, I took MANAGER to NO SQUEEZING and then opened up all the SQZ Nodes. 

At this point I notice the OPO_LR node which had been green is actually also cycling (but not as fast as the MANAGER & LO_LR.  And OPO_LR was also  showing a User Message, BUT it displayed too quick that I had to take a screenshot really quick to catch it.  This node would hang out at the NOMINAL state (LOCKED CLF DUAL), drop out from this, display a message(s) for 1sec and then return to nominal state....all within a minute or so.  There were actually 2 messages, they were to the best I could see (also attached):

1. pump fiber rej power in ham7 high, nominal 35e-3, align fiber pol on sqzt0
2. Disabling pump iss after 10 lockloss counter.  Going to through LOCKED CLF_DUAL_NO_ISS to turn on agai

At this point, I started diving into medms for anything related to the shy messages above.  "pump fiber rej power" gave me the most info for where to look, but at this point, I also wanted to do an alog search.

Then Alog Search

I searched OPO_LR, where I focused on an alog from (1) RyanS which referenced a Tony alog and (2) RyanC.  These three alogs narrowed my focus and had me looking at:

1. HAM7 Fiber Rejected Power signal
2. SQZT0 waveplate

The Rejected power has been trending for the last month+ up to a max value around 3.0 (where it starts from about 0.03).  I wasn't totally sure I had the scenario right, so at this point I reached out to Daniel (main contact for SQZ on the Call List).

Phoning Daniel For Help

After giving a summary the best I could, Daniel logged in and started looking.  Some of the main points he mentioned:

• Fiber polarization
• Did not have enough green power
• Running out of range
• Crystal may be degrading
• Need more Green power!

With all that said,m he was able to get the LO_LR node to its nominal state, but he mentioned we could also fix the Fiber Rejected Power (atleast until the new year or about a week!).  So he started to work on this.  And this ended up being what was done by RyanS in his alog.  See attached for image showing which waveplate/picomotor he adjusted on SQZT0 along with the HAM7 Fiber Rejected Power signal he watched & reduced.  See attached for zoom in of the rejected power signal adjusment he made. 

Daniel mentioned there will probably be a nead to make adjustments out on the table, but this change can get us through the holiday break.

Return to OBSERVING at 2056utc

nds1:

The /var file system on h1daqnds1 (the default NDS) became 100% full yesterday. This is discovered by the daily central syslogs for this machine increasing from roughly 3GB per day to 19GB yesterday.

For a short term fix, I first removed the syslog.3.gz and syslog.4.gz files to make enough room so I could compress syslog.1 and daemon.1 This reduced the disk down to 50% usage.

alarms:

To "green up" the alarms system which had been permanently RED since Thursday due to HAM6's gauge going offline I temporarily removed the PT110 channnels from the alarms system and restarted at 10:33 Sun 29dec2024.

As a place holder to remind me to return these channels back to alarms when the gauge is fixed, FRS32957 will remain open.

Channels removed:

H0:VAC-LX_Y0_PT110_MOD1_PRESS_TORR

H0:VAC-LX_Y0_PT110_MOD1_PRESS_TORR_ERROR

Sun Dec 29 10:16:20 2024 INFO: Fill completed in 16min 16secs

TITLE: 12/29 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 160Mpc
OUTGOING OPERATOR: Ryan C
CURRENT ENVIRONMENT:
    SEI_ENV state: USEISM
    Wind: 13mph Gusts, 7mph 3min avg
    Primary useism: 0.04 μm/s
    Secondary useism: 0.42 μm/s
QUICK SUMMARY:

H1 has been locked for 10+hrs!  Secondary microseism is mostly under the 95th percentile line, and winds are only around 10mph (with light rain on the drive in).