TITLE: 12/10 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Earthquake
INCOMING OPERATOR: Oli
SHIFT SUMMARY:

IFO is LOCKING and in ENVIRONMENT due to EARTHQUAKE.

We had a few good hours of locking today but as with yesterday, earthquakes have been rampant. Here are details:

• Lockloss 1: EQ caused and interrupted comissioning.
  • Tripped ISI HAM7 WD after LL (during high motion)
  • alog 81694
• Lock Re-acquisition:
  • FSS glitchy during initial alignment, resulting in DOWN until FSS relocked and I was able to re-do the initial alignment. After that, locking was fully auto.
  • screenshot attached
• Lockloss 2: EQ caused
  • Tripped many WDs
  • alog 81709
• SQZ Recovery following interlock trip
  • HAM7 interlock tripped, turning off High Voltage and unlocking SQZ. Fil turned it on in the morning and we were able to lock SQZ again. Cause of trip yet unknown but VAC knows about it - there was no immediate culprit or pressure spike when the trip happened.
  • SQZ was successfully brought back using Ryan's alog 79887
  • relevant alogs: alog 81689, alog 81697
  • SDF diffs attached
• Range dips hour into observing
  • We've been seeing our range dip down by ~5-10MPc during observing and have been trying to troubleshoot. This is not a new occurrence.
  • Went into comissioning for 10 mins to turn SQZ off (22:36 - 22:46 UTC) to see the effect on range - we saw no positive range change but seemed to stabilize the noise of the plot.
  • Camilla and TJ turned off CO2 to see if this was a potential cause but they saw no positive range change. alog 81705
  • Investigation ongoing.
• FMCSSTAT Red on CDS Overview
  • Dave noticed FMCSTAT going red on CDS overview and found that the Zone2A LVEA temps were going up very quickly. He informed FAC. Alog 81701
  • Tyler confirmed that it was not an issue, just that the temperature differential today was high and that the Zone 2A thermometer is in the wall, making it more susceptible to such excursions.

LOG:

Lockloss @ 12/09 23:12UTC due to large earthquake from Nevada. Sitting in DOWN until ground motion goes down

Camilla C, TJ S

This morning we had another period where our range was fluctuating almost 40Mpc, previously seen on Dec 1 (alog81587) and further back in May (alog78089). Camilla and decided to turn off both TCS CO2s for a short period just to completely rule them out, since previously there was correlation with these range dips and a TCS ISS channel. We saw no positive change in DARM during this short duration test, but we didn't want to go too long and lose lock. CO2s were requested to have no output power from 16:12:30-16:14:30UTC

The past times that we have seen this range loss, the H1:TCS-ITMX_CO2_ISS_CTRL2_OUT_DQ channel and the H1:SQZ-FC_LSC_DOF2_OUT_DQ channel had noise that correlated to the loss, but theISS channel showed nothing different this time (attachment 2). We also were in a state of no squeezing at the time. So it's possible that this is a completely different type of range loss.

DetChar, could we run Lasso or check on HVETO for a period during the morning lock with our noisy range?

SEI seismometer mass check - Monthly - Closes FAMIS 26496. Last Checked alog 80959

T240

Averaging Mass Centering channels for 10 [sec] ...
2024-12-09 14:00:13.262669

There are 14 T240 proof masses out of range ( > 0.3 [V] )!
ETMX T240 2 DOF X/U = -0.751 [V]
ETMX T240 2 DOF Y/V = -0.833 [V]
ETMX T240 2 DOF Z/W = -0.352 [V]
ITMX T240 1 DOF X/U = -1.576 [V]
ITMX T240 1 DOF Y/V = 0.388 [V]
ITMX T240 1 DOF Z/W = 0.488 [V]
ITMX T240 2 DOF Y/V = 0.308 [V]
ITMX T240 3 DOF X/U = -1.657 [V]
ITMY T240 3 DOF X/U = -0.762 [V]
ITMY T240 3 DOF Z/W = -2.097 [V]
BS T240 1 DOF Y/V = -0.329 [V]
BS T240 3 DOF Z/W = -0.41 [V]
HAM8 1 DOF Y/V = -0.415 [V]
HAM8 1 DOF Z/W = -0.7 [V]

All other proof masses are within range ( < 0.3 [V] ):
ETMX T240 1 DOF X/U = 0.049 [V]
ETMX T240 1 DOF Y/V = 0.002 [V]
ETMX T240 1 DOF Z/W = 0.005 [V]
ETMX T240 3 DOF X/U = 0.029 [V]
ETMX T240 3 DOF Y/V = -0.001 [V]
ETMX T240 3 DOF Z/W = 0.014 [V]
ETMY T240 1 DOF X/U = 0.092 [V]
ETMY T240 1 DOF Y/V = 0.202 [V]
ETMY T240 1 DOF Z/W = 0.263 [V]
ETMY T240 2 DOF X/U = -0.032 [V]
ETMY T240 2 DOF Y/V = 0.233 [V]
ETMY T240 2 DOF Z/W = 0.071 [V]
ETMY T240 3 DOF X/U = 0.286 [V]
ETMY T240 3 DOF Y/V = 0.127 [V]
ETMY T240 3 DOF Z/W = 0.184 [V]
ITMX T240 2 DOF X/U = 0.182 [V]
ITMX T240 2 DOF Z/W = 0.29 [V]
ITMX T240 3 DOF Y/V = 0.175 [V]
ITMX T240 3 DOF Z/W = 0.134 [V]
ITMY T240 1 DOF X/U = 0.111 [V]
ITMY T240 1 DOF Y/V = 0.129 [V]
ITMY T240 1 DOF Z/W = -0.006 [V]
ITMY T240 2 DOF X/U = 0.059 [V]
ITMY T240 2 DOF Y/V = 0.218 [V]
ITMY T240 2 DOF Z/W = 0.147 [V]
ITMY T240 3 DOF Y/V = 0.099 [V]
BS T240 1 DOF X/U = -0.127 [V]
BS T240 1 DOF Z/W = 0.157 [V]
BS T240 2 DOF X/U = -0.014 [V]
BS T240 2 DOF Y/V = 0.072 [V]
BS T240 2 DOF Z/W = -0.057 [V]
BS T240 3 DOF X/U = -0.136 [V]
BS T240 3 DOF Y/V = -0.284 [V]
HAM8 1 DOF X/U = -0.298 [V]

Assessment complete.
 

All other proof masses are within range ( < 0.3 [V] ):
ETMX T240 1 DOF X/U = -0.133 [V]
ETMX T240 1 DOF Y/V = -0.112 [V]
ETMX T240 1 DOF Z/W = -0.159 [V]
ETMX T240 3 DOF X/U = -0.1 [V]
ETMX T240 3 DOF Y/V = -0.226 [V]
ETMX T240 3 DOF Z/W = -0.103 [V]
ETMY T240 1 DOF X/U = 0.022 [V]
ETMY T240 1 DOF Y/V = 0.09 [V]
ETMY T240 1 DOF Z/W = 0.148 [V]
ETMY T240 2 DOF X/U = -0.098 [V]
ETMY T240 2 DOF Y/V = 0.152 [V]
ETMY T240 2 DOF Z/W = 0.049 [V]
ETMY T240 3 DOF X/U = 0.155 [V]
ETMY T240 3 DOF Y/V = 0.041 [V]
ETMY T240 3 DOF Z/W = 0.089 [V]
ITMX T240 1 DOF Z/W = -0.092 [V]
ITMX T240 3 DOF Z/W = -0.289 [V]
ITMY T240 1 DOF X/U = -0.25 [V]
ITMY T240 2 DOF X/U = -0.08 [V]
ITMY T240 2 DOF Y/V = -0.185 [V]
HAM8 1 DOF X/U = -0.287 [V]
Assessment complete.

STS

Averaging Mass Centering channels for 10 [sec] ...

2024-12-09 14:00:32.260633
There are 1 STS proof masses out of range ( > 2.0 [V] )!
STS EY DOF X/U = -2.37 [V]

All other proof masses are within range ( < 2.0 [V] ):
STS A DOF X/U = -0.472 [V]
STS A DOF Y/V = -0.961 [V]
STS A DOF Z/W = -0.448 [V]
STS B DOF X/U = 0.281 [V]
STS B DOF Y/V = 0.954 [V]
STS B DOF Z/W = -0.387 [V]
STS C DOF X/U = -0.777 [V]
STS C DOF Y/V = 0.737 [V]
STS C DOF Z/W = 0.654 [V]
STS EX DOF X/U = -0.145 [V]
STS EX DOF Y/V = -0.044 [V]
STS EX DOF Z/W = 0.139 [V]
STS EY DOF Y/V = 0.035 [V]
STS EY DOF Z/W = 1.273 [V]
STS FC DOF X/U = 0.216 [V]
STS FC DOF Y/V = -1.054 [V]
STS FC DOF Z/W = 0.66 [V]

Assessment complete.
 

J. Freed,

SR2 M1 stage bosems (speciffically T2, T3) show strong coupling below 20Hz

Today I did damping loop injections on all 6 BOSEMs on the  M1. This is a continuation of the work done previously for ITMX, ITMY, PR2, PR3, PRM

As with PRM, gains of 300 and 600 were collected (300 is labled as ln or L).

The plots, code, and flagged frequencies are located at /ligo/home/joshua.freed/bosem/SR2/scripts. While the diaggui files are at /ligo/home/joshua.freed//bosem/SR2/data.

Neil and Jim swapped HS-1s on the LVEA floor last week Tuesday morning (2024-12-03). Repeated huddle test and both vertical HS-1s (serial no. 1151 and 1152) work brilliantly. The two huddle images attached are not of HS-1-1151 nor HS-1-1152, but of HS-1-LHO-Brian-Lantz; the setup for the huddle tests for all 3 HS-1 units was the same. The plot shows a seismic signal (blue) and noise floor of HS-1-1152 (green).

FAMIS 31063

No major events of note this week, except that PMC REFL seems to have been consistently on the rise over the past week. Total power increase over that time has been about 0.5W with no majorly corresponding decrease in PMC TRANS.

FMCS STAT is red on the CDS Overview due to a warming of the LVEA Zone2a area. Attached 7 day trend shows it did something similar yesterday (Sunday) morning.

Mon Dec 09 10:21:24 2024 INFO: Fill completed in 21min 20secs

Gerardo confirmed a good fill curbside. Long fill with temps starting positive, hence the strange looking trend.

Summary of Week:

• Range was consistently between 150 and 165 Mpc when in observing mode
• Low frequency noise found in PEM LVEA Floor accelerometers appears to be coupling to the chambers. (See Fig 1)
• Reoccurring low frequency noise in strain around 30-40 Hz may be ringing up DARM. (See Fig 2)

More detailed information can be found on the DQ Shift Report page.

After the SQZ HV tripped off and was brought back 81689, the PSAMS servos (80685) railed. Attached plot shows the output of SAMS_SERVO at the limit and Strain gauge not at PSAMS target.

To fix this, on the  H1:AWC-ZM{2,4,5}_M2_SAMS_SERVO (small box to the left of REQUESTED PZT(V)),  I increased ramp time form 2 seconds to 5 seconds, ramped gain to 0, cleared history, ramped gain to 1 and then put ramp time back.

After these were back to their nominal, the alignments of the ZMs went back close to nominal and we were able to lock the FC.

It appears that we have lost the gauge for Y2-8, nothing to do for now since this is a monitoring tool.  We will assess tomorrow, but if the system needs to be charged, then maybe there will be enough sunlight today.

Lost the gauge early Saturday morning.

Earthquake caused Lockloss.

TITLE: 12/09 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 97 Mpc
OUTGOING OPERATOR: Tony
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 4mph Gusts, 2mph 3min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.36 μm/s
QUICK SUMMARY:

IFO is in NLN and OBSERVING without squeezing.

There were issues with the squeeze high voltage, so observing without squeezing until this is investigated today. (per Camilla's alog 81689)

We also have planned comissioning time today from 8AM-11AM (30 mins earlier than usual due to an all-hands meeting).

Other than that, it seems the wind is low, the microseism is slowly coming down and the barrage of aleutian islands earthquakes has ended.

TITLE: 12/09 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Earthquake
INCOMING OPERATOR: Tony
SHIFT SUMMARY: Currently Observing without squeezing and have been Locked for 1 hour.
All thoughout the day we haven't been able to lock because of many earthquakes, but in the last hour I was finally able to relock the IFO without us losing lock from ground motion.
When we got to NLN we couldn't go into Observing because the squeezer PMC wouldn't lock - the message said "Cannot lock PMC. Check SQZ laser power". Last time this happened we were just able to get it to lock by re-requesting LOCKED for the PMC guardian, but this time it didn't work. I called Camilla, and we saw that although the PZT thought it was scanning to try and lock, the PZT volts were not changing and were just sitting out of range, around -12 volts, and that 10 hours ago something had happened to the PZT volts where they suddently glitched down to over -200, which is much lower than they ever go.
We weren't able to get it back working so we used Ryan Short's script to change the nominal sqz states. I accepted the sdfs, and we went to Observing at 06:31 UTC. Here is Camilla's alog with an ndscope. tagging SQZ
LOG:

22:32 Started an initial alignment
22:58 Initial alignment done, relocking
    - Lockloss from LOCKING_ARMS_GREEN
    - Lockloss from OFFLOAD_DRMI_ASC
    - Lockloss from ENGAGE_ASC_FOR_FULL_IFO
    - Lockloss from RESONANCE due to large EQ hitting
00:23 Put IFO in DOWN to wait out earthquake
- HAM2, HAM3, HAM5 ISI tripped
- IM2 tripped

02:58 Started an initial alignment
03:22 Initial alignment done, relocking
    - Lockloss from PREP_DC_READOUT_TRANSITION due to earthquake
03:43 Sitting in DOWN while earthquake passes

04:48 Trying to relock
05:46 NOMINAL_LOW_NOISE
    - Issues getting PMC to lock - see above
06:31 Observing without squeezing

It appears that the SQZ HV (PMC/SHG/OPO PZT, ZM2,4,5 PSAMS) went down at 12/08 20:33UTC. Trends attached. 

Oli found that the SQZ PMC wasn't locking. We tried enabling the PZT manually and although it thought it was scanning, the PZT Volts remained at -12, Oli found it had been like this since 12/08 20:33UTC. Then realized all PSAMS volts are off and other PZTs. Oli has taken the IFO into observing without SQZ for tonight do the CDS/EE team can look at this tomorrow.

I've written a script, /opt/rtcds/userapps/release/sqz/h1/scripts/switch_nom_sqz_states.py, which will change the nominal states of the SQZ Guardian nodes to be used when it's decided that H1 will observe without SQZ, a process adapted from the ObservationWithOrWithoutSqueezing wiki. This is intended to save operators time so the switch to a configuration where H1 can observe without SQZ is quick and brings us to observing promptly.

The script first commits any uncommitted changes in the SQZ Guardians to svn with an "as found" message before the nominal states are changed. Once changed, the nodes are then loaded and SQZ_MANAGER is requested to NO_SQUEEZING, which should make sure all nodes are in the state they should be.

This script also adds the 'syscssqz' model to the EXCLUDE_LIST in the DIAG_SDF node when going to the no-SQZ configuration. This is to facilitate any SDF diffs that may appear as a result of SQZ misbehaving and will allow H1 to go to observing while ignoring any diffs in this model. More models to exclude can be added in the script as desired.

Since the pysvn package used by the script to commit to svn is a Debian package not found in conda, all conda environments must be deactivated for this script to work. Hence, when running this script, utilize the 'noconda' bash script wrapper found in userscripts. Calling this script to change to the configuration where H1 will observe without SQZ would look as follows:

noconda python switch_nom_sqz_states.py without

This script can also be used to go back to the configuration where H1 is observing with SQZ; simply replace the 'without' argument when running it with 'with' and the nominal SQZ Guardian states will be changed back to normal and the SQZ models will be removed from DIAG_SDF's exclude list.