Mon CP1 Fill

Mon Dec 09 10:21:24 2024 INFO: Fill completed in 21min 20secs

Gerardo confirmed a good fill curbside. Long fill with temps starting positive, hence the strange looking trend.

DQ Shift Report, LHO: 25 November 2024 00:00 UTC - 01 December 2024 23:59 UTC

Summary of Week:

• Range was consistently between 150 and 165 Mpc when in observing mode
• Low frequency noise found in PEM LVEA Floor accelerometers appears to be coupling to the chambers. (See Fig 1)
• Reoccurring low frequency noise in strain around 30-40 Hz may be ringing up DARM. (See Fig 2)

More detailed information can be found on the DQ Shift Report page.

SAMS servos railed after HV trip. Needed to clear histories.

After the SQZ HV tripped off and was brought back 81689, the PSAMS servos (80685) railed. Attached plot shows the output of SAMS_SERVO at the limit and Strain gauge not at PSAMS target.

To fix this, on the  H1:AWC-ZM{2,4,5}_M2_SAMS_SERVO (small box to the left of REQUESTED PZT(V)),  I increased ramp time form 2 seconds to 5 seconds, ramped gain to 0, cleared history, ramped gain to 1 and then put ramp time back.

After these were back to their nominal, the alignments of the ZMs went back close to nominal and we were able to lock the FC.

Y2-8 Pressure Gauge is Not Reporting

It appears that we have lost the gauge for Y2-8, nothing to do for now since this is a monitoring tool.  We will assess tomorrow, but if the system needs to be charged, then maybe there will be enough sunlight today.

Lost the gauge early Saturday morning.

Lockloss 16:39 UTC

Earthquake caused Lockloss.

OPS Day Shift Start

TITLE: 12/09 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 97 Mpc
OUTGOING OPERATOR: Tony
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 4mph Gusts, 2mph 3min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.36 μm/s
QUICK SUMMARY:

IFO is in NLN and OBSERVING without squeezing.

There were issues with the squeeze high voltage, so observing without squeezing until this is investigated today. (per Camilla's alog 81689)

We also have planned comissioning time today from 8AM-11AM (30 mins earlier than usual due to an all-hands meeting).

Other than that, it seems the wind is low, the microseism is slowly coming down and the barrage of aleutian islands earthquakes has ended.

Ops Eve Shift End

TITLE: 12/09 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Earthquake
INCOMING OPERATOR: Tony
SHIFT SUMMARY: Currently Observing without squeezing and have been Locked for 1 hour.
All thoughout the day we haven't been able to lock because of many earthquakes, but in the last hour I was finally able to relock the IFO without us losing lock from ground motion.
When we got to NLN we couldn't go into Observing because the squeezer PMC wouldn't lock - the message said "Cannot lock PMC. Check SQZ laser power". Last time this happened we were just able to get it to lock by re-requesting LOCKED for the PMC guardian, but this time it didn't work. I called Camilla, and we saw that although the PZT thought it was scanning to try and lock, the PZT volts were not changing and were just sitting out of range, around -12 volts, and that 10 hours ago something had happened to the PZT volts where they suddently glitched down to over -200, which is much lower than they ever go.
We weren't able to get it back working so we used Ryan Short's script to change the nominal sqz states. I accepted the sdfs, and we went to Observing at 06:31 UTC. Here is Camilla's alog with an ndscope. tagging SQZ
LOG:

22:32 Started an initial alignment
22:58 Initial alignment done, relocking
    - Lockloss from LOCKING_ARMS_GREEN
    - Lockloss from OFFLOAD_DRMI_ASC
    - Lockloss from ENGAGE_ASC_FOR_FULL_IFO
    - Lockloss from RESONANCE due to large EQ hitting
00:23 Put IFO in DOWN to wait out earthquake
- HAM2, HAM3, HAM5 ISI tripped
- IM2 tripped

02:58 Started an initial alignment
03:22 Initial alignment done, relocking
    - Lockloss from PREP_DC_READOUT_TRANSITION due to earthquake
03:43 Sitting in DOWN while earthquake passes

04:48 Trying to relock
05:46 NOMINAL_LOW_NOISE
    - Issues getting PMC to lock - see above
06:31 Observing without squeezing

SQZ High Voltage seems to be off. Oli taken IFO to observation without SQZ.

It appears that the SQZ HV (PMC/SHG/OPO PZT, ZM2,4,5 PSAMS) went down at 12/08 20:33UTC. Trends attached. 

Oli found that the SQZ PMC wasn't locking. We tried enabling the PZT manually and although it thought it was scanning, the PZT Volts remained at -12, Oli found it had been like this since 12/08 20:33UTC. Then realized all PSAMS volts are off and other PZTs. Oli has taken the IFO into observing without SQZ for tonight do the CDS/EE team can look at this tomorrow.

I logged into h0vacly and looked up the status of PT153. See "Latched Device Error Details" in the attached screenshot. According to the manual, this appears to translate to "Electronics failure, Non-volatile memory failure, invalid internal communication".

In-Lock SUS Charge Measurements for December 3rd FAMIS + script issue fixed

Closes FAMIS#28382, last checked 80571

Here are December 3rd's plots for ITMX, ITMY, ETMX, and ETMY. There are no new points for ITMX since the coherence for the bias drive bias off measurement was 0.07, which is below the threshold of 0.1 that we have set. We had previously concluded (79597) that this may be because of no charge build-up on the test mass, assuming that we are calculating our coherences correctly.

Previous issue with measurement processing script:

There had been some issues with processing the In-Lock SUS Charge Measurements for this past week, so I looked at the script and was able to fix the issue.
Originally, the way that the script would figure out whether it needs to run a measurement or whether it had already been processed is that it would make a list of all the measurements that occurred during the entered time period (default is the past month) and a list of all processed measurements that occurred during the entered time period, and then uses nested for loops to check whether a processed file existed for each data file. That normally works fine, but in this case, the last time we had gotten and processed measurements was over one month ago, so when the script tried making a list of the processed files from within the last month, there weren't any and the list was blank. Then when it tried to iterate over each data file and then over each processed file, it would skip the entire processed file iteration because the list was empty.

The range over which it looks for unprocessed files can be changed or set to look at all files in each directory, but that shouldn't be required in this/most cases, so I edited the script. I added in a catch for when the coefficient file list is empty, for it to run all measurements in the data directory list. This update works and has been committed to svn.

Ops EVE Midshift Status

Currently in DOWN due to more earthquakes(nuc5). I was able to run an initial alignment and start relocking, but we had more earthquakes come in and we lost lock at PREP_DC_READOUT_TRANSITION a few minutes ago because of the ground motion. Since we just had a couple more earthquakes announced and just entered earthquake mode, I'll have us sit in DOWN for a bit while the earthquakes roll though and hopefully we can get back to NLN soon. Thankfully it does look like secondary microseism is slowly starting to go down.

OPS Day Shift Summary

TITLE: 12/09 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Earthquake
INCOMING OPERATOR: Oli
SHIFT SUMMARY:

IFO is DOWN due to Earthquake

H1 was only able to lock for 36 minutes today (lockloss alog) in between earthquakes. It's been a very quakey day with:

• 3 6+ mag EQs (all 3 in the aleutian islands in Alaska.
• 8 5+ mag EQs
• 6 4+ mag EQs

We tired locking after the first 6.3 magnitude EQ died down only to have another 6.3 magnitude EQ 3 hrs later (and then another 6.3 10 mins after that). I've attached a screenshot of our microseism. The secondary microseism is also over the 90% threshold, which makes us more succeptible to lower mag EQs. We've been in useism and earthquake mode all day essentially.
 

LOG:

None

Ops Eve Shift Start

TITLE: 12/09 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Earthquake
OUTGOING OPERATOR: Ibrahim
CURRENT ENVIRONMENT:
    SEI_ENV state: USEISM_EARTHQUAKE
    Wind: 6mph Gusts, 3mph 3min avg
    Primary useism: 1.11 μm/s
    Secondary useism: 0.86 μm/s
QUICK SUMMARY:

Currently in DOWN due to an earthquake in Japan. Microsseism is still pretty high so we were having a hard time getting back up anyway.

Suspension watchdog trips during the large earthquake

Back in March (76269) Jeff and I had updated all the suspension watchdogs (besides OFIS, OPOS, and HXDS since those were already up to date) to use better blrms filtering and to be output into um. We set the suspension watchdog thresholds to values between 100 and 300 µm, but these values were set arbitrarily since there was no way to previously see how far the stages move during different scenarios. We had upped a few of the thresholds after having some suspensions trip when they probably shouldn't have, and this is a continuation of that.
During the large earthquake that hit us on December 5th, 2024 18:46 UTC, all ISI watchdogs tripped as well as some of the stages on several suspensions. After a cursory look, all suspensions that tripped only had either the bottom or bottom+penultimate stage trip, meaning that with the exception of the single suspensions, the others' M1 stage damping should have stayed on.

We wanted to go through and check whether the trips may have just been because of the movement from the ISIs tripping. If that is the case, we want to raise the suspension watchdog thresholds for those stages so that these suspensions don't trip every single time their ISI trips, especially if the amount that they are moving is still not very large.

Suspension stages that tripped:

Triples:
- MC3 M3
- PR3 M2, M3
- SRM M2, M3
- SR2 M2, M3

Singles:
- IM1 M1
- OFI M1
- TMSX M1

MC3 (M3) (ndscope1)

When the earthquake hit and we lost lock, all stages were moving due to the earthquake, but once HAM2 ISI tripped 5 seconds after the lockloss, the rate at which the OSEMs were moving quickly accelerated, so the excess motion looks to mainly be due to the ISI trip (ndscope2).

Stage	Original WD threshold	Max BLRMS reached after lockloss	New WD threshold
M1	150	86	150 (unchanged)
M2	150	136	175
M3	150	159	200

 

PR3 (M2, M3) (ndscope3)

Looks to be the same issue as with MC3.

Stage	Original WD threshold	Max BLRMS reached after lockloss	New WD threshold
M1	150	72	150 (unchanged)
M2	150	162	200
M3	150	151	200

 

SRM (M2, M3) (ndscope4)

Once again, the OSEMs are moving after the lockloss due to the earthquake, but the rate they were moving at greatly increased once HAM5 saturated and the ISI watchdogs tripped.

Stage	Original WD threshold	Max BLRMS reached after lockloss	New WD threshold
M1	150	84	150 (unchanged)
M2	150	165	200
M3	150	174	225

 

 SR2 (M2, M3) (ndscope5)

Again, the OSEMs are moving after the lockloss due to the earthquake, but the rate they were moving at greatly increased once HAM4 saturated and the ISI watchdogs tripped.

Stage	Original WD threshold	Max BLRMS reached after lockloss	New WD threshold
M1	150	102	150 (unchanged)
M2	150	182	225
M3	150	171	225

 

IM1 (M1) (ndscope6)

Again, the OSEMs are moving after the lockloss due to the earthquake, but the rate they were moving at greatly increased once HAM2 saturated and the ISI watchdogs tripped.

Stage	Original WD threshold	Max BLRMS reached after lockloss	New WD threshold
M1	150	175	225

 

OFI (M1) (ndscope7)

Again, the OSEMs are moving after the lockloss due to the earthquake, but the rate they were moving at greatly increased once HAM5 saturated and the ISI watchdogs tripped.

Stage	Original WD threshold	Max BLRMS reached after lockloss	New WD threshold
M1	150	209	250

 

TMSX (M1) (ndscope8)

This one seems a bit questionable - it looks like some of the OSEMs were already moving quite a bit before the ISI tripped, and there isn't as much of a clear place where they started moving more once the ISI had tripped(ndscope9). I will still be raising the suspension trip threshold for this one just because it doesn't need to be raised very much and is within a reasonable range.

Stage	Original WD threshold	Max BLRMS reached after lockloss	New WD threshold
M1	100	185	225

OPS Day Midshift Update

While the microseism has gone down a small bit, locking has still been difficult due to EQs.

First, there were two back to back EQs (mag 4s and 5s) that killed a potential lock at near NLN (LASER_NOISE_SUPPRESSION)

Then, we got to NLN and were OBSERVING for 36 mins but another two back to back (mag 4s and 5s) caused a lockloss.

Finally, just now while relocking, another two higher 6.5 mag and 6.3 mag EQs from Japan are coming through.

All of this is made much worse due to the already over 1 um/s secondary microseism.

As such, I will bring H1 to DOWN and ENVIRONMENT and wait until the Earth rings down a bit.

alarms issue 01:00 Sunday 08dec2024

Jonathan, Dave:

The alarms service on cdslogin stopped reporting around 1am this morning. Symptoms are status file was not being updated (caused alarm block on CDS Overview MEDM to turn PURPLE) and the report file was not being updated. Presumably no alarms would have been sent from this time onwards.

At 08:10 I restarted the alarms.service on cdslogin. A new report file was created but not written to, the /tmp/alarm_status.txt file was not changed (still frozen at 01:00) but I did get a startup text. Then 14 minutes later the files started being written. I raised a test alarm and got a text, but no email.

At 09:38 after not getting a keepalive email at 09:00 or any SSH login emails I rebooted cdslogin. Same behavior as 08:10; report file created not written, tmp file not created, startup text sent successfully. After 14 minutes alarms starts running, writes to file system, test alarms are texted but no emails at all.

Jonathan is going to check on bepex.