Mon Jan 20 10:08:53 2025 INFO: Fill completed in 8min 50secs
TCmins [-75C, -73C] OAT (-1C, 30F). deltaTemp trip time 10:08:55.
TITLE: 01/20 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: Ryan C
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 5mph Gusts, 3mph 3min avg
Primary useism: 0.03 μm/s
Secondary useism: 0.26 μm/s
QUICK SUMMARY:
H1 has been locked for about 2hrs on this chilly (~22degF) morning with low winds and microseism between the 50th & 95th percentile.
12:19 UTC lockloss, IFO_NOTIFY says its was in NLN with SDF diffs, but we lost lock moments before I logged in.
13:58 UTC Observing, I adjusted the OPO temperature before I went into observing.
Something weird happened with the alert system last night - there were multiple times where IFO_NOTIFY went into ALERTS_ACTIVE, but it didn't call Ryan every time that happened.
The squeezer was having trouble locking/staying locked in FDS, so once we were out of Observing for 8 minutes, IFO_NOTIFY went into ALERT_ACTIVE, during which it should have called Ryan, but maybe it didn't get to because only 20 seconds later, it got back to FDS.
However, it dropped back out a few seconds later, and once those 8 minutes had gone by, IFO NOTIFY went into ALERT_ACTIVE again, and it did call Ryan at 07:21 UTC.
When the SQZer got back to FDS for a few seconds two minutes later, it again took us back out of ALERT_ACTIVE, and then waited the 8 minutes again before going into ALERT_ACTIVE again and sitting in there for 23 minutes while the SQZer tried to relock. During these 23 minutes, Ryan did not get called.
Then the same thing happened again where the SQZer got to FDS for a few seconds, taking us out of ALERT_ACTIVE and resetting that 8 minute timer. This time it stayed in ALERT_ACTIVE for over 4 hours and did not call Ryan until more than 4 hours later.
I've attached an ndscope and the IFO_NOTIFY log from last night, complete with my commentary.
It seems like besides the times where the call did not go out like it was supposed to, there might also need to be repeat calls made during times where we sit out of Observing for longer periods of time, maybe once per hour or per 30 minutes?
Tagging SQZ. 2025/01/20 07:02:12 UTC for 5 hours we had the SQZ FC struggling to lock. OPO, CLF and PMC stayed locked during this time, plot.
Initially unlocked with message "IR unlocked?" and then repeatedly locked back to IR_FOUND where it lost lock with message "GR lost lock??", checker is return ezca['SQZ-FC_TRANS_C_LF_OUTPUT'] > sqzparams.fcgs_trans_lock_threshold (60) which makes sense, we are at 100 when locked so not near this threshold. Unsure on the cause of FC green unlocking, maybe something in the FC servo dragged the lock away unless we are just seeing the unlock itself, plot and zoom.
TITLE: 01/20 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Ryan C
SHIFT SUMMARY: Relocking from the lockloss and currently at PREP_DC_READOUT_TRANSITION. I didn't change the damping for ETMY mode1 officially, but it seems like it will need to be made permanent. Besides the lockloss everything was quiet, I ran an initial alignment and helped ALSY a bit and we have had no issues relocking.
LOG:
21:20 Changed damping for ETMY mode1 to be only FM1 and FM10 (nominal is FM1 FM8 FM10), and gain to -0.2 (same as TJ did yesterday 82340)
04:37 Lockloss
04:40 Initial alignment
Lockloss @ 01/20 04:37 UTC after over 13 hours locked
Observing and have been Locked for 10 hours now. Nothing to report.
TITLE: 01/19 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 157Mpc
INCOMING OPERATOR: Oli
SHIFT SUMMARY: Quiet day with H1 observing for the duration and one candidate superevent. H1 has now been locked for almost 6 hours.
TITLE: 01/19 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 159Mpc
OUTGOING OPERATOR: Ryan S
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 10mph Gusts, 7mph 3min avg
Primary useism: 0.04 μm/s
Secondary useism: 0.24 μm/s
QUICK SUMMARY:
Observing and have been Locked for almost 6 hours.
Sun Jan 19 10:04:19 2025 INFO: Fill completed in 4min 16secs
TCmins [-65C, -64C] OAT (-1C, 30F). I'm testing delta-temperature based trip code in parallel, it tripped at 10:04:21.
At 02:31 PST this morning we had a slight vacuum glitch at EX originating at X2-8 (beam tube 220m from EX). Gas levels were low, well below VACSTAT trip levels. Pump down time was about 6 minutes. Attached plot show a coincident drop in H1 range at this time.
Second plot shows a similar correlation for an EX glitch on Sat 11 Jan 05:45 PST.
TITLE: 01/19 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 146Mpc
OUTGOING OPERATOR: Ryan C
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 5mph Gusts, 2mph 3min avg
Primary useism: 0.07 μm/s
Secondary useism: 0.34 μm/s
QUICK SUMMARY: H1 has been locked and observing for 20 minutes. Range is lower than I'd expect, but perhaps it will come up as things thermalize. Looks like H1 lost lock at 13:41 this morning from an unknown cause (no sign of ETM glitch that I can see) and relocked itself after running an initial alignment.
TITLE: 01/19 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 157Mpc
INCOMING OPERATOR: Ryan C
SHIFT SUMMARY: Currently Observing at 157 Mpc and have been Locked for almost 40 hours. Quiet night where the only thing that happened is that I got my third superevent candidate of the weekend!!
LOG:
02:51 Superevent candidate S250119ag
Observing at 155Mpc and have now been Locked for 35.5 hours. Secondary microseism is on the rise but not too bad yet. Nothing to report.
TITLE: 01/18 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 161Mpc
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 3mph Gusts, 1mph 3min avg
Primary useism: 0.06 μm/s
Secondary useism: 0.25 μm/s
QUICK SUMMARY:
Observing at 161 Mpc and have been Locked for almost 31 hours. I'll keep an eye on ETMY mode1, but it's looking good right now.
TITLE: 01/18 Day Shift: 1530-2100 UTC (0730-1300 PST), all times posted in UTC
STATE of H1: Observing at 160Mpc
INCOMING OPERATOR: Oli
SHIFT SUMMARY: Locked for 30.5 hours. Violin mode ETMY mode 1 was rung up this morning, but changing the damping to no phase and -0.2 gain has damped the mode down nicely. Coordinated calibration measurements were run and other than that we have been observing the whole time.
LOG:
Ran the usual coordinated calibration sweep following the wiki. The IFO has been locked for 29 hours.
Simulines start:
PST: 2025-01-18 11:39:53.237752 PST
UTC: 2025-01-18 19:39:53.237752 UTC
GPS: 1421264411.237752
Simulines end:
PST: 2025-01-18 12:05:08.721297 PST
UTC: 2025-01-18 20:05:08.721297 UTC
GPS: 1421265926.721297
Files:
2025-01-18 20:05:08,547 | INFO | File written out to: /ligo/groups/cal/H1/measurements/DARMOLG_SS/DARMOLG_SS_20250118T193956Z.hdf5
2025-01-18 20:05:08,565 | INFO | File written out to: /ligo/groups/cal/H1/measurements/PCALY2DARM_SS/PCALY2DARM_SS_20250118T193956Z.hdf5
2025-01-18 20:05:08,576 | INFO | File written out to: /ligo/groups/cal/H1/measurements/SUSETMX_L1_SS/SUSETMX_L1_SS_20250118T193956Z.hdf5
2025-01-18 20:05:08,587 | INFO | File written out to: /ligo/groups/cal/H1/measurements/SUSETMX_L2_SS/SUSETMX_L2_SS_20250118T193956Z.hdf5
2025-01-18 20:05:08,598 | INFO | File written out to: /ligo/groups/cal/H1/measurements/SUSETMX_L3_SS/SUSETMX_L3_SS_20250118T193956Z.hdf5
Sat Jan 18 10:04:21 2025 INFO: Fill completed in 4min 18secs
TCmins [-69C, -67C] OAT (0C,32F)
TITLE: 01/18 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 160Mpc
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 10mph Gusts, 7mph 3min avg
Primary useism: 0.03 μm/s
Secondary useism: 0.23 μm/s
QUICK SUMMARY:
Observing and have been Locked for over 10 hours. Range dipped a bit earlier due to unknown reasons, but looks to be slowly coming back up.
I had forgotten to post my summary for Friday DAY due to some parts-searching, so here is what I had:
TITLE: 01/17 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 158Mpc
INCOMING OPERATOR: Oli
SHIFT SUMMARY:
H1 locked entire shift! There was a short 1-min break for a calibration change.
LOG:
D. Davis, E. Capote, O. Patane
There was a discussion recently in the Detchar tools channel about how to interpret the cumulative range plots generated on the summary pages, such as today's cumulative range plot. Specifically, it seems incorrect that we could accumulate 30% of our range below 30 Hz.
Derek has pointed out that this is so misleading because the calculation of cumulative range in this manner is actually performed somewhat incorrectly. In short, range can be thought as analogous to SNR, which is a quantity that must be added in quadrature. Therefore, the order matters when calculating a cumulative range, i.e. the range acquired from 10-20 Hz, then 10-30 Hz, 10-40 Hz, etc. Therefore, the total cumulative range number, as in the one we think about all the time (160 Mpc, for example) is correct, but determining the range over a subset of the band (such as 10-30 Hz) needs to be done more carefully so it is not misleading.
Once we started discussing this, I pointed out that this means that the way we compare ranges is also misleading, as in when we run our DARM integral comparison scripts, we are subtracting the cumulative range of two different DARM PSDs, but we subtract it in amplitude (Mpc) and not in quadrature (Mpc^2).
Derek has created an improved way to calculate cumulative range, which they have coined to be the "cumulative normalized range". To get right to the point: it is better to normalize the cumulative range squared by the total range. This is an example plot showing how these two differ. This plot shows that for a given DARM PSD, the cumulative normalized range better estimates the sensitivity gained over a particular range of frequency. The low frequency portion is still very important (this results from the f^(-7/3) dependence in the range calculation), but indeed we gain very little sensitivity between 10-20 Hz, for example. You can also see that, when using the normalized method, the curve where you integrate up in frequency and the curve where you integrate down in frequency intersect at about 50% of the range, which is what you would expect.
In equation form, this image attachment defines the total cumulative range, and this image attachment shows our defintion of the normalized cumulative range.
In order to more sensibly compare two sensitivities by frequency, we have also derived a way to calculate the cumulative normalized range difference. The derivation is slightly more complicated, but the result is that you subtract the two cumulative normalized quantities, and then normalize by the sum of the two ranges.
This image attachment shows the equation form of this.
To make sense of why this method is better than the method we use now, you can imagine that we have two PSDs, one with 100 Mpc of range, and one that is exactly the same, except that between 10-20 Hz there is an additional gain of 20 Mpc, such that the total range is now 120 Mpc. If you compare these two bizarre PSDs, you would expect that the cumulative range difference between the two from 10-20 Hz is 20 Mpc, and then zero thereafter. This is an example plot showing how the cumulative range difference would appear, using the method where you subtract the two cumulative ranges, and then the method where you apply this normalized range method. The normalized range calculation behaves as expected, while the method that straightforwardly subtracts the two cumulative ranges overshoots the range gain from 10-20 Hz, and then misleadingly indicates the range is decreasing above 20 Hz to make up for it.
There is a lot of information to grasp here, and Derek and I will be posting a document to the DCC soon with a fuller explanation and full derivations. Oli has taken the time to implement these new methods in our DARM comparison scripts, and they will follow up here with more information about that.
As a start, I've only corrected these things in the range_compare script that I previously made based off of the Hanford Noise Budget darm_integral_compare script (81015). This script that I made is a simplified version of the script used for creating NoiseBudget plots so I thought it would be a good start to making these changes. There are also plans to correct the the calculations in other places (summary pages and the official NoiseBudget scripts for example).
All changes have been committed to git and are up to date in gitcommon/ops_tools/rangeComparison/. In addition to the changes necessary to correct the cumulative range plots, I also swapped out the way we were grabbing data so it now uses GWPy, and I added in an additional plot that shows the cumulative sum of the range over frequency. Here's an comparison of the old vs new cumulative range
Derek and I have just updated a document to the DCC with a full workup of this change and some fun examples, see P2500021.
