We had a lull in the heavy winds during which I was able to get an initial alignment in and then up to DRMI locked.
The wind is now back up to the 30s mph and the IMC glitch is happening quite frequently (both on the rocky start-up to DRMI and thereafter).
Thus, I will be staying in DOWN until conditions improve.
Meanwhile, the HAM6 SHUTTER red alert came on with guardian telling me not to power up due to SHUTTER_FAILURE. I am investigating this.
Sun Nov 17 10:10:23 2024 INFO: Fill completed in 10min 20secs
TITLE: 11/17 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Wind
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 44mph Gusts, 33mph 3min avg
Primary useism: 0.12 μm/s
Secondary useism: 0.32 μm/s
QUICK SUMMARY:
IFO is IDLE due to ENVIRONMENT.
In the last hour since TJ's OWL alog, the wind has not died down, and the microseism has only gone up.
With 50mph gusts, we are unable to lock. Will stay in idle until we can re-attempt lock.
On the bright side, the IMC is locked.
The IMC has been struggling to stay locked for the last few hours, interrupting locking and restarting the process. Ground motion has also not been kind with a 6.1M earthquake from Japan a few hours ago, and useism and wind on the rise. Since the wind is getting over 40mph gusts, I'm switching the Observatory Mode to Environment - Windy.
TITLE: 11/17 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: TJ
SHIFT SUMMARY:
3:01 UTC GRB-Short E 525935 We were not locked for the begining of this stand down event.
2024-11-17 3:27:53 NOMINAL_LOW_NOISE reached
3:30:27 OBSERVING reached
Violin ITMY Mode 5 is still rising on this lock.
I changed the Gain to 0 and waited 15 minutes to see if it will stop going up or even turn around.
I then tried about 3 minutes with the gain at -0.02 ( Nominal gain just negative) but it was obvious that the no gain is better than opposite gain.
I ended up just turnin goff the gain.
05:11 UTC Lockloss https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/index.cgi?event=1415855535
This was a very fast lockloss , it doesn't look like the same IMC /FSS locklosses we have been seening. But the PSL-FSS_FAST_MON_OUT_DQ was still the first signal to change behavior.
Relocking notes:
Relocking has been made very difficult with the IMC constantly unlocking. Most of the Locklosses are in locking Green arms, but everyfew attempts it gets a little further like DRMI, then the IMC stops working again.
I tried turning the ISS off again, but it hasn't stopped it from unlocking the IMC.
I turned back on the ISS so just in case H1 manager gets to about ISC_LOCK state 500 it will auto lock for him.
I couldn't get Past DRMI 1f, and ended up in a PRMI & Check Mich loop.
I tried to do an Intial_Alignment, But IMC would not stay locked for long enough.
I eventually waited for the IMC to lock then turned off the ISS Auto Locker for the duration of the Initial alignment.
Initial alignment went well with the ISS turnt off.
Trying to relock again.
TITLE: 11/17 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 159Mpc
OUTGOING OPERATOR: Ibrahim
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 15mph Gusts, 11mph 3min avg
Primary useism: 0.02 μm/s
Secondary useism: 0.24 μm/s
QUICK SUMMARY:
H1 Has been locked for 1 Hour and 45 minutes.
DCPD's are diverging, Which I usually attribute to the Violins growing.
Turns out it's one of the usual suspects..... ITMY Mode 5 increasing.
I have turned the Gain off of ITMY m5 to see if it stays at that level or decreases.
The IFO stays locked until 1:25 UTC then I can do a Calibration sweep are the calibration time early today was missed by a lockloss.
Lockloss https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/index.cgi?event=1415840300
This looks like there is some IMC issue that cause this Lockloss.
The Lockloss plots show that at -906ms there was blip on the PSL-FSS_FAST_MON _OUT_DQ channel.
Relocking the IMC was difficult as the IMC kept going into fault. I took ISC_LOCK to idel and took the IMC to idle for 10 minutes.
Relocking started fine, except I had to manual green arms.
TITLE: 11/16 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 157Mpc
INCOMING OPERATOR: Tony
SHIFT SUMMARY:
IFO is in NLN and OBSERVING as of 22:35 UTC
There was one IMC-caused Lockloss which ended our 13hr and 36 min lock acquisition. (alog 81310)
Stayed locked for the majority of this shift with one somewhat troublesome lock acquisiton for a number of reasons:
I was not able to take a calibration measurement due to the Lockloss, and we're not thermalized yet. I've informed Tony, the EVE operator, and he will take a calibration sweep once thermalized (01:24 UTC).
LOG:
None
Lockloss seemingly caused due to the IMC with MC2 losing lock within 1ms of ASC (attached).
Sat Nov 16 10:10:51 2024 INFO: Fill completed in 10min 47secs
TITLE: 11/16 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 164Mpc
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 12mph Gusts, 9mph 3min avg
Primary useism: 0.02 μm/s
Secondary useism: 0.20 μm/s
QUICK SUMMARY:
IFO is in NLN and OBSERVING as of 05:34 UTC (10 hr lock!)
TITLE: 11/16 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Aligning
INCOMING OPERATOR: TJ
SHIFT SUMMARY:
Most of the meat of the shift has already been posted earlier.
Had a return of the FSS/IMC glitches this afternoon/evening. Have made it back to Observing at the end of the shift though.
For the night we are set for locking for the OWL. But if locking becomes an issue that lasts over 2hrs, plan is to take ISC LOCK to IDLE for the night.
LOG:
Had about 20hrs with no IMC/FSS glitches from 9pm Thurs night until about 340pmPT Fri afternoon. But have now had several for the last 5hrs.
Also for one ~60min stretch, H1 almost made it to NLN, but had rung up violins which kept us at OMC Whitening for 20min until the next IMC glitch.
Here are some running notes for the rough 5hrs filled with glitches (currently have H1 in IDLE).
Notes During Return of IMC/FSS Glitch Locklosses
2340utc (340pmPT) LOCKLOSS with IMC tag was the beginning of IMC/FSS glitch locklosses (where we went almost 20hrs since the last glitch around 5utc (9pmPT).
0111-0208: OBSERVING
0208utc (608pmPT): Attached screenshot shows a coincident glitch lockloss seen on FSS & IMC. Here is the LOCKLOSS page also tagged with IMC glitch.
Starting an FRS Ticket to mark the DOWNTIME (FRS-32644) for 3.5hrs of DOWNTIME so far. Where there was notable part of downtime due to IMC not locking easily.
VIOLINS Rung Up A Bit: (haven't had this since my shift on Tues night!) So were stuck at OMC WHITENING for over 20min before H1 had....
So for the last 3hrs:
Not sure what is more useful. Keep having ISC LOCK continue to locking, or to leave it in IDLE for IMC/PMC/FSS monitoring....
For FAMIS #26340: All looks well for the last week for all site HVAC fans (see attached trends).
TITLE: 11/16 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Aligning
INCOMING OPERATOR: Corey
SHIFT SUMMARY:
IFO is LOCKING in ENGAGE_ASC_FOR_FULL_IFO
Just had a lockloss and trying to get back to observing. IFO has been quite well behaved today. Both lock acquisitions were fully automatic with H1 needing no intervention.
That being said, we did have two locklosses.
Lockloss 1: ETM glitch caused.
Lockloss 2: IMC caused.
LOG:
None
TITLE: 11/16 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Aligning
OUTGOING OPERATOR: Ibrahim
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 14mph Gusts, 11mph 3min avg
Primary useism: 0.03 μm/s
Secondary useism: 0.28 μm/s
QUICK SUMMARY:
Walked in to a locking h1, which then has moved on to INITIAL ALIGNMENT. Ibrahim shared how H1 has had a better time--so am cautiously optimistic for the night---with the goal of OBSERVING!
Environmentally we are becoming quieter with µseism dropping below the 95th percentile and winds not too breezy.
Ibrahim, Ryan S
Seems like an IMC lockloss. Interestingly, there was no obviously glitchy behavior (as previously observed) before or after the lockloss. Attached screenshot of ASC and IMC losing lock.
Adding more channels for the sake of investigation. As Ibrahim mentions, no glitchy behavior was seen leading up to this IMC lockloss, so this was either a large fast glitch out of nowhere or something else giving out. Seemingly all at once (or at least very close to each other), the FSS_FASTMON, RefCav trans, NPRO power, PMC high voltage, PMC mixer, and all IMC signals quickly change, so it's hard to tell which is truly happening first as there is lots of feedback between these signals.
Sheila, Vicky.
We changed the low pass filtering on the ADF SQZ angle servo to be lower, going from 1 Hz low pass to 0.1 Hz low pass on the ADF I/Q demod filter banks. I think this reduced the sqz angle readback noise a bit (trends before/after). It didn't make a huge impact on the control signal that goes to the CLF_RF6 demod phase (bottom purple trace), and it might be worth considering low pass filtering that control signal too. I accepted the SDFs for this filter bank LPF change.
Fast shutter worked and is working. (I was worried that the mirror was broken off but that's not the case, the shutter still opens and closes without an issue.)
From the guardian log (attached), it failed because AS_B_DC_NSUM was 20.3 when the shutter was closed and the threshold is 15. AS_B was 0.28. Fundamentally, this is because I reduced the gain of this DC path by 20dB and digitally compensated in search for an existing sensor that won't rail when lock is lost, and the same fast shutter test failure was observed (see alog 79104). Back then, I wrote "I adjusted the dark offset while IMC was unlocked but we can probably increase the threshold to 30 or so if it continues to bother us."
The dark offset has drifted enough to bother us at times (i.e. it is about 12 counts or so on average now), so I told Ibrahim to increase the thresholds only for AS_B, i.e.
PdBDarkLo=-40 # dark level low
PdBDarkHi= 40 # dark level high
in FAST_SHUTTER.py.
Since we now have a dedicated diode to look at lock loss power spikes, we can turn down the analog gain of AS_B_DC by 20dB and turn off +20dB filter in FM4 of H1:ASC-AS_B_DC_SEG[1234] filter, and adjust the dark offset again on Tuesday.
The mirror is still attached to the fast shutter and is moving. It actually closed using HV (see GS13 signal) when the test failed (2nd attachment), and later when I opened/closed it with low voltage drive it responded correctly (3rd attachment).