at 00:29 Fri 30 Aug 2024 PDT there was a short vacuum glitch in HAM6 detected by PT110_MOD1.
The pressure increased from 1.90e-07 to 2.06e-07 Torr.
Gitch was detected by VACSTAT.
VACSTAT is in testing mode, and the MEDMs still needs some polishing. The Overview and PT110 MEDM are attached.
Gitch time doesn't appear to be related to H1 locking or unlocking. 24 hour PT110 and H1 range trend attached.
The smaller, wider glitch on left was at 16:51 Thu and is known about (pump operations as part of noise hunting). The 00:29 glitch is the larger, sharp one to the right.
I've promoted VACSTAT from a H3 test system to a H1 pre-production system. This allows me to add the channel H1:CDS-VAC_STAT_GLITCH_DETECTED to the alarms system (alarms was restarted 10:15). For testing it will send alarms to my cellphone.
Yesterday in the instrument air trend there was a brief period of pressure loss which was caused by the air dryer being turned off for several minutes while a line voltage switch was added to the unit. The bypass on the dryer was not opened, so additional air was not fed into the system. Once power was restored the dryer functioned correctly and allowed air back in.
Fri Aug 30 08:11:24 2024 INFO: Fill completed in 11min 20secs
TITLE: 08/30 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 146Mpc
OUTGOING OPERATOR: Oli
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 9mph Gusts, 7mph 5min avg
Primary useism: 0.01 μm/s
Secondary useism: 0.08 μm/s
QUICK SUMMARY:
IFO was locked and Observing for 8 Hours before I walked in to the Control room.
Looks like the IFO locked it's self last night with out calling Oli after losing lock at LASER_NOISE_Suppression.
H1's Current Status is LOCK at NOMINAL_LOW_NOISE and OBSERVING for 8 Hours and 27 Min.
H1EDC is reporting a NUC33 error :
Total Channels : 56980
Connected Channels : 56968
Disconnected Channels : 12
H1:CDS-MONITOR_NUC33_CPU_LOAD_PERCENT
H1:CDS-MONITOR_NUC33_CPU_COUNT
H1:CDS-MONITOR_NUC33_MEMORY_AVAIL_PERCENT
H1:CDS-MONITOR_NUC33_MEMORY_AVAIL_MB
H1:CDS-MONITOR_NUC33_PROCESSES
H1:CDS-MONITOR_NUC33_INET_CONNECTIONS
H1:CDS-MONITOR_NUC33_NET_TX_TOTAL_MBIT
H1:CDS-MONITOR_NUC33_NET_RX_TOTAL_MBIT
H1:CDS-MONITOR_NUC33_NET_TX_LO_MBIT
H1:CDS-MONITOR_NUC33_NET_RX_LO_MBIT
H1:CDS-MONITOR_NUC33_NET_RX_ENO1_MBIT
H1:CDS-MONITOR_NUC33_NET_TX_ENO1_MBIT
Did some troubleshooting on NUC33.
couldn't VNC or SSH into it.
Got a Keyboard and Mouse and couldn't get NUC33 to do anything, completely frozen.
I tried the REISUB linux rescue and no response. So I gave it a hard shutdown via the power button.
Once it powered back on everything came back up as normal. H1EDC went back to functioning as normal.
TITLE: 08/29 Eve Shift: 2300-0800 UTC (1600-0100 PST), all times posted in UTC
STATE of H1: Observing at 141Mpc
INCOMING OPERATOR: Oli
SHIFT SUMMARY:
H1 had a late-shift lockloss, and was pretty much automated (need INCREASE FLASHES & PRMI), but near the end there was a lockloss at LASER NOISE SUPPRESSION...most likely due to an EQ from Russia. H1's already giving it a go.
LOG:
Here is an investigation that might give us insight into the PRCL/CHARD/DARM coherences.
Today, we ran a PRCL injection for the feedforward where we injected directly into the PRCL loop. I took this injection time and looked at how the CHARD P and Y error signals changed, as well as their respective coherences to PRCL and DARM (figure). When injecting about 15x above ambient in PRCL from 10-100 Hz, there is a 2x increase in the CHARD P error signal and 4x increase in the CHARD Y error signal. The coherences of CHARD P and Y to PRCL increase as well, and the coherences of CHARD P and Y to DARM also increase. This injection allows us to measure a well-defined CHARD/PRCL transfer function. In the attached screenshot of this measurement, all the reference traces are from the injection time, and all the live traces are during a quiet time.
Meanwhile, I looked back at the last time we injected into CHARD P and Y for the noise budget, on June 20. In both cases, injecting close to 100x above ambient in CHARD P and Y did not change either the CHARD/PRCL coherence or PRCL/DARM coherence. There is some change in the PRCL error signal, but it is small. Again, in the attachments, reference traces are injection time and live traces are quiet time. CHARD P figure and CHARD Y figure.
I think that this is enough to say that the PRCL/DARM coupling is likely mostly through CHARD P and Y. This would also make sense with the failure of the PRCL feedforward today (79806). However, we may want to repeat the CHARD injections since there have been many IFO changes since June 20.
As a reminder, the previous work we did adding a PRCL offset did reduce the PRCL/CHARD coupling: read 76814 and the comments. We are currently not running with any PRCL offset.
I decided to compare the PRCL injection times back in March when I set the PRCL offset to reduce the coherence of DARM with LSC REFL RIN (76814, 76805). One conclusion of these tests was that a PRCL offset can reduce the REFL RIN/DARM coherence, but not necessarily improve the sensitivity. Also, the offset reduced the PRCL/CHARD Y coupling and increased the PRCL/CHARD P coupling.
A bruco shows that there is once again significant coherence with REFL RIN and DARM. I compared the PRCL injection time from yesterday with the PRCL injections with different offsets. The PRCL/ CHARD couplings have increased for both pitch and yaw: plot. I also included the coherences of CHARD to DARM for these times but then realized that data might actually be confusing to compare. However, the PRCL offset has an effect on the PRCL/CHARD coupling, so it could be one strategy to combat this coupling. Unfortunately, it has opposite effects for pitch and yaw.
There was a lot of work done to move the alignment of the PRC around in May; here are some alogs I found to remind myself of what happened: 77736, 77855, 77988. Seems like the goal was to reduce clipping/center on PR2. I wonder if this alignment shift caused the increase in the PRCL/CHARD coupling from March to now.
We should consider checking the PRCL and CHARD coupling while adjusting the PRC alignment. The yaw coupling is stronger, maybe because the beam is much further offcenter of PR2 in yaw than in pitch?
Overall, I think the benefit of this investigation would be a reduction of the noise in CHARD, which would help improve the sensitivity.
TITLE: 08/29 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 142Mpc
INCOMING OPERATOR: Corey
SHIFT SUMMARY:
IFO is in NLN and OBSERVING as of 19:03 UTC (with some small drop-outs to reload ISC lock and test some filters)
We only had one lockloss today and it was during our 9-12 PST comissioning. Lock acquisition was fully automatic.
Other than this, comissioning went well. Some highlights:
LOG:
Start Time | System | Name | Location | Lazer_Haz | Task | Time End |
---|---|---|---|---|---|---|
23:58 | SAF | H1 | LHO | YES | LVEA is laser HAZARD | 18:24 |
15:50 | FAC | Karen | Optics, Vac Prep | N | Technical Cleaning | 16:22 |
16:27 | FAC | Tyler | Patio | N | Forklift + Heat Pump Moving | 18:04 |
16:27 | FAC | Karen | PCAL Lab | Local | Technical Cleaning | 17:23 |
16:28 | PCAL | Francisco | PCAL Lab | Local | Escorting | 17:08 |
17:23 | PEM | Sheila, Robert | LVEA | YES | Looking for mystery light on ISCT1 | 17:48 |
17:58 | PEM | Robert | LVEA | YES | Check no-light and let PSL light back out ISCT1 | 18:09 |
18:43 | FAC | Karen | Wood Shop | N | Technical Cleaning | 18:43 |
22:28 | PCAL | Francisco | PCal Lab | Local | Measurements | 22:54 |
22:44 | PEM | Sam, Carlos, Robert | EY | N | PEM Measurements | 22:54 |
22:58 | VAC | Janos | MY | N | Measurements | 23:58 |
TITLE: 08/29 Eve Shift: 2300-0500 UTC (1600-2200 PST), all times posted in UTC
STATE of H1: Observing at 141Mpc
OUTGOING OPERATOR: Ibrahim
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 7mph Gusts, 5mph 5min avg
Primary useism: 0.02 μm/s
Secondary useism: 0.10 μm/s
QUICK SUMMARY:
H1 has been locked 5.5+hrs, with range hovering under 150Mpc. Environmentally we are really quiet.
There has been mention of taking H1 down briefly for a PRCL FF test/change (ElennaC).
Here is a comparison of DRMI and CHARD control signal spectra from July 11th to this afternoon, after Elenna's new HAM1FF.
The control room FOM seems to indicate that there is some huge excess noise in our DRMI signals, but this comparison shows that isn't the case. Also, the CHARD control signals are largely similar, with an improvement in pitch which is likely from the HAM1 FF tuning.
Robert, Sheila
Robert and I went to ISCT1 before the beam diverters closed in this morning's lock acquisition. We saw that there was a beam from a POP path beam splitter that was dumped on a razor blade dump that was hitting the top of the dump, so that a small part of the beam was visible on the ALSX camera. We adjusted the beam dump to fully catch the beam. Once the beam divereters are closed this beam isn't there, so this shouldn't have been a noise issue in nominal low noise.
Robert has seen coherence between an accelerometer on this table and DARM in observing, we tried closing the ALS light pipe and Robert saw that didn't change the coherence, so he opened the light pipe again.
We did a 10 minute on/off HAM1 FF test today. I determined from that test that improvement could be made to the feedforward to CHARD P and INP1 P, so I used that off time to train new filters (time here: 79792).
I took a screenshot of the DTT results. The golden traces represent the various loop spectra with the feedforward OFF. Blue represents the previous feedforward, and red represents the new feedforward I fit today. First, look at the top and bottom plots on the left, showing CHARD P and INP1 P. You can see that the old feedforward was having minimal benefit (gold to blue), and the new feedforward is performing much better (gold to red).
Next, look at the middle plot on the left showing CHARD Y. This plot shows that the feedforward is making CHARD Y WORSE (blue worse than gold). Therefore, I just turned it off. I am not yet happy with the fitting results for CHARD Y, so I will continue to work on them.
You can also see in the middle right plot, PRC2 P current feedforward is performing well (gold to blue), so I made no change, red still matches blue.
Note! This means the significant change here must be related to RF45- PRC2 is only sensed on RF9, while INP1 and CHARD use RF45.
Finally, DARM on the right hand side shows improvement from blue to red. It looks like HAM1 FF off is better below 15 Hz, maybe due to CHARD Y.
The new CHARD P and INP1 P filters are all saved in FM9 of their respective banks. I SDFed the new filters, and the CHARD Y gains to zero, second screenshot.
I made one mistake which is that I did not SDF these changes in SAFE, only in OBSERVE, which means that needs to be done before an SDF revert, since they are not guardian controlled!
Naoki, Sheila
We changed SRCL offset to see if it can improve the bucket sensitivity. Every time we changed the SRCL offset, we ran SCAN_SQZANG. We modified the SCAN_SQZANG to optimize the BLRMS6 at 2 kHz instead of BLRMS3 at 350 Hz. We reverted it after the measurement.
The attachment shows the DARM with different SRCL offset. The nominal SRCL offset is -175. The SRCL offset of -100 is worse than nominal in the bucket, but others are similar. So we left the SRCL offset at nominal value. Compared to O4b start, squeezing is good above 1kHz, but worse below 1kHz.
I ran the a2l_min_multi.py script, again not with a thermalized IFO though (Monday same story alog79714). This time there were some changes to ISC_library.py that made it so I was running into an import error with cdslib. I temporarily removed the dependence on cdslib by commenting it out, and I'll figure out how to get around this later. I did all quads and both dofs at the same time, ETMX P having the largest change of -0.14 but the rest were similar. I ended up unmonitoring all of these gains since they are saved in guardianl I updated lscparams.py with these new values, but I have no loaded ISC_LOCK yet.
RESULTS
*************************************************
ETMX P
Initial: 3.13
Final: 2.99
Diff: -0.14
ETMX Y
Initial: 4.87
Final: 4.9
Diff: 0.03
ETMY P
Initial: 4.48
Final: 4.48
Diff: 0.0
ETMY Y
Initial: 1.13
Final: 1.13
Diff: 0.0
ITMX P
Initial: -0.98
Final: -1.0
Diff: -0.02
ITMX Y
Initial: 2.87
Final: 2.87
Diff: 0.0
ITMY P
Initial: -0.37
Final: -0.39
Diff: -0.02
ITMY Y
Initial: -2.48
Final: -2.45
Diff: 0.03
I was able to new SRCL and MICH feedforward using the iterative method. Measurements were taken in alog 79693.
The current MICH FF is performing well except for below 30 Hz, so that's where I targeted in my fit. SRCL needed some improvement everywhere.
The new MICH filter is saved and loaded in FM8 and the new SRCL filter is saved and loaded in FM5. They are both labeled with the date '8-27-24'.
To compare how these filters are performing, I would use the templates Oli saved in the above alog, but first save the live trace as a reference to compare against.
I have not yet done the PRCL fit because I want to see how SRCL performs, and maybe redo the PRCL injections before trying the fit.
Here is some more information about the fits.
First, my MICH fit caused a lockloss this morning because I forgot to the check the phase on the filter. Sometimes, the phase gets flipped during the fitting. Usually, I compare the sign of the phase with the previous filter in foton and adjust accordingly, but I forgot to do it this time (rookie mistake). I have double checked the new SRCL filter phase and fixed the MICH phase.
Attached are two screenshots of the fittings. First, the MICH fitting compares the current filter, labeled as "reference 1" with the red trace which represents the new fit. The bottom right plot compares the fit residuals. You can see from this plot that the most improvement occurs at low frequency, with some small improvement at mid frequency. The SRCL fitting has many more traces, but compare the orange "current fit" trace with the trace labeled "best with less HF gain". This has more improvement almost everywhere. There is an increase in gain at high frequency, but it is less than an order of magnitude, so I think it's ok. The new fit also has reduced the high Q feature around 300 Hz that was potentially injecting noise. There is a factor 5-10 improvement between 10-50 Hz that will help the most.
These have now been tested, SRCL passes, MICH fails.
The SRCL screenshot compares Oli's SRCL measurement from four days ago with the "current" filter, and the new "trial" filter that I applied today. There is clear improvement everywhere (except for a small worsening between 100-200 Hz), so I think we should use this filter.
The MICH screenshot compares Oli's MICH measurement from four days ago with the "current" filter, along with "no FF" and "trial". The trial did what I promised, and reduced the coupling between 10-20 Hz, but clearly at the sacrifice of the noise everywhere else. I think we should stay with "current".
I am changing the guardian to select this new SRCL filter (FM5) in "lownoise length control" and the gain back to 1. There will be an SDF observing diff for SRCLFF1 that can be accepted by the operator.
Some thoughts about MICH FF:
In my opinion the hardest region to fit is between 10-20 Hz because of the presence of some high Q features, such as around 17 Hz. It would be worth considering what is causing those features- perhaps some bounce/roll notches. Do we still need those notches, or could they be briefly turned off during a feedforward injection? That might make the low frequency portion easier to fit and therefore easier to achieve good subtraction 10-30 Hz.
Looks like these are maybe BS M2 LOCK L FM10. We can try turning them off I suspect. Foton says they have a 2 second ramp, so should be okay to turn off just before the measurement (I'm not sure if we need them all the time, but maybe we do).
Today Sheila took another injection of PRCL for me so I could fit a new feedforward. The fit looked promising, however once it engaged it apparently caused oscillations everywhere, and I turned it off fast enough to avoid lockloss (thanks Corey and Ibrahim!). I checked the phase and gains beforehand, no high Q features, etc so I don't know what could be the issue.
I ran A2L while we were still thermalizing, might run again later. No change for ETMY but the ITMs had large changes. I've accepted these in SDF, I reverted the tramps that the picture shows I accepted. I didn't notice much of a change in DARM or on the DARM blrms.
ETMX P
Initial: 3.12
Final: 3.13
Diff: 0.01
ETMX Y
Initial: 4.79
Final: 4.87
Diff: 0.08
ETMY P
Initial: 4.48
Final: 4.48
Diff: 0.0
ETMY Y
Initial: 1.13
Final: 1.13
Diff: 0.0
ITMX P
Initial: -1.07
Final: -0.98
Diff: 0.09
ITMX Y
Initial: 2.72
Final: 2.87
Diff: 0.15
ITMY P
Initial: -0.47
Final: -0.37
Diff: 0.1
ITMY Y
Initial: -2.3
Final: -2.48
Diff: -0.18
I am not sure how the A2L is run these days, but there is some DARM coherence with DHARD Y that makes me think we should recheck the Y2L gains. See attached screenshot from today's lock.
As a reminder, the work that Gabriele and I did last April found that the DHARD Y coupling had two distinct frequency regimes: a steep low frequency coupling that depended heavily on the AS A WFS yaw offset, and a much flatter coupling about ~30 Hz that depended much more strongly on the Y2L gain of ITMY (this was I think before we started adjusting all the A2L gains on the test masses). Relevant alogs: 76407 and 76363
Based on this coherence, the Y2L gains at least deserve another look. Is it possible to track a DHARD Y injection during the test?
Since I converted this script to run on all TMs and dofs simultaneously, its performance hasn't been stellar. We've only run it a handful of times, but we definitely need to change something. One difference between the old version and the new is the frequencies the injected lines are at. As of right now, they range from 23-31.5Hz, but perhaps these needs to be moved. In June, Sheila and I ran it, then swapped the highest frequency and the lowest frequency to see if it made a difference (alog78495) and in that one test it didn't seem to matter.
Sheila and I are talking about the AS WFS offset and DHARD injection testing to try to understand this coupling a bit better. Planning in progess.