LVEA has been Swept.
Ready to start locking now.

J. Oberling, R. Short, M. Pirello, F. Clara, F. Mera

Summary: We're pretty much back where we started, not sure where the problem is but sure there is a problem somewhere.

Since mid-September there have been an increasing number of glitches in the FSS.  These are seen in the FAST channel (signal to NPRO PZT), the PC channel (signal to phase-correcting EOM), and the TEMP channel (NPRO crystal operating temperature).  In addition, these glitches are the suspected cause of many locklosses in the last couple of weeks and have delayed relocking due to these glitches causing FSS oscillations.  In light of this we took a look at the FSS today.

At the start of maintenance we unlocked the IMC, leaving the FSS locked to only the Reference Cavity (RefCav).  I tweaked up the alignment then went out on the floor with Fernando and Ryan to take a look at the FSS transfer function.  I set up the Agilent network analyzer as normal, but the TF looked like garbage; I had to up the source power to 0 dBm (from our usual -20 dBm) to get a clear signal, and still the TF looked wrong.  See first attachment.  Seeing this, we filed a WP (12114) and pulled the TTFSS box from the PSL enclosure and took it to the EE shop.

The current TTFSS is old, and we were unable to find a test plan for it.  With Marc, Fil, and Fernando we began stepping through the circuit diagram, testing each component to make sure it was functioning as expected from the diagram (for example: "The gain should be x, is it?  No, but we might be injecting too high a frequency, see the filter in the feedback?  Lower the frequency, now is the gain OK?  Yes.").  This was somewhat time consuming, but we could find no single component that was clearly bad or not operating properly.  Visually everything looked ok, nothing was obviously blown or damaged.  Marc did clean some excess flux from some of the solder pads.  We then checked the LO and everything looked OK there as well; signal amplitude might be a little high but Fil didn't see anything worrisome.  Not knowing what else to do, I reinstalled the TTFSS box.

Once the box was installed I took a TF, this time from inside the enclosure using the Agilent unit we keep in there.  The TF looked just fine and as expected, see 2nd attachment.  With a common gain of 15 and a fast gain of 9 the FSS has a UGF of 452 kHz with 65° of phase margin; the peaks around 770 kHz are still present like normal.  We decided to check the crossover (where the NPRO PZT and the phase-correcting EOM meet) at ~20 kHz (this is set with the fast gain).  The 3rd attachment shows the crossover as found, with a fast gain of 9.  We tried raising the fast gain, but this made the crossover peak larger.  Lowering the fast gain made the peak smaller but also starts to raise the overall signal (see 4th and 5th attachments, with a fast gain of 7 and 6, respectively).  We decided to leave the fast gain at 7 for now, will see how things behave (in the 3 hours since we made this change and the time of writing this alog behavior seems unchanged).  The picture with the fast gain at 7 does looke a little worse than the intial one with fast gain at 9; not sure why, we went back and forth between them a couple times and 9 looked worse than 7 each time (maybe I got "lucky" with the camera shot?).  This done we transitioned the enclosure back to Science Mode.

Before leaving the LVEA we tried one more thing.  Since we had a successful TF measurement inside the enclosure, we hooked up the Agilent unit by the PSL/ISC racks again and took a TF (since I had an apparent junk measurement with this same unit earlier in the day, see that 1st attachment again).  The TF looked perfectly fine, matched the measurement done inside the enclosure.  Not sure where the garbage TF from this morning came from, at this point I can only call it "user error of unknown origin."

So we've ended where we started, sure there's a problem but not sure where it is.  The current TTFSS appears to be operating correctly, but the FAST signal is still glitching; we even saw glitching with the IMC unlocked, although Ryan reports that it "seems" like the glitching is more frequent when the IMC is locked (maybe IFO feedback is exacerbating an existing issue...).  Marc and Fil have been testing a newer gen of TTFSS (not the same as used in SQZ, as that one is not compatible with the PSL as currently wired), so once that is qualified we will try switching to it (Peter King had it working with the RefCav in the optics lab, so it should work with the PSL).  We'll try that next.

TITLE: 10/01 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Microseism
OUTGOING OPERATOR: Ibrahim
CURRENT ENVIRONMENT:
    SEI_ENV state: MAINTENANCE
    Wind: 21mph Gusts, 15mph 3min avg
    Primary useism: 0.08 μm/s
    Secondary useism: 0.81 μm/s
QUICK SUMMARY:
Secondary Micro Seism is elevated,
The Vacuum team is still working on an ION pump annulus swap out on the floor of the LVEA.
Locking attempts will start once the VAC team is done and a sweep of the LVEA has been completed.
 

FAMIS 21312

pH of PSL chiller water was measured to be just above 10.0 according to the color of the test strip.

TITLE: 10/01 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Microseism
INCOMING OPERATOR: Tony
SHIFT SUMMARY: Maintenance ran a little over again from the TTFSS investigation. The ground is moving today, elevated microseism alog80392 and a 6.6 from Tonga tripped some ISI WDs. The DAQ was not restarted. The LVEA is now is laser SAFE.
LOG:                                                                                                                                                       

Some of the work that was completed is as follows:

• Laser Safe transition
• ALS investigation cont. at both ends
• EE/CDS UPS checks
• TTFSS investigation
• ISC_EY front end replaced alog80395
• Reheat coil checks alog80401
• HAM1 ION pump investigation/swap

A 6.6 from Tonga rolled through around 20:20UTC, it started tripping watchdogs when the R wave came through (peaked at just over 10 on NUC10)

20:49 UTC ISI BS stage 2, ITMY stage 1 & 2 watchdogs tripped

21:23 ISI HAM7 wd trip, maybe from Jim moving stuff arond the LVEA? Verbal called out a glitch beforehand (all the cps sensor saw it)

DIAG_MAIN message was flashing occasionally - TIDAL LIMITS: X_ARM_CTRL_OUTPUT within 10 percent of limit

WP12099 h1iscey computer swap

Erik:

Recenly we discovered that h1iscey (W2245) has a bad PCIe slot-7 on its motherboard, which is the cause of the A2 Adnaco issue in its IO Chassis. We have been skipping this slot in the past weeks.

h1iscey's front end computer was swapped with one from the test stand. See Erik's alog for details.

No model changes were required, no DAQ restart.

WP12108 laptop upgrade

Erik:

End station CDS laptops were upgraded.

Alarms reconfiguration

Dave:

The FMCS reverse osmosis (RO) has become very stable. The alarm settings were changed to alarm after 1 hour (was set to 1 day).

New CDS Overview

Dave:

New overview was released, see alog for details.

I have release a new CDS Overview, image is attached.

The previous overview was a hybrid of computer generated and hand drawn content. Specifically the upper 80% (the FE and DAQ section) was computer generated and the lower 20% was a mix of external files and hand drawn.

The new overview is completely genenerated by python code (generate_cds_overview_medm.py). This is reflected in its name change from H1CDS_APLUS_O4_OVERVIEW_CUST.adl to H1CDS-CDS_OVERVIEW_MACHINE_GEN.adl.

To open the new CDS overview, you need to kill all existing MEDMs and restart the sitemap from scratch, otherwise you will see a strange overview-within-an-overview screen.

New features:

Lower area has better layout, rows and columns align.

New IFO range "7 segment LED" section, hopefully range can be more easily read from afar. Currently it has a static orange-on-black color scheme, later version will turn green when in observe.

Color standardisation: related display buttons are light-blue on navy, shell execution buttons are light-green on dark-green.

Most areas can be clicked to open related displays.

To Do:

Complete the slow-controls and FMCS areas

The LVEA temperature trends will stabilize throughout the day today. The excursions are a result of the work I did this morning to the zone reheats. 

The TTFSS investigation is still ongoing, microseism has greatly risen to above the 90th percentile, the wind is also rising but its only ~20 mph.

There was an issue today where the VM hypervisor running alog, svn, awiki, ldap, and some services lost its network.  It has been restored.

The microseism today has reached fully above the 90th percentile, recreating Roberts plot from alog74510 the seismometers seem to show the largest phase difference is between EX and the CS, and the 2nd highest is with EY which suggests its because of the motion from our coast? From windy.com there's currently a 10 meter wave low pressure system off the coast of WA that's pretty much along the axis of the Xarm. These seismometers are also dominated by a ~0.12 Hz oscillation.

Yesterday afternoon until 2 am, we had low range because the squeezing angle was not well tuned.  As Naoki has noted 78529 this can happen when the OPO PZT is at a lower voltage than we normally operate at. This probably could have been improved by running SCAN_SQZANG. 

I've edited the OPO_PZT_OK checker, so that it requires the OPO PZT to be between 70 and 110 V (it used to be 50 to 110V).  This might mean that sometimes the OPO has difficulty locking, (ie, 76642), which will cause the IFO to call for help, but that will avoid running with low range when it needs to run SCAN_SQZANG.

No new damage noted. First pic is EX, second is EY.

h1iscey frontend server was replaced because of a possible bad PCI slot that was preventing connection to one of the Adnaco PCIe expansion boards in the IO Chassis , see here: https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=80035

The new frontend is running without issue, connected to all four Adnacos.

The new frontend model is Supermicro X11SRL-F, the same as the old frontend.

Tue Oct 01 08:09:24 2024 INFO: Fill completed in 9min 20secs

VACSTAT issued alarms for a BSC vacuum glitch at 02:13 this morning. Attachment shows main VACSTAT MEDM, the latch plots for BSC3, and a 24 hour trend of BSC2 and BSC3.

The glitch is a square wave, 3 second wide. Vacuum goes up from 5.3e-08 to 7.5e-08 Torr for these 3 seconds. Neighbouring BSC2 shows no glitch in the trend.

Looks like a PT132_MOD2 sensor glitch.

There is a bump disturbance in the H1 strain spectrum that is degrading the noise in the vicinity of the Crab pulsar. Attached are zoomed-in ASDs from recent dates (drawn from this CW figure of merit site). That band was pretty stable in O4a, but has been unstable in O4b. I can see hints of the bump as early as the end of March 2024 (see calendar navigation to sample dates).

I have poked around in Fscan plots and tried the handy-dandy Carleton coherence mining tool, but saw no obvious culprit PEM channels. Is there a new motor running somewhere (or an old motor with a drifting frequency)? 

Attachments:

Sample ASD snippets from August 1, September 1 & 2, along with a snippet from the end of O4a on January 16.

The red curve is from one day of averaging, and the black curve is the run-average up to that date.

The vertical bars show the approximate band of the Crab pulsar during the O4 run (taking into account Doppler modulations and long-term spin down).

Edit : Neil knows how to make links, now. Tony and Camilla were instrumental in this revelation.

This is an update to my previous post (Neil’s previous post on this topic) about looking for possible explinations why similar seismic wave velocities on-site may or may not knock us out of lock.

The same channels are used in addition to:

SEI- * ARM_GND_BLRMS_30M_100M
SEI- * ARM_GND_BLRMS_100M_300M
Where * is C, D, X, or Y.

I have looked at all earthquake events in O4b, and only ones which knocked us out of lock. This is to simplify the pattern search, for now. Here are the results.

Total events : 29

Events with ISI y-motion dominant : 11 (30-100 Hz) : 8 (100-300 Hz)
Events with ISI x-motion dominant : 3 (30-100 Hz) : 0 (100-300 Hz)
Events with ISI z-motion dominant : 9 (30-100 Hz) : 19 (100-300 Hz)

Events with ISI xy-motion dominant : 1 : 0 (Both axes are similar in amplitude.)
Events with ISI yz-motion dominant : 0 : 1
Events with ISI xz-motion dominant : 0 : 1

Events with IS xyz-motion dominant : 5 : 0

Total SEI- * ARM recorded events : 8
CARM dominant events : 7 : 8
C/XARM dominant events : 1 : 0

Conclusion is that in the 30-100 Hz band, it is equally likely to have either z- or y-axis motion be dominant. In the 100-300 Hz band, the ratio is about 1:2 for z and y motion being dominant during lockloss.

Clearly, common modes are a common (^_^) cause of lockloss.

Note that velocity amplitudes should be explored more.