TITLE: 08/21 Eve Shift: 2300-0800 UTC (1600-0100 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
OUTGOING OPERATOR: Ryan S
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 15mph Gusts, 11mph 5min avg
Primary useism: 0.02 μm/s
Secondary useism: 0.07 μm/s
QUICK SUMMARY:
Walked in to H1 being at DARM to DC. TJ and Camilla went out to center the OMC Trans camera, but then H1 lost lock. Ryan has been working on getting them an OMC single-bounce so they can align the camera.
[Sheila, TJ, Ryan S, Elenna]
Here we go!
Ryan S began by running the dither scripts. The ITM and TMS dither scripts worked well, the ETM dither scripts did NOT. Note for future: do not run ETM dither scripts.
ALS X and Y locked well, X was faster than Y as usual. Beatnotes look good!
Input Align successful.
Sheila and I checked SR3 oplevs, and noticed while pitch was very similar to July 12th, pre-vent locking, the yaw oplev had moved by about 20 microrad. We checked that the sliders are about the same. Note: SR2 had to be moved significantly in yaw to find the beam at AS. We spoke to Jim and TJ; they pointed out that HAM5 had been moved (see 79183). Jim moved the HAM5 ISI back by 65 microradians, and this recovered the SR3 yaw oplev alignment to match pre-vent. Note for everyone: 65 micrad of HAM5 ISI motion compared to about 20 microrad of SR3 oplev motion. We should figure out this discrepancy!
Since this step fixed the SR3 yaw alignment, Sheila reverted SR2 back to the July alignment using the sliders.
Ryan began moving PRM by hand to get AS_A alignment better- after this PRC align was successful. However, as PRC1 converged, the AS_A_DC_NSUM decreased. The sensor for PRC1 in this step is REFL B. We did not offload this alignment and instead went to DOWN, and then reran the alignment of PRM by hand. Good alignment onto AS_A for PRM does not correspond to a converged PRC1 alignment in the PRC align state right now.
While struggling with MICH Bright align, Sheila realized that the AS centering was likely the problem. We went to down, turned on AS WFS centering, and saw a huge improvement in the signals, indicating that our issue was that the AS signals were no good. MICH Bright align occured quickly after this change. This may also explain the PRC align issue.
Ryan took us back to PRC align. Indeed, the PRC1 alignment signal converged while maximizing the AS A NSUM. We ran PRC align offloaded this time.
Ryan ran SR2 align successfully.
Sheila had to do a big move of SRM to find the fringing, otherwise, no issues with SRC align. (-577 urad yaw, +300 urad in pitch compared to early July slider positions)
We then paused so Camilla and TJ could realign the AS air camera on the floor.
***********
Lock failure at DRMI ASC- SRC ASC is probably no good, the PRMI ASC ran well. Sheila changed the flag so only MICH ASC runs for DRMI.
Next lock attempt failed at "resonance". Sheila checked the PRG based on the REFL dip and arm transmission, and it is between 48 and 54 just before lockloss (34% off resonance and 1230 arm transmission, see 62110)
Next lockloss towards the end of "engage asc for full ifo". Hard to tell the cause, but the corner alignment decayed as the various loops engaged.
At the next lock attempt, we stepped engage ASC by hand. All ASC is fine to engage except SRC ASC. Both the SRC1 and SRC2 error signals are far off, and it appears that they are not converging properly to maximize the buildups. This is extra confusing given that SRC2 actuates both SRM and SR2 (using AS_C), however even SRC1 is not very good (SRM actuation with AS RF72). We left the SRC ASC off and engaged the soft loops. By moving SRM and SR2 by hand we can keep the buildups good, but the ASC error signals are still bad.
After the soft loops converged, Sheila reset the green references so our alignment is saved. SDF screenshot attached.
We still don't know what to do about SRC ASC, but we have decided to go to DC readout and try damping some violins while we think. DARM offset is on, and we have made it to prep DC readout transition.
In order to lock the OMC, I needed to edit two lines in OMC_LOCK.py. I lowered the minimum peak height from 6 to 5 on line 328 and the "wrong mode?" check threshold in the OMC_LOCKED state also from 6 to 5 on line 512.
When we get to a better alignment or raise the DARM offset we're using, these should be raised again.
Accepted SQZ alignments sdfs from Monday 79592, plot attached. SQZ and ASCSQZFC are fine as the guardian will take control of them.
Turned ZM2 SAMs back on to 90.
Turned on OFI TEC loop back on. Was at 21fdeg, the set point is closer to 25deg. We will later check what temperature is best.
GV7 was opened (from soft-close) at 09:15 PDT. I've reconfigured the vacuum alarms to now sound on GV7 close.
DITTO for GV5
We have investigated the occurrence and correlation of specific states of the ISC_LOCK and INIT_ALIGN during O4a. We downloaded minute averaged data for these channels, taking the minimum for ISC_LOCK and the mean for INIT_ALIGN for every minute. The total number of minutes in O4a has been 341’281, we report the amount of time spent in each state as a percentage of the total time.
The left panel on the first plot shows the percentage of time spent in each ISC_LOCK states, while the right panel excludes the time spent in the ‘Low noise’ state.
We then looked into detail at the states occupied while the ISC_LOCK was in ‘prep’, ‘initial alignment’ or ‘als’ which make up the biggest contribution when we do not consider the ‘Low Noise’ state, i.e. when the detector is down.
The third plot shows the percentage of time out of the total time spent in ‘prep’ states for each of the sub-states.
The fourth plot shows the same for the ‘als’ states. It is interesting to notice that the total time spent in the state “Locking_arms_green” for O4a has been ~10 days, this includes the ISC_LOCK state 12 and the INIT_ALIGN state 8 while in the ISC_LOCK state 7 (Initial alignment).
In the fifth plot we show the percentage of time for each INIT_ALIGN state when the ISC_LOCK is on ‘Initial_alignment’ computed over the total time spent in this state. One can note that the most common states are ‘Locking_green_arms’, ‘SRC_aligning’, ‘Prc_aligning’ and ‘Init_align_complete’. The total time spent in this latter stage over the course of O4a is 12.5 hours. The last plot is an histogram of the occurrences of the consecutive minutes spent in the ‘init_align_complete’ status by INIT_ALIGN. The mean consecutive time spent in this state is 31 minutes. The outlier at 278 minutes corresponds to 3 am to 7 am on the 22nd September 2023.
Wed Aug 21 08:08:22 2024 Fill completed in 8min 18secs
Gerardo confirmed a good fill curbside. Text alerts did not come through following cdslogin reboot yesterday. I have re-enabled this for tomorrow's fill.
I rebooted the seismon server (seismon2), which tracks incoming earthquakes. A slow memory leak was causing it to run out of memory. It will need another reboot in a few months.
TITLE: 08/21 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
CURRENT ENVIRONMENT:
SEI_ENV state: MAINTENANCE
Wind: 6mph Gusts, 5mph 5min avg
Primary useism: 0.01 μm/s
Secondary useism: 0.07 μm/s
QUICK SUMMARY: Corner volume pumpdown continues; HAM5/6 HEPIs are still locked but are ready to be unlocked again.
Today's activities: - After the replacement of the PT-154 gauge (https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=79599), and after the small volume of the gauge was pumped overnight, now it has been valved together with the main volume - The small Pfeiffer turbo, along with the leak checker was taken off from the gauge's volume - Then, the HAM7-BSC3 small FCT section was valved together with HAM7 - FCV-1 opened - The 2 aux carts have been taken off from the HAM5/HAM6 annulus system - An aux cart was added to GV7 for safety reasons, to protect its annulus system during opening it up - The HAM6 turbo was valved out. The reason for valving it out so early is that the backing scroll pump had horrendous noise, so it needs rebuilding, and would have not been a good decision to let it run overnight. Especially, that the backing pressure was slowly growing. The hoses, cables, and the cart was also put away - RGA scans were collected at the corner, before opening up the large gate valves - GV5 and GV7. The results will be posted to the comment section
Corner RGA Scans
RGA Model: Pfeiffer Prisma Plus
EM Voltage: 1200V
Dwell Time: 500ms, 0-100 AMU
Valve States: GV5 & GV7 closed, RV1 closed, FC2 & FC3 closed (Relay Tube & FC Tube valved out of corner)
Pumping: 4x 2500 l/s Ion Pumps, 1x 500 l/s ion pump (HAM6), 3x 2000 l/s Turbo pumps
RGA Volume had been pumped with 80 l/s turbo & aux cart throughout duration of vent, but these were not valved in to RGA volume during scan collection, RGA filament on for ~1 week before scans collected
Corner Pressure at time of scans: 1.15E-7 Torr (PT120B)
Raw data files will be collected at T2400297
Last attachment is a comparison to the March pumpdown, which had GV5 and GV2 closed, 4x 2500 l/s IPs, 1x 500 l/s IP, 2x 2000 l/s TPs, March scan collected after ~12 days of pumping
August pumpdown started 8/6/24, and turbopumps were turned on on 8/7/24, corner has been pumping for 13 days, GV1 & 2 opened on 8/12/24
Yesterday morning I unlocked HAM5&6 HEPIs, unaware that VAC needed back up on that chamber to disconnect turbos. This afternoon, Mitch and Ibrahim helped me re-lock those HEPIs. Because we didn't think any alignment work would be needed before unlocking again, we didn't use the normal procedure to try to preserve HEPI position, I just turned off the loops and we engaged the stops. Even so, the biggest offsets from operating position are less than 50 um/urad, most are more like 10.
Jim, the vacuum team finished with all the works around HAM5 and HAM6, so feel free to unlock the HEPIs.
Unlocked now.
TITLE: 08/20 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
SHIFT SUMMARY: Just some light maintenance activities today as pumpdown continues. HAM5/6 HEPIs were locked so VAC team could valve out the turbopump on top of HAM6.
LOG:
Start Time | System | Name | Location | Lazer_Haz | Task | Time End |
---|---|---|---|---|---|---|
14:41 | CDS | Erik | CR | n | Restarting Picket Fence | 14:48 |
15:30 | FAC | Nelly | FCES | n | Technical cleaning | 16:22 |
15:40 | CAL | Dave | remote | n | restart h1calcs | 17:15 |
15:40 | DAQ | Dave | remote | n | Restart NDS1 | 17:15 |
15:42 | FAC | Karen | LVEA | n | Technical cleaning | 17:10 |
15:50 | FAC | Kim | EX | n | Technical cleaning | 16:49 |
16:14 | PEM | Genevieve, Sam | MY | n | Checking access | 16:30 |
16:23 | FAC | Tyler, Eric | LVEA | n | Moving cabinets | 17:01 |
16:42 | OPS | RyanC | LVEA | n | Restarting dust monitor | 16:50 |
16:45 | PSL | RyanS | CR | n | FSS RefCav alignment | 17:59 |
16:49 | FAC | Kim | LVEA | n | Technical cleaning | 17:50 |
16:58 | VAC | Jordan | MY, EY | n | Turbopump check | 19:15 |
16:59 | VAC | Travis | MX, EX | n | Turbopump check | 19:46 |
17:10 | FAC | Karen | MY | n | Technical cleaning | 18:16 |
17:14 | FAC | Richard | LVEA | n | Safety walkabout | 17:31 |
17:52 | ISC | Elenna | CR | n | IM transfer functions | 18:49 |
17:59 | FAC | Kim | MX | n | Technical cleaning | 18:51 |
18:33 | VAC | Janos | LVEA | n | VAC checks | 18:41 |
19:42 | SEI | Jim, Mitchell, Ibrahim | LVEA | n | Locking HAM5/6 HEPI | 19:58 |
20:08 | PEM | Genevieve, Sam | MX, EX, MY, EY | n | Taking pictures of PEM equipment | 21:43 |
20:38 | VAC | Janos, Jordan, Gerardo | LVEA | n | VAC checks | 21:38 |
22:04 | PEM | Robert, Genevieve, Sam | EBay | n | PEM things | 22:27 |
22:21 | VAC | Jordan, Gerardo | FCES | n | Move turbopump on HAM8 | Ongoing |
22:28 | PEM | Robert, Genevieve, Sam | LVEA | n | Taking pictures of PEM equipment | 23:13 |
22:30 | CAL | Tony | PCal Lab | local | PCal checks | 23:03 |
23:08 | VAC | Janos | LVEA | n | Valving out HAM6 turbo (on-chamber) | Ongoing |
MiriamR, RickS, TonyS, FranciscoL
On Monday, August 19, we moved the X-End pcal inner beam to be centered on the Rx input port. The previous configuration had the inner beam to the right of the Rx input port by 5mm. We centered the beam to the desigated position before the interferometer resumes the obsevation run.
Note: Screwing the yaw-adjuster in a clock-wise motion moves the beam from the right side of the target to the center (goes to the left).
The voltage from BothBeamsBefore.jpg was of 3.383 V as measured with a hanheld voltmeter, after moving the beams (see BothBeamsAfter.jpg) the voltage was of 3.384. EndStationLog.pdf has a list of voltage values with the corresponding times (local timezone) at which they were measured.
This morning, I manually ran through the usual magnetic injection suite (21 total injections across all stations) that's run before Tuesday maintenance during normal observing times to verify things were working. The only hiccup I encountered was that none of the injections at EX seemed to run; the amplifiers turned on correctly, the correct filter modules were applied, and Guardian showed no errors, but the magnetometers gave no response when the injection was supposed to start and I saw no signal in the appropriate _EXCMON channels. After conversing with Dave and Erik and seeing there was no problem starting a manual excitation, they suggested that the PEM_MAG_INJ Guardian likely needed to be restarted to re-establish connection with the frontends at EX since some of them were restarted in June. After restarting the node, I ran through the suite again and there were no issues.
The last successful test ran from 1408203575 to 1408204731 (about 22 minutes), and I've left the Guardian in WAITING.
Naoki, Camilla.
After SQZ laser was turned on laser week, today we locked PMC and SHG with no issues. We struggled with OPO locking (CLF dual). In the end Naoki could lock by hand but the trigger wasn't triggering. We will continue to troubleshoot tomorrow. The OPO dither lock worked well but only after we swapped H1:SQZ-OPO_IR_LSC_SERVO_GAIN sign from -10 to 10: strange!
I put ZM4/5 PSAMS and ZM4/5/6, ZM2/3 and FC1 alignments back to the same values as in alog79193. Todays plot and July 17th plot attached. There was only around 0.3mW on the IR trans, when we expect 1mW. So may need some more alignment moved. We have the SQZT7 iris to help if needed.
CLF input power was lower than before. After I increased CLF input power, OPO can be locked by guardian.
After Camilla changed ZM2 PSAMS in 79625, we can see 1.3mW seed trans in SQZT7 with 75mW seed. However, seed dither lock did not work. It seems that the OPO PZT offset after scan by guardian does not match well with the seed resonance. I found that there is an intentional PZT offset for hysteresis in line 728 of OPO guardian as follows.
ezca['SQZ-OPO_SERVO_SLOWOUTOFS'] += 0.08 # hysteresis
I flipped the sign of this offset from 0.08 to -0.08 and the dither lock works. I am not sure if this is a correct way to fix this issue, but I tested the dither lock several times and all worked.
The dither lock gain needs to be reverted to -10.
Camilla, Oli
We looked into the weird results we were seeing in the ITMX In-Lock SUS charge measurements. The coherence when the bias is on for both the bias and length drives is always good, but the coherences are bad (usually below 0.09) for both bias and length drives when the excitations run with the bias off.
I compared the differences in the coherence value outputs in the python script versus the matlab script(screenshot), and although the values are calculated slightly differently and are not exactly the same, they are resonably close enough that we can say that there is not an issue with how we are calculating the coherences.
Next, we used ndscope to look at the latest excitations and measurements from July 09th (ndscope-bais on, ndscope-bias off). Plotting L3_LOCK_BIAS_OUTPUT, L3_DRIVEALIGN_L2L_OUTPUT, L3_LVESDAMON_UL_OUT_DQ, and L3_ESDAMON_DC_OUT_DQ for both ITMX and ITMY, if there was an issue with the excitations not going through, we would expect to see nothing on the ESDAMON and LVESDAMON channels, but we do see them on ITMX.
We were still confused as to why we would see the excitations go through the ESDAMON channels but still have such low coherence, so we compared the ITMX measurements to ITMY on dtt for the July 09th measurements looking at how each excitation showed up in DARM and what the coherence was. When the bias was on, both the bias drive and Length drive measurements look as we expect, with the drive in their respective channels, a peak seen in DARM at that frequency, and a coherence of 1 at that frequency(bias_drive_bias_on, length_drive_bias_on). However, in the comparisons with the bias off, we can see the excitations in their channels for both ITMX and ITMY, but while ITMY has the peak in DARM like the bias on measurements, ITMX is missing this peak in DARM(bias_drive_bias_off, length_drive_bias_off). The coherence between DARM and the excitation channel is also not 1 on ITMX.
We showed these results to Sheila and she said that these results for ITMX with the bias off make sense if there is no charge built up on the ITM, which would be the first time this has been the case! So there are no issues with the excitations or script thankfully.
We will be making changes to the analysis script to still run the analysis even if the coherence is low, and will be adding a note explaining what that means.
Hey LHO, the in lock charge measurement script is another script, other than A2L scripts and calibration scripts, that I overhauled last year due to my deep dissatisfaction with the existing code.
I'll point you to the aLog when I ran it: [68310].
Among the huge amount of behaviors that I correct I implemented many lessons I learned in implementing simuLines for LHO: your DARM units are a very small number and you must explicitly cast DARM data to np.float64 in order to have the TFs and coherences (in particular coherences) calculate correctly. I've had to repeat this lesson to lat least 4 people writing code for LHO now in the calibration group because it trips up people again and again and it is not an obvious thing to do, and something I solved through sheer brute force (took Louis a lot to convince since he initially refused to believe it).
In particular inside the ''digestData
" function of the "/ligo/svncommon/SusSVN/sus/trunk/QUAD/L1/Common/Scripts/InLockChargeMeasurements/process_single_measurement.py
" you will see me casting the gwpy data to float64 on lines 50 and 51; followed by some sampling rate tricks to get coherence to calculate correctly with gwpy's coherence
call as well as gwpy handles average_fft
calls.
Hope it helps!
Thanks Vlad, we'll have a look at that.
While looking at these measurements we realized that we were not using the same bias setting for all the quads (ITMY around half bias). We want to change this using the attached code but first will run the charge measurements to directly compare before and post vent.
I turned the CO2s back on today and CO2X came back to its usual 53W, but CO2Y came back at 24W. We've seen in the past that it will jump up a handful of watts overnight after a break, so maybe we will see something similar here. If not, we will have to investigate this loss. Trending the output of this laser, it has definitely been dropping in the last year, but we should be higher than 24W.
Sure enough, the power increased overnight and we are back to 34W. This is still low, but in line with the power loss that we've been seeing. Camilla is looking into the spare situation and we might swap it in the future.
We searched for home on both CO2 lasers, took them back to minium power and then asked CO2_PWR guardian to NOM_ANNULAR_POWER. This gave 1.73W on CO2X and CO2Y to 1.71W (1.4W before bootstrapping).
/psl/h1/scripts/RotationStage/CalibRotStage.py
), so we should change to this method next time.