TITLE: 12/0 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Planned Engineering
OUTGOING OPERATOR: None

QUICK SUMMARY: Still in the midst of recovery from the power outage yesterday. Looks like all suspensions stayed damped overnight except the BS which tripped at 0422UTC. SEI is still in a non-nominal state as Jim left it yesterday (alog88373). Many medm screens, scripts, other processes are running *very* slow. 

Recovery will continue, one forward step at a time.

Recovery of the front end systems was hampered by a slow /opt/rtcds file system, which still persists. There are no logs on h1fs0 indicating the problem, and the ZFS file system itself appears to be fully functional. Hourly backups by h1fs1 continue normally. At this point we cannot rule out a network issue.

The alarms system is running but does not seem to be operational, again no errors are being logged and channel access appears to be working.

Jim reported a bad model initialization with its safe.snap, most probably caused by a slow /opt/rtcds. I had to restart many models several times to get through a "burt restore" timeout for similar reasons.

We may need to restart h1fs0 in the morning and remount /opt/rtcds on all the NFS clients

The positive 18V power supply which provides power to Rack SUS-C1 and SUS-C2 failed after the power outage. Fan seized and power supply tripped off. Power supply was replaced.

New Power Supply  installed - S1300291

F. Clara, M. Pirello

WP 12915
Corner Station Controls Chassis 4 Wiring Diagram - E1101222
EtherCAT Corner Station Chassis 4 - D1101266
Modifications to the Beckhoff chassis - E2000499

The EtherCAT Corner Station Chassis 4 was modified per E2000499. New Beckhoff terminals were installed to support JAC and BHD. A new rear panel was installed to accommodate the new rear adapter boards and shutter control connectors. The din rail power terminal blocks were relocated making the exising power cables too short. New power cables installed to the EtherCAT couplers and power terminals. All field cabling was reconnected, expect for ISC_313. The Beckhoff software will need to be updated.

EtherCAT Corner Station Chassis 4 - Serial Number S1107450

F. Clara, D. Sigg

Seems like all of the models are running, but many of the guardians are still down. What I was able to get up and running:

HEPI Pump stations for all 3 buildings. Patrick had to help some with the end station beckhoff. I had to adjust the float switch for the overflow tank at EX, we were ridding just at the trip level there. The corner station Athena looked dead when I powered on the power supply on the mezzanine, but I think some of the leds are just dead. Good thing we plan to replace that soon.

Both BRS are running. Patrick had to power cycle the EX beckhoff, we were then able to restart the software. The damping settings got reverted (highthreshold and lowthreshold), so I had to fix those through the cli. Heater settings also got lost, both brs cooled off a bit over the several hours we were blind, and will take a little while to warm back up. Probably will keep watching for a while to make sure everything is okay, but I think we are good.

Some of the chambers are able to be isolated through guardian, but many have dead guardian nodes. I manually engaged damping where I couldn't get the ISI isolated via guardian.

None of the blend, sensor correction or cps diff is alive enough to recover yet. I have manually disabled sensor correction for the corner, the ends can't be isolated yet.  I guess that will wait for tomorrow. 

Oli, Ryan S, Rahul

Except ETMX and ETMY (timing error, work currently ongoing), we have recovered all other suspensions by un-tripping the WD and setting it to SAFE for tonight. The inmons looks fine - eyeballed them all (bosems and aosem), nothing out of the order.

For ETMX and ETMY, Dave is currently performing a computer restart, following which they will set to safe as well.

TITLE: 12/04 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Planned Engineering
INCOMING OPERATOR: None
SHIFT SUMMARY:

After CDS & H1-Locking work of yesterday, today we transitioned to starting prep work for the upcoming vent of HAM7.

However, there was more CDS work today which was related to ISC Slow Controls....but in the middle of this...

There was a 90min power outage on site!!
 

LOG:

• early-1615 Randy was at the FCES putting stickers on items
• 1539-1900 pre-vent technical cleaning begins (kim.nellie)
  • 1716 SR3 + HAM5 tripped; gave it some time, untripped/redamped @1805
• 1545 SEI_ENV transitioned to MAINTENANCE for vent
• 1609 TCSx Laser off (TJ contacted; prob due to cleaning near the rack)
• 1624-1745 BSC2 platform & then sorting parts (randy)
  • Tripped BS, ISI, HEPI---leaving tripped until randy finishes in about an hr
  • Once he was done, BS was taken to ALIGNED & SEI_BS taken to DAMPED_HEPI_OFFLINE
  • 1745-1845 Sorting through rail parts in North Bay
• 1626 quick assistance for cleaning (chris)
• 1628-1635 lvea walkthru (richard)
• 1630 cer to pull slow control chassis for ISC (fil)
  • This affects a PZT for the IMC, so it lost lock.  Keeping IMC_LOCK at OFFLINE during this beckhoff work.
• 1718-1810 Unboxing the JAC in the optics lab (jennie, jason)
• 1728 Taking Integrating sphere (had been at KAGRA) into PCal Lab (tony)
• 1730-1855 Grabbing parts at offsite vendor (mitch)
• 1751-1900 Building 2nd CHETA enclsure in the CHETA/JAC Lab (matt)   
• 1803-2012 OPO Testing cont (KarMeng.Eric)
  • 1812 Shiela joining but will be in/out with them + looking for parts
• 1813-1820 chamber measurements for JAC (jason)
• 1814 Kobelco (purge air compressor) being turned on (travis,jordan)
• 1848-2000 Inventory work at EX + EY (randy)
• 2005 Grabbing wipes in Cleaning Room (ibrahim)
• 2023 CHETA Viewport part search (gerardo)
• 2025 Power Outage ON Site!
  • Ended up down for over 90min
• See oli's alog for a summary of recovery.

R. Short, P. Thomas, J. Oberling

We have recovered the PSL after today's power outage.  Some notes for the future:

• When recovering the PSL Beckhoff PC and control software, make sure both Control Boxes are ON before trying to start the software
• The PSL's calibration data was lost, again (this seems to happen after every software restart, we're still not sure why)

I've attached a picture of the Settings table for PSL sensor calibration and operating hours for future reference.  Again, our persistent operating hours (that track total uptime of PSL laser components; OPHRS A in the attached picture) will continue to be wrong as we cannot update this value.  The current operating hours, which tracks operating hours of currently operating components (i.e. we've been running this specific NPRO for X hours; OPHRS in the attached picture) are correct.

We have the PMC and FSS RefCav locked, but have left the ISS OFF overnight while the PMC settles.  The PMC requires a beam alignment tweak (normal after an extended time off, like a 90 minute power outage) but we don't yet have Beckhoff so we don't have access to our picomotor mounts.  I'll tweak the beam alignment tomorrow once Beckhoff has been recovered.

[Sheila, Eric, Karmeng]

We checked the NLG of the OPO without cavity lock, threshold power is roughly at 3.7mW, at the same order as the test performed at MIT (E2500270)

Checked the crystal alignment, found 9 good dual resonant position, from -2689um to 1223um, with 490um separation between each point. Everything seems to be in a good position. 

Plus a clipped dual resonance position at -3190um. Crystal edge is roughly at 1500um and -3200um.

Red alignment does not shift between the various crystal position.

It's been a few minutes so far.  There is emergency lighting.  Luckily since it was lunchtime there were no people out on the floor.  Recovery Begins!

FAMIS 38886, last checked in alog88025

Both HAM5 and the BS chambers were in 'ISI_DAMPED_HEPI_OFFLINE' at the time this was run, but all other chambers were nominal.

There are 15 T240 proof masses out of range ( > 0.3 [V] )!
ETMX T240 2 DOF X/U = -1.565 [V]
ETMX T240 2 DOF Y/V = -1.633 [V]
ETMX T240 2 DOF Z/W = -0.886 [V]
ITMX T240 1 DOF X/U = -2.256 [V]
ITMX T240 1 DOF Z/W = 0.467 [V]
ITMX T240 3 DOF X/U = -2.386 [V]
ITMY T240 3 DOF X/U = -1.073 [V]
ITMY T240 3 DOF Z/W = -2.895 [V]
BS T240 1 DOF Y/V = -0.323 [V]
BS T240 2 DOF Z/W = 0.327 [V]
BS T240 3 DOF X/U = -0.547 [V]
BS T240 3 DOF Z/W = -0.401 [V]
HAM8 1 DOF X/U = -0.638 [V]
HAM8 1 DOF Y/V = -0.729 [V]
HAM8 1 DOF Z/W = -1.11 [V]

All other proof masses are within range ( < 0.3 [V] ):
ETMX T240 1 DOF X/U = 0.007 [V]
ETMX T240 1 DOF Y/V = -0.049 [V]
ETMX T240 1 DOF Z/W = -0.017 [V]
ETMX T240 3 DOF X/U = -0.005 [V]
ETMX T240 3 DOF Y/V = -0.056 [V]
ETMX T240 3 DOF Z/W = -0.051 [V]
ETMY T240 1 DOF X/U = 0.045 [V]
ETMY T240 1 DOF Y/V = 0.198 [V]
ETMY T240 1 DOF Z/W = 0.249 [V]
ETMY T240 2 DOF X/U = -0.073 [V]
ETMY T240 2 DOF Y/V = 0.218 [V]
ETMY T240 2 DOF Z/W = 0.037 [V]
ETMY T240 3 DOF X/U = 0.268 [V]
ETMY T240 3 DOF Y/V = 0.076 [V]
ETMY T240 3 DOF Z/W = 0.147 [V]
ITMX T240 1 DOF Y/V = 0.254 [V]
ITMX T240 2 DOF X/U = 0.151 [V]
ITMX T240 2 DOF Y/V = 0.258 [V]
ITMX T240 2 DOF Z/W = 0.223 [V]
ITMX T240 3 DOF Y/V = 0.104 [V]
ITMX T240 3 DOF Z/W = 0.094 [V]
ITMY T240 1 DOF X/U = 0.074 [V]
ITMY T240 1 DOF Y/V = 0.123 [V]
ITMY T240 1 DOF Z/W = -0.013 [V]
ITMY T240 2 DOF X/U = 0.017 [V]
ITMY T240 2 DOF Y/V = 0.22 [V]
ITMY T240 2 DOF Z/W = 0.148 [V]
ITMY T240 3 DOF Y/V = 0.091 [V]
BS T240 1 DOF X/U = 0.184 [V]
BS T240 1 DOF Z/W = -0.223 [V]
BS T240 2 DOF X/U = 0.094 [V]
BS T240 2 DOF Y/V = -0.224 [V]
BS T240 3 DOF Y/V = 0.004 [V]

There are 2 STS proof masses out of range ( > 2.0 [V] )!
STS EY DOF X/U = -4.625 [V]
STS EY DOF Z/W = 2.321 [V]

All other proof masses are within range ( < 2.0 [V] ):
STS A DOF X/U = -0.498 [V]
STS A DOF Y/V = -0.915 [V]
STS A DOF Z/W = -0.485 [V]
STS B DOF X/U = 0.154 [V]
STS B DOF Y/V = 0.939 [V]
STS B DOF Z/W = -0.337 [V]
STS C DOF X/U = -0.685 [V]
STS C DOF Y/V = 0.77 [V]
STS C DOF Z/W = 0.505 [V]
STS EX DOF X/U = -0.195 [V]
STS EX DOF Y/V = -0.111 [V]
STS EX DOF Z/W = 0.112 [V]
STS EY DOF Y/V = 1.271 [V]
STS FC DOF X/U = 0.071 [V]
STS FC DOF Y/V = -1.254 [V]
STS FC DOF Z/W = 0.567 [V]
 

Last night I took a calibration measurement (broadband and simulines) after the IFO was locked for about 3.5 hours. This generated calibration report 20251204T035347Z.

The results from broadband and simulines on the front page look reasonable. The plot label indicates that these are from PCAL X and Y but I think the broadband and simulines are run on PCAL Y only, so I don't know if PCAL X agrees with Y. I believe as of last night, the DAC for PCAL X had been swapped.

I made no changes to the pydarm ini file to account for the DAC change before running the report.

Thu Dec 04 10:15:45 2025 INFO: Fill completed in 15min 42secs

Plot has zoomed y-scale to reduce number of divisions in order to show detail.

FAMIS 27645

pH of PSL chiller water was measured to be between 10.0 and 10.5 according to the color of the test strip.

TITLE: 12/04 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Planned Engineering
OUTGOING OPERATOR: None
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 3mph Gusts, 1mph 3min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.33 μm/s 
QUICK SUMMARY:

H1 has been locked almost 15hrs....but H1 is going DOWN in prep for the HAM7 vent and will be set to IDLE. 

There will be more CDS (have heard Beckhoff) work, but no "proof of life" for H1 after that work will be done. 

Now the focus is on prep to vent with Kim & Nellie commencing shortly with thorough technical cleaning of HAM chambers.  Other possible items on the list for today:  

• Turn on Kobelco purge
• Forklifting furniture
• HAM7 vent prep
• Prep for HAM7 -Y door removal

It seems like it would be useful to have this recipie written in one place.  After the gate valves are opened and closed, we often have a hard time relocking because of alignment.  

Here's what we think we should do to recover after gate valves open.  

1. Do initial alignment without using green cameras in the loop, as the cameras are right next to the gate valves. 
   1. Start by using the baffle PD dither alignment script to set the TMS alignment: use the suspension guardians to misalign both ITMs and ETMs, run dither align script for both TMSs found on the INIT ALIGN MEDM, (dither align scripts, TMSX and TMSY, can run at the same time).  Open the ITM baffle PD scope and watch the script find the pointing of TMS that sets the beam to PD1, PD4, and finally centered on the optic.  When this finishes sucsesfully for both TMSs, there should be a dim green beam visible on both ALS cameras, if there is not check the alignment of PR3 (affects both arms) and the BS (only moves the Y arm beam).  This TMS alignment has to be set before you can move onto the next step, otherwise the ITM alignment won't be right. 
   2. Use the suspension guardians to align both ITMs, but leave the ETMs misaligned.  Run the dither alignment script for ITMX and ITMY, this time watching the ETM baffle PD scope to watch the script's progress.  Once this is finished, the ITMs should be set, and we should not move them based on the cameras. 
   3. Use the suspension guardians to set the ETMs to aligned, and you will hopefully see mulitple beams or cavity flashes on the ALS cameras.  You can manually align the ETMs to increase the flashes until they are good enough to lock the arm.  Use the ALS guardians (in auto mode might be helpful) to set them to ETM_TMS_WFS_OFFLOADED, this will not run the green camera servos, it runs green WFS to the ETM and TMS alignment only to increase build up. Don't run INITIAL_ALIGNMENT or INITIAL_ALIGNMENT offloaded. 
   4. Align the vertex to these ITMs: set the ALIGN_IFO guardian to manual, init it to control the suspension guardians.  Choose INPUT_ALIGN_OFFLOADED, once that finishes PRC_ALIGN_OFFLOADED, then MICH_BRIGHT_OFFLOADED, then SRC_ALIGN_OFFLOADED.  
2. Make two guardian changes before you start locking the full IFO
   1. Don't use the SRC1 + SRC2 ASC loops in DRMI, these don't seem to work until the alignment is set well for full lock.  You can turn these off by going to lscparams, and searching for use_DRMI_ASC, setting SRC1+ SRC2 to false.  Load ISC_DRMI for this change to take effect. 
   2. Don't shutter the ALS beams once they are no longer used for locking.  In ISC_LOCK, find the edge     ('IDLE_ALS',  'CARM_OFFSET_REDUCTION', 5), and reduce the weight which is normally 5 to 1, so that this path which avoids shuttering the ALS becomes the default path.  Load ISC_LOCK for this change to take effect.
3. Start to relock as usual, going through PRMI.  Stop at CHECK_DRMI_ASC, where all the DRMI ASC loops will be running other than SRC1+SRC2, and manually adjust SRM sliders to reduce POP90 and increase POP18. 
4. You can try to lock all the way to ENGAGE_ASC for full IFO, but this may not work.  If it does not, you may need to manually align DHARD, CHARD, and SRM. 
   1. If you are adjsuting alignments by hand, you want to keep the green beams locked to the arms as the alignment changes.  To get the green alignment to follow any changes you make, stop at IDLE_ALS and turn on the scripts that servo the green TMS QPD offsets based on the green WFS.  You can access this script by going to either ALS end station overview and clicking the button that says QPD OFFSETs, then using the pink drop down button to start the servo for each arm, and open the ndscope that lets you watch the progress of these servos (screenshot)
   2. If the power in the two arms is mismatched (visible on the PRMI SB4 min ndscope on the front wall), this could be a sign that DHARD is misaligned. 
      1. If you notice this in a state before DHARD WFS are closed, RF_DARM or CARM_OFFSET reduction , you can use the move_arm_dev script to step DHARD pitch and yaw to bring the arms closer to each other.  userapps/asc/h1/scripts/move_arm_dev.py DH P 0.01, or similar.  
      2. If you notice the arm powers diverging from each other after DHARD WFS are closed, this might be fixed by moving SRM while watching to make sure POP 90 stays low.  
   3. If the power recycling gain is low, you may need to adjust CHARD.  You can step through CARM_OFFSET reduction or stop after it finishes and use 62110 to check if the refl power is too low for the arm transmission that you are at, and try to move CHARD while trying to increase the refl power (watch build_up_slow ndscope linked from the ISC guardians overview and other places).  
5. Once you are all the way locked (congratulations), stop at PREP_ASC_FOR_FULL_IFO before you try turning on ASC.  It may be a good idea to note the green QPD offsets with a screenshot, if you loose lock they wil be reverted with SDF revert but you would need to reset them to these values to relock.   Also look at the ITM green camera offsets, from each ALS end station overview in the green WFS section there is a button that says CAM ITM YAW or PIT, you can take a screenshot of this screen for both ITM cameras and compare the current location of the beam centroid to the setting (screenshot)
6. Use a guardian shell (guardian -i ISC_LOCK) to copy and paste lines from ENGAGE_ASC_FOR_FULL_IFO manually, skipping SRC1 +SRC2.  (you might grab another screenshot of the green QPDs and cameras at this point)
7. Then you can manual the guardian to TMS_SERVO, engage_soft_loops, and DARM offset, and take another green QPD offset + camera screenshot.  
8. Having the DARM offset on as well as the soft loops seems to help the SRC ASC work, so try engaging those once those steps are done.  (Congratulations on getting all the ASC controlled)
9. Important step: reset the green references.  In the past we have reset the green QPD offsets by accepting them in SDF as well as the camera offsets, but it doesn't make sense to be changing the QPD offsets as they shouldn't be impacted by the gate valve moving the camera.  Today once we had all the ASC engaged, we stepped the QPD offsets back to their original settings from SDF, watching the arm transmissions in the ndscope.  The arm transmissions seemed fine with the original QPD offsets, so we left those there and only updated the green camera offsets.  I think this should be the way we do things in the future, not reseting the QPD offsets by default.  
10. undo the gaurdian changes from step 2 above.  After the next lockloss, run initial alignment as usual, letting the green cameras be used to point the ITMs.  DRMI ASC should work now that the alignment has been set.