Following from the airflow changes yesterday at the end station Adnaco chassis, the h1cdsrfm cards have maintained their new temps of (ey, ex, cs) = (64C,75C,61C).

Scanning all the cards this morning, their temps are in the range 61C to 79C (full list below).

61C = h1cdsrfm,h1omc0

79C = h1seib2,h1seih45

Addressed TCS Chillers (Wed [Aug7] 754-804am local) & CLOSED FAMIS #27795:

• TCSx Chiller:  Had level of 29.8 cm.  Added 170mL for 30.1cm.
• TCSy Chiller:  Had level of 10.2 cm.  Added 0mL.  
• There was no water in the small cup under the slow leak.

FAMIS: 26280
Laser Status:
    NPRO output power is 1.829W (nominal ~2W)
    AMP1 output power is 64.06W (nominal ~70W)
    AMP2 output power is 136.4W (nominal 135-140W)
    NPRO watchdog is GREEN
    AMP1 watchdog is GREEN
    AMP2 watchdog is GREEN
    PDWD watchdog is GREEN

PMC:
    It has been locked 28 days, 21 hr 1 minutes
    Reflected power = 19.8W
    Transmitted power = 107.1W
    PowerSum = 126.9W

FSS:
    It has been locked for 20 days 17 hr and 33 min
    TPD[V] = 0.6431V

ISS:
    The diffracted power is around 2.2%
    Last saturation event was 21 days 18 hours and 46 minutes ago

Possible Issues:
    AMP1 power is low
    PMC reflected power is high
    FSS TPD is low

Wed Aug 07 08:10:45 2024 INFO: Fill completed in 10min 41secs

Travis confirmed a good fill curbside.

Ran the Check_Dust_Monitor script and one of the PSL dust monitor is stuck at the same value & will need to be checked.

edit:  this is a KNOWN issue.  It's just that incursions into the PSL are somewhat rare & I had forgotten about this one.  This is a dust monitor in the anteroom--the dust monitor in the main PSL room is still operational.

TITLE: 08/07 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
OUTGOING OPERATOR: n/a
CURRENT ENVIRONMENT:
    SEI_ENV state: MAINTENANCE
    Wind: 4mph Gusts, 3mph 5min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.07 μm/s
QUICK SUMMARY:

Pumpdown of the Corner volume started last night---Janos just went out to continue Corner Pumping this morning.  (HAM7 also started to be pumped down yesterday.)

Les Guthman and film crew are here for day 2 of filming.

1,7 and 30 day trends of the INST AIR pressure signal can be viewed on the Ndscope Trend Web Page

1 Day

1 Week

1 Month

Today's activities:
- The relay tube was installed, and was leak checked: no leaks bigger than 1E-10 Torr have been detected
- The HAM5/HAM6 annulus system is being pumped down. The Turbo backing line is already at around ~4E-5 Torr
- The Mobile pumping station with 2 pcs. of ISP1000 pumping carts and with the mobile transformer has been installed next to the OMC Turbo station (see picture, and see the PID according to LIGO-E2300443-v1 Main volume Double Mobile Scroll Turbo Startup Procedure, Appendix II.)
- The checklist for pumpdown has been done according to LIGO-E2300169-v2
- The dew point at the OMC venting port in the chamber was measured -38.9 deg C, which is surprisingly good (details: https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=79483)
- The pumpdown of the corner has started. With the Double ISP-1000 carts it is pretty quick: the pumpdown started at 16:15, and at 19:07 the pressure is already 167 Torr, which means 1900 l/min (at the Turbo station, as the pressure is already <100, it is well over 2000 l/min, as it should be)
- The pumpdown of HAM7 has also started, it is pretty fast, as it was anticipated, since no doors were opened there. It is already at 1.2E-5 Torr. See details here: https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=79482

As you can see in both pictures, the HAM5-HAM6-HAM7 area is very busy with vacuum equipment, so please only go there if it is absolutely necessary, and if you do, be very careful. Especially the Pfeiffer pumping carts can tip over easily (remember last pumpdown), and in general, Turbo pumps don't like movement - they can easily trip off. Now there are 2 Pfeiffer pumping carts, 3 more small Pfeiffer turbos, a leak checker, a supersucker cart, the new double ISP-1000 mobile pumping station, and the HAM7 turbo up and running.

There were a couple of other issues, too:
- The PT-154 (MKS) gauge, between HAM7 and BSC3 on the FCT, falsely shows 1E-3 Torr. It needs replacement, which will be done after O4
- The instrument air compressor failed to switch on one more time at around 16:45. The aLog about the last failure is here: https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=79443. It was restarted with some struggle, but it seems that the only real solution is to replace it. A Nitrogen and a compressed air bottle was staged in the mechanical room, just in case

Dew point measurement taken this morning read at -43.7 ^oC. No demand on system, HAM5 and HAM6 all doors on.

HAM7 pump down went very smooth, no issues encountered during this process.

Blowdown dew point measurement was at -40.0 ^oC.

Pump down cart, SS500, was set to manage the pump down, now that the turbo pump is on, the cart can manage the selenoid valves in case of an emergency.

Pressure at HAM7 is 1.19X10^-05 Torr.

In response to recent failures of the air compressor that's holding open some of the gate valves:

There's now a 6-hour trend of the air pressure on the lower left corner of the top screen for nuc28.

If the compressor is working, there should be a saw tooth pattern between 60 and 85 PSI, at 90+ minutes per tooth. 

When the pressure drops below 40, the pneumatic gate valves are in danger of closing.

TITLE: 08/06 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
INCOMING OPERATOR: n/a
SHIFT SUMMARY:
Big DARPA tour, Relay Tube was installed, and other prep work for pumping down, and then Les Guthman did some filming work.

LOG:

• 1445 Prep for morning tours (karen, kim, nelly, richard, tony,...)
  • 1512-1518 Turning on light in the H1 PSL room for the tour (rick)
  • 1555 Escorting Eric to Squeezer station for tour (Sam & later spelled out by sheila)
  • 1545 karen/kim/nelly out
• 1535-1640 Acme pest control escort (chris)
• 1545 3IFO checks in lvea (tyler)
  • 1831-1851 3IFO checks at MX & MY (tyler)
• 1545 DARPA tours begin
• 1545-1628 installing shutter for PCal at EY--requiring quick laser hazard transition (Tony, Shingo)
• 1600 OFI TFs on HAM5 (jeff)
• 1619-1704 Filter Cavity End Station cleaning (kim, karen, nelly)
  • 1704-1801 EX cleaning (kim)
  • 1704 MY cleaning---they skipped EY cleaning due to swarming wasps at the EY outhouse (karen, nelly)
• 1632 Relay Tube install/pump prep (travis)
  • 1645 Janos joined (gerardo also went out there at some point)
  • 1729 Meeting with VAC group about test of receptacles (richard)
  • 1737 Janos grabbing items at EX
• 1756-1940 LN2 truck at MX CP6 
  • CP1 (LVEA y-beam manifold, (but didn't catch arrival))
  • CP1 top-off caused aa VAC SDF overfill diff (it should return to green after LN2 overfill stops.
• 1907-1920 lvea checks (betsy)
• 2003-2030 EX computer racks to check overheating dolphin computers. (marc, erik)
  • 2055 Now visiting EY to put a plate on the computer chasis to cool it down (dave, erik)
  • 2145-2211 Visiting EX to put a plate on the computer chasis to cool it down (dave, erik)
• 2014 EPO crew grabbed LEGO ifo from ops station.
• 2011 Pcal measurements (dripta, cervane, tony joining later)
  • 2326-2333 Heading back into the lab
• 2028 FAMIS checks in LVEA & then moved on to arms (chris)
• 2058 Les Guthman film crew scoping out Filter Cavity Tube Enclosure (Oli escorting)
  • 2245-2330 interviewing Betsy in Control Room

EJ, Marc, Jonathan, Erik, Dave:

EJ noted that the Dolphin PCIe cards located at the endstations and read out by h1cdsrfm in the corner are running much warmer than the rest of the Dolphin cards.

h1cdsrfm has 3 Dophin IX cards on its PCI bus, one is plugged into its local PCIe bank, the other two are in Adnaco expansion chassis which are located at the end stations, and are accessed by a 4km single mode fiber link connected to local adnaco adapter cards.

The local Dolhin card, cooled by h1cdsrfm's fans, runs at 62C. The EY card runs at 74C and the EX card runs at 90C. The end station temps are significantly higher than the manufacturer's recommendation.

The Adnaco expansion chassis were inspected at both end stations:

   both chassis side mounted, resting on a shelf

  both chassis have their cover installed

  both chassis fan is operational

The only difference between the two end stations is the PCIe slot the Dolphin card is installed in. At EY (74C) it is in the slot closest to the fan (closest to center of chassis), at EX (90C) it is the slot furthest from the fan (closest to the edge of chassis).

The chassis has one airflow outlet, the fan itself blowing outwards at the rear. There are several airflow inlets: circular hole pattern on top lid, center of the chassis; rectangular hole pattern on top lid, directly above the four PCIe slots; series of slots on left side, from front of chassis to half way back; ditto on right side.

At 14:06 at EY we put a plate on the top lid which blocks the circular hole pattern. Now air comes in from the holes directly above the PCIe slots, and the left and right slot array. This cooled the Dolphin card down from 74C to 65C in 30mins.

At 15:00 at EX we put a plate on the circular pattern, and blocked the right side slot array with tape. Now input air is directly above the PCI connectors and on the left side array only (the one closest to the cards)

Attached trend shows EX and EY temps over this period, started with 90C,74C and now at a steady 76C,66C

We removed the high voltage bypass which turns off the high voltage power to Ham 6, Ham 7, and the ESD drivers to prepare for pumpdown.

Janos, Marc, Richard

The PCAL End Y Shutter stopped working.
I had Fil check it out and he simply replaced the shutter motor with RS Pro part number 238-9709.
Fil showed me how to replace the motors that go bad in them.
I went out to EY to replace the shutter today, ran a few functionality tests for local operation and remote opperation. It is working well.
Closing FRS ticket 31730

Noticed that GV5 had hard closed on Saturday at 19:59 utc time, no human intervention.  It appears as if the instrument air pressure left for soft close GV5 was enough to hard close it.  According to PEM instruments near GV5 made lots of noise, see attached plot, accelerometer and microphone data.

I checked the dial on the gauge to measure the instrument air at GV5, and indeed there was only about 10 psi, this value is within the "usual" pressure used for soft clossing pneumatic gate valves and per procedure.

No interaction is required at this moment from the vacuum team.

Tagging ISC, since sensitive equipment is hanging near this gate valve, like cameras and OpLevs.