Displaying reports 1921-1940 of 83272.Go to page Start 93 94 95 96 97 98 99 100 101 End
Reports until 08:42, Monday 07 April 2025
H1 PEM
ryan.crouch@LIGO.ORG - posted 08:42, Monday 07 April 2025 - last comment - 08:13, Tuesday 08 April 2025(83766)
HAM1 dust monitor weekend trend

The counts were pretty low over the weekend, peaking at ~ 30 counts of 0.3s and 10 counts for 0.5s.

Images attached to this report
Comments related to this report
ryan.crouch@LIGO.ORG - 09:21, Monday 07 April 2025 (83770)
Link

The EY dust monitor died with the power outage and has not come back with a process restart, it's having connection issues.

ryan.crouch@LIGO.ORG - 08:13, Tuesday 08 April 2025 (83804)
Link

The EY dust monitor came back overnight.

H1 CDS
david.barker@LIGO.ORG - posted 08:12, Monday 07 April 2025 - last comment - 09:13, Monday 07 April 2025(83765)
Systems which are still down from power outage:

By no means a complete list and in no particular order:

FMCS EPICS

CER Timing Fanout 14th port

Diode room PSL Beckhoff

HEPI pump controller, corner station

Slow controls DEV4 terminal errror

h1cdsrfm long range dolphin

DTS

ncalex

BRS EX, EY

HWS

EX mains mon

 

Comments related to this report
jonathan.hanks@LIGO.ORG - 09:13, Monday 07 April 2025 (83767)
Link 

FMCS-EPICS and the DTS have been recovered
H1 General
oli.patane@LIGO.ORG - posted 07:41, Monday 07 April 2025 (83764)
Ops Day Shift Start

TITLE: 04/07 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Planned Engineering
OUTGOING OPERATOR: None
CURRENT ENVIRONMENT:
    SEI_ENV state: MAINTENANCE
    Wind: 18mph Gusts, 10mph 3min avg
    Primary useism: 0.04 μm/s
    Secondary useism: 0.29 μm/s
QUICK SUMMARY:

The CDS team is currently working on getting the control room screens and other systems back up from the power outage last night.

Today we have a vent coordination meeting at 15:30UTC.

Plans for today:

H1 CDS
david.barker@LIGO.ORG - posted 20:14, Sunday 06 April 2025 - last comment - 14:58, Friday 25 April 2025(83758)
Recovery from power outage

Richard powered up h1seih23 from the front panel, it is now running.

We power cycled h1psl0, it is back.

Erik reconfigured EX Dolphin front ends to communicate with port 8 of the EY Dolphin switch to get them to get the models going. CDSRFM at EX points to port 7 of EY's switch.

All models are running, only IPC errors are long-range-dolphin.

Should someone from operations start activating the SEI and SUS systems?

Images attached to this report
Comments related to this report
jordan.vanosky@LIGO.ORG - 23:09, Sunday 06 April 2025 (83762)
Link

After the power outage, I went out to the site ~10pm Pacific to check status of purge air skid. Kobelco, water pump and drying towers were all running upon arrival. I verified the dew point was still ok (-41.8C measured at HAM1) and verified the drying towers switched normally. Note: Purge air was only connected to HAM1 since doors are off, corner volume is vented but not actively purging with no doors off.

While there I also restarted the controller for 3 of the FCT ion pumps and reset solenoid valve on aux cart connected to GV5 annulus. All looks ok. Left site ~10:40pm.

david.barker@LIGO.ORG - 20:15, Sunday 06 April 2025 (83759)
Link

h1guardian1 did not reboot, here is the guardian MEDM

Images attached to this comment
david.barker@LIGO.ORG - 20:49, Sunday 06 April 2025 (83760)
Link

Systems currently down as of 20:49:

FMCS/BACNET

Corner Station HEPI Pump Controller

PSL Diode Room slow controls

CP1 overfill

NCAL

BRS

HWS

DTS test stand

david.barker@LIGO.ORG - 21:30, Sunday 06 April 2025 (83761)
Link

To continue running vac alarms overnight I have bypassed the currently active alarms for the next 24 hours.

Bypass will expire:
Mon Apr  7 09:28:54 PM PDT 2025
For channel(s):
    H0:FMC-CS_CY_H2O_PUMPSTAT
    H0:FMC-CS_CY_H2O_SUP_DEGF
    H0:FMC-CS_FIRE_PUMP_1
    H0:FMC-CS_FIRE_PUMP_2
    H0:FMC-EX_CY_H2O_PUMPSTAT
    H0:FMC-EX_CY_H2O_SUP_DEGF
    H0:FMC-EY_CY_H2O_PUMPSTAT
    H0:FMC-EY_CY_H2O_SUP_DEGF
    H1:CDS-DTS_ENV_TEMPERATURE_DEGF
    H1:CDS-STAT_NUMBER_OF_ERRORS
 

gerardo.moreno@LIGO.ORG - 14:58, Friday 25 April 2025 (84134)VE
Link

Regarding the 4UHV ion pump controller, the autostart setting is enabled now, thus the unit should power ON after a power glitch.  See some of the instructions to access the general menu.

Images attached to this comment
H1 CDS
david.barker@LIGO.ORG - posted 18:25, Sunday 06 April 2025 - last comment - 19:54, Sunday 06 April 2025(83753)
Power event, most front ends down, Beckhoff slow controls down, DAQ down.

From UPS emails it appears the site has had mutiple power issues starting 18:06 PDT extending to 18:20

Images attached to this report
Comments related to this report
david.barker@LIGO.ORG - 19:36, Sunday 06 April 2025 (83754)
Link

Richard is on his way to the site. Erik and I remotely brought h1susauxb123 (edc) back online, which got the DAQ to restart itself so we are trending vacuum channels again. Data gap for VE data is 18:05 - 18:40.

Current CDS issues:

h1seih23 ipmi not working, computer is still down.

IX Dolphin switch at EX is not working correctly. h1susex, h1seiex, h1iscex are down. Erik is trying a workaround.

h1psl0 is down, it didn't actually power off, we are not sure we should get that going again.

Images attached to this comment
david.barker@LIGO.ORG - 19:41, Sunday 06 April 2025 (83755)
Link

MSR UPS messages:

Name     : ups-msr-0
Location : LHO MSR
Contact  : CDS Administrators
http://ups-msr-0.cds.ligo-wa.caltech.edu
http://10.99.3.10
Serial # : ZA0522010987
Device Ser #: PD0615240264
Date: 04/06/2025
Time: 18:20:42
Code: 0x0002
Critical - System: Warmstart.

 

Name     : ups-msr-0
Location : LHO MSR
Contact  : CDS Administrators
http://ups-msr-0.cds.ligo-wa.caltech.edu
http://10.99.3.10
Serial # : ZA0522010987
Device Ser #: PD0615240264
Date: 04/06/2025
Time: 18:20:50
Code: 0x0101
Informational - UPS: Restored the local network management interface-to-UPS communication.

david.barker@LIGO.ORG - 19:51, Sunday 06 April 2025 (83756)
Link

GC OSB UPS messages (times out by one hour)

Name : lho-gc-osb-ups
Location : LHO OSB Mechanical Room (rm 160)
Contact  : lho-gc-root@ligo-wa....
Serial Number : 5A1315T04169
Device Serial Number : PS1316131451
Date : 04/06/2025
Time : 17:06:10
Code : 0x0235
Critical - UPS: A site wiring fault exists.

Date : 04/06/2025
Time : 17:06:10
Code : 0x020F
Critical - UPS: An input voltage or frequency problem prevents switching to bypass mode.

Date : 04/06/2025
Time : 17:06:26
Code : 0x010A
Informational - UPS: No longer on battery power.

Date : 04/06/2025
Time : 17:06:44
Code : 0x0236
Informational - UPS: A site wiring fault no longer exists.

Date : 04/06/2025
Time : 17:06:44
Code : 0x0210
Informational - UPS: An input voltage or frequency problem no longer prevents switching to bypass mode.

 

david.barker@LIGO.ORG - 19:54, Sunday 06 April 2025 (83757)
Link

GC LSB (one hour out)

Time : 17:18:14
Code : 0x0109
Warning - UPS:  On battery power in response to an input power outage.

Critical - UPS: An out of tolerance input voltage or frequency , prevents switching to bypass mode.

Time : 17:18:33
Code : 0x0210
Informational - UPS: An out of tolerance input voltage or frequency no longer prevents switching to bypass mode

Date : 04/06/2025
Time : 17:18:36
Code : 0x010A
Informational - UPS: No longer on battery power.

 

LHO VE
david.barker@LIGO.ORG - posted 10:37, Sunday 06 April 2025 (83752)
Sun CP1 Fill

Sun Apr 06 10:06:44 2025 INFO: Fill completed in 6min 40secs

 

Images attached to this report
LHO VE
david.barker@LIGO.ORG - posted 10:25, Saturday 05 April 2025 (83750)
Sat CP1 Fill

Sat Apr 05 10:05:42 2025 INFO: Fill completed in 5min 38secs

 

Images attached to this report
H1 CDS
david.barker@LIGO.ORG - posted 09:17, Saturday 05 April 2025 - last comment - 10:36, Saturday 05 April 2025(83747)
h1daqnds1 having issues serving data, please use h1daqnds0 for now while we work on this

Note that the web trends use nds0 and are updating.

Comments related to this report
david.barker@LIGO.ORG - 10:20, Saturday 05 April 2025 (83748)
Link

FRS33784 ticket created for this issue.

david.barker@LIGO.ORG - 10:23, Saturday 05 April 2025 (83749)
Link

Jonathan restarted rts-daqd on h1daqnds1 at 09:37 Sat 05apr2025PDT which cleared the immediate issue.

It appears that nds1 had a connection burst of over 300 connections requests at 23:30 Fri 04apr2025PDT which caused the issue. The daqd process continued to run after that but was no longer accepting any connection requests until the restart this morning.

david.barker@LIGO.ORG - 10:36, Saturday 05 April 2025 (83751)
Link

I rebooted nuc5 at 10:28 to get the Seismic BLRMS FOM running again. I think the other FOMs can wait till Monday.

LHO VE
janos.csizmazia@LIGO.ORG - posted 18:30, Thursday 03 April 2025 - last comment - 01:31, Monday 07 April 2025(83738)
2025 April vent - VAC diary 
Today's activities:

- HAM1's Y+ and Y- doors have been taken off (thanks to Tyler, Randy for the help in the door removal, and to Tony and Corey for breaking the bolts earlier)
- All 5 HAM1/HAM2 viewport covers have been installed
- The 16.5" flange was taken off from the top of HAM1 - this will be the basis of the new vacuum interlock gauge
- A high number of leak checking was done, in detail:
 - Y-manifold: Y+ X+ side flanges (3 pcs.); Turbo flange (1 pc.)
 - BSC8: Y+ X+ corner (3 pcs.); Y- X+ corner (3 pcs.); X+ HEPI bellows (2 pcs.); X+ collar 16.5" flanges (2 pcs.)
 - BSC8-BSC1 spool: Y+ X+ flanges (3 pcs.)
 - BSC1: Y+ X+ side flanges (3 pcs.); X+ collar flanges (2 pcs.); X+ HEPI bellows (2 pcs.); X+ door flanges (5 pcs.); Y- X+ side flanges (3 pcs.)
 - BSC2: Y+ X+ side flanges (3 pcs.)
 - BSC3: Y+ HEPI bellows (2 pcs.); Y+ X- side flanges (3 pcs.); Y+ collar flanges (2 pcs.); Y+ door flanges (5 pcs.); Y+ X+ side flanges (3 pcs.)
 - HAM2-HAM3 spool: vent port (1 pc.); X- flanges (11 pcs.); Y- side flanges (2 pcs.); X+ Y- flanges (5 pcs.)
 - HAM3: Y- door flanges (5 pcs.)
Comments related to this report
gerardo.moreno@LIGO.ORG - 01:31, Monday 07 April 2025 (83763)VE
Link

(Travis, Tyler, Randy, Jordan, Janos, Gerardo)

Something to note during the removal of the -Y door for HAM1, a hair (see attached photo) was found embeded on one of the annulus o-rings, good eye Travis.  Maybe we are due for a contamination control refresh.

Images attached to this comment
Displaying reports 1921-1940 of 83272.Go to page Start 93 94 95 96 97 98 99 100 101 End