We have insulated ~100 feet of beam tube as a trial. In the near future vibration measurements will be taken in order to understand the damping performance of this configuration.
Most recent attempt at this had no effect as I had made the programming changes to the 7000V-5000V step instead of the 5000V-3000V step
So we ran overnight at ETMY with the HEPI position loops on. I've revisited the data I posted yesterday morning. In the attached plot I've added some of last night's overnight running with the loops on. The dashed lines continue to be the off loop reference from ~2AM Tuesday, The solid line references(REF17+) are with the loops on but starting about 8am Tuesday, possibly a noisier time. The current traces are from 2AM Wednesday with the loops on. It is involved but my simple analysis says that some places are worse with it on and at another time it is better suggesting it is just dependent on the ground motion at the time.
When I started the loops yesterday I just put the current positions into the Target. From my log I put in 12 March, the Target positions should zero. The differences are 57 to 30 ums in position, 11urads tilt, and <7 urads Rz. The HEPI Position at ETMY is ready for controlling as needed; so if the current position is not close enough, go for it.
Partway through our accumulation on Y2 - strange CC gauge behavior on one gauge and different rate of rise on another.
We might have a small leak at our RGA setup - will check at the end of accumulation.
The default version of Matlab for the control room is now 2012b, as it was previous to Tuesday's internet outage.
So this actuator got 17+hours of bleeding, hopefully sufficient. Shoot, I forgot to grab the serial number again. Replacement SN is 22952
no restarts reported.
Report from Richard McCarthy:
The site contractor was able to make repairs to the fiber and our
network services and phones have been restored as of ~3 AM today.
Before we understood that the noise we were seeing was the bounce and roll modes of ETMX (see alexa's alog) we were looking around at Osem signals, and we saw that there is a large peak at around 25 Hz in L1_VOLTMON_LR_DQ. It was there before the bounce and roll modes were rung up, an it is there with damping on and damping off.
(Sheila, Alexa)
We have been trying to search for the 1/f feature present at 1-50Hz in the noise plots. We tried a few different things as described below. The noise spectrum under nominal conditions is represented in REF0-2.
All these plots can be found under /ligo/home/sheila.dwyer/ALS/HIFOX/COMM/COMM_COherences_April8.xml. In the set of three, the first reference is always the power spec, the second the RMS, and the third a coherence plot. I have attached an image which shows the nominal power spectrum and the cases with the HEPA fan on and one with the power reduced (NoiseSearchPic1).
For future improvement of the QPD servos, we adjusted the input beam onto the QPDs with the servo off, such that the beam was fairly centered onto both QPDS. We did not get a chance to work on the actual servos though.
We proceeded to do some more noise hunting, but halted the effort when we saw two new spikes in the noise spectra. One spike was at 9.8Hz and the other at 13.8Hz (see second png). Turns out these are the bounce and roll modes of the suspension that must have got rung-up during an ISI trip. (Refer to Sheila's alog).
(Sheila, Chris, Alexa)
Since we had seen a difference in the 1/f noise by decreasing the laser power at EX, we decided to try the following -- we placed an ND filter in the path of the x-arm green transmission on ISCT1. With the ND filter the ALS-C_TRX counts dropped from 0.94 to 0.471. We saw an increase by a factor of 2 in the 1/f noise. We would expect a factor of Sqrt(2). We should repeat this measurement by decreasing the power further. The red line in the attached figure is the nominal spectra; while the light blue line (REF 27) is with the ND filter in place. (Overall these spectra look higher then those posted earlier in the attached alog; it's possible that this change came from alignment drift).
Subsequently, we decided to look into the noise at the COMM BBPD.
Signal from the PD w/ no RF pre-amp @ 100Hz BW:
With PD blocked and no RF pre-amp @ 100Hz BW:
With PD blocked and RF pre-amp (+13dBm) @ 100Hz BW:
Instead of using the beatnote signal from the PD, we used a signal generator (-31dBm at 79.8Mhz) to lock the PLL and measured the in loop noise spectrum (with the input gain of the PLL at -25dB w/ the two compensation filters off). We will posts plots tomorrow. At first glance; however, this does not appear to be the 1/f noise source we have been hunting.
I have attached the plots and data we collected last night. The noise spectrum was measured out of IMON of the PFD with the PLL locked to a signal generator instead of the PD signal. This noise spectra is loop suppressed by the COMM PLL loop. The COMM PLL had an input of -25dB with the two common filters off (unlike the nominal settings). We took a OLTF of this modified loop.
The corner station controller medm was found white boxed. The computer (h1hpipumpctrll0) was running--we did not have to restart it. However, the servo program was not running. It has been down since early Sunday morning. The VFD was running at 31.xx Hz, the last value it received. System was recovered in the best way possible:
Restarted servo--see wiki. This database does not process any records when started and begins in manual mode, so even though its output is zero, it actually hasn't written anything to the VFD at this point and it continues driving the pump motors at 31hz.
The Servo motor speed slider was then rapidly dragged to the nominal running output for 80 psi. You can look in the trends at the CONTROL_VOUT channel, for the LVEA system it is ~1080. If you are dexterous enough, this will only slightly glitch the system if at all. I got lucky and saw no pressure drops.
Once pressure is steady, tweek the output til 80psi. Put 80 psi in the Setpoint (it is zero'd with the database restart.) Engage the Servo (push the On button.)
Following the h1isiham5 model change last friday to correct the BIO card assignments, Jim W reported continuing binary output switching problems with h1isiham5.
We have found the problem.
The drawing D1000298 shows the following:
The IO Chassis has two BIO cards installed. Each card has an input and output connector, each connector pig tails to two cables (lower 32 channels and upper 32 channels).
The CER rack has two binary input chassis installed (one for ham4 and one for ham5) and only one binary output chassis shared between the two chambers (ham4 uses lower 32 channels, ham5 uses upper 32 channels).
So the inputs are straight forward, CARD0 both cables go to the back of ham4 binary input chassis, CARD1 both cables go to the back of the ham5 binary input chassis.
The problem is with the binary outputs. The drawing shows that CARD0 lower 32 cable goes to the lower 32 of the binary output chassis and the upper 32 cable is not used. With CARD1, lower 32 cable goes to the upper 32 of the binary output chassis and the upper 32 cable is also not used.
We found that CARD1 binary output is missing its cable, and both cables from CARD0's output go to the output chassis. In other words ham4 gets two cables and ham5 gets none. The fix is to run a fourth cable to the CARD1 output, and use its lower32 for the output chassis upper 32 connector.
The h1isiham5 model is in agreement with the drawing, so no software change is needed.
(Rich M, Sheila, Alexa)
HEPI and ISI ETMX tripped because we were playing with the suspension watchdogs.
We had a hard time recovering from this trip which caused several more trips, the wd plotting script is not working this afternoon, possibly beause of our network problems so we don't have any plots. Once HEPI was isolated, we tried letting guardian isolate the ISI. It tripped immediately on CPSs (stage1) after we cleared this and tried again it tripped on GS13 (stage2).
We then paused the guardian, moved all the blends to start, and gave the gaurdian another try. (note, the ground motion is fairly quiet today). This tripped again, GS13s.
We then puased the guardian, and tried the command script, stage 1 level 3, tripped on the actuator limit. I then stopped taking notes.
The rough procedure that Rich used to bring it back was damping both stages, turning on 1 DOF at a time RX, RY, Z, RZ, X, Y (stage 1 first, then stage 2). Then Alexa moved the blends over to Tbetter without anything tripping.
We don't know why the things that used to work don't anymore, but now ETMX ISI is harder to untrip and neither the guardian nor the command script seem to be able to do it anymore.
Rich may not have mentioned in the alog, but after our trip the other day while trying to switch blends, he changed the filters from zero crossing to ramp, which should make it easier to switch blends without tripping the ISIs.
One thing about guardian, its not clear to me how we should unpause if we need to pause it for some reason. TO pause it we paused all three, stage 1, stage2 and the manager. Wen unpausing the subpodinates they don't come back in managed mode. I was able to get them back to managed mode by going to init on all three, but in the meantime they were doing their own thing, but happily didn't trip the ISI. What is the best way to do this? Also, maybe the right way to pause and unpause a manager with subordinates is something that needs to be made more obvious to a user.
Alexa also tried pausing, she only paused the manager, bringing that back was not a problem but it doesn't pause the subordinates.
quoth Sheila:
We don't know why the things that used to work don't anymore, but now ETMX ISI is harder to untrip and neither the guardian nor the command script seem to be able to do it anymore.
But you did mention one likely important thing that changed: new blend filters (Tbetter?). What happens if we go back to Tcrappy? Do we have the same problems?
also quoth Sheila:
One thing about guardian, its not clear to me how we should unpause if we need to pause it for some reason. TO pause it we paused all three, stage 1, stage2 and the manager. Wen unpausing the subpodinates they don't come back in managed mode. I was able to get them back to managed mode by going to init on all three, but in the meantime they were doing their own thing, but happily didn't trip the ISI. What is the best way to do this? Also, maybe the right way to pause and unpause a manager with subordinates is something that needs to be made more obvious to a user.
Alexa also tried pausing, she only paused the manager, bringing that back was not a problem but it doesn't pause the subordinates.
This is definitely sounds like something that could be improved. It is true that all nodes, manager and subordinates, need to independenty be paused and unpaused. I'll try to think about how to make that smoother. In the mean time, here's the procedure that should be used:
The manager INIT state resets all the works, restarts them, and puts them into managed mode. That should be the most straightforward way to do things for the moment.
Thanks Jamie.
We did try both the command script and guardian with all the blends on start, which used to work. I don't think we tried turning sensor correction off, which is another change.
I heard that rumors were spread that this alog was written during an earthquake that we hadn't noticed, so there is no reason to worry about our inability to use guardian or the command scripts to reisolate ETMX. That is false, there was no earthquake. PS, this is sheila, still loged in as Stefan.
Sheila, Chris, Alexa, Stefan
This morning we looked for coherences with many signals with our COMM noise. There weren't any surprises, we see the same coherence with the end station PDH control signal around 1kHz that we are used to, ISCT1 accelerometers around 70-100Hz, and a little coherence (-0.7-0.9) with the oplevs at low frequencies (pitch at 0.5Hz, yaw around 0.1). We don't see much coherence at all with the TMS QPDs, just a small bit at low frequencies. We lose coherence with the in loop sensor below around 100Hz.
Chris and I went out to End X to look into the beam quality problems. We didn't find much that was promising. We checked rather carefully for clipping on the way into the chamber again. We also set up the nanoscan again, and we see that there really isn't anything coming back to the WFS when the ITM and the ETM are both misaligned (so it isn't a second beam that is directly reflecting a lens or some optic on transmon.)
We also tried Daniel's suggestion of closing down an iris in the path. For WFS B this only made things worse. WFS A scans will be attached to this alog soon, but we didn't get any improvement there either.
We came back and tried out some of Rich's blends. We were able to get relatively low noise once without using WFS, 30z down to 0.02 Hz. The attached plot shows the COMM noise (measured by the IR) with different blends on. We are curently seeing a lot of noise on ETMX T240s at around 3Hz, (100 times more) Rich is looking into this.
