RyanS, Jonathan, Dave:

The EPICS IOC cds_aux_ioc.py was crashing when Guardian started writing to its PVs this afternoon. 

This IOC was discovered to not be running after the power outage, its channels were missing from EDC.

I took the opportunity to "do the right thing" and add cds_aux_ioc to cdsioc0's systemd using puppet earlier in the week. Everything looked to be working, for example EDC connected to all 160 channels.

As mentioned, as soon as guardian violin node attempted to write to a PV it crashed the IOC. Systemd then restarted the service 10 minutes later and the cycled repeated.

To further confuse us, attempting to read PVs from the guardian machine failed. If we set the CA_ADDR vars we could get a connection with cdsioc0, but these were not normally set. We then thought that an EPICS gateway must be missing, server on FELAN client on CDSLAN. Jonathan found and started the gateway and we got to the point where Guardian could read the PVs, but crashed the IOC on write.

I could not find any alog/wiki description of where cds_aux_ioc had been running, but on reflection the fact that it is python2 code hinted at an old script machine.

I tried running the code on h1fescript0 and success,  it is not crashing on writes.

For now cds_aux_ioc.py is running in a TMUX session as user controls on h1fescript0

We then undid all the good work we did this week: stopped the gateway, stopped and disabled the systemd service on cdsioc0. I now need to take it out of puppet and delete it from cdsioc0.

TODO: convert script to python3, run on an updated FE script machine.