Some more numbers from Alba, last 3 months

Some time a go, I wrote about numbers that are around this facility. I have say some summary, in the update at the end of the post, about the operation calendar. But this calendar has been changed by the management (unilaterally) after the problems we've faced after eastern until June.

I didn't post anything about those almost 3 months of unexpected shutdown, because I saw what happens as surrealistic.

After the week shutdown of eastern (April 2nd), it was not possible to start up the machine. Some of the collected data from the latest days shows that in March 27th the flow on certain places of the water cooling system went down due to an unknown reason.

Ok, at the beginning all of us thought that this will be a short problem, easy to be fixed in commissioned and well known system. But it wasn't the case. Day after day, the system was not recovered. Shifts of beam time where cancel. During April very shy, beam time cancel by weeks (but working shifts not). First week many people was purging the water cooling system in shifts. Popularly we start calling Alba as the purgatory (as a joke word combination).

Well, days pass and the origin of the problem was not found. In May the beam time of the month was full cancelled, one complete run. Even that, the work shifts wasn't. The workers on shifts gets the notification of the shift cancel by half a week, on average (but the spread makes people notified that the night shift is cancelled when they arrive to the facility).

Finally, past June was found a working point of the cooling system (meta)stable. Many things has been changed and from the information given to all the workers, it looks that is not fully understood what had happen, neither know what or which of the things made has make the system running.

But this was the end of the water cooling system issue, but the beginning of some other collateral problems. One of the things is that some dirt was found in the water circuit. As far as I know previously, this was a circuit of deionized pure water, but looks not compatible with what was found in there.

Another problem faced was a poor materials quality in some hoses. During one of the affected weekends, it has been explained to the workers, one PLC that manages the cooling system was weirdly hanged (no new values measured, but the last read was send with a new time-stamp as if it was really read). Because many equipments where in test, specially magnets, they heat up the water temperature and some hoses vulcanizes. Water leaks started, the circuit gets half empty, pump cavitates, and on Monday morning the problem was faced as burned pumps.

After 2 months trying to realise what was happening, this last was slap. Undermining the moral. But wasn't the latest issue. Even if you cannot believe, a TLD placed in a insertion device (ID) with standard stick to measure the dose in some encoders, fells on the cavity of the ID and when the gap was close it dent the cavity: enough to hit the beam, having to change the section.

By today, the synchrotron is up again and giving beam time for experiments. But the new calendar has magically appeared:

What has special this calendar? Well it sets the unexpected shutdown as "warm up" not as "off" as a shutdown is. Well, yes, it can be think like that. Many people has work very very hard this time, and no one of us was something like "off". (I'm not saying that "off" is no work because many maintenance tasks waits from those "off" periods).

One of (the many) issues of this calendar is that it is trying to recover the lost time telling the workers as "we have to complain with this unexpected problem". But is forgetting the efforts already made, and is asking renewed efforts foreseeing the problem as unpredictable. It wasn't an accident, this was a lack of knowledge! We still don't know what causes the problem, neither what fixes it (or looks like it fixes). There is no proves that this will not happen again.

But how the calendar has changed?

The expected 4992h of machine up time (57% of the year) it reschedules to 4592h (52%, converting the 2.5 months of shutdown in only a 5% of the year time). From the experimentation point of view, the beam time was scheduled to be available for 3600h (41.10%) and now would be 3312h (37.80%) only 3.30% of reduction.

But what has been lost in this unexpected shutdown is 1008h of the scheduled beam time (1176h of the machine up time).

It we lost the 11.50% of the beam time, how can be possible that at the end of the year this drops down to only a 3.30%? Easy, stilling the rest time. Now some of the "off" days and what was originally scheduled as "warm up" is now beam or machine time.

It's easy to realize that the 6 extra weekends that now are planned to work in shifts will not be worked by the rescheduled calendar...

Even all of those things, this morning, planned to have beam for users, the "solved" problem peep again:

No beam...