Usually august is a good time to think on refactorings. During the year normal operation there is never enough time to look back. The sentence "I want all, and I want it now" it's a classical sentence from clients and bosses (many times followed with a "I want it fix, but don't touch any thing").
Also is good because too many managers doesn't care about refactoring. They don't really understand what this means and didn't give to it the real value of this action. For them, often, the maintenance of some code is sort of magic, they don't know how this can be made. Furthermore they don't care, and only think in do it fast an cheap.
In the case of the scope, we are currently using the scope waveforms in a Filling Pattern calculation using the Fast Current Transformer signal. This was the fpFCT project. In this case, we went far away from the original design of the access to the scopes. This original design was already improved long a go, and looks that now it's time to remade it base on the newer requirements.
The current bottleneck is the Visa middlemen. The agent that shows the user the interface of an instrument like an scope doesn't know about protocols. This agent knows the language to talk with the instrument, but not the channel used to have bidirectional communication with it.
From Max IV, there is a proposal to talk directly via socket. This will remove from the scene one of the actors, and for sure will improve.
The second proposal, should be helped by a new feature requested in tango: give the information to the device that someone is listening (or not) over the events on an attribute. This is, avoid to request information to the instrument if noone is listening.
Current idea of the refactoring is to use proposal one, and by commands (until the mentioned feature is available) configure the attributes that are being updated by an isolated thread, optimizing the scope access, in an approach to proposal two.
Update 20130806: Max IV has contributed with their nice idea in a branch.
Showing posts with label synchro. Show all posts
Showing posts with label synchro. Show all posts
2013/07/29
2013/07/09
Some more numbers from Alba, last 3 months
Some time a go, I wrote about numbers that are around this facility. I have say some summary, in the update at the end of the post, about the operation calendar. But this calendar has been changed by the management (unilaterally) after the problems we've faced after eastern until June.
I didn't post anything about those almost 3 months of unexpected shutdown, because I saw what happens as surrealistic.
After the week shutdown of eastern (April 2nd), it was not possible to start up the machine. Some of the collected data from the latest days shows that in March 27th the flow on certain places of the water cooling system went down due to an unknown reason.
Ok, at the beginning all of us thought that this will be a short problem, easy to be fixed in commissioned and well known system. But it wasn't the case. Day after day, the system was not recovered. Shifts of beam time where cancel. During April very shy, beam time cancel by weeks (but working shifts not). First week many people was purging the water cooling system in shifts. Popularly we start calling Alba as the purgatory (as a joke word combination).
Well, days pass and the origin of the problem was not found. In May the beam time of the month was full cancelled, one complete run. Even that, the work shifts wasn't. The workers on shifts gets the notification of the shift cancel by half a week, on average (but the spread makes people notified that the night shift is cancelled when they arrive to the facility).
Finally, past June was found a working point of the cooling system (meta)stable. Many things has been changed and from the information given to all the workers, it looks that is not fully understood what had happen, neither know what or which of the things made has make the system running.
But this was the end of the water cooling system issue, but the beginning of some other collateral problems. One of the things is that some dirt was found in the water circuit. As far as I know previously, this was a circuit of deionized pure water, but looks not compatible with what was found in there.
Another problem faced was a poor materials quality in some hoses. During one of the affected weekends, it has been explained to the workers, one PLC that manages the cooling system was weirdly hanged (no new values measured, but the last read was send with a new time-stamp as if it was really read). Because many equipments where in test, specially magnets, they heat up the water temperature and some hoses vulcanizes. Water leaks started, the circuit gets half empty, pump cavitates, and on Monday morning the problem was faced as burned pumps.
After 2 months trying to realise what was happening, this last was slap. Undermining the moral. But wasn't the latest issue. Even if you cannot believe, a TLD placed in a insertion device (ID) with standard stick to measure the dose in some encoders, fells on the cavity of the ID and when the gap was close it dent the cavity: enough to hit the beam, having to change the section.
By today, the synchrotron is up again and giving beam time for experiments. But the new calendar has magically appeared:
What has special this calendar? Well it sets the unexpected shutdown as "warm up" not as "off" as a shutdown is. Well, yes, it can be think like that. Many people has work very very hard this time, and no one of us was something like "off". (I'm not saying that "off" is no work because many maintenance tasks waits from those "off" periods).
One of (the many) issues of this calendar is that it is trying to recover the lost time telling the workers as "we have to complain with this unexpected problem". But is forgetting the efforts already made, and is asking renewed efforts foreseeing the problem as unpredictable. It wasn't an accident, this was a lack of knowledge! We still don't know what causes the problem, neither what fixes it (or looks like it fixes). There is no proves that this will not happen again.
But how the calendar has changed?
The expected 4992h of machine up time (57% of the year) it reschedules to 4592h (52%, converting the 2.5 months of shutdown in only a 5% of the year time). From the experimentation point of view, the beam time was scheduled to be available for 3600h (41.10%) and now would be 3312h (37.80%) only 3.30% of reduction.
But what has been lost in this unexpected shutdown is 1008h of the scheduled beam time (1176h of the machine up time).
It we lost the 11.50% of the beam time, how can be possible that at the end of the year this drops down to only a 3.30%? Easy, stilling the rest time. Now some of the "off" days and what was originally scheduled as "warm up" is now beam or machine time.
It's easy to realize that the 6 extra weekends that now are planned to work in shifts will not be worked by the rescheduled calendar...
Even all of those things, this morning, planned to have beam for users, the "solved" problem peep again:
I didn't post anything about those almost 3 months of unexpected shutdown, because I saw what happens as surrealistic.
After the week shutdown of eastern (April 2nd), it was not possible to start up the machine. Some of the collected data from the latest days shows that in March 27th the flow on certain places of the water cooling system went down due to an unknown reason.
Finally, past June was found a working point of the cooling system (meta)stable. Many things has been changed and from the information given to all the workers, it looks that is not fully understood what had happen, neither know what or which of the things made has make the system running.
But this was the end of the water cooling system issue, but the beginning of some other collateral problems. One of the things is that some dirt was found in the water circuit. As far as I know previously, this was a circuit of deionized pure water, but looks not compatible with what was found in there.Another problem faced was a poor materials quality in some hoses. During one of the affected weekends, it has been explained to the workers, one PLC that manages the cooling system was weirdly hanged (no new values measured, but the last read was send with a new time-stamp as if it was really read). Because many equipments where in test, specially magnets, they heat up the water temperature and some hoses vulcanizes. Water leaks started, the circuit gets half empty, pump cavitates, and on Monday morning the problem was faced as burned pumps.
After 2 months trying to realise what was happening, this last was slap. Undermining the moral. But wasn't the latest issue. Even if you cannot believe, a TLD placed in a insertion device (ID) with standard stick to measure the dose in some encoders, fells on the cavity of the ID and when the gap was close it dent the cavity: enough to hit the beam, having to change the section.
By today, the synchrotron is up again and giving beam time for experiments. But the new calendar has magically appeared:
One of (the many) issues of this calendar is that it is trying to recover the lost time telling the workers as "we have to complain with this unexpected problem". But is forgetting the efforts already made, and is asking renewed efforts foreseeing the problem as unpredictable. It wasn't an accident, this was a lack of knowledge! We still don't know what causes the problem, neither what fixes it (or looks like it fixes). There is no proves that this will not happen again.
But how the calendar has changed?
The expected 4992h of machine up time (57% of the year) it reschedules to 4592h (52%, converting the 2.5 months of shutdown in only a 5% of the year time). From the experimentation point of view, the beam time was scheduled to be available for 3600h (41.10%) and now would be 3312h (37.80%) only 3.30% of reduction.
But what has been lost in this unexpected shutdown is 1008h of the scheduled beam time (1176h of the machine up time).
It we lost the 11.50% of the beam time, how can be possible that at the end of the year this drops down to only a 3.30%? Easy, stilling the rest time. Now some of the "off" days and what was originally scheduled as "warm up" is now beam or machine time.
It's easy to realize that the 6 extra weekends that now are planned to work in shifts will not be worked by the rescheduled calendar...
Even all of those things, this morning, planned to have beam for users, the "solved" problem peep again:
No beam...
(more) frustraited @ work
Alba is broken like since eastern. When we come back from that shutdown at the end of March. Water cooling problems, the water flow was unstable; sponges were found in the "clean water" cooling pipes, together with other dust; a weekend instrument hang causes a temperature increase and bad quality of hoses become vulcanized (below the specs) and caused a disaster in one of the two main power supplies of the booster; a TLD wrong placed in a weird space and when an Insertion Device was moved, it bends the beam pipe blocking the beam orbit, requiring to vent and replace a section. It's like we stomp shit.
This is one of the issues in this accelerator complex, but there are others that may interact.
Panic reaches the management at the beginning of this years when 12 people announces their new positions in other institutes. If you take into account that we are 145 workers, you can realizes the magnitude of the problem. From the people that says goodbye, 8 were from the computing division (around 50) and from those 4 were from the controls section (of 14). This panic didn't mention the 1 or 2 per month that has announced that the left during the past two years.
Management has announced 3 months later the creation of a commission from work behaviour improvements. Now, in July, this commission has made the first meeting by last week.
As I mention in the previous post, another reason of personal sadness: my bosses didn't case about a worker that has been called to do a talk in the university.
I didn't write here since then. I didn't found any reason to write about software design here.
But last reason to get one step further in frustration went last week.
Starting 2 months a go, in the tango meeting, there was a request to the tango community to introduce security embedded inside the tango implementation. From the last 4 years I've been trying to start a PhD in cryptology (but too busy due to the work at Alba), and I've been getting closer and closer to the field of the RFIDs and the smartcards. During the presentation where this was requested, I thought that many of the schemas to ensure RFID can be also valid to be applied in between the communication of the agents in a distributed system, like tango.
I thought it can be interesting for Alba and the tango community, to exploit that one of the workers has a hobby in cryptology and security. In this terms I've talked to my boss.
For that, I need 3 weeks to talk to my boss (this is already one problem). During this time, I've been able to, out side working hours, explain this idea to the research group in the university (they are in another city, 150 km away). I found a great acceptance about "Ensuring Tango control system", even more I've seen enthusiasm about the idea of a PhD based on an application of all the cryptological work made by the whole group (many things about public key -specially elliptic curves- symmetric cyphers, stream ciphers, secret sharing, homomorphic encryption and field like that).
At work, the response wasn't that good. The answer was like: "do what ever you like in your free time, but this must not affect your current duties at all. It's not possible to dedicate any of your working hours in such a thing". Clearly have said, if I do a PhD, is not meaning anything for Alba at all. Ooh!! This is a very clear way to motivate a worker. For my partners, also fun when was said "if you do, others would ask to do this", what's the problem on workers training?
Went I talked to my boss, I said many time to him and have to point here, that my duties have been increase every time that a partner left. Last Friday, there was a presentation about the Alba's controls system, and when the subsystems of the machine was listed, half of the elements on that list are on my behalf (and not all my duties were there).
When I went to talk about this with my boss, I went there offering my free time to work in a industrial PhD. My proposal was not to stop doing my job and disappear for full time dedication to this, my proposal was mostly the free time, but being realistic that the brain thinks at any time. As a collaboration between the university and the industry, all the involved has to put something, specially the PhD candidate.
Well done, with this deal, the simpler solution is that Alba will not appear at all in my PhD. Or if it appears will be to be mention explicitly that they haven't contribute at all; even worst Alba's position was opposition to doing a work like that.
This is one of the issues in this accelerator complex, but there are others that may interact.
Panic reaches the management at the beginning of this years when 12 people announces their new positions in other institutes. If you take into account that we are 145 workers, you can realizes the magnitude of the problem. From the people that says goodbye, 8 were from the computing division (around 50) and from those 4 were from the controls section (of 14). This panic didn't mention the 1 or 2 per month that has announced that the left during the past two years.
Management has announced 3 months later the creation of a commission from work behaviour improvements. Now, in July, this commission has made the first meeting by last week.
As I mention in the previous post, another reason of personal sadness: my bosses didn't case about a worker that has been called to do a talk in the university.
I didn't write here since then. I didn't found any reason to write about software design here.
But last reason to get one step further in frustration went last week.
Starting 2 months a go, in the tango meeting, there was a request to the tango community to introduce security embedded inside the tango implementation. From the last 4 years I've been trying to start a PhD in cryptology (but too busy due to the work at Alba), and I've been getting closer and closer to the field of the RFIDs and the smartcards. During the presentation where this was requested, I thought that many of the schemas to ensure RFID can be also valid to be applied in between the communication of the agents in a distributed system, like tango.
I thought it can be interesting for Alba and the tango community, to exploit that one of the workers has a hobby in cryptology and security. In this terms I've talked to my boss.
For that, I need 3 weeks to talk to my boss (this is already one problem). During this time, I've been able to, out side working hours, explain this idea to the research group in the university (they are in another city, 150 km away). I found a great acceptance about "Ensuring Tango control system", even more I've seen enthusiasm about the idea of a PhD based on an application of all the cryptological work made by the whole group (many things about public key -specially elliptic curves- symmetric cyphers, stream ciphers, secret sharing, homomorphic encryption and field like that).
At work, the response wasn't that good. The answer was like: "do what ever you like in your free time, but this must not affect your current duties at all. It's not possible to dedicate any of your working hours in such a thing". Clearly have said, if I do a PhD, is not meaning anything for Alba at all. Ooh!! This is a very clear way to motivate a worker. For my partners, also fun when was said "if you do, others would ask to do this", what's the problem on workers training?
Went I talked to my boss, I said many time to him and have to point here, that my duties have been increase every time that a partner left. Last Friday, there was a presentation about the Alba's controls system, and when the subsystems of the machine was listed, half of the elements on that list are on my behalf (and not all my duties were there).
When I went to talk about this with my boss, I went there offering my free time to work in a industrial PhD. My proposal was not to stop doing my job and disappear for full time dedication to this, my proposal was mostly the free time, but being realistic that the brain thinks at any time. As a collaboration between the university and the industry, all the involved has to put something, specially the PhD candidate.
Well done, with this deal, the simpler solution is that Alba will not appear at all in my PhD. Or if it appears will be to be mention explicitly that they haven't contribute at all; even worst Alba's position was opposition to doing a work like that.
2013/04/30
untitled
Yesterday I gave a talk in the college where I studied the Bachelor and the Master degrees and where I am now starting the PhD. What stuck me most is the lack of interest from my bosses in this informative talk. After tell them about the invitation received from the university the answer wasn't further that "oh, that's nice".
But what ever. I take this opportunity to present to the students how an engineer can work in a place that often is not in the main path of jobs that one have.
Starting with how I got this job, and explaining that 6 months before start working here I didn't know what a synchrotron was, I have explained the heterogeneous different jobs I had in the past. I've put the accent in the English, because it has been told to me that many students didn't evaluate as necessary to have a good English level. Well, is not a thing to have a absolutely very nice level, but starting with a communicative sufficient level (and with practice the rest will come).
I have explained to them, with some youtube videos, has is not a synchrotron. Why start the other way around? The idea behind this was more a way to list other kind of scientific facilities where an engineer have job opportunities and sometimes are not in the main target because is not known.
Two basic ideas can summarize my talk: internationalization and search for jobs in more unexpected fields. And an important thing in the job search it to apply to those offers. Without this step it's impossible to get it!
But what ever. I take this opportunity to present to the students how an engineer can work in a place that often is not in the main path of jobs that one have.
Starting with how I got this job, and explaining that 6 months before start working here I didn't know what a synchrotron was, I have explained the heterogeneous different jobs I had in the past. I've put the accent in the English, because it has been told to me that many students didn't evaluate as necessary to have a good English level. Well, is not a thing to have a absolutely very nice level, but starting with a communicative sufficient level (and with practice the rest will come).
I have explained to them, with some youtube videos, has is not a synchrotron. Why start the other way around? The idea behind this was more a way to list other kind of scientific facilities where an engineer have job opportunities and sometimes are not in the main target because is not known.
Two basic ideas can summarize my talk: internationalization and search for jobs in more unexpected fields. And an important thing in the job search it to apply to those offers. Without this step it's impossible to get it!
2013/02/27
Some numbers from Alba
Yesterday, I've placed some numbers in the post and I think there are other important numbers on this facility.
From the operation calendar, some numbers can be extracted. The synchrotron will be running 5184h (60% of the year time).
Well this is more the number of hours that can be running, because the 27% of this time (1416h, 16% of the year) are machine studies and accelerator test. This means that during this time is not necessary that we have beam in the storage.
For the users, it's plan that we will deliver 3768 beam hour. This represent the 72% of the available running time (or the 43% of the year time).
The rest of the year is dedicated to maintenance and improvements (984h, 11% of the total) and warm-up (2432h, 27.7%). There are special time slots dedicates to the Personal Safety System (64h, 0.7%) and the CSN (spanish acronym of the Nuclear Security Council).
This give a total amount of 8760h (365 days).
More numbers?
The storage ring has a perimeter of about 270m, designed to have an energy of 3GeV and the accumulated current at 400mA. In this circumference there are:
More numbers? There are many more fields where numbers can be written here. This may starts a series of articles.
Update 20130228: Summary:
From the operation calendar, some numbers can be extracted. The synchrotron will be running 5184h (60% of the year time).
Well this is more the number of hours that can be running, because the 27% of this time (1416h, 16% of the year) are machine studies and accelerator test. This means that during this time is not necessary that we have beam in the storage.
For the users, it's plan that we will deliver 3768 beam hour. This represent the 72% of the available running time (or the 43% of the year time).
The rest of the year is dedicated to maintenance and improvements (984h, 11% of the total) and warm-up (2432h, 27.7%). There are special time slots dedicates to the Personal Safety System (64h, 0.7%) and the CSN (spanish acronym of the Nuclear Security Council).
This give a total amount of 8760h (365 days).
More numbers?
The storage ring has a perimeter of about 270m, designed to have an energy of 3GeV and the accumulated current at 400mA. In this circumference there are:
- 32 bending magnets that generates a magnetic field up to 1.42T each and a magnetic gradient (G) of 5.5T/m.
- 112 quadrupoles
- 120 sextupoles with B=1.12T
- Those sextupoles have additional coils to apply corrector dipole field.
- 40 bending magnets with a B=0.89 T, and G=2.2T/m
- 60 quadrupoles
- 16 sextupoles
- 72 correctors
More numbers? There are many more fields where numbers can be written here. This may starts a series of articles.
Update 20130228: Summary:
| Off | 984h | 11% | |||
| Warm-up | 2432h | 27.3% | |||
| Beamtime | 5184h | 60% | Machine | 1416h | 27% (16% of the year) |
| Beamlines | 3768h | 72% (43% of the year) | |||
| CSN | 96h | 1% | |||
| PSS | 64h | 0.7% | |||
| Total | 8760h |
2013/02/26
Why some of our cameras die
From the 54 cameras that we have in side the tunnel, there is one location that is registering more failures than the other positions. Years a go, to find the cameras on their places (and the power source for all of them) I did by hand a map:
You have to have on mind that, following the current information in our cabling data base, we have 18767 cables (with a total length of 168km, on average 11.33m), with 6683 equipments of 728 different types, located in 374 racks.
But back to the reason of this post, there is one ccd camera location, where the hardware dies too often. The technical service from the manufacturer, Basler, have answered to us, something like "what the hell did you did to those cameras!?". They didn't saw this level of damages before.
It is on the top of the map image, but a detail will de useful:
The circles indicates locations of cameras. In red the "sr16/di/fs-01" that to often dies, in blue other cameras. The one near is the "bt/di/fsotr-03" who is not failing (at least not by now). A picture of the place is:
On this picture, the camera is the black box (with white letters saying Scout) on top with a grey network cable on the top.
It's a very important camera, due to the long list of uses it has, specially because when an injected beam to the storage ring is seen here means that one turn is completed (no obstacles in the path).
But why it dies? We know that we are placing this equipments in the tunnel and they can suffer mal functions due to the radiation, but why this is the only place where this cameras die? It looks that we found the reason.
The first hypothesis was: synchrotron light from the transferline. This camera is almost aligned with the second bending magnet from the booster transferline. But is not convincing us.
Basically this camera is 40cm up to the beam pipe. Like, in radians from the aperture, the blue circle at the exit of this bending, where a synchrotron radiation is placed and is not failing.
A secondary hypothesis is neutron scattering. But this is neither easy because the last camera of the booster transferline is also there. Its blue dot is near. The only difference is that the one in the transferline is down the beam pipe and perhaps those particles could have more obstacles in the middle. Maybe yes, maybe not.
Lest look on the storage injection in more detail. Remember the drawing with the coloured circles, there where two boxes besides the red dot, after a bigger box and followed by two more boxes from the first size. The four with the same size are kicker magnets, and the bigger one represents the septum magnet.
Those are pulsed magnets that when there is no injection they don't act, but when there is a beam injection the kickers divert the beam trajectory in four squares and the septum is placing the incoming (blue) and the stored (red) beams very very close:
A very schematic view of what is happening in the septum can be (yes, it can be draw by a kid):
The green line in between both beams is a metallic shielding to isolate the magnetic field of the septum for the booster incoming beam (blue), that the stored beam (red) should not see this field.
The current hypothesis now is that we maybe heating this shielding, scattering back "something" (probably neutrons) and they find the camera in the path. The last camera of the booster transferline is not in the same plane than this shielding.
2013/02/14
fpFCT: bugfixing
After the development of this study of the filling pattern using the fast current transformer it was time to search for bugs. From a result like:
This calculation doesn't have good sense and looks like there is a bug. This only 2 groups doesn't look nice. Even worst when we execute the code under an scripting way, the result was different. And we found the error: the number of giga samples used in the calculation. We've seen a 10th factor and we found that using the scripting mode, it reads the attribute from the scope, but from the device server it uses a default initial value of 4e10.
We've been lucky that the default value wasn't the same than what was expected, or we would get crazy if some day it changes. For example if the scope samples at 20GS/s instead of 40.
After last bug search we can see results like:
There are things to review, we didn't finish completely this development. But what is true is that the scientist already have usable data for their studies.
This calculation doesn't have good sense and looks like there is a bug. This only 2 groups doesn't look nice. Even worst when we execute the code under an scripting way, the result was different. And we found the error: the number of giga samples used in the calculation. We've seen a 10th factor and we found that using the scripting mode, it reads the attribute from the scope, but from the device server it uses a default initial value of 4e10.
We've been lucky that the default value wasn't the same than what was expected, or we would get crazy if some day it changes. For example if the scope samples at 20GS/s instead of 40.
After last bug search we can see results like:
There are things to review, we didn't finish completely this development. But what is true is that the scientist already have usable data for their studies.
2013/01/25
fpFCT: with beam
A month a go I have work in simulate the beam of the storage ring by its bunches in the Fast Current Transformer (FCT). Well now we are in a run of 3 weeks (the operation calendar is available on the web site), and we can see the signal in the scope from the FCT and apply the algorithm developed in accelerators using the device server developer by the controls group.
To see those inputs and those outputs we need our Tango Graphical User Interface (GUI) called Taurus. And with a "simple" command line like:
$ taurusform sr/di/FillingPatternFCT-01/{nAcquisitions,StartingPoint,Threshold,ScaleH,OffsetH,TimingTrigger,resultingFrequency,FilledBunches,SpuriousBunches} sr02/di/sco-01/{Channel1,CurrentSampleRate,ScaleH,OffsetH} sr/di/FillingPatternFCT-01/{BunchIntensity,State,Status} &
This command line, looking so complicated is only listing all the attributes (and their sorting) to be set in a taurusform, by using a feature of bash to write the (almost) minimum.
For example, first parameter is: sr/di/FillingPatternFCT-01/{nAcquisitions,StartingPoint,Threshold,ScaleH,OffsetH,TimingTrigger,resultingFrequency,FilledBunches,SpuriousBunches} An this means to get from the device server sr/di/FillingPatternFCT-01, the attributes listed.
That command will open a small window like:
There are two button in the attribute space because they are 1D data (spectrum in tango), Clicking them a taurusplot will be launched for each. One for the FCT signal:
And another for the Bunch Intensity calculated:
With Taurus, it's so easy to have a little simple gui to start with. Also the General Taurus Gui is neither so difficult.
To see those inputs and those outputs we need our Tango Graphical User Interface (GUI) called Taurus. And with a "simple" command line like:
$ taurusform sr/di/FillingPatternFCT-01/{nAcquisitions,StartingPoint,Threshold,ScaleH,OffsetH,TimingTrigger,resultingFrequency,FilledBunches,SpuriousBunches} sr02/di/sco-01/{Channel1,CurrentSampleRate,ScaleH,OffsetH} sr/di/FillingPatternFCT-01/{BunchIntensity,State,Status} &
This command line, looking so complicated is only listing all the attributes (and their sorting) to be set in a taurusform, by using a feature of bash to write the (almost) minimum.
For example, first parameter is: sr/di/FillingPatternFCT-01/{nAcquisitions,StartingPoint,Threshold,ScaleH,OffsetH,TimingTrigger,resultingFrequency,FilledBunches,SpuriousBunches} An this means to get from the device server sr/di/FillingPatternFCT-01, the attributes listed.
That command will open a small window like:
With Taurus, it's so easy to have a little simple gui to start with. Also the General Taurus Gui is neither so difficult.
2013/01/18
Brain drain
After many years working at Alba this January 2013 has exposed one of its biggest weakness: a brain drain. There is a general talk in the corridors of this excellence Scientific&Technician Singular installation, the number of people that is renouncing to the job to go to a better place.
In a place that work about 140 people, between the confirm number of people that say good bye and the rumours, there are between 9 and 14 people. And this is only this month! That is a coincidence, because the search for a new job is not a magical thing that happens fast. This people has been searching for a period of time and I can bet to say that others are on the way.
The trickle of people that renounce has been set almost constant this last years. Almost every month there where one or two that did a farewell celebration. In the past, when someone left, there where a diner with some of the coworkers. It was something special, it happens sporadically, but this last years this have become monotony, repetitive, a routine... Further than that, there where overlapped farewells.
I hope there will be someone thinking on the reasons of this and thinking in solutions for that... This is not the normal movement between research institutes. Many alba's workers have left to go work to the ITER, fusion4energy, cern, esrf, maxlab, desy, and others, but scientist, engineers or technicians from those places are not coming here!
Update 20130124: By today, the list of 9 people has been increased up to 11, and if the rumours are taken in the count the 14 raises up to 16.
In a place that work about 140 people, between the confirm number of people that say good bye and the rumours, there are between 9 and 14 people. And this is only this month! That is a coincidence, because the search for a new job is not a magical thing that happens fast. This people has been searching for a period of time and I can bet to say that others are on the way.
The trickle of people that renounce has been set almost constant this last years. Almost every month there where one or two that did a farewell celebration. In the past, when someone left, there where a diner with some of the coworkers. It was something special, it happens sporadically, but this last years this have become monotony, repetitive, a routine... Further than that, there where overlapped farewells.
I hope there will be someone thinking on the reasons of this and thinking in solutions for that... This is not the normal movement between research institutes. Many alba's workers have left to go work to the ITER, fusion4energy, cern, esrf, maxlab, desy, and others, but scientist, engineers or technicians from those places are not coming here!
Update 20130124: By today, the list of 9 people has been increased up to 11, and if the rumours are taken in the count the 14 raises up to 16.
2012/12/31
To start 2013
As I've recently said, many projects are open. But the job isn't the only source of projects. It's more the bottleneck of my projects. Often is this job requires too many extra-hours under an uncompensable way: the time dedicated as a workers representative.
It looks far from now, but it's not that far, when we've been having enormous problems in this synchrotron. I remember when the corridors were not save, and at any time you can cross someone that screams you in there. As an example: accusing you of sabotage and in the same conversation telling you that you aren't skilful to do whatever you are being accused, but unknowing about what the hell you are being accused. There were a time that this kind of things have happen once at a month or two.
I remember when a Director interrupt in a workers assembly, screaming and saying that workers doesn't have do decide anything. When two days later, a worker has been fired (technically invited to left the company) being a workers advice. In my opinion this has been the trigger (or at least one of the triggers) of what have happen later on.
Event that the situation scares too much, some of the workers have started to self-organise to change this. We where clear that the things on that institution must change radically. Yes, in my opinion, was the old director who generates the situation of its fall (sorry: ascension to direct a Basque spallation source)
Even that, not all the problems have left this excellence installation. Still too far from being a normal laboratory, there are problems that persist. Hindrances of the past that are a burden in the institution, playing the same roles than with the old director.
We have change the direction but still not the heading. I have hope over 2013 that this institution will follow a good heading.
It looks far from now, but it's not that far, when we've been having enormous problems in this synchrotron. I remember when the corridors were not save, and at any time you can cross someone that screams you in there. As an example: accusing you of sabotage and in the same conversation telling you that you aren't skilful to do whatever you are being accused, but unknowing about what the hell you are being accused. There were a time that this kind of things have happen once at a month or two.
I remember when a Director interrupt in a workers assembly, screaming and saying that workers doesn't have do decide anything. When two days later, a worker has been fired (technically invited to left the company) being a workers advice. In my opinion this has been the trigger (or at least one of the triggers) of what have happen later on.
Event that the situation scares too much, some of the workers have started to self-organise to change this. We where clear that the things on that institution must change radically. Yes, in my opinion, was the old director who generates the situation of its fall (sorry: ascension to direct a Basque spallation source)
Even that, not all the problems have left this excellence installation. Still too far from being a normal laboratory, there are problems that persist. Hindrances of the past that are a burden in the institution, playing the same roles than with the old director.
We have change the direction but still not the heading. I have hope over 2013 that this institution will follow a good heading.
2012/12/20
fpFCT: simulating the bunch current
fpFCT is an smaller name of the Filling Pattern from Fast Current Transformer. As I've said I didn't know how to simulate the signal that is in coming from the scope, and I found a solution.
This is a reused image of the scope signal from the storage ring FCT and what I found how to simulate using PySignalSimulator is:
There are only 4kpoints instead of the 40k that the scope is giving but it looks quite similar.
But how this has been set up in the PySignalSimulator properties? The initial idea was the amplitude modulation but with this what is represented looks like:
But what if the carrier wave, better than a sinusoid looks like a quadratic waveform. With this I have started to play with harmonics. The formula that likes more to be a carrier has been:
In red is plot the carrier over the modulated signal in blue. The signal that is being modulated has been omitted because is almost a sinusoid.
The formula of the carries is:
With parameters values:
The formula of the modulated signal is:
With a_2 = 1; w_2=0.065. The x_2 is a value that is changing from time to time to introduce a drift in the signal in order to have a "dancing" bunches like in the scope.
Lets see the result:
This is a reused image of the scope signal from the storage ring FCT and what I found how to simulate using PySignalSimulator is:
There are only 4kpoints instead of the 40k that the scope is giving but it looks quite similar.
But how this has been set up in the PySignalSimulator properties? The initial idea was the amplitude modulation but with this what is represented looks like:
But what if the carrier wave, better than a sinusoid looks like a quadratic waveform. With this I have started to play with harmonics. The formula that likes more to be a carrier has been:
In red is plot the carrier over the modulated signal in blue. The signal that is being modulated has been omitted because is almost a sinusoid.
The formula of the carries is:
Lets see the result:
2012/12/18
Fast Orbit FeedBack
One of the necessary improvements of Alba is the top up system. With this, the beam accumulated in the storage ring can be restored under a continuous re-injection precisely on the buckets here it bunch is losing charge. Recovering an old snapshot of the machine status:
This is an example where the stored beam had decay from 60mA to a bit more than 30 in around 6 hours (this was a year a go, by now we are having more current and a longer beam live time). This lost of current requires re-injections like what is in the middle of the red plot.
This re-injection, afaik, produces perturbations on the beam orbit that at the end means drifts in the x-ray beam in the beamlines. This re-injection, due to radiation safety reason have to be done with the frontends closed. That is with the beamlines in a standby, without light and waiting that this re-injection finishes. This re-injection causes an interruption on the experiments, and the longest experiment you can do is the one that can be performed between re-injections. You can imagine the consequences if the most insignificant thing fails in the middle of an experiment.under this conditions.
Well, to make this re-injection possible with the frontends open, the orbit of the beam must be very static. The beam has to be thine characterised and its oscillations below some microns.
How the orbit is corrected? In the sextupoles, there are the coils for each of the poles, but it is also another pair of thinner coils that sets the corrector component. I imagine it like a tiny little bending (they have like 3 orders less of magnitude in terms of current).
Those correctors can change its current very fast for a small modifications. That can be, in the order of the hundreds of miliamps can change it with a 10kHz frequency.
With this, together with a beam position monitoring at this speed, and if you can process all this data, the orbit perturbations can be controlled good enough to allow to re-inject with the necessary safeness to implement the top up system and have the plot of the current as an straight horizontal line (negligible ripple).
Main issue: with in a ring of around 400m of perimeter share between 88 beam position monitors all their sets of data, plus the sniffers we have to take this data to calculate those corrections, and stablish the new setpoints to another 88 corrector power supplies. All this at 10kHz: read, calculate, write: simple operations.
Filling Pattern From Fast Current Transformer
As I explain the a previous post we have a Fast Current Transformer in the storage ring and we are able to see the signal in an oscilloscope. Many years a go I have started a project to have the features of those scopes we have in the control system.
In the next video, the oscilloscope signal has been simulated using this simulator. You will see it as the top-right blue plot with an amplitude that oscillates between 0 and a maximum. The variables of the formula that simulates this are below this plot, on the left side.
On the left side of the screen some data are revealed about the cpu, memory, network use and so one. After send the Start() command the image below the plot (on its middle) start to collect each one of the waveforms from above. And already with non complete buffer the data is being processed. On the right of this cyclic buffer image representation, there are the attributes to adjust the algorithm and then the output. The green plot on the top is the filtered output.
It has been started as a Tango Device server in C++ many years a go, but from a refactoring it have changed to be a python device using PyTango. Also this new release have suffered many changes, specially the one that expands this server to more classes to be able to control, not only scopes, but also Spectrum analysers, Signal generators, and later on Radio frequency generators.
But this device have been more used in this "extensions" than in the original scope class. The Signal generator has been used in the booster tune excitation together with the spectrum analyser in the booster tune measurement (I didn't have time to write about those projects here, I hope soon). This devices works fine and the quality was good enough to change the storage tune measurements to be like the booster ones.
Due to Linac's needs to archive some scope measurements, the scope tango class was improved giving as an output here three posts. This was the real trigger that have launched this device server class.
A few weeks a go, more requests have been receiver about scope acquisition: It has been requested to have available the signal from an FCT in the control system, Also this required to improve the acquisition of the waveform from the scope and after some tries help has been requested to the manufacturer. With this I have reduced the network transition of 40kpoints as 300kB to 80kB or 40kB depending on the float precision. This allows to start polling the scope waveform at 3Hz.
This have boosted the project of the Filling Pattern studies using this FCT signal and a device server is available to perform this calculation. Just now that there is no beam, there is no signal in the FCT, only electronic noise like:
)Instead of the usual signal we have when there is beam:
Like often happens I had to ask what this mean and I had try to explain. But, to test the development of the FCT signal analysis, it's necessary to simulate a bit it. No necessary to have the same signal but I should complain that have to change from time to time to have something alive to be analysed.
The solution is to use a simulator of signals, and the best way is using the incredibly versatile PySignalSimulator device server. Creating a device with one property "DynamicAttributes" with something like:
A1=float(READ and VAR('A1') of WRITE and VAR('A1,VALUE)This sets an scalar attribute with a float value that than be read or write. Combining this with an attribute that contains an array of elements:
Channel1=[ (sin(VAR('W1')*x)*sin(VAR('W2')*x)) for x in range(4096)]With those simple lines you can have an spectrum value with a amplitude modulated signal. It's not the same than the beam signal, but it can acts to show how this bunch analyser works.
In the next video, the oscilloscope signal has been simulated using this simulator. You will see it as the top-right blue plot with an amplitude that oscillates between 0 and a maximum. The variables of the formula that simulates this are below this plot, on the left side.
On the left side of the screen some data are revealed about the cpu, memory, network use and so one. After send the Start() command the image below the plot (on its middle) start to collect each one of the waveforms from above. And already with non complete buffer the data is being processed. On the right of this cyclic buffer image representation, there are the attributes to adjust the algorithm and then the output. The green plot on the top is the filtered output.
Projects cross check
Today, we have a blackout in the Alba's synchrotron. From 8h to 15h we have received a recommendation from the director to avoid to go to the facility before 15h. Some intermediate bosses have tell their workers to not come and what ever they want, others say to work from home, and some of them say they don't expect that us comes after 15h. Ok, this is like an almost free day, as usual in this facility, not using clear works and giving different orders to different workers.
Well, when they do this confusion in diffusion (don't confuse with Shannon's principle) I remember Sun Tzu words: “If words of command are not clear and distinct, if orders are not thoroughly understood, then the general is to blame. But, if orders are clear and the soldiers nevertheless disobey, then it is the fault of their officers.”
I'll dedicate the day to a projects cross check to have them reviewed, and I'll share them using this blog channel.
Update: Only the Filling Pattern FCT and something about the FOFB, has been possible to write by now. The new plugin to acquire with the Basler Ace Coloured cameras (by now we are using monochrome Basler Scout models in many of the beam diagnostics stations) and the refactoring of the graphical user interface of the imaging acquisitions, are also very important an big project that we need to proceed with. Also this imaging tools must be integrated with in sardana. With out forget the StateCodeInterpreter project, very necessary to trace back what had happen with a power supply. Another day it will be explained.
And the things wouldn't finish where, there are also projects that have a title but not yet received the Start() command... (like the linac's control & gui).
Well, when they do this confusion in diffusion (don't confuse with Shannon's principle) I remember Sun Tzu words: “If words of command are not clear and distinct, if orders are not thoroughly understood, then the general is to blame. But, if orders are clear and the soldiers nevertheless disobey, then it is the fault of their officers.”
I'll dedicate the day to a projects cross check to have them reviewed, and I'll share them using this blog channel.
Update: Only the Filling Pattern FCT and something about the FOFB, has been possible to write by now. The new plugin to acquire with the Basler Ace Coloured cameras (by now we are using monochrome Basler Scout models in many of the beam diagnostics stations) and the refactoring of the graphical user interface of the imaging acquisitions, are also very important an big project that we need to proceed with. Also this imaging tools must be integrated with in sardana. With out forget the StateCodeInterpreter project, very necessary to trace back what had happen with a power supply. Another day it will be explained.
And the things wouldn't finish where, there are also projects that have a title but not yet received the Start() command... (like the linac's control & gui).
2012/11/21
How to see the beam in the storage ring
I have written some times about how a synchrotron works (underlining that I am not a physicist and what I write is what I understood from the explanation of other people). Also I have wrote about how can we saw the beam or its effects: visible beam, first xrays or fluorescent screens
But there is another way to see the beam: using an scope.
Today I have ask to know a little bit more what we can see from an scope that we are reading the signal:
This signal is get from a Fast Current Transformer (FCT) located in sector 02:
Very close to the dcct that is giving us the current of the beam in the machine status. This FCT is an inductor surrounding the vacuum chamber and what we see in the scope is the inducted current by the primary current given by the electron bunches.
In the scope picture we can see the bunches in one turn (about 900ns). But a bunch is not one of this eleven groups in the signal. They are composed by bunches.
May can help to see that by plots taken with taurus:
Zooming on one of this groups:
With this last picture we can see the bunches themselves. The bunches are separated by 2ns, and the linac is shutting in groups of 32 consecutive bunches and a gap of 24ns.
Also I have ask why this groups of bunches have different shapes like:
This 3 examples has been taken over different times during today's start up of the machine. This shapes can be explained because fine configuration or how the equipments works and their efficiency. In the best case, in a world of the theory, this groups would be absolutely squared.
But there is another way to see the beam: using an scope.
Today I have ask to know a little bit more what we can see from an scope that we are reading the signal:
vncviewer of an Agilent DSO80804B
But, 2 questions: who this signal is get and what does this mean?This signal is get from a Fast Current Transformer (FCT) located in sector 02:
In the scope picture we can see the bunches in one turn (about 900ns). But a bunch is not one of this eleven groups in the signal. They are composed by bunches.
May can help to see that by plots taken with taurus:
Equivalent to the vnc, but getting the data from a tango device server.
Also I have ask why this groups of bunches have different shapes like:
Alba synchrotron recovered
Yesterday, after the accident and the weekend works to recover it, we had beam again in the storage ring:
Having beam during more than 2 hours means that, after the accident, nothing have fell down and block the path in the pipe.
Subscribe to:
Posts (Atom)
