(re)install sage(math) in aws

Some time a go, I followed a Scientific computing course and I've learn the basics to configure a Free tier machine in Amazon web services to have my own installation of Sage(math). But I've made a mistake in the configuration (because I didn't know it can be changed) and that machine has only 8GB of storage. The consequences of it has been to have an almost full disk and updates where almost impossible (uninstall required to update, and reinstall later on). In practice it has mean that i've been stack in sage 5.11 when today it's 6.11.

But this Free tier has up to 30GB of hard disk space! And I have configured a new machine with enough space and the service are being moved to that new machine.

As this machine has been set up from scratch, I like to write what I did in command line for my records. And if it would get useful for others, great.

After having a fresh ubuntu 13.10 64bit AMI (Amazon Machine Image) on a t1.micro, I've:

$ sudo apt-get update
$ sudo apt-get dist-upgrade
$ sudo apt-get install ipython python-numpy python-scipy python-matplotlib python-dev git gfortran openmpi-bin liblapack-dev apache2 gnutls-bin

This is to have an up to dated machine and some basic needed packages. Then is necessary to add the ppa (personal package archive) where sage's packages are stored:

$ sudo apt-add-repository -y ppa:aims/sagemath
$ sudo apt-get update
$ sudo apt-get install sagemath-upstream-binary
$ sudo apt-get install python-openssl openssl libssl-dev libssl-doc python-openssl-doc python3-openssl
$ sudo sage -i pyopenssl-0.13.p0

Once installed the main package and some optional packages first execution can be called:

$ sage -notebook interface=\'\' secure=True

In here, a command line interactive process will ask for the sage's admin user password. And after that the script for the init.d can be prepared and started, like described in a previous post.

I see threads everywhere...

A very long time a go, when I was a cluster scheduler developer, I've seen my record of concurrent threads. In the stress test of the system, the scheduler had up to 480 threads consuming a 1% of the cpu. That was good and the objective because an scheduler shall not be doing anything else than planning (with in different levels an abstractions, managing restrictions and complains). The code was full of semaphores. events and signals. The tests were succeed without deadlocks. Since then this has been my record in number of threads.

In my code I use very often threading. In a object oriented design many threads can be doing different things with those objects and it is fun when many of them are interacting and the application works as expected with a good performance. I've play with OpenMP and MPI, but I've never raised my record of concurrent threads, until now.

I'm following, in coursera, a subject about heterogeneous parallel programming with CUDA and this has broken the record in very very small pieces. In one of the exercises, a matrix multiplication, the data set is: A_{200,100}*B_{100,256} where the result is C_{200,256}. That is 200*256 cells and one thread each to do the underlying basic operations.

Using CUDA blocks and grids, where often the data doesn't fit exactly and you exceed a bit, this matrix, in my case I've used 16*16 blocks on a 17*12 grid of blocks. This rises my record of parallel threads up to 52224 threads. More than a hundred times...

Event this numbering, imho there are categories on this threading count. In this record, those threads where doing almost the same with different data. In my previous record, there where only a few of them doing the same thing. That splits also the threading count in two categories...

Update 20140202: Did I set my threading record to 52.224? Doing another exercise on this cuda course, I've break it again: a convolution of a 2k*2k coloured image (3 channels, and a 5*5 mask) 12.582.912 parallel threads did the job in 19ms...

cloud.sagemath.org

After to follow the coursera's "High Performance Scientific Computing" I've learn about the Amazon Web Services and how to use them. Specially in the hook for new customers with 12 month free tier. I have mention it about a month a go.

I have there some sage sheets that in other alternatives didn't finish the calculation (there is a sheet timeout when there is no activity). With my own installation, I can be aware of when this instance is being crazy and forget this timeout to continue the calculations.

But there is something more with sage in the cloud. Further than install sage in a machine in the cloud, there is a new sage project to take advantage from the cloud: cloud.sagemath.com

I have started using it, not much yet, but I like specially to mention one of the talks than William Stein did:

I recommend this talk very much.

Sage(math) in the Amazon Web Service

A few days a go a new version of Sage has been released, the 5.11.

When I was doing a coursera's High Performance Scientific Computing, I've learn how to have a small machine in the Amazon Web Services. In particular I've set up, as the course shows but was just one of the options I would do, an Ubuntu 13.04 Amazon Machine Image (It look that the ami I used is no longer available).

With this very small virtual machine (64bits, Up to 2 EC2 Compute Units (for short periodic bursts), 613 MiB memory, 8GB Hard disk, I/O Performance: Low). And for the first year the service is for free.

What to do with a machine in the cloud? Lets be sage...

There is a easy way to have your sage installation and someone else will maintain the sagemath package. With 3 commands you can have an installation in your ubuntu.

The thing I did by hand was and script to insert this as a service in the init.d:

#!/bin/bash
# configuration
SAGE_OPTS="-notebook interface='' secure=True"
SAGE_LOG="/var/log/sage/sage.log"
SAGE_USER='ubuntu' #default in aws machine
# commands
CMD_ECHO="/bin/echo"
CMD_GREP="/bin/grep"
CMD_SLEEP="/bin/sleep"
CMD_KILLALL="/usr/bin/killall"
sage_start() {
        $CMD_ECHO "Starting Sage..."
        (su - $SAGE_USER -c "sage $SAGE_OPTS" 2&>1 ) >> $SAGE_LOG &
}
sage_stop() {
        $CMD_ECHO "Stopping Sage..."
        # CAUTION : maybe you kill something important in your server
        #$CMD_KILLALL python
        ps -ef |grep "sage -notebook" |grep -v grep|awk '{print $2}' |xargs kill -9
}
case $1 in
        start)
                sage_start
                ;;
        stop)
                sage_stop
                ;;
        restart)
                sage_stop
                $CMD_SLEEP 5
                sage_start
                ;;
        *)
                $CMD_ECHO "Use: $0 {start|stop|restart}"
                exit 1
                ;;
esac
exit 0

After that, a simple:
$ /etc/init.d/sage_daemon start
is enough to have the notebook  up and running. Perhaps you would prefer to insert it to the boot process
$ sudo update-rc.d sage_daemon defaults 95
(this is only an example)

Best of all is that since I installed it (it was version 5.9) to the current 5.11, those two updates have been as simple as:
$ sudo apt-get dist-upgrade


Incredible, I'm really happy with this, the only point I'm missing is to have those binaries also available for debian's installations.

working with git(hub)

After so many years of work with cvs and later with svn, many of the project I work with uses git. It's not so easy to start with git, but once you did, you will fall in love.

I only can imagine what a young developer can start using version control directly from git, but my knowledge is conditioned by my previous experience. First time I've work with cvs, I never remember how the commands works, what they mean, which order was the correct. But you learn. Migration to subversion, was good. Many of the things starts to work the way you think.

But git is a big step further. It has been a big change for me. To mention to books have helped:

• Magic Git (pdf) and,
• pro-git (pdf)

Shorted than books are videos, and the one that I prefer about git is:

Power Your Workflow With Git

Recently I've seen another video that helps me on improve my way to work with git: the git flow.

This, together with the decentralized repository schema, make me happy. A long a go I was trying to find a way to have code in more than one repository and have them synchronized.

Nowadays this feature is even documented!

For example, the work with the Lima project. The main sources are on github but instead of have all the developers working in parallel and all of us committing to the same trunk (specially because I know a very short number of people that works with subversion with branches), each of the developers have their own branch by make a fork. That is great! You have your own branch, and with the sync feature mention before it's incredibly easy to get the official changes to your repository:

The first time you do it, from your clone of your fork:

$ git remote add esrf-bliss https://github.com/esrf-bliss/Lima.git

And when you like to take those changes:

$ git fetch esrf-bliss

$ git checkout master

$ git merge esrf-bliss/master

And with this you can upload the result of your synchronization:

$ git push origin master

Further than that, you can have as many remotes as forks it has. And you can repeat this with any of the branches that the other developers can do (remember that the master is only one branch more with an special name, meaning what you think is the trunk).

Scopes refactoring

Usually august is a good time to think on refactorings. During the year normal operation there is never enough time to look back. The sentence "I want all, and I want it now" it's a classical sentence from clients and bosses (many times followed with a "I want it fix, but don't touch any thing").

Also is good because too many managers doesn't care about refactoring. They don't really understand what this means and didn't give to it the real value of this action. For them, often, the maintenance of some code is sort of magic, they don't know how this can be made. Furthermore they don't care, and only think in do it fast an cheap.

In the case of the scope, we are currently using the scope waveforms in a Filling Pattern calculation using the Fast Current Transformer signal. This was the fpFCT project. In this case, we went far away from the original design of the access to the scopes. This original design was already improved long a go, and looks that now it's time to remade it base on the newer requirements.

The current bottleneck is the Visa middlemen. The agent that shows the user the interface of an instrument like an scope doesn't know about protocols. This agent knows the language to talk with the instrument, but not the channel used to have bidirectional communication with it.

From Max IV, there is a proposal to talk directly via socket. This will remove from the scene one of the actors, and for sure will improve.

The second proposal, should be helped by a new feature requested in tango: give the information to the device that someone is listening (or not) over the events on an attribute. This is, avoid to request information to the instrument if noone is listening.

Current idea of the refactoring is to use proposal one, and by commands (until the mentioned feature is available) configure the attributes that are being updated by an isolated thread, optimizing the scope access, in an approach to proposal two.

Update 20130806: Max IV has contributed with their nice idea in a branch.

(more) frustraited @ work

Alba is broken like since eastern. When we come back from that shutdown at the end of March. Water cooling problems, the water flow was unstable; sponges were found in the "clean water" cooling pipes, together with other dust; a weekend instrument hang causes a temperature increase and bad quality of hoses become vulcanized (below the specs) and caused a disaster in one of the two main power supplies of the booster; a TLD wrong placed in a weird space and when an Insertion Device was moved, it bends the beam pipe blocking the beam orbit, requiring to vent and replace a section. It's like we stomp shit.

This is one of the issues in this accelerator complex, but there are others that may interact.

Panic reaches the management at the beginning of this years when 12 people announces their new positions in other institutes. If you take into account that we are 145 workers, you can realizes the magnitude of the problem. From the people that says goodbye, 8 were from the computing division (around 50) and from those 4 were from the controls section (of 14). This panic didn't mention the 1 or 2 per month that has announced that the left during the past two years.

Management has announced 3 months later the creation of a commission from work behaviour improvements. Now, in July, this commission has made the first meeting by last week.

As I mention in the previous post, another reason of personal sadness: my bosses didn't case about a worker that has been called to do a talk in the university.

I didn't write here since then. I didn't found any reason to write about software design here.

But last reason to get one step further in frustration went last week.

Starting 2 months a go, in the tango meeting, there was a request to the tango community to introduce security embedded inside the tango implementation. From the last 4 years I've been trying to start a PhD in cryptology (but too busy due to the work at Alba), and I've been getting closer and closer to the field of the RFIDs and the smartcards. During the presentation where this was requested, I thought that many of the schemas to ensure RFID can be also valid to be applied in between the communication of the agents in a distributed system, like tango.

I thought it can be interesting for Alba and the tango community, to exploit that one of the workers has a hobby in cryptology and security. In this terms I've talked to my boss.

For that, I need 3 weeks to talk to my boss (this is already one problem). During this time, I've been able to, out side working hours, explain this idea to the research group in the university (they are in another city, 150 km away). I found a great acceptance about "Ensuring Tango control system", even more I've seen enthusiasm about the idea of a PhD based on an application of all the cryptological work made by the whole group (many things about public key -specially elliptic curves- symmetric cyphers, stream ciphers, secret sharing, homomorphic encryption and field like that).

At work, the response wasn't that good. The answer was like: "do what ever you like in your free time, but this must not affect your current duties at all. It's not possible to dedicate any of your working hours in such a thing". Clearly have said, if I do a PhD, is not meaning anything for Alba at all. Ooh!! This is a very clear way to motivate a worker. For my partners, also fun when was said "if you do, others would ask to do this", what's the problem on workers training?

Went I talked to my boss, I said many time to him and have to point here, that my duties have been increase every time that a partner left. Last Friday, there was a presentation about the Alba's controls system, and when the subsystems of the machine was listed, half of the elements on that list are on my behalf (and not all my duties were there).

When I went to talk about this with my boss, I went there offering my free time to work in a industrial PhD. My proposal was not to stop doing my job and disappear for full time dedication to this, my proposal was mostly the free time, but being realistic that the brain thinks at any time. As a collaboration between the university and the industry, all the involved has to put something, specially the PhD candidate.

Well done, with this deal, the simpler solution is that Alba will not appear at all in my PhD. Or if it appears will be to be mention explicitly that they haven't contribute at all; even worst Alba's position was opposition to doing a work like that.

fpFCT: bugfixing

After the development of this study of the filling pattern using the fast current transformer it was time to search for bugs. From a result like:

This calculation doesn't have good sense and looks like there is a bug. This only 2 groups doesn't look nice. Even worst when we execute the code under an scripting way, the result was different. And we found the error: the number of giga samples used in the calculation. We've seen a 10th factor and we found that using the scripting mode, it reads the attribute from the scope, but from the device server it uses a default initial value of 4e10.

We've been lucky that the default value wasn't the same than what was expected, or we would get crazy if some day it changes. For example if the scope samples at 20GS/s instead of 40.

After last bug search we can see results like:

There are things to review, we didn't finish completely this development. But what is true is that the scientist already have usable data for their studies.

fpFCT: with beam

A month a go I have work in simulate the beam of the storage ring by its bunches in the Fast Current Transformer (FCT). Well now we are in a run of 3 weeks (the operation calendar is available on the web site), and we can see the signal in the scope from the FCT and apply the algorithm developed in accelerators using the device server developer by the controls group.

To see those inputs and those outputs we need our Tango Graphical User Interface (GUI) called Taurus. And with a "simple" command line like:

$ taurusform sr/di/FillingPatternFCT-01/{nAcquisitions,StartingPoint,Threshold,ScaleH,OffsetH,TimingTrigger,resultingFrequency,FilledBunches,SpuriousBunches} sr02/di/sco-01/{Channel1,CurrentSampleRate,ScaleH,OffsetH} sr/di/FillingPatternFCT-01/{BunchIntensity,State,Status} &

This command line, looking so complicated is only listing all the attributes (and their sorting) to be set in a taurusform, by using a feature of bash to write the (almost) minimum.

For example, first parameter is: sr/di/FillingPatternFCT-01/{nAcquisitions,StartingPoint,Threshold,ScaleH,OffsetH,TimingTrigger,resultingFrequency,FilledBunches,SpuriousBunches} An this means to get from the device server sr/di/FillingPatternFCT-01, the attributes listed.

That command will open a small window like:

There are two button in the attribute space because they are 1D data (spectrum in tango), Clicking them a taurusplot will be launched for each. One for the FCT signal:

And another for the Bunch Intensity calculated:

With Taurus, it's so easy to have a little simple gui to start with. Also the General Taurus Gui is neither so difficult.

fpFCT: simulating the bunch current

fpFCT is an smaller name of the Filling Pattern from Fast Current Transformer. As I've said I didn't know how to simulate the signal that is in coming from the scope, and I found a solution.

This is a reused image of the scope signal from the storage ring FCT and what I found how to simulate using PySignalSimulator is:

There are only 4kpoints instead of the 40k that the scope is giving but it looks quite similar.

But how this has been set up in the PySignalSimulator properties? The initial idea was the amplitude modulation but with this what is represented looks like:

But what if the carrier wave, better than a sinusoid looks like a quadratic waveform. With this I have started to play with harmonics. The formula that likes more to be a carrier has been:

In red is plot the carrier over the modulated signal in blue. The signal that is being modulated has been omitted because is almost a sinusoid.

The formula of the carries is:

With parameters values:

The formula of the modulated signal is:

With a_2 = 1; w_2=0.065. The x_2 is a value that is changing from time to time to introduce a drift in the signal in order to have a "dancing" bunches like in the scope. 

Lets see the result:

Fast Orbit FeedBack

One of the necessary improvements of Alba is the top up system. With this, the beam accumulated in the storage ring can be restored under a continuous re-injection precisely on the buckets here it bunch is losing charge. Recovering an old snapshot of the machine status:

This is an example where the stored beam had decay from 60mA to a bit more than 30 in around 6 hours (this was a year a go, by now we are having more current and a longer beam live time). This lost of current requires re-injections like what is in the middle of the red plot.

This re-injection, afaik, produces perturbations on the beam orbit that at the end means drifts in the x-ray beam in the beamlines. This re-injection, due to radiation safety reason have to be done with the frontends closed. That is with the beamlines in a standby, without light and waiting  that this re-injection finishes. This re-injection causes an interruption on the experiments, and the longest experiment you can do is the one that can be performed between re-injections. You can imagine the consequences if the most insignificant thing fails in the middle of an experiment.under this conditions.

Well, to make this re-injection possible with the frontends open, the orbit of the beam must be very static. The beam has to be thine characterised and its oscillations below some microns.

How the orbit is corrected? In the sextupoles, there are the coils for each of the poles, but it is also another pair of thinner coils that sets the corrector component. I imagine it like a tiny little bending (they have like 3 orders less of magnitude in terms of current).

Those correctors can change its current very fast for a small modifications. That can be, in the order of the hundreds of miliamps can change it with a 10kHz frequency.

With this, together with a beam position monitoring at this speed, and if you can process all this data, the orbit perturbations can be controlled good enough to allow to re-inject with the necessary safeness to implement the top up system and have the plot of the current as an straight horizontal line (negligible ripple).

Main issue: with in a ring of around 400m of perimeter share between 88 beam position monitors all their sets of data, plus the sniffers we have to take this data to calculate those corrections, and stablish the new setpoints to another 88 corrector power supplies. All this at 10kHz: read, calculate, write: simple operations.

Filling Pattern From Fast Current Transformer

As I explain the a previous post we have a Fast Current Transformer in the storage ring and we are able to see the signal in an oscilloscope. Many years a go I have started a project to have the features of those scopes we have in the control system.

It has been started as a Tango Device server in C++ many years a go, but from a refactoring it have changed to be a python device using PyTango. Also this new release have suffered many changes, specially the one that expands this server to more classes to be able to control, not only scopes, but also Spectrum analysers, Signal generators, and later on Radio frequency generators.

But this device have been more used in this "extensions" than in the original scope class. The Signal generator has been used in the booster tune excitation together with the spectrum analyser in the booster tune measurement (I didn't have time to write about those projects here, I hope soon). This devices works fine and the quality was good enough to change the storage tune measurements to be like the booster ones.

Due to Linac's needs to archive some scope measurements, the scope tango class was improved giving as an output here three posts. This was the real trigger that have launched this device server class.

A few weeks a go, more requests have been receiver about scope acquisition: It has been requested to have available the signal from an FCT in the control system, Also this required to improve the acquisition of the waveform from the scope and after some tries help has been requested to the manufacturer. With this I have reduced the network transition of 40kpoints as 300kB to 80kB or 40kB depending on the float precision. This allows to start polling the scope waveform at 3Hz.

This have boosted the project of the Filling Pattern studies using this FCT signal and a device server is available to perform this calculation. Just now that there is no beam, there is no signal in the FCT, only electronic noise like:

)Instead of the usual signal we have when there is beam:

Like often happens I had to ask what this mean and I had try to explain. But, to test the development of the FCT signal analysis, it's necessary to simulate a bit it. No necessary to have the same signal but I should complain that have to change from time to time to have something alive to be analysed.

The solution is to use a simulator of signals, and the best way is using the incredibly versatile PySignalSimulator device server. Creating a device with one property "DynamicAttributes" with something like:

A1=float(READ and VAR('A1') of WRITE and VAR('A1,VALUE)

This sets an scalar attribute with a float value that than be read or write. Combining this with an attribute that contains an array of elements:

Channel1=[ (sin(VAR('W1')*x)*sin(VAR('W2')*x)) for x in range(4096)]

With those simple lines you can have an spectrum value with a amplitude modulated signal. It's not the same than the beam signal, but it can acts to show how this bunch analyser works.

In the next video, the oscilloscope signal has been simulated using this simulator. You will see it  as the top-right blue plot with an amplitude that oscillates between 0 and a maximum. The variables of the formula that simulates this are below this plot, on the left side.

On the left side of the screen some data are revealed about the cpu, memory, network use and so one. After send the Start() command the image below the plot (on its middle) start to collect each one of the waveforms from above. And already with non complete buffer the data is being processed. On the right of this cyclic buffer image representation, there are the attributes to adjust the algorithm and then the output. The green plot on the top is the filtered output.

How to see the beam in the storage ring

I have written some times about how a synchrotron works (underlining that I am not a physicist and what I write is what I understood from the explanation of other people). Also I have wrote about how can we saw the beam or its effects: visible beam, first xrays or fluorescent screens

But there is another way to see the beam: using an scope.

Today I have ask to know a little bit more what we can see from an scope that we are reading the signal:

vncviewer of an Agilent DSO80804B

But, 2 questions: who this signal is get and what does this mean?

This signal is get from a Fast Current Transformer (FCT) located in sector 02:

Very close to the dcct that is giving us the current of the beam in the machine status. This FCT is an inductor surrounding the vacuum chamber and what we see in the scope is the inducted current by the primary current given by the electron bunches.

In the scope picture we can see the bunches in one turn (about 900ns). But a bunch is not one of this eleven groups in the signal. They are composed by bunches.

May can help to see that by plots taken with taurus:

Equivalent to the vnc, but getting the data from a tango device server.

Zooming on one of this groups:

With this last picture we can see the bunches themselves. The bunches are separated by 2ns, and the linac is shutting in groups of 32 consecutive bunches and a gap of 24ns.

Also I have ask why this groups of bunches have different shapes like:

This 3 examples has been taken over different times during today's start up of the machine. This shapes can be explained because fine configuration or how the equipments works and their efficiency. In the best case, in a world of the theory, this groups would be absolutely squared.

DevEncoded and Basler Ace in Taurus

Sometime a go, I have received the request to control a new model of cameras The Basler Ace acA1300-30g{m,c}. Yes both types: monochrome and colour. The most common cameras to look at the beam are the Basler Scout scA1000-30gm and the scA1300-32gm (yes until now we have only supported monochrome cameras in the service of beam diagnostics by imaging.

To take the pictures of this Basler Scout we have used and contribute using the Soleil's device ImgGrabber, and to analyse those taken pictures we have used the ImgBeamAnalyzer (also originally from Solei).

How we look at the images? Long time a go we have start working with Qt and as a result we have used Qub widget from esrf. This widget has been integrated in tau that has been renamed to taurus.

As an example, I have saved hundreds of snapshots of the gui with beam. The latest publish here was to celebrate the first x-ray beam at alba, but I can share much more of those pictures:

But the software we used doesn't support the Ace cameras. There is a newer software that makes deprecated this software and many of the synchrotrons that are using tango are migrating to LImA (Library for Image Acquisition). This is a very powerful software specially focussed in detectors.

This ccds that I'm talking here are not usually "detectors" in a synchrotron sense. They are not the end of the beamline from where the data of the experiment is collected. The most often activity of this cameras is beam diagnostics previous to the experiment to make sure that everything is on the expected place and the shape of the beam have the desired characteristics. Then they are used very often as a live video image in the graphical user interface (gui) from which I have placed images before.

Another situation is during beam commissioning when this cameras are used much more like a detector during a scan using Sardana. On this situation is not that important the image availability during the measurement, the most important this is the beam study of each picture and the statistical characteristics from the beam.

Then we have two very different user cases. In the upgrade to LImA both must be implemented, and yes they already are.

In the LImA Basler device the Ace cameras are already supported and there are some attributes that allows the possibility to have a gui like we have. But what was the problem? This attribute are from the "new" tango data type DevEncoded. Our system didn't support yet this data type and this has been an august development (almost the best month to work, almost every one on holidays and no one is bothering your development).

What make this data type special? This data type can contain almost anything: basically is a list of an string together with a list of bytes, where the string says what and how the things are codified in the list of bytes.

In the case of the LImA's "video_last_image" attribute, the DevEncoded is given as a "VIDEO_IMAGE" format. An issue of this flexibility is the naming, we are starting putting names without a rule and in the future this will be very difficult to unify and harmonise. 

LImA is hosted in a git repository and in the file "applications/tango/LimaCCDs.py" there is a method 'read_video_last_image()' where this "VIDEO_IMAGE" is codified.

Recently this DevEncoded format has been integrated in taurus.

And as we want to see the images using guiqwt who is integrated in taurus using an extension of this software, we have integrated this DevEncoded as a interpretable data from taurus.

The result: we've been able to take and show images from a monochrome Basler ace camera:

Using a fake beam (a image of the beam printed in a paper and placed in front of a ccd) we can check how it behaves with different beam types:

Very glad and satisfied when you see this images. It doesn't look special, but knowing the effort behind that the view is different.

Refactoring

The software has its live cycle during which the development and bug fixing are link to several variables, some under control or the developer but many are out of this control. I'm not talking much about the time dedicated to the development during a project, but too. Those variables are out of control specially in the bug fixing time, when the speed and extreme programming are more present.

When some day, some how, some one decides that something is urgent, or more urgent than the other urgent things. At that time, the development will done because of an heroic effort and all bugs will be fixed fast (hopping that the bug fix will not produce any other side effect).

How old must be a code to consider refactoring? It could depend on the number of requirements coming from the heaven. I mean when the number of times that some one is very urgent to codify, the developer does it, but with a price. And this price is a quality loss, until it becomes untenable. When any bug fix requires longer and longer time due to the side effects of the modification.

But there is a moment, that triggers an alert for the project manager that says refactoring is mandatory: when the design have almost nothing in common with the code: when code smell.

Then I found a book that looks nice for this task: "Refactoring: Improving the Design of Existing Code" Fowler, Martin (1999). More than ten years old book, older than the code I want to refactor, live irony.

Generalized Rijndael, schematics

Before to enter in deep on the questions remarked in the previous post about this series, I like to post some schemas that some day would be useful.

The Rijndael symmetric cryptosystem build by iteration using a network of permutations who follows the basic shannon's properties of "confusion and diffusion". The bits in the plain text are mixes and substituted by a group of operations do by an order. To decrypt, what have to be done is to do the same things but in the opposite way. For sure, there is a key to introduce here the secret to be able to undo the encrypt operation.

Diagram 1: Flow of the Rindael encrypt/decrypt

As in the diagram 1 shows, using only 4 operations (subBytes, shitfRows, mixcolumns, addRoundKey, and its inverses). But those operations are in a certain order to maintain a set of properties.

• subBytes: word substitution, where each element in the state matrix is replaced by its inverse and an affine mapping. Operations in
• shiftRows: cyclic left shift of the elements of the i'th row by i words.
• mixColumns: column linear transformation of the state matrix, where each column is given as an element of a polynomial ring, where the coefficients of this polynomial are polynomials in .
  This polynomial ring is: 
• addRoundKey: XORed transformation between the state matrix and the round key.

Is necessary to emphasize the use of a part of the key in the 'addRoundKey()' operation. This round key is much longer than the given key to encrypt/decrypt, and the process to generate this key expansion can be described in a iterative way:

Diagram 2: Iterator schema of the Rijndael key expansion

The first 4x4 matrix (the ) is the original key given with in a structure of a matrix of elements in the wordsize (8 bits, a byte in rijndael, AES). This example is using the 128 key option but the key matrix can have more columns: To build the key expansion its the same way, but remark that "#c" represents the number of columns for the message, not the key.

To build the following columns to have each round keys, an iterator is good to see how this is made. I have tried to get an schema from other webs sites, but the ones that I found haven't convinced to me. I hope this would help to someone who search on internet for a diagram of the rijndael key expansion.