Wednesday 22 April 2015

Pilgrimage into the world of Tarkovsky: through the eyes of hope and suffering

[Level N]

The world is not perfect. It has given us scientists, authors, artists and politicians - and I have lived enough to know none of them were really perfect. Among these, we have personal heroes, personalities that have made great discoveries, built wonderful things or have lived extraordinary lives. Whether it is Obama, Einstein or George Orwell, they have their deficiencies.

I am saying this because the word Pilgrimage in the title can put you off. In fact it puts me off. But ... it is there for a reason, and I hope by the time you finish reading - if you hang on long enough - you would see it.

*  *  * 

A stuttering boy who finally mutters a few words with no pause after a session of hypnotherapy, and then leading to a black screen of titles with the music of Bach, is not a typical opening scene. But this for me has been the most memorable opening among all the films I have seen. If you are looking to describe the body of work by the late director Tarkovsky, look no further, it is all there in the opening scene of The Mirror (1975). This scene somehow encapsulates Tarkovsky's view of himself. A timid lad who can barely speak two words in sequence without constantly stuttering but with the help of "supernatural" powers can speak and tell us his stories. And the process is painful for him, it is only achieved with determination and sacrifice.

*  *  * 

Stumbling a few times along the way, I find my way with difficulty through the aisles of the dark cinema. I think I have missed the first few minutes but that should be OK.

I am lucky to be here. After queueing several hours in a cold sunny day on February 1988, I have managed to buy a ticket to Tarkovsky's Stalker (1979) in Fajr Film Festival. A special section of the Festival is dedicated to the memory of late Tarkovsky who died the previous year and they are showing all his films - with understandable cuts when it does not meet with "the code", at the end of the day Iran is run as an Islamic country. These are the films that intellectuals go to - and I should go to since I am planning to become one!

And I sit there in the dark, watching this 220 minute epic where very few things actually happen. And the film is in fluent Russian with no subtitles!

And through the confusion of barely knowing the storyline, and not getting any of the dialogues, as a young 19-year-old student, I am mesmerised. The film works its way through me, somehow, precipitates deep marks that are ingrained with me until this day. The film communicates with a strange language which I feel I have known but very remotely, as if in a previous life. It is hazy, sublime and next to impossible to translate to words.

And next thing I know, I am sitting watching Mirror (this time it is the public screening and is translated) and incoherent images and storylets come and go, with apparently no relationship. And yet, by the end, I cannot control myself and my eyes are wet. And again, I have no explanation, when being accused of pretentious intellectualism or sentimentalist.

My journey (or Pilgrimage) has started. These films, I have lived with. They grew with me, and gradually, over quarter of a century, made sense. And this post is about why and how.

*  *  * 

It was not a coincident that in the same Fajr Festival 1988 there was a screening of Parajanov's Colour of Pomegranates. It is generally believed that films of Tarkovsky and Parajanov are very similar. Tarkovsky indeed was a fan of Parajanov works and I later found out they were in fact friends. I did manage to watch it later on the public screening but when it was even more bizarre, I did not quite like it. Form is the vehicle to deliver the meaning and not the meaning itself. Parajanov felt overly concerned with form and while narrative and a story of love is there, the meaning did not quite live up to the novelty of the form - I don't know maybe I will think differently.

Going back to Tarkovsky, "the meaning/message" is not easy to grasp. Commonly there are different interpretations and even it is said that his films are meant to take us to a personal journey to understand hence all interpretations are correct because they are true to ourselves - so post-modern!

Did Tarkovsky hide specific messages for us to grasp in his often difficult and unusual films? Most works of art (and even more so for the music and modern art) are open to personal interpretation. Abstract paintings famously invite us to find our personal comprehension of the work of art. But how about Tarkovsky?

Only he can answer us. And he did.

*  *  * 

It is very rare for a director to uncover his tricks and spoil the meaning of his films in a book. Well, he did not quite do that in Sculpting in time but he did reveal his vision of cinema as an art form. And more importantly, why he made his films. While for many, making film is a means of gaining fame, a career, or a vehicle to project one's intellectual viewpoints, or (as Tarkovsky refutes) as a means of self expression, for Tarkovsky it was a selfless and painful endeavour to fulfil a responsibility he was trusted with. While for some, making on average one film every 7 years means they were striving for perfection, for him it was painstakingly ensuring his duty in this world gets fulfilled.

What do we mean by responsibility? Hard to explain in words but easier to point you to his films. We get to meet Tarkovsky himself in his films - whom do you think Andrei Rublev was then?! An artist monk, sick of the decadence of the world, taking a vow of silence only to understand at the end that he cannot forfeit his duty as an artist in order to stay pure. His work will involve suffering but that is the sacrifice he is meant to make. An artist is not free, despite the theories of modern art, artist is not solely responsible to himself and his art. Tarkovsky shunned the modern art:
"Modern art has taken a wrong turn in abandoning search for the meaning of existence in order to affirm the value of the individual for its own sake."
Tarkovsky sees the process of making art as a consummation of the artist for the cause - he called artists "sufferers". Artist is a martyr and artistic creation a sanctimonious sacrifice:
"Artistic creation demands that he 'perish utterly' in the full tragic sense of those words."
The word self-expression, this inner looking for fulfilment, utterly made him frustrated with the artistic culture of the day. Artist himself is the last person to gain from the artistic creation - very much like the character Stalker that could not benefit himself from "The Room", nor any of the other Stalkers.
"The artist is always a servant, and is perpetually trying to pay for the gift that has been given to him as if by miracle."
Also, the artist is not merely an intellectual concerned with the abstract notions of his art form, but he is an evangelist (in its literal meaning) making his art for everyone:
"Art addresses everybody, in the hope of making an impression, ... of winning people not by incontrovertible rational argument but through the spiritual energy with which the artist has charged the work."
And oh boy, that spiritual energy that sets you on fire, making you look for the answer - in my case for quarter a century, Now probably it makes a lot more sense to think of this man as a prophet.

*  *  * 

Tarkovsky films are slow - for some, painfully slow. They contain many long takes, and this by itself does not signify a technique but it is a by-product of his vision and language for the cinema as an art form. This vision was used later by Bela Tarr, a true student of this vision.

On the surface, it could appear that this is a stylistic decision to come up with a unique formalism, a pretentious intellectual gesture. But Tarkovsky himself disdained pure experimentation to come up with a new formalism:
"People talk about experiment and search above all in relation to the avant-garde. But what does it mean? ... For the work of art carries within it an integral aesthetic and philosophical unity; it is an organism ... Can we talk of experiment in relation to the birth of a child? It is senseless and immoral."
And this again reminds of the burden of responsibility he felt in making his films. On the other hand, he is regarded as one of the proponents of "poetic cinema", a term that Tarkovsky himself find almost offensive:
"I find particularly irritating the pretensions of modern 'poetic cinema', which involves breaking off contact with fact and with time realism."
Tarkovsky talks of the works of arts that have inspired him and have shaped his artistic language. These range from late middle ages icons, Italian paintings of the renaissance period to the works of literature by Dostoevsky, Tolstoy, Goethe and finally to the films by Dovzhenko, Bresson and others. In an effort to describe an ideal piece of work, he brings an example from a relatively obscure painter of the renaissance period, whose painting had a deep effect on him. In contrast to Raphael's "Virgin and Child", he was captivated by the inexplicability of the works of Carpaccio.

Preparation of Christ's Tomb (1505) - Vittorio Carpaccio

Back to cinema, he believed that the ideal film is countless meters of celluloid capturing entire life of a person. This probably make it easier to understand why his films were usually longer than 2 hours and in case of Stalker 3 hours and 40 minutes! Tarkovsky believed that a work of art need to be true to life. And when we think of life, there are fast cuts and edits: it is a very long take.
"I want to make the point yet again that in film, every time, the first essential in any plastic composition ... is whether it is true to life."
Tarkovsky explains some of the techniques he used in order to make his scenes have a deeper impression on the viewer. These techniques move away from the cinematic languages of the time, from the cliches symbolisms of the common cinematic cinema. They usually enhance and magnify the image and make it imprint on our psyche. An example, is the scene from Mirror where the Doctor meets Mother and at the end of the scene a strong wind blows, making the Doctor look back towards the house.

All in all, for Tarkovsky, a masterpiece is a work of art that you cannot remove anything from it without completely destroying the work. And that is exactly what he saw in the works of Carpaccio - a unity of that cannot be broken. As such, it is really difficult to pinpoint what makes the masterpiece exceptional as it is.

*  *  * 

So where does Tarkovsky get his inspiration from? Where is his true role model?
It might come as a surprise for some but Tarkovsky was a devout Christian. He knew two of the gospels by heart and would recite them in conversations. His book is full of quotations of the New Testament (1 Corinthians a favourite of him) and phrases that can only mean he truly believed. He was not after happiness (remember Stalker - the Black Dog of depression):
"Let us imagine for a moment that people have attained happiness ... Man becomes Beelzebub."
He saw a strong similarity between art and religion:
"In art, as in religion, intuition is tantamount to conviction, to faith. It is a state of mind, not a way of thinking."
He felt a deep connection in his role to that of an evangelist:
"Art ... expresses its own postulate of faith."
And his role model: of course, Jesus: Selfless sacrifice, Servant, For everyone, Winning people. All his films and writings points to him. It is not accidental that we hear John's Revelation in Stalker. As apocalyptic as the film is, this could have not been more literal. No accident that we meet God in Solaris (Ocean), or Stalker is so stricken by the lack of faith of others, and in Sacrifice, one can save everyone.

Commonly people ask why he made his films so difficult under layers of meanings. Why? Exactly for the same reason Jesus, as a teacher, used parables to convey messages, not plainly.

*  *  *

And my quest is not finished, but surely it has eased off. After believing in Jesus in 2001, I revisited Tarkovsky again lately. Now all symbols and meanings crystal clear. I feel very close to what he tried so hard to shape into images. It just makes sense.

Messages and meanings ... and what are those? It will be clear by the process of your personal pilgrimage. And it could begin now...

Friday 3 April 2015

Utilisation and High Availability analysis: Containers for Microservices

Microservices? Is this not the same SOA principles repackaged and sold under different label? Not this time, I will attend this question in another posts. But if you are considering Microservices for your architecture, beware of the cost and availability concerns. In this post we will look at how using containers (such as Docker) can help you improve your cloud utilisation, decrease costs and above all improve availability.

Elephant in the room: most of the cloud resources are under-utilised

We almost universally underestimate how long it takes to build a software feature. Not sure it is because our time is felt more precious than money, but for hardware almost always the reverse is true: we always overestimate hardware requirements of our systems. Historically this could have been useful since commissioning hardware in enterprises usually a long and painful process and on the other hand, this included business growth over the years and planned contingency for spikes.
But in an elastic environment such as cloud? Well it seems we still do that. In UK alone £1bn is wasted on unused or under-utilised cloud resource.

Some of this is avoidable, by using elasticity of the cloud and scaling up and down as needed. Many cloud vendors provide such functionality out of the box with little or no coding. But many companies already do that, so why waste is so high?

From personal experience I can give you a few reasons why my systems do that...

Instance Redundancy

Redundancy is one of the biggest killers in the computing costs. And things do not change a lot being in the cloud: vendors' availability SLAs usually are defined in a context of redundancy and to be frank, some of it purely cloud related. For example, on Azure you need to have your VMs in an "availability set" to qualify for VM SLAs. In other words, at least 2 or more VMs are needed since your VMs could be taken for patching at any time but within an availability zone this is guaranteed not to happen on all machines in the same availability zone at the same time.

The problem is, unless you are company with massive number of customers, even a small instance VM could suffice for your needs - or even for a big company with many internal services, some services might not need big resource allocation.

Looking from another angle, adopting Microservices will mean you can iterate your services more quickly releasing more often. The catch is, the clients will not be able to upgrade at the same time and you have to be prepared to run multiple versions of the same service/microservice. Old versions of the API cannot be decommissioned until all clients are weaned off the old one and moved to the newer versions. Translation? Well some of your versions will have to run on the shoestring budget to justify their existence.

Containerisation helps you to tap into this resource, reducing the cost by running multiple services on the same VM. A system usually requires at least 2 or 3 active instances - allowing for redundancy. Small services loaded into containers can be co-located on the same instances allowing for higher utilisation of the resources and reduction of cost.

Improved utilisation by service co-location

This ain't rocket science...

Resource Redundancy

Most services have different resource requirements. Whether Network, Disk, CPU or memory, some resources are used more heavily that others. A service encapsulating an algorithm will be mainly CPU-heavy while an HTTP API could benefit from local caching of resources. While cloud vendors provide different VM setups that can be geared for memory, Disk IO or CPU, a system still usually leaves a lot of redundant resources.

Possible best explained in the pictures below. No rocket science here either but mixing services that have different resource allocation profiles gives us best utilisation.

Co-location of Microservices having different resource allocation profile

And what's that got to do with Microservices?

Didn't you just see it?! Building smaller services pushes you towards building ad deploying more services many of which need the High Availability provided by the redundancy but not the price tag associated with it.

Docker is absolutely a must-have if you are doing Microservices or you are paying through the nose for your cloud costs. In QCon London 2015, John Wilkes from Google explained how they "start over 2 billion containers per week". In fact, to be able to take advantage of the spare resources on the VMs, they tend to mix their Production and Batch processes. One difference here is that the Live processes require locked allocated resources while the Batch processes take whatever is left. They analysed the optimum percentages minimising the errors while keeping utilisation high.

Containerisation and availability

As we discussed, optimising utilisation becomes a big problem when you have many many services - and their multiple versions - to run. But what would that mean in terms of Availability? Does containerisation improve or hinder your availability metrics? I have not been able to find much in the literature but as I will explain below, even if you do not have small services requiring VM co-location, you are better off co-locating and spreading the service onto more machines. And it even helps you achieve higher utilisation.

By having spreading your architecture to more Microservices, availability of your overall service (the one the customer sees) is a factor of availability of each Microservice. For instance, if you have 10 Microservices with availability of 4 9s (99.99%), the overall availability drops to 3 9s (99.9%). And if you have 100 Microservice, which is not uncommon, obviously this drops to only two 9s (99%). In this term, you would need to strive for a very high Microservice availability.

Hardware failure is very common and for many components it goes above 1% (Annualised Failure Rate). Defining hardware and platform availability in respect to system availability is not very easy. But for simplicity and the purpose of this study, let's assume failure risk of 1% - at the end of the day our resultant downtime will scale accordingly.

If service A is deployed onto 3 VMs, and one VM goes down (1%), other two instances will have to bear the extra load until another instance is spawned - which will take some time. The capacity planning can leave enough spare resources to deal with this situation but if two VMs go down (0.01%), it will most likely bring down the service as it would not be able cope with the extra load. If the Mean Time to Recovery is 20 minutes, this alone will dent your service Microservice availability by around half of 4 9s! If you have worked hard in this field, you would know how difficult it is to gain those 9s and losing them like that is not an option.

So what's the solution? This diagram should speak for more words:

Service A and B co-located in containers, can tolerate more VM failures

By using containers and co-locating services, we spread instance more thinly and can tolerate more failures. In the example above, our services can tolerate 2 or maybe even 3 VM failures at the same time.


Containerisation (or Docker if you will) is a must if you are considering Microservices. It helps you with increasing utilisation, bringing down cloud costs and above all, improves your availability.