The Painting Fool

Meera - The emotionally aware Painting Fool, sounds exciting, but what does it do? Well that's the name of the winning machine from the 2007 Machine Intelligence Awards. It's an artificial intelligence system that's made up of two pieces of software. The first is an emotion detecting system that videos your face and analyses it to see what emotion you're feeling. The second part produces and abstract painting of you on the screen using colours and styles to reflect the emotion that it think you're feeling.  These combine to produce a system that could revolutionise the art world either by giving artists some competition or helping them to design their masterpieces. I met up with the creators Simon Colton, Michel Valstar and Maja Pantic this week to find out more and also to see what emotions the machine thought I was feeling. I spoke to Maya first about how the emotionally aware Painting Fool works.

Maja - It is a system which is combined of two systems: one is the visual part to the system and the other is the painting part. The visual part detects automatically the emotion of the sitter from live video while the painting part of the system uses this information in order to choose the paint and then highlights this emotion in the painting of the sitter.

Meera - When you say sitter, you just mean the person standing in front of the video?

Maja - Exactly.

Meera - So you've already set up the equipment ready for me to have a go. I'm just going to go and stand in front of it and portray my emotions.They've taken a video clip of me looking neutral and then smiling afterwards and now the first part of the machine is processing it. What's going on Michel?

Michel - At this point it has found in the first frame where exactly your face is and next it will find 20 facial points such as the points on your eyebrows, the corners of your mouth and the corners of your eyes.

Meera - It's saying I'm very happy. How's it found that out?Michel: Basically we tracked those 20 points through the image sequence and based on the distances between the points and the positions of the points and the speeds of those points we found out which facial muscles you were using.

Meera - So once you know the facial muscles in use, how does this then translate to it knowing the emotion, Maja?

Maja - We are trying to detect six basic emotions which are: surprise, fear,sadness, happiness, anger and disgust. For each of these emotions there are certain facial muscles which are activated. For surprise your eyebrows are raised and your eyes are wide open and your mouth is wide open. So based on these kind of rules we are able to recognize which emotion is shown in animage and in this case as can be expected the smile triggered actually happiness. This point as you see the painting will start its work.

Meera - Ok yeah, so it has started painting me now. Simon, how has itstarted producing this painting?

Simon - Well it's been given three important bits of information from the vision software. It's been given the emotion, happiness. It's been told where the eyes, nose and mouth are and it's been told which picture within the video sequence where you're expressing your emotion the best, where were you smiling the most. First of all it's going to paint that picture.It's going to spend more detail on your eyes, nose and mouth so it makes itl ook more like you and it's going to choose its painting style according to the happiness emotion. It's turned your face into a number of regions of colour then for each paint region it chooses the colour for that paint region. It chooses whether to enlarge or make it smaller. It can do that with the eyes for example to emphasize the emotion in the eyes and it chooses how to simulate the painting in of that paint region.

Meera - If someone else was to have stood in front of that camera, would these colours still have come up?Simon: I've trained it specifically to use a colour palette of about 15colours, all of which are bright, and it does use different ones depending on the facial colours of the person using it. The bigger difference would have been had you not smiled and had you shown fear, or had you shown disgust, it would have produced a completely different picture. It would have stretched your face, it would have painted you in greys and kind of rotten tones and it would have used a very loose painting style so that it would really emphasise the distortion of the face.

Meera - How have you given it the information so that it knows how to appreciate its subject?

Simon - I trained it like a human painter would train an apprentice so I've told it, for example, that if it uses greens and reds then that's likely to heighten the anger in a picture. I've told it that if it uses a stretched face then certain people will associate that with Edvard Munch's The Scream: that famous painting where there is disgust going on. What I'm aiming for more with this is to get the software accepted as a creative artist in its own right, independent of me as the software originator. The next thing I wanted to do is to imagine completely new scenes and paint from its imagination.

Meera - You have obviously combined together for this project and created the emotionally aware Painting Fool. Where do you want to continue this merger, Michel?

Michel - Well what I think we want to do is make it even more aware of its subject matter. There is so much more information in the face that we could use. For instance, gender: we could paint it differently depending on whether it's a man or a woman. But also maybe age category and even we might look into beauty aspects of the face.

Meera - My painting is now finished and considering how abstract it is, itr eally does look like me I think but also you can tell that's a girl. How will you make it distinguish between males and females even more?

Michel - Well of course it looks like you because it's based on your portrait, your image, and therefore it will always resemble a girl. We could actually enhance the feeling that you get when you look at a painting if we know what the actual gender of the sitter was.

Anger	Disgust	Fear
Sadness	Surprise	Happiness