Gauging Photo Fakery

A system capable of spotting when images have been digitally retouched has been published by researchers at Dartmouth College. Chris Smith heard from study author Hany Farid about the new system and why they created it...

Hany -  We were motivated by legislation in the UK that was considering mandating that altered or retouched photos be labelled as such and that of course was in response to the body of literature that is linking eating disorders, body dissatisfaction with the overexposure to extremely retouched photos. One of the criticisms of that legislation was that publishers were arguing that a simple label saying this has been retouched would not, in fact, distinguish between very simple types of manipulations like colour correction and cropping, and the more extreme forms of photo retouching.  And we wanted to create the ability to distinguish between those types of manipulations.

Chris -   So there would be almost like a sort of "Richter" scale of the degree of photo manipulation which the law could require people who publish photos to publish with the photo but because it’s an industry standard, everyone’s on an even footing and it would enable people to make a value judgement as to whether that picture is faithful to reality or not.

Hany -   That's exactly right.  So it’s a scale of 1 to 5.  One means the image is largely unchanged from the original photograph and 5 means there's been a radical shift in the underlying appearance of the person.  And I’ll also add - in addition to being useful in a legislative sense - it can also be also be used in a voluntary sense.  I mean, photo retouchers often just – it gets away from them.  They do one little tuck here and a little nip there, and pretty soon you're looking at a Barbie doll.  And so, the ability to, in real time, get feedback to what your manipulations are doing we think would also be very useful for the photo retoucher.

Chris -   So how did you actually approach this?  How did you go about writing a computer programme that would generate an objective readout of how the pictures were different but which and this is the real rub – would be meaningful to a human eye?

Hany -   That's exactly what the rub is.  So we, as mathematicians and computer Photo manipulation scientists know how to model computer alterations, but how do we predict or model how a person would perceive changes?  So there were two basic stages.  The first stage was the mathematics.  How do we mathematically model the alterations to a photograph and that includes geometric changes - things that alter the shape of the body, the length of the neck, the width of the hips and so on - and photometric changes - things that change the skin tone, the wrinkles, the colour.  And those are the types of manipulations that we know photo retouchers do, and after about a year of effort we found a way to mathematically model all of that.  Now of course that mathematical model is not exactly what we want, so the second stage was to collect many, many images of the original and retouched and asked human observers to rate them on a scale of 1 to 5 and for that, we used a crowd sourcing tool where we were able to collect data from hundreds and hundreds of people from around the world.  And then the magic was linking up the mathematical measurements with the human measurements and for that, we use some very nice statistical machine learning tools that allow you to learn a mapping between a bunch of numbers, which are the mathematical measurements, and how people perceive; and as it turned out, although it didn’t have to be this way, but they correlated extremely well with each other.

Chris -   So, the people give their impression, the computer gives its impression and you can use the people’s impression to validate the computer model, so you know that your computer is returning a value which is meaningful not just to another computer, but is the way that a person would perceive the changes to that picture.

Hany -   That's exactly right and if the mathematical model was not properly constructed, that may not have been true.  I mean, there are plenty of mathematical models that have nothing to do with human perceptions.  So the magic was, how do you develop the mathematical model in a way that you have at least a chance of doing that modelling.

Chris -   And Hany, the idea would be that you could apply your model across the globe and say to people, “Right, if you're going to do some photo retouching, you have to subject it to analysis by our system.”  But, say I was a nifty artist, are we not going to see a game of sort of graphical cat and mouse here where people will begin to manipulate photos in a way that actually can fool your algorithm in just the same way that people make web pages that fool Google’s algorithm?

Hany -   Absolutely, this is a cat and mouse game and I equated to the same of the spam, anti-spam and the virus and the anti-virus and the search internet.  And there is a game, and people will try to game the system, and that does mean that the technology will have to evolve to play that game. I think the most powerful use of the technology is at the voluntary basis as I've said.  I do think that photo retouchers sometimes just get a little carried away without knowing it and I think even just that amount of information will be very useful.  But obviously in the legislative sense, there is going to be a game here and we’re going to have to play that game like we would in any other field.

Chris -   Terrific!  We’ll have to leave it there but thank you for joining us to explain how it works.  That was Hani Farid from the Computer Science Department of Dartmouth College.