How do computer graphics work?

Whether you’re playing games, watching films or simply doing work - how does the computer generate the images you see on screen? Chris Smith spoke to Alan Blackwell from Cambridge University, asking him firstly to cast his mind back to the early days of computer graphics...

Alan - The early computers mostly were calculating equations and they were outputting numbers as text, and mostly they had something like a typewriter connected to them. The text that they were outputting, mechanical hammers would be going up and down just like on a typewriter. And when I started working on computers, if I made a mistake in my programme and wanted to run it again, roles of paper would be rolling out and filling up the office while I worked. So there was a huge saving when the text started to come out on the screen which we called a “glass teletype,” because it didn’t have paper coming out of it.

Chris - Is that why it was called graphics because it’s graphos writing and was literally generating paper output?

Alan - Well yes, although certainly for a long time we talked about text and graphics being different. Graphics was pictures and it was very unusual in the 1960s and 70s to ever see a picture coming out of a computer. If you did, it would be made out of typewriter characters that made up the dots of somebody’s face.

Chris - Who made the leap then to a screen from bits of paper?

Alan - The first person to do that was Ivan Sutherland who, in 1963 at MIT, used displays that previously were being used in the Cold War as radar screens to possibly show incoming missiles. Ivan Sutherland had the idea of drawing directly onto the screen of the computer. The ideas that he came up with in order to do that required a huge amount of hardware because we needed to think about where all the individual lines were going, where the individual pixels were going.

Those sorts of graphics didn’t really become available in your home really until the Apple Macintosh or the II were created, the point at which you had one location and memory for every pixel on the screen. Finally you could start putting a picture together pixel by pixel in a bitmap display.

Chris - In two colours because those computers were green weren’t they?

Alan - They did. The Apple Macintosh, yes, every pixel was either on or off because they could only afford to use one bit of memory for each of those thousands of pixels. If you think of the megapixels, that’s what your camera or your phone would use to take a photograph nowadays, we didn’t have enough memory in the computer to store that many. If you wanted to be not just black and white, but to have different amounts of red, green, and blue for each pixel, you needed three bytes for every pixel.

Chris - Is that where we’re at now then? Is that still how it’s done when you’re representing an image on the screen is there literally an addressing to each pixel, so it knows where it is and it’s got a different colour signature for every pixel?

Alan - That’s right. Effectively, every frame of the picture you see on your computer screen, the computer is making up a red picture, a blue picture, and a green picture and then it’s merging them all together at the same time and putting them together so that you get the impression that you’re seeing colours.

Chris - But there are various cheats and sneaky things that clever people like you can resort to to make it so that it’s not so laborious for the computer, isn’t it? There are sort of ways of compressing images so that you actually save space, save memory, and do things faster?

Alan - That’s absolutely right. We can display the pixels quite fast but we don’t have enough memory to store them. If you’re watching a digital video film, then you’ll be seeing 50 frames per second and every one of those frames has got millions of pixels in it. There’s no way that we could store, even with our terabyte of data we might have on our home computer, in order to fit the movies that we see in a terabyte they have to be compressed. If there’s large amounts of blue sky in your picture, then the computer can say well, there’s lots of blue pixels, the next one is pretty much the same as the last one so just imagine that everything’s the same until I tell you differently. And you can see sometimes when this doesn’t work. For example, if you’re using Skype and you don’t have enough bandwidth to send the picture, you see the pictures start to break up because…

Chris - It looks like shoot minecraft, doesn’t it?

Alan - Exactly. It’s assuming that all those blocks are the same. If it had a little bit more data communication then it would be able to say no, hang on there’s more detail inside of here.

Chris - When they plug in the graphics card or the function is taken over by the processor, it’s doing that sort of compression, it’s doing that addressing of the pixels to the right place on the screen, that’s what’s going on under the hood?

Alan - The graphics card is spending some of its time on compression but most of its time, if you’ve got a really powerful graphics card of the kind that’s used for video games, then it’s spending most of its time calculating the geometry of three-dimensional scenes. It’s trying to say all of these objects that might be in your Minecraft world, for example, or perhaps in your shoot-em-up, if you played Minecraft you will have noticed that things in the far distance it starts filling them and when it’s got a little bit more because it takes a long time to calculate all those three dimensional geometry.