Dalek War - DVD Boxset

It is the restoration of episode three of 'Planet of the Daleks' that presented a new challenge to the team - and one which would rely on a combination of several techniques, including one brand-new process fresh from the bleeding edge of image processing technology. Therefore it's only fair that this article devotes the majority of its space to discussion of the fascinating story behind the episode's return to full colour.

Since the mid-seventies, the master broadcast copy of 'Planet of the Daleks':3 has been a 16mm film recording negative, originally created for overseas sales exploitation by BBC Enterprises. In 2006, the Restoration Team approached 2entertain via our commissioning editor, Dan Hall, with a view to having the episode re-coloured by a computer colourisation process developed and in commercial use by Legend Films of San Diego, California. In their previous guise of American Film Technologie, the company had been employed by us in the mid-nineties to artificially colourise a ninety-second section of 'Doctor Who and the Silurians', to bridge a section missing from the off-air domestic recordings used in the colour restoration of that story. Since the re-emergence of the company's main players with the creation of Legend Films in 2001, they have colourised archive film for TV and retail release, as well as providing colourisation technology to the movie industry for such high profile films as 'The Aviator'.

The Legend colourisation begins by the selection of a colour palette for each individual scene - the colour of walls, clothes, skin-tones, objects etc and the creation of single reference frames which are sent for client approval. Once the palette has been agreed, colour is applied manually to a single frame within the shot using a computer-based painting system and then powerful tracking algorithms in the software apply the colours to subsequent frames, tracking the outlines of moving objects through shot automatically. At certain points the system will fall down - when a new object enters frame for the first time or if the shape of the object changes so much that the software loses track - and manual intervention by the operator and the manual creation of a new reference frame are required. It is a painstaking job and one that takes several weeks, even for a single 25 minute television episode.

At that time, computer colourisation was the only option available if we wished to release the story in colour for the first time and Legend seemed to be the most experienced company in the field. The exchange rate was also very favourable at the time, so Dan Hall was successfully able to lobby 2entertain for the funds to pay for the episode to be processed. Ralph Montagu was tasked with looking after the project from our end.

The 16mm monochrome negative was transferred and graded to Digital Betacam videotape by Jonathan Wood, using the Spirit telecine at BBC Resources. During transfer we noticed high levels of residual PAL colour subcarrier  in the picture.  The reason why this gets a special mention here will become clear later.  The residual colour subcarrier manifests itself as a pattern of fine dots ("chroma dots") overlaid on the picture.  These are remnants of the colour information contained within the original colour video signal.  In a video signal, colour is encoded and piggy-backed onto the luminance signal at a high spatial frequency so that a 'composite' signal containing both the monochrome and colour information together in a single video signal can be produced. At this point in the restoration work, the residual subcarrier was nothing more than an annoyance for us - it can form annoying coloured interference patterns on a PAL monitor due to random decoding of the residual colour signal, so as part of the grading work, this 'useless' interfering chroma dot signal was filtered off to provide a clean monochrome image.  There was no system to decode it properly and any system that could would have to overcome seemingly insurmountable problems caused by geometrical distortions and stability problems in the film recorder and the lack of a stable decode reference (which in the original signal would have been provided by a 'colour burst' containing ten cycles of subcarrier, but which is outside of the area recorded onto film).

The film transfer was sent up to SVS, where the usual film dirt and scratch cleanup was undertaken. The clean episode was then rendered out as a numbered frame sequence of approximately 33,000 individual uncompressed 10-bit  720 x 576 pixel frames onto a firewire hard drive and sent off to Legend Films, along with sample colour material from the same sets in the other episodes for them to use as colour references. After client approval of colour palette, a few test scenes were created to our satisfaction and Legend started the task of colourising the entire episode (minus of course the opening titles and credits, which could be created from scratch by us as normal). It should be pointed out that Legend are used to working with 35mm master film material with much more dynamic range than they were presented with here and they admitted to finding the relatively black-crushed film-recorded pictures much more difficult to work with than usual.

Eventually however a second firewire drive containing the colourised frame sequences winged its way back across the Atlantic to SVS, where it was reassembled with new opening and closing titles to form a complete colour episode, and the VidFIRE process applied to bring back the interlaced video motion of the original studio video sequences.

It was always the intention that the Legend version was to undergo a full grade to try to achieve a closer match to the other colour episodes, a process which took four or five hours back at BBC Resources, again with Jonathan Wood. The major problem with computer colourisation is that the colour is generally quite 'flat', losing the subtle hue variations within a person's face, for example. Legend's job was not made any easier by the relatively low dynamic range of film-recorded video, which crushes both black and highlight detail, unlike the movie negatives they are used to dealing with. But at the end of the grade, the finished version was considered to be more than acceptable by all present - and would have been the version released on DVD, if it hadn't been for the presence that day of BBC Information and Archives preservation specialist James Insell and the remarkable contents of his laptop...

James Insell's name will be familiar to anyone who has followed the history of our restoration work in the last decade. Formerly an engineer at the BBC's R&D facility at Kingswood Warren in Surrey, he had been instrumental in bringing Jim Easterbrook's 'Reverse Standard Conversion' process to a form that could be used in the real world - see the article on 'The Claws of Axos' for more details. Some years ago, whilst watching a UK Gold broadcast of another Doctor Who episode that now only exists as a monochrome film recording, he had seen strong patches of spurious colour breaking through and realised that it meant that at least some of the colour subcarrier was still present in the film recording. He became interested in exploring the possibility that it might be possible to extract this information and decode it back to the original colour - see the article on 'The Ambassadors of Death'. After conducting some initial investigations, he attempted to launch an R&D project at Kingswood Warren, but the idea met with some scepticism and available R&D effort was scarce.  Undeterred, he set up an external working group based around a Wiki. Calling themselves the Colour Recovery Working Group, he invited interested parties to join him with ideas for how such a process might be realised. To aid the research work, a high resolution scan of a section of a 'Top of the Pops' film recording was made available to researchers, along with a surviving clip of the same sequence from colour VT to provide a reference.

Eventually, group member Andrew Browne was able to demonstrate a technique which could recover some of the colours in the picture with reasonable accuracy, and - importantly - without needing to recover the original line structure. This attracted some press publicity, and also a new member to the group - Andrew Steer. He started his own experiments and was soon able to improve markedly on the results Andrew Browne had achieved. However, the technique was still able only to reproduce a subset of the colour gamut, in particular blues and purples eluded it. Encouraged by the progress made, the idea was taken forward by another member of the group, in the form of ex BBC R&D engineer, Richard Russell, who realised that there was a way to recover the full gamut of colours from the film recording - an astonishing achievement. The implementation of Richard's technique was written in BBC Basic for Windows, a language he had developed as a continuation of his professional work on the popular home and educational BBC Microcomputer produced in association with Acorn in the eighties. Richard had also been responsible for taking Jim Easterbrook's research and simulation work on the Transform PAL Decoder and turning it into a working hardware solution - and in doing so provided us with a device which has greatly improved the picture quality of the Doctor Who DVD range over the years.

The intricacies of his process go well beyond the scope of this article, but the basics are these: A 1920 x 1080 horizontally-stretched HD video scan of the film recording is made in order to fully extract all the information contained in the standard definition film image. The chroma components are separated away from the underlying luminance, and the frames are analysed to provide a map of geometric errors in the film recording, which is vital to the correct colour decoding. By cleverly exploiting a mathematical property of PAL colour encoding, Richard's system can identify and correctly decode the colour in most of the picture automatically, at a rate of about 15 frames per minute. Manual intervention during the decoding process allows areas of colour that the software is unsure about to be corrected into the correct colour quadrant.  Areas which are subsequently found to be incorrectly decoded can then, with use of a Quadrant Editor tool, be manually flipped into one of three other possible states, one of which will be the correct one. The output is standard definition, down-scaled from the HD files using Richard Russell's own scaling algorithms. The process has been named Colour Recovery and will be referred to as CR for the rest of this article. 

It should be noted that the accuracy and stability of the output from the CR process is highly dependent on the individual properties of the film recording, sometimes in ways that are not yet fully understood, but will involve variables such as film recorder focus, spot-wobble, and stability. Low saturation colours may not decode correctly or indeed at all, requiring much operator intervention in the CR stage or manual retouching further down the line. Cross-colour effects (in which genuine luminance detail is misinterpreted as chroma information) is a problem, and positional differences between objects in the field-pairs in the original video (for example somebody turning their head quickly) can cause decoding errors in those areas - so moving objects can appear more colourful than they were when stationary. Focus fluctuations in the film recording or telecine process produce a variation in the level of recovered subcarrier, which again translates to a variation of chroma saturation. However, none of these problems are insurmountable given enough time and skilled retouching further down the line.

So, back to James Insell and the laptop he had brought with him to the grade of the Legend version of the episode... It contained a video file of the output of Richard Russell's CR software applied to a HD scan of the 'Planet of the Daleks':3 negative. At this point, for quick demonstration, it was the automatic pass through the CR process, before any manual editing to correct incorrectly decoded colours. However, it provided an excellent colour reference for some of the more obscure objects in the picture - for example the true colour of the Doctor's screwdriver and the flashing indicator lights on the Dalek control panels. It also became clear that, although quite noisy, the colour information contained in objects such as people's faces contained a lot of subtle hue variation that was missing from the flat, computer-generated colour coming out of the Legend process.

Clearly we now had two entirely different sources of colour to work with. Legend's colour was clean and stable, but lacked the subtle tonal variation that would really sell the image. The CR colour was unstable and noisy, but it contained much of the subtlety that was missing. Richard Russell was asked to CR process the episode and use his skills and Quadrant Editor to produce a better version that Peter Crocker could use to manually correct any mistakes he found in  Legend's colourisation (such as the aforementioned screwdriver), but also to try to blend some sections of the CR into the Legend version to enhance realism in shots such as close-ups of faces if possible. At least, that was the original intention. In then end, blending processed CR material back into the flat, stable Legend pictures proved so successful and added so much realism to the pictures that Peter used it to enhance just about every shot in the episode! The finished version takes the best features of two entirely separate colour processes and couples them with the VidFIRE process and the beautifully restored soundtrack to produce a final output that most people would be hard pushed to distinguish from genuine videotape.

Below are two examples - click for a full-sized image. On the left is the graded Legend colourisation, in the middle picture is the CR version (ungraded with arbitrary saturation) and on the right is the final graded composite of the two.

On this image, the amount of cross-colour generated by the fine patterning on Jo's hair and jacket can clearly be seen. This demonstrates one of the difficulties in using CR chroma alone - it can be problematic in areas containing fine luminance detail. However, unlike the Legend process, it can provide excellent soft tonal variations such as in the fleshtones of Jo's face and the different shades in her hair.

The second example is a revised version of the one originally posted in this article. Initially, the red light pouring down from the chimney had been much reduced in order to match better to the same set at the beginning of episode four. However, it was pointed out that the red lamp was almost certainly removed or repositioned between the recording of these episodes to allow the actors to ascend the chimney. Therefore we revisited that sequence and blended back much more red from the CR process.

We were amused to note that Legend's artists had obviously put a great deal of time and effort into creating a beautifully variegated jungle, complete with purple flowers, however when we checked the CR version of the same shots we found that the original BBC designer had put rather less effort in - it was all simply green!

Much of the colour that we were able to put back in from the CR version was in the form of non-obvious variations that really helped to sell the realism of the shot - the green hues of the jungle reflecting on the Thal's faces, the red-light spilling down out of the cooling plant chimney in the final scenes, for example.

• The Space War (dur. 17' 57") - cast and crew look back at the making of the story. Featuring actors Katy Manning, Vera Fusek and Michael Hawkins, visual effects designers John Friedlander and Mat Irvine. Produced by Steve Broster.

• Roger Delgado: The Master (dur. 32' 35") - a biography of actor Roger Delgado, most famous for playing the Doctor's arch-nemesis The Master from his introduction in 1971 until his tragic death two years later. Featuring previously unseen photographs, rare excerpts from his many BBC TV appearances and interviews with those who knew and loved him. With his wife, Kismet Marlowe, producer Barry Letts, script editor Terrance Dicks, director Christopher Barry, actors Katy Manning, Linda Thorson, Harry Towb, Frazer Hines, William Gaunt and Damaris Hayman, stunt coordinator Derek Ware. Narrated by Stephen Greif. Produced by John Kelly.

• Stripped for Action: The Third Doctor (dur. 16' 05") - the ongoing series looking at the Doctor's comic book adventures reaches his third incarnation. With former Doctor Who Magazine editors Gary Russell and Alan Barnes, comic historians John Ainsworth and Jeremy Bentham, author Paul Scoones.

Planet of the Daleks:

• Commentary wth actors Katy Manning, Prentis Hancock and Tim Preece, producer Barry Letts and script editor Terrance Dicks.

• Perfect Scenario: The End of Dreams (dur. 30' 04") - Continuing his search for inspiration, scenariosmith Zed resumes his studies of twentieth century television's Doctor Who. What he finds will have a profound effect on the lives of all of the remaining sleepers in The Field of Dreams… Featuring interviews with actors Katy Manning, Jane How, Janet Fielding, Bernard Horsfall and Tim Preece, producer Barry Letts, script editor Terrance Dicks. With Rich Batsford, Mick Broster, Tony Broster and Paul Ewing, and the voices of Steve Broster, David Harley and Cathryn Miller. Written by David Harley.

• The Rumble in the Jungle (dur. 16' 46") - cast and crew look back at the making of the story. Featuring actors Katy Manning, Jane How, Bernard Horsfall and Tim Preece, director David Maloney and designer John Hurst. Produced by Steve Broster.

• Multi-colourisation (dur. 10' 47") - a look at the colour restoration of episode three. With producer Barry Letts, Colour Recovery Working Group leader James Insell, 2entertain commissioning editor Dan Hall and colourist Jonathan Wood. Narrated by Glen Allen. Produced by Ed Stradling.

• Stripped for Action: The Daleks (dur. 13' 54") - the ongoing series looking at the Doctor's comic book adventures focuses on his deadliest foes - the Daleks! With TV producer Gerry Anderson, former Doctor Who Magazine editors Alan Barnes and Clayton Hickman, comic historian Jeremy Bentham and author Paul Scoones.

• Blue Peter (dur. 12' 34") - two items from the childrens' magazine programme, featuring an appeal for any information on the theft of two Daleks from the BBC and their subsequent return.

• Plus of course the usual PDF materials (including Radio Times listings, BBC Enterprises sales materials and production drawings), Coming Soon trailer, Programme Subtitles, Subtitle Production Notes and Photo Galleries.