The Art and Science of Digital Visual Media Forensics

One eye-witness weighs more than ten hearsays. Seeing is believing, all the world over.- Plautus

Mankind has a long history of using pictorial representations of information as a way of disseminating and assimilating knowledge and among all pictorial representations; nothing bears more verisimilitude than photographs and videos. As direct expressions of reality, photographs and videos present an image of the world that influences human opinion the same way real-world stimuli do.

Photographs and videos are regarded as the most accurate ways of documenting people and objects, and the visual information they provide acts as an unbiased and universal record of occurrence of events, which is often used as evidence for making critical decisions and judgments in highly sensitive areas such as journalism, politics, civil litigations, criminal trials, defense planning, surveillance and intelligence operations. The forensic applications of photographs were actually recognized almost as soon as photography was invented.

After the invention of heliography and the creation of the first permanent photographic image in 1826 or 1827 by French inventor Joseph Nicéphore Niépce, several attempts were made to commercialize photography. But it wasn’t until the invention of daguerreotype cameras by French artist and photographer Louis-Jaques-Mandé Daguerre, that camera manufacturing became an industrial procedure [1]. When these cameras became available to the general public in 1839, photography’s potential for identification and documentation of the criminal classes was recognized, making photographs a widely acceptable forensic means of identification. The earliest evidence of photographic documentation of prisoners dates back to 1843 in Belgium. By 1848, police in Liverpool and Birmingham were photographing criminals and by the mid 1850s, English and French authorities had begun encouraging their law enforcement agencies to photograph prisoners, mostly to prevent escapes and document recidivism [2]. The forensic photographic process was standardized in 1888 when Alphonse Bertillon, a French police officer and biometrics researcher, suggested anthropological studies of profiles and full-face shots (which later came to known as mug shots) to identify criminals [3].

A photograph captures a single moment in time and is essentially an arrested representation of reality. A video however, presents a record of an event over a period of time, making it a much more potent forensic evidence that still photographs.

In a world with a plethora of multimedia capture devices and ubiquitous surveillance, video data has emerged as a truly indispensable source of information. During a criminal trial for instance, video evidence, if available, is considered particularly inculpatory. Unlike other kinds of forensic evidence like DNA and fingerprints, which are circumstantial in nature and require further inference, videos provide a first-hand eyewitness account of an event.

The first documented instance of use of video footage during a criminal trial was the Rodney King case of 1991. King, a taxi driver in Los Angeles, was beaten by four LAPD officers following a high-speed car chase on March 3, 1991. A witness named George Holliday videotaped much of the beating from his balcony and sent the footage to a local news station [4] (Figure 1).

Figure 1: Snapshot from the video footage shot by Holliday showing four LAPD offices assaulting King. [Picture Courtesy of PBS].

Holliday’s footage was used as evidence during the ensuing trial where the officers were charged with assault with a deadly weapon and use of excessive force. Despite the video evidence, the officers were acquitted and within hours of the acquittals, the six-day 1992 LA riots broke out, in which 63 people lost their lives and over 2,000 received non-fatal injuries [4].

While the Rodney King case demonstrated an instance of use of a home video as evidence, these days, most of the video evidence admitted in courts comes from CCTV cameras and while CCTV surveillance had begun as early as the 1950s in minor capacity in the UK and US, it wasn’t until the Bulger case of 1993 that CCTV’s investigative benefits were truly recognized.

On February 12, 1993, James Patrick Bulger, a two-year-old boy from Merseyside, England, disappeared from the New Strand Shopping Centre in Bootle, where he had been shopping with his mother. After he was reported missing, tapes from CCTV cameras around the crime scene were examined, one of which showed Bulger being led out of the shopping centre by two ten-year-old boys, Robert Thompson and Jon Venables, who murdered him later that day [5] (Figure 2).

There have been numerous cases over the past years where video evidence has led to successful convictions and has brought justice to those in need. From the David Copeland (London Nail Bomber) case of 1999 [6] to the Manchester Arena bombing of 2017 [7], surveillance videos have demonstrated their forensic utility time and again.

Video evidence has been equally beneficial for the exposition of social injustice. Whether it’s a video of child soldiers or police torture of prisoners, or footage of human rights violations or wildlife abuse, video evidence can accomplish a lot from initiating investigations to expediting cases in courts. For instance, footage of the 2012 Houla massacre in Syria called attention to the incident and led the UN to call for a special investigation [8]. Video evidence played a critical role in the Endorois Welfare Council‘s human rights violation case against the Kenyan government in 2009 [9]. Similarly, videos documenting human rights violations and war crimes during the Sri Lankan Civil War in 2014 were used by the UN High Commission to initiate several investigations into the matter [10].

All these and numerous other cases are demonstrative of the fact that when used in an evidentiary capacity, photographs and videos can be remarkably influential, and in a world where visual evidence holds such power over our decision-making faculties, we must ascertain that this evidence is in actuality what it purports to be, because as compelling as photographs and videos are, the picture they paint may not always be accurate.

Digital content is inherently susceptible to manipulation, and the possibility of tampering is especially worrisome in situations where this content is being used as evidence to make decisions and judgments that have far-reaching consequences.

Content tampering is not a recent trend; earliest known instances of photo tampering can be traced back to 1860, i.e., within half a century of the invention of photography itself (Figures 3 & 4).

Figure 4: In this doctored photograph.

(a)(c.1937), Joseph Goebbels (second from the right) was removed from the original picture.
(b) After he fell out of favor with Hitler. [Pictures Courtesy of Four and six Technologies Inc.].

 

These examples illustrate pre-digital era tampering. When photos were recorded on film, manipulation was a task usually performed by skilled photographers in specialized photo labs. In the digital era, the widespread availability of inexpensive yet sophisticated image editing software like Adobe Photoshop, Photo Scape, and Phantasmagoria, has made content manipulation even easier to perform, even by novice individuals with minimal skills. Figures 5 and 6 depict some instances of tampered digital content that surfaced in the media world in the more recent times.

Figure 5: In July 2011, the Associated Press had to withdraw a news photograph supplied by the Korean Central News Agency, after it was discovered to be a digital composite. The photograph, which depicted North Korean citizens walking through high floodwaters, was almost instantly debunked when it was noticed that despite all the water, people’s clothes were not wet. It was speculated that this photograph was an attempt to gain sympathy for North Korea so that it could receive more international aid. [Picture Courtesy of Four and six Technologies Inc.].

Figure 6: In November 2014, a pro-Palestinian Facebook group posted a doctored photograph of gaunt inmates of a Nazi concentration camp holding photo shopped signs bearing messages that castigated Israel and expressed support for Palestinians in Gaza. [Picture Courtesy of Four and six Technologies Inc.].

All these examples of content manipulation bear witness to the fact that while a picture may be worth a thousand words, those words may not necessarily be true. Moreover, this issue of content infidelity is not restricted to digital images alone; integrity of video data too cannot be taken for granted any more. In the past, we had fewer reservations about accepting videos as truthful representations of reality, partly due to the inherent complexity of video processing and partly because of the general lack of readily available high-tech video processing tools. But lately, the proliferation of easily attainable and powerful video editing software like Adobe Premier, Light works and Video Edit Magic, has made it possible for digital videos to be altered without any significant effort, even by non-darkroom experts.

The missing footage cases of July 2005 police shooting of Jean Charles de Menezes in London [11] and Kendrick Johnson murder case of January 2013 in Georgia, US [12], to the cases of footage tampering as witnessed during the trial of Bosnian war criminal Radovan Karadžic? in 2006 [13] and in the Sandra Bland case of 2015 [14], bring to pass the somber realization that video evidence, like photographic evidence, is not infallible.

In this day and age, use of visual evidence in one capacity or another is simply inevitable, and the susceptibility of this content to manipulations does not preclude it from being admitted as evidence. The indispensability of digital visual evidence is best exemplified by two rulings: “Merely raising the possibility of tampering is insufficient to render evidence inadmissible” (US v. Allen 106 F.3d 695, 700 - 6th Cir. 1997) and “The fact that it is possible to alter data contained in a computer is plainly insufficient to establish untrustworthiness” (US v. Bonallo, 858 F. 2d 1427, 1436 - 9th Cir. 1998).

Thus, in situations where dependence on photographic or video evidence is unavoidable and where reliance on tampered evidence could be detrimental, it becomes paramount to first examine the trustworthiness of the given evidence, before placing complete faith in the fidelity of its contents. And since subjective inspection fails to provide the desired degree of conviction regarding content authenticity, specialized digital forensic tools and techniques have to be relied upon, which are provided by the research field known as digital visual media forensics.

Evidence collection

Figure 7: Primary tasks constituting the process of visual media forensic analysis.

Forensic evidence enhancement

Evidence authentication and the concept of identifying traces

Evidence provenance:

Visual Content Authentication:

Content enhancement versus content tampering:

Variants of Digital Visual Media Forensics:

So far, we have considered digital visual media forensics in a general setting without making any assumptions about the kind of access (if any) the forensic analyst has to the components of the image/video generation process, or the availability of any a priori knowledge vis-à-vis the processing history of the given content. Depending on the characteristic features exhibited by a particular forensic scenario, digital visual media forensics can be categorized according to the scheme presented in figure 8.

Figure 8: Categories of digital visual media forensic techniques.

Active and Passive Forensics: 

• Active Forensics: In active forensics, certain identifying traces are either attached to the content in the form of metadata (such as a hash or signature), or are inserted directly into the content (such as a watermark), at an early stage of its generation. Active forensic methods are implemented directly in the acquisition device and the components of the content generation process cannot be inferred until the identifying traces are inserted into the content. This implies that active forensic methods do not allow for the evaluation of the trustworthiness of arbitrary photographs/videos of unknown origin.
  
  
• Passive Forensics: In passive forensics, the analyst has no control over the image/video creation process or the type and/or appearance of the identifying traces. The analyst is also unaware of the specifics of the content generation process and its processing history, and is confined to authenticating the content by inspecting the characteristics it displays. Passive forensic methods rely on two kinds of identifying traces: device characteristics and processing artifacts.

Device characteristics refer to the intrinsic variations in different acquisition devices, which can occur for a variety of reasons, such as different camera manufacturers using different components in their devices, or due to them adjusting the parameter settings of their devices in different ways. Variations can also occur due to unwanted technological imperfections such as sensor defects. Consequently, every acquisition device leaves unique identifying traces on the content it generates, and by studying these traces, inferences can be made about the device itself.

Processing artifacts refer to those traces that are introduced by the various processing operations that an image/video undergoes after it has been generated. Such artifacts are unique to each processing operation, and thus serve as identifying traces for the discernment of that particular operation.

It is important to understand that while both active and passive forensics rely on the detection of identifying traces, in case of active forensics, these traces are embedded or attached to the data intentionally, whereas in case of passive forensics, these traces are introduced by the inherent characteristics of the content acquisition process or the various post-production operation it undergoes during its lifetime.

Blind and non-blind forensics

• Blind forensics: A digital forensic process is called blind if the analyst works in the complete absence of any a priori knowledge regarding the recording device, the content generation process, the original scene captured by the image/video, or any of the processing operations that may have been applied to it. Blind forensic schemes examine the identifying traces present (or absent) in the given content and try to infer the identity of the device that may have been used to capture it or the post-production operations it may have been subjected to.
  
  
• Non-blind forensics: Unlike blind forensics, non-blind forensic methods use additional information about the characteristics of the content generation and processing operations. Such additional information may include knowledge about the identity of the recording device or information regarding the processing history of the content. Identifying traces embedded at the time of content creation are also utilized during non-blind forensics.

Though non-blind approaches help mitigate some of the uncertainty the analyst may face (regarding whether or not the image/video has undergone any post-production processing), they are often infeasible in practical situations. For instance, a non-blind approach cannot help determine the origin of an image/video of unknown origin, because for such an image/video, there is no additional information available aside from the contents of the image/video itself.

Identification and interpretation

As documented representations of reality, photographs and videos facilitate post-event analysis by offering potent forensic evidence, which we are becoming increasingly dependent on for making sensitive and momentous decisions in politics, journalism, surveillance, intelligence, civil and criminal litigations. Photographs and videos are especially influential because they provide visual evidence of occurrence of a real world event, which has a kind of authority over our perception that is surpassed only by firsthand real-time assimilation of that event. In a world where visual evidence holds such significance, it is imperative that we be certain of its integrity and trustworthiness before accepting it as an accurate portrayal of reality.

Digital visual media forensics is a chain of scientific inquiry that supports digital image and video authentication and integrity verification, the fundamental goal of which is to preserve the given content in its most original form, all the while conducting a structured investigation that enables validation and interpretation of the information it conveys.

1. John Hannavy (2008) Encyclopedia of nineteenth-century photography: A-I index (Vol1). Taylor & Francis, Abingdon, UK.
2. Robin Lenman (2005) Police and Forensic Photography, the Oxford Companion to the Photograph. Oxford University Press, Oxford University Press, England, UK.
3. Rhodes HTF (1956) Alphonse Bertillon, father of scientific detection. Abelard-Schuman Ltd., London, UK.
4. Independent Commission on the Los Angeles Police Department (1991) The Rodney King Beating. In: Christopher W (ed.). Report of the Independent Commission on the Los Angeles Police Department. Diane Publishing, ollingdale, USA.
5. Smith DJ (1994) The Sleep of Reason: The James Bulger Case. Century Arrow Books, UK.
6. Hopkins N and Hall S (2000) David Copeland: a quiet introvert, obsessed with Hitler and bombs. The Guardian,UK.
7. Sephton C (2017) Manchester Arena explosion: What we know so far. Sky News, UK.
8. Jim Muir and Leana Hosea (2012) Syria crisis: Houla massacre leaves 88 dead. BBC, London, UK.
9. Witness (2016) Video as Evidence: Field Guide. Witness, Brooklyn, New York, USA.
10. UN News Centre (2010) Deeming Sri Lanka execution video authentic, UN expert calls for war crimes probe. UN News Centre, USA.
11. Morris N, Bennetto J, Clement B (2005) Police and Tube firm at odds over CCTV footage of innocent Brazilian's shooting. The Independent, London, UK.
12. Blackwell V (2013). Kendrick Johnson footage released; expert finds it 'highly suspicious. Cable News Network, Georgia, USA.
13. Blaszczyk T (2010) The investigation of the Srebrenica videos made by Belgrade journalist. Sense Tribunal, The Hague, Netherlands.
14. Worland J (2015) Sandra Bland Arrest Video Appears Edited, Time, UK.
15. Petherick WA, Turvey BE, Ferguson CE (2010) Forensic Criminology. Elsevier Academic Press, Netherlands.
16. ABC News (2006) Nanny Cleared of Violently Shaking Baby. ABC News, Australia.