Commentary on: Hak. Evaluation of the Forensic Science Regulator’s recommendations regarding image comparison evidence. Forensic science international: Synergy 2019; 1(1)

Abstract

Sir Hak [1] considers the role of the Forensic Science Regulator and whether the Regulatory Notice [2] regarding provision of opinion evidence on image enhancement and image comparison intrudes into the purview of the court. The four principles set out in the notice are as follows: Principle 1 The evidence containing opinion must be admissible in this jurisdiction as expert evidence. Principle 2 The person proposing to give opinion evidence must be an expert in all relevant aspects they intend to give an opinion on. Principle 3 The person giving evidence must comply with all legal obligations including setting out limitations on the evidence. Principle 4 If the expert’s opinion relies on the results of any method the report shall take proper account of matters such as the degree of precision or margin of uncertainty, affecting the accuracy or reliability of those results. All are a restatement of established legal principles in England and Wales and as such, do no more than remind the practitioner of their legal obligations. Of these, only Principle 2, which calls upon an expert to be expert in all relevant aspects they are giving opinion on, appears to be disputed; in particular, subsections b. and c., which are reproduced below: b. Expertise in CCTV, video, imaging, enhancement etc does not equate to expertise on the content of the image. c. Unless they are also an expert in the content of the images, imagery experts must not attempt to give expert opinion evidence on the meaning of a comparison between the objects in question. There is a whole canon of case law on expert opinion. The common theme is that evidence of opinion is only admissible where the judge and jury require the assistance of evidence which depends on the application of specialist skill or knowledge. For example, R v. Cooper [1998] EWCA Crim 2258 states an “expert’s opinion is admissible to furnish the court with scientific information which is likely to be outside the experience and the knowledge of a judge or jury. If, on the other hand, on the proven facts or on the nature of the evidence, a judge or jury can form their own conclusions without help, then the opinion of an expert is unnecessary”. So far as expert witness competence is concerned, the Criminal Practice Directions (CrimPD), reflecting the common law position, make clear that in one of the conditions that govern the admissibility of expert evidence is that “the witness is competent to give that opinion” [3]. The Criminal Procedure Rules (CrimPR) impose obligations upon experts to provide opinions within their areas of expertise [4], to define these areas both in reports and when testifying [5] and, when testifying, to draw the court’s attention to questions the answers to which would fall outside the ambit of those areas [6]. Moreover, the CrimPD, recognising that some expert evidence may lack a sufficiently scientific basis to justify its admission [7], indicate that when the court is determining the reliability of expert evidence the matters it may take into account include “the extent to which the expert’s opinion is based on material falling outside the expert’s own field of expertise” [8]. Finally, the CrimPR require a party serving an expert report to serve alongside it, “notice of anything of which the party serving it is aware which might reasonably be thought capable of— (i) undermining the reliability of the expert’s opinion, …” [9]. These provisions of the CrimPR and of the CrimPD make clear that the English and Welsh courts take the issue of experts giving evidence that falls outside their field of expertise very seriously and expect both experts and instructing parties to make clear where this is the case. This information is important both when the court is determining the admissibility of expert evidence and when it is evaluating the weight of such evidence. In the absence of such transparency from experts, it is more difficult for lawyers, judges and juries to challenge or determine the true scope of expert competence. The Regulatory Notice is intended to align with and support the attainment of these principles in a specific context by helping experts to identify issues that feed into the CrimPR and CrimPD requirements outlined above. R v. Atkins & Atkins [2009] EWCA Crim 1876, when discussing the issue of comparing facial feature in the absence of a statistical database, was clear that an “expert who spends years studying this kind of comparison can properly form a judgment as to the significance of what he has found in any particular case. It is a judgment based on his experience. A jury is entitled to be informed of his assessment. The alternative, of simply leaving the jury to make up its own mind about the similarities and dissimilarities, with no assistance at all about their significance, would be to give the jury raw material with no means of evaluating.” Hak agrees with the conclusion of the Forensic Science Regulator that facial image comparison is an area where specialist subject knowledge is required rather than expertise in any general form of comparison; why, in principle, are other subject areas different? There are many objects within CCTV footage where failure to have specialist knowledge of the similarities and dissimilarities and their significance would render an imagery expert no more qualified than the jury to give an opinion on the meaning of a comparison. In an example known to the Forensic Science Regulator, an imagery expert concluded that two images showed the same vehicle, but an individual with expert knowledge of that type of vehicle was able to identify a difference in the light housing between the two images. This difference meant that the two images could not have been of the same vehicle. An imagery expert who presents opinions on such issues when they do not have the specialist knowledge to do so risks misleading the court and, by being no more qualified than the jury, they could be said to be usurping the role of the jury. R v Coles [2018] EWCA Crim 407 was cited to show that the issue of whether a subject matter expert was required had been recently considered. The prosecution witness gave evidence that she recognised the defendant who she knew sufficiently well for this purpose; this is not expert opinion and is governed by the rules in PACE Code D [10]. From the judgment, it would appear that only in cross examination did the defence expert discuss similarities in the clothing. An item’s colour under particular lighting or camera conditions is an area where imagery experts have specific knowledge and expertise which may be able to assist the court in some cases. In this case, however, the judgment concluded that “the question of linking the appearance and colour of an object which the jury had - the jacket - with the appearance and colour on the video does not require, in [the court’s] view, expert evidence”. This echoes subsection (c) of Principle 2 of the Regulatory Notice, which has mistakenly been interpreted by Hak as a requirement to have a second expert present. If there is a subject matter expert who can “furnish the court with scientific information which is likely to be outside the experience and the knowledge of a judge or jury” [11], their evidence may be required. However, in some cases, once any imagery artefacts have been accounted for, the court may decide that no further expertise is required; this is the position in R v Coles. Hak describes the work of the expert in forensic video analysis, using an example of images of vehicles to describe their work “addressing such issues as aspect ratio, compression artefacts, motion blur, lighting, resolution and other technical considerations”: we concur that these activities are within the expertise of reputable experts in forensic video analysis and are invaluable in enabling the court to understand technicalities that are not within the expertise of lay judges or jurors. Without this expertise, a lay person would be in danger of mistaking a technical artefact for a difference between two images or could fail to appreciate the impact that, for example, missing frames could have made. Hak goes on to describe the comparison phase, again using the example of vehicle comparison, thus: “The expert would then conduct a scientific comparison of the questioned and known vehicles and provide a qualitative opinion as to the relationship between the vehicles.” A scientific comparison is of course what should happen, by a suitably qualified expert. However, in many examples of such work seen by the Forensic Science Regulator, the approach deployed is not a scientifically robust process. Continuing the vehicle example, although noting that similar problems exist across other comparison types, many analysts currently start from a position of attempting to exclude the known vehicle as being the same as the vehicle in the questioned footage, by checking if there are any differences that cannot be accounted for by technical issues. Only if the vehicle cannot be excluded as being that in the questioned footage does the analyst attempt to provide a “qualitative opinion as to the relationship between the vehicles”. This two-stage interpretation is not the best way to carry out an evaluation of evidential strength. One of the problems with first attempting to exclude is that the poorer the quality of the imagery, the less likely is it that the analyst will be able to find sufficient differences to exclude the known vehicle as being that in the questioned footage. Wording such as “despite making every effort to do so, I have not been able to exclude [item x] as being that observed in the [CCTV footage from the crime]” has the potential to be highly biasing, yet it may be that there are very few features in poor quality footage that would be capable of making an exclusion. When the analyst is unable to exclude, they currently then attempt to qualify, using a verbal scale, the strength of support for the two vehicles being the same. This assessment appears to be based only on the quality of the imagery (what features can be seen) and the level of similarity between the vehicles and to be prone to large (and unquantified) levels of variation within and between analysts. There appears, in the majority of instances the Regulator has seen, to be no consideration of the probability of the observed level of similarity if the items are in fact different. Neither does there appear to be any form of calibration of the opinion produced in this way. In order to make a balanced and meaningful scientific interpretation, a scientist must address two or more competing propositions within the framework of circumstances of the case; for each, the scientist must consider the probability of the findings, given the proposition [12]. Returning to the vehicles example, where the propositions could take the form: The vehicle in the questioned footage is the same vehicle as that in the known footage. The vehicle in the questioned footage is some other unknown vehicle. The findings here will consist of a collection of observations on the questioned footage, some of which will be similar to observations on the known footage and some of which will be different. It is necessary to address questions of the form: What is the probability that those observations would have been made if the first proposition were true? What is the probability that those observations would have been made if the second proposition were true? It is the ratio of those two probabilities that is central to the evaluation. We do not expect that the scientist would necessarily have the means to provide quantitative probabilities (and this is the state in many forensic comparative disciplines as they are currently practised, including handwriting, toolmarks and fingerprints) but we do consider it reasonable for the scientist to form a qualitative view about which of the two probabilities is the larger. The greater the ratio between the scientist’s assignments of the two probabilities, the greater the weight of evidence in favour of the former proposition (and, of course, vice versa). The evaluation will be conditioned heavily by the scientist’s knowledge of the domain including, in particular, the proportion of vehicles on the roads that would give rise to features observed in the questioned image. Opinions formed in this way should be subject to calibration by regular participation in proficiency tests using CCTV footage of similar quality to that observed in casework, but where the true identity of the objects in question is known by the organiser. If the expert has no such domain knowledge, he or she is not in a position to provide a balanced evaluation to the court: evaluating only the probability of the findings if the vehicle in the questioned footage is the same vehicle as that in the known footage is unbalanced and has little meaning, even supposing that minor differences between variants of vehicle models are successfully identified. Referring to an unbalanced evaluation of only one proposition as a scientific comparison or opinion will mislead the court. Hak argues that the expert is trained to understand how the imaging process affects the applicable subject matter and to look at class and individual characteristics for both similarities and dissimilarities (much of which we agree) but he then concludes that “the Forensic Scale is used by image comparison experts to properly describe the strength of their opinion.” As set out above, this opinion does not currently appear to be “properly described” or based on scientific principles of interpretation. It lacks, for classes of item not within the expertise of the video analyst, knowledge of the independence or otherwise of class characteristics or the probability of observing the findings under competing proposition(s). Applying a “Forensic Scale” without the appropriate scientifically justified reasoning and then referring to “scientific comparison” risks giving a false impression that the evaluation is scientific when it is not. That is why, as is entirely proper, the Regulator has focussed on the science. Research has demonstrated that an expert in comparison of one class of object does not perform as an expert when comparing different classes of objects (e.g. Ref. [13] for an overview [14]; for a specific example concerning vehicles). It is also essential to learn the lessons of comparison evidence from disciplines such as microscopic hair comparison, where examiners reached conclusions about questioned and known hair samples which went beyond any scientific justification [15]. We have yet to see scientific evidence for the conclusions reached by imagery analysts in respect of vehicles, clothing or any other subject matter with the exception of facial comparison, where there have been a number of studies into the effectiveness of trained and untrained individuals in carrying out facial comparison [16]. Similar research has been carried out on fingerprint comparison [17]. Hak refers to the use of comparison evidence in court for decades, but in no other area of forensic science does a reputable expert attempt to provide an interpretation of a comparison when they have no expertise within that discipline: a fingerprint expert will not conduct comparison of footwear marks unless he or she has been specifically trained in footwear marks and vice versa. The conclusions reached by such experts depend not only on their evaluation of similarities and differences between marks but also on their knowledge of the subject matter: the fact that no differences are observed between two footwear marks does not mean that the marks were made by the same shoe; there must also be consideration of the probability of observing indistinguishable marks if they were left by different shoes. Notwithstanding the disagreements above, in the final analysis, we differ from Hak on only one of three points in his suggested approach. It is indisputable that judges hold the responsibility of determining admissibility but in order that they can do so, the expert is obliged to disclose any and all limitations of their evidence and their expertise. All experts have a legal obligation to confine their opinion to their own area of expertise; it should not be left to the court to uncover a lack of expertise that has not been made clear. We agree that more training is indeed required. We are in agreement that a subject matter expert is required for facial comparison and for determining the type of vehicle, clothing, or other object shown. It is only in relation to a general comparison that we differ. Here, it is possible that a content expert may be required or, depending on the case circumstances, it is possible that all the court requires is for the imagery expert to explain the technical issues with the footage that could make an object appear more similar or more different than a lay person might expect. Beyond that, it may be that the expert is no more qualified than the juror to assess whether or not the items are the same. The difference may in the end be one of emphasis: Hak argues that the issue is one of competence and that the Regulator should mandate training and certification standards. We are in agreement that only those with the correct expertise, using an appropriate scientific method, should give opinion evidence in comparison. Therefore, the Regulator continues to press for compliance with the required standards, which require both demonstration of competence and scientific validity of methods. To reiterate, the proper method for scientific evaluation of evidence is balanced, evaluating the probability of the evidence under at least two alternate propositions. Without knowledge of the subject area, it is difficult to see how an “expert in general comparison” could properly evaluate the probability of their observations under the proposition that the items being compared are not the same; that leaves an unbalanced evaluation based solely on the level of similarity between the items and the level of detail that can be seen in the footage. In conclusion we note that, prior to publication, the Regulatory Notice was subject to detailed consideration by the Forensic Science Advisory Council, which includes representatives from the across the CJS. Declaration of competing interest Gillian Tully is the Forensic Science Regulator.