What we see when we look at medical images

Discussions of medical imaging technologies often leave the connection between imaging, visualization, and vision ambiguous, which fails to explain the visual and technological nature of the images. I argue that this is because the standard way that account of image interpretation are constructed gets the explanatory order wrong; rather than looking to image use to explain interpretation, image use should be explained by image interpretation. An account of imaging that takes visual interpretation as primary has more explanatory power and provides new tools for analysis and critique of medical imaging.


MEDICAL IMAGING TECHNOLOGIES
Visual representations have come to play a central role in contemporary medical practice.Unlike the hand drawn illustrations, diagrams, and graphs that dominated 17 th to early 20 th century science, imaging today is most often technologically produced, computationally processed, and digitally rendered.Contemporary technologies are also different from other diagnostic and exploratory tools in that they rely neither on global observation the way a clinical examination does, nor on measurement of a single dimension of the body the way biopsy and blood tests do.Imaging technologies are used to investigate bodily states through the production and interpretation of visual representations of the body, which I will henceforth call images.
Images can play investigative and diagnostic roles, but can also be used to perform procedures and track the effectiveness of those procedures; image types and uses are diverse.X-ray, electroencephalogram (EEG), magnetic resonance imaging (MRI), functional MRI (fMRI), sonography, Doppler, and computed tomography (CT) can be differentiated because they use different probes to measure different states of the body.Anthony Wolbarst describes the relationship: they share a fundamental communality of approach: they create medical images by following and recording, by some means, the progress of suitable probes that are attempting to pass through a patient's body.The body must be partially, but only partially, transparent to the probes.If the probes slip right through bones and organs without interacting with them, like light through a pane of clear glass, no differences among the tissues can be visualized.Different probes, different interactions with the tissues, and different means of detecting the probes give rise to different images conveying different types of clinical information.(Wolbarst 1999) They are furthermore used in different contexts to perform different imaging tasks and so the images produced have to be understood as products of the technology in use (Lopes 2005).For this reason, images cannot be interpreted in exactly the same manner even if they look similar.CT images and radiograms both use x-ray, but radiograms image the total absorption of rays through tissue while CT imaging reconstructs absorption rates in a slice of the body which helps to get around the skull.This does not yet tell us what images are, or what makes them different from biopsies and other medical exams?What relationship does the image have to the body as represented, and as visualized in medical practices?An explanatory account of medical imaging should explain how images are made, interpreted, and used; but it should also account for the differences between image types, and between images and other kinds of representations.What images have in common is that they require primarily visual, rather than numerical or linguistic interpretation.How we understand the bodily state represented in the images depends somehow on what we see in the images.

Imaging as a visual technology
Often in discussions of imaging, seeing is treated elliptically, as if it were a metaphor without explanation, or is placed inside scare quotes, "see," to signify a non-literal use of the term.This convention crosses disciplines; it is neither a philosophical nor a scientific peculiarity.Unfortunately, it does nothing to explain the relationship between imaging technologies, images, the bodies visualized, and our visual experiences of them.The sense in which "seeing" is like or unlike ordinary seeing is left only ambiguously related to vision, which serves to muddy related concepts such as 'visualization'.
What is required is an account of how images mediate between perceivers and bodies.We cannot make sense of the role of imaging in medicine without making sense of how making visual representations of certain kinds allows us to extend our abilities and to perform new tasks.Image use and image interpretation are closely connected, our ability to grasp the content of an image allows us to understand the image as being about its subject, and to thereby have certain thoughts, ideas, beliefs, or attitudes about that subject.The account that I develop aims to clarify the relationship between vision and imaging with minimal mutilation of our intuitions, while preserving as much of the literal sense of seeing as possible.

IMAGE INTERPRETATION
Image diversity can partially explain the disconnect described above between seeing the image and visualizing the body.The differences in appearance, use, and interpretive practice between EEG and MRI makes it seem a stretch to say that we see anything similar when we look at them, even if both are representations of the same brain.
Yet diversity is not the lone culprit.If the EEG in Figure 1 is compared with the MRI in Figure 2, the difference in imaging is not just in the kind of technology that produced the image, but in how we grasp their contents.We see the brain in the MRI in a different way than we see it in the EEG.The technological nature of images, and the startling visual clarity of some of them simultaneously demand and reject their relation to vision.The difference between seeing a brain in the two image types needs to be explained.
A feature that all images share is that they use spatial properties to convey information about something else (Perini 2005).Image interpretation, in general, is a matter of understanding what an image represents, or how the visible form of the image relates to the content of the image.Image content is generally understood as what an image attributes to or says about its subject.It isn't enough to say Image X represents subject S, since any image also represents its subject as being some way or another.We can take the general interpretive goal for images to be grasping how an image represents its subject as thus-or-so, given the imaging task and context of use.In a medical context this is often done to extract diagnostically or clinically useful information.

Two views of interpretation
There are two general approaches in the study of visual representation.One is resemblance or similarity; an image represents its subject in virtue of resembling it or being similar to it in some way.The second is convention; an image represents its subject in virtue of there being practices that establish how one thing comes to be about another, or what the relation between form and content is (Perini, 2005).
Resemblance is problematic, because there are numerous ways in which things can resemble each other, most of which are not helpful for interpretation.While it can seem intuitive that a portrait of someone sitting in their easy chair depicts them thus by resembling them in appearance, explaining the content in terms of "likeness" faces well-established problems (Goodman 1976).First, the flat picture surface does not in fact resemble a three dimensional scene, and neither do the marks on the surface.Second, whether the portrait is black and white or coloured it is unlikely to resemble the scene in colour.We would not say that a picture depicts its subject as sitting in their easy chair because it resembles it in colour.These basic differences can be overlooked due to how easily we are able to grasp the contents of some images."Naturalistic" images, photographs, and similar images seem to present their contents in a straightforward way, but our facility in interpreting them should not be taken as evidence of an easy relation.Experiencing a picture as like its subject involves ignoring numerous dissimilarities, and following criteria that establish likeness, realism, and misrepresentation.
Convention accounts of pictorial representation argue that certain picture types acquire their meanings through correspondence rules that determine conventions of realism or naturalism.The realism that we experience some pictures as having is a convention of how detailed marks on the surface are related to certain subjects and contents (Goodman 1976).It is no more visually necessary than other conventions, there are detailed, naturalistic images of dragons and unicorns that undermine the connection between realism and reality.Different correspondence rules can produce different meanings of the same image, depending on how it is being used.

Interpretation from use
In discussions of medical imaging, image interpretation is often explained in terms of image use.Images are used to represent subjects as thus-and-so, by establishing conventions in which certain image features come to stand for features of the subject.Radiologic signs are characteristic appearances of bodily states under imaging conditions, which radiologists use to perform differential diagnosis.Part of the practice of radiology is being able to recognize these signs and knowing the conditions under which they appear.The "bamboo spine" sign is indicative of Ankylosing spondylitis, as you see in Figure 3.In radiographs a spine with this condition has the appearance of a strand of bamboo, and textbooks even display images of bamboo beside images of the radiograph in order to emphasize the resemblance (Mulligan 1997).The use of signs by radiologists gives us a reason to treat image interpretation in terms of understanding how certain series of marks become correlated with disorders.

Figure 3: Later lumbar spine x-ray showing Ankylosing spondylitis. Image Wikimedia Commons
There is more to this understanding of interpretation, as we can see in the debates over obstetric sonography.The swirling grey scale marks of an obstetric sonogram track echoes within a womans body, and these marks have been used to signify a pregnant body, gestational age of a foetus, a baby, or even "your beautiful baby girl" (Mitchell 2004).A blurry series of light and dark areas on a screen does not "look like" a baby, so how it comes to accept that interpretation must be understood in terms of the conventions for relating the form to the content of the image.

Figure 4: Obstetric sonogram at 12 weeks gestation. Image credit Kelly Gray
What we see in debates over ultrasound is that the subject itself is contested.The echoes measured along two-dimensional planes of a woman's body are translated into a grey-scale image that only makes the interior of the body comprehensible through understanding the technology of ultrasound, and the translation of sound into visible marks.The marks on the surface coming to be about the foetus and not the woman whose body is being scanned depends on a number of historical and social contingencies concerning pregnancy, women's bodies, and images of the foetus (Duden 1984;Petchesky 1987).For the marks on the screen to resemble a foetus or a baby, requires a specific kind of interpretation of the image surface.
The mapping between form and content differs depending on what the subject is taken to be.Antiabortion films such as The Silent Scream take this to one extreme, interpreting the movements on the screen to be evidence of pain and fear in a way that signifies personhood.The same movements could be interpreted as non-intentional jerks and kicks (Petchesky 1987).
The control of image interpretation is contested, and infused with power relations.The current implication of sonography in reproductive politics in the United States speaks to these differences in interpretation.What does a foetal sonogram mean?Who should have one performed, and who should see what in it?Just as states are passing laws requiring women to undergo and view sonograms as part of the informed consent for abortion, expectant mothers are fighting to see more.Debates over the 'entertainment' aspect of obstetric sonography concern women's anticipation of the procedure and simultaneously dismiss women's desires for information and for visual access to their bodies by reducing them to entertainment values (Simonsen, Branch, and Rose 2008;Burthaw 2004;Mitchell 2001).
How images are presented to patients, how many images someone has done and how the quality of those images is determined, betray the extent to which interpretation is not straightforward.Sonographers can withhold certain interpretations from women that they think are not "good" mothers, or can guide their ability to resolve the surface marks into "baby's first picture" (Mitchell, 2004).
Like other representations, images are not natural in the sense that they do not merely replicate the objects of visual awareness in a novel medium.As such, their production and use is often understood in terms of a multitude of "texts" that place certain demands on viewers or interpreters.Visual texts relate images to other kinds of artefacts and bodies of knowledge, the way obstetric sonograms have been tied to images of astronauts in space (Petchesky 1987).Medical texts privilege vision over other senses and that normalizes conceiving of the body in slices, the medical gaze transforms an individual into a patient and shifts authority and autonomy from their lived experience to that of medical science.
Social texts define certain objects as matters of concern for certain groups, and constrain how images should be read, and moral texts normalize certain findings and certain responses.Women are expected to be pleased to view their babies on ultrasound screens, even if they are disappointed that they could not resolve the swirling blobs into a view of their baby (Mitchell 2001).Patients should continue with hope in the case of positive exams for cancer.All of these factors can influence the way that image interpretation is shaped and constrained, including how it is normalized in medical practice.This approach to image interpretation considers how the content of images is contingent upon a social context.Images are understood in terms of how they are used, and their content is understood as likewise dependent on the particular convention of visuality.Interpretation works from how images are used in contexts of diagnosis, social control, moral critique to what subjects the images take and what they say about those subjects.This process of interpretation explains what different people see when they look at medical images.An expectant father might see an ultrasound as a blur when he tries to use it to see the foetus inside his partner's body, while an MD might see a telling sign of disorder; one is viewing within the medically established correspondences for visualization and the other is not.
Though it rigorously captures the contexts of imaging tasks and image use in a way that connects the form and the content of diverse images, I think that there are reasons to resist accepting this as adequate for an account of medical imaging.For one, it doesn't establish how seeing in medical images relates to our ordinary use of 'see'; it explains many ways in which our experience and interpretation can differ but does not establish what we see.Furthermore, treating interpretation in this way can lose sight of the fact that images are inseparable from the technologies of imaging, which are not merely about image production but have their own histories and contexts of development.Examining these technologies allows us to approach image interpretation in a different way.

Use from interpretation
Medical imaging technologies produce two different things.They produce measurements of nonvisual features of the body such as brain waves, echogenicity, and tissue density, which can be clinically important.They also produce images: marked surfaces whose differential visual patterns represent states of the body measured.Since the features measured are not visual, we do not straightforwardly see brainwaves, tissue density, or echogenicity when we look at the images.What we do see is a surface with a pattern of marks that is an outcome of measuring.These marks are counterfactually dependent on the measured features of the body: if the brain waves, echogenicity, or density had been different, the marked surface would have been different.Medical image interpretation is the process of understanding the marked surfaces as representing states of the body determinable by the features the technology measures.If image interpretation involves seeing certain things in the image, then understanding what we see will involve seeing the patterns of marks on the surface of the image.
The sensitivity with which the marked surface can come to provide information about the body relies of the counterfactual sensitivity between grey-scale marks and measured tissue values.However, while the precise counterfactual relationship between the marked surface and the quantitative values of the property measured in the body might constitute the image, its relationship to the content of the image is more complicated.A book that contained the exact correspondence values between, say, tissue density and grey-scale could provide any reader with full information about the density of tissue at any point in a radiogram.Such a book would render the image completely comprehensible as a measurement of the body, even to those with no medical expertise.In fact a list of correspondence values could render image interpretation otiose, and even render the image itself unnecessary.This sort of point-by-point information would not be useful in a medical context.Technology that printed out lists of numerical or other values would not thereby enable a reader of that list to make inferences from that information about the state of the body measured.Medical imaging is useful precisely because it allows practitioners to make judgements about bodily states that they couldn't from one-dimensional tests.It is not just the marks as measures, but the arrangement of those marks on the image surface that give imaging its power.Images need to be understood as measuring instruments, but they also have to be understood as representing states of the body.These images are not just measurement outcomes, but neither are they just pictures.
If we approach image interpretation based on how images are used it does not account for the fact that images are made as representations, and that, in some sense, their representational content is fixed by the technology.In the analysis in the previous section, the content was taken to be contested because the subject was taken to be contested.However, if we consider imaging technologies as measuring devices we can approach their content in a different way.In this section I argue that the various conventions of interpretation are made possible by images having a particular content that must be understood in terms of its relation to vision.Imaging technologies take advantage of our visual systems, and have developed in response to certain visual abilities.The uses of imaging technologies depends on the kinds of interpretations that images afford.

VISUALIZATION AS INSTRUMENTALLY AIDED VISION
While anything could be made to represent a brain, by fiat, and to represent it as being thus-and-so, not all representations are counterfactually dependant.By examining the counterfactual relation between the patterned surface of images and the objects imaged we can see that the patterns of luminosity values depend on measured properties of tissue.All imaging technologies are representational media, and all are visual media.The patterned surfaces correspond to patterns of tissue having certain properties, but the properties measured on a voxel to pixel basis are not visual.Neither are the properties measured necessarily diagnostically useful on a voxel to pixel basis.
Attempts have been made in the case of most imaging technologies to find a quantitative correspondence between measured tissue properties and states of the body.MRI and ultrasound were both initially developed as nonsurgical biopsies to test for cancer, on the basis that tumour tissue was measurably different than normal tissue.MRI (as well as CT) machines were originally designed to print out numbers, and the images were not considered their primary importance.The images produced by early MRI machines were in colour, with differences in colour corresponding to differences in proton density in hydrogen in tissue.

Figure 5: Early colour MRI of a human finger
In both cases it turned out that there is too much diversity in tissue for a quantifiable measure to be possible (Joyce 2008).It also turned out that colour was not helpful in developing medical uses for MRI.Removing numerical values from the display, and using grey-scale instead of colour made MRI into a diagnostically useful technology.This is because what is useful in these images is seeing differences between types of tissue, and within tissue.I argue that this is because displaying tissue in this way allowed for the visual identification of areas of the body, and localization of differences in tissue.
The different imaging modes in MRI show tissue differently, in imaging the brain they are identified by differences in the appearance of white matter, grey matter, bone, and cerebrospinal fluid.In ultrasound, denser tissue has higher echogenicity and so bounces back soundwaves faster.Denser tissue like bone appears white on the images.In most imaging technologies bone appears white.This choice could be considered arbitrary, except that it makes images easier to navigate.It makes images easier to navigate because it exploits our ability to see three-dimensional objects in twodimensional surfaces.

Seeing patterns, seeing bodies
Systems of representation differ in their "natural generativity" (Schier, 1986), or how easily one can figure out how to understand novel representations within the system.Pictorial representations are more generative than others, once we are familiar with photographs we can easily grasp the subjects of photographs we haven't seen before.Indeed we often learn what things look like through photographs.Pictures, of course, are not the real, and some systems of pictorial representation are more familiar to us than others.Pictures do not involve a belief that the thing is in front of us, they are not illusory; however, pictures concern appearances, the appearances that concern us in daily life.Picture perception has elements of convention, some styles and perspectives are more generative than others; it also relies on our visual systems being as they are.
All imaging relies on our visual abilities, but not all imaging takes advantage of the same aspects of our vision.In any case of imaging we see the marked surfaces, but some cases also involve doubled seeing-we do not just have a visual experience of the marks on the surface.EEG and MRI can both represent the brain, but they do so by measuring and displaying different properties of the brain.For this reason, they are capable of only certain kinds of representational content.For instance, an EEG might represent your brain as being in a state correlated with epilepsy.Hypsarythmia is an example of this that has a characteristic appearance in EEG, when a brain produces this pattern it is taken as diagnostic evidence of certain diseases.This kind of recognition is different than that involved with recognizing the amygdala in an MRI and being able to assess whether it looks normal.
Besides being marked surfaces with subjects and contents, pictures have designs; the design is the set of marks on the image surface by which it represents its subject as thus-and-so.Pictures present us with objects and allow us to have visual experience of an object that is not proximal, called seeing-in.Seeing-in usually involves a kind of doubled vision; we see the object, but we also see the marked surface.A pluralistic account of seeingin differentiates between different kinds of seeing that could double with seeing-in (Lopes 2006).In design seeing the two 'folds' of our experience are seeing the image depicted and seeing the design of the surface that gives rise to it.In many cases of picture perception we are not, nor do we need to be, aware of the design properties as design.We can simultaneously have a visual experience of the subject and see the two-dimensional, coloured, and textured, surface of the picture without the need to see that surface as design.When seeing-in is doubled with design seeing, the experience is twofold.When seeing-in is doubled with surface seeing the experience can be illusionistic, but it does not have to be.
If trompe l'oeil images trick us into ignoring their surfaces, then this is purely illusion without doubled seeing.
Illusionistic pictures separate the experience of seeing-in from either design seeing or surface seeing.This does not, however, mean that there is no design or that we cannot ever see it.We can examine the marked surface to locate design features even if, in doing so, our experience ceases to be illusionistic.If this were not the case it would be difficult to learn about different styles of visual representation, let alone draw them.To see the surface as the design is, in part, to see how the picture works.Some images are vehicles for seeing-in, they are produced so as to support and sustain objectperceiving experiences.This is what differentiates MRI, fMRI, ultrasound, CT, and related technologies from EEG, and other graphical images.It also allows us to explain other features of visual imaging.EEG, charts, graphs, and other visual images can indeed bear resemblance relations to their subjects without engendering subject seeing.In these cases we see the surface marks as design separate from subject seeing.These cases are best explained in terms of our developing tools that capitalize on our visual abilities, in this case visual organization.It is also the case that such data visualization techniques could allows us to see patterns, connections, and relations in the data that we could not see before.
We are able to see the data in these images in ways it would not be were it presented in other graphical formats.That is to say that we could extract information about the data from the image that was not (perhaps could not be) available were the data presented differently.This is important for a number of reasons having to do with pedagogy and public relations to data analysis, such as surveying trends, and identifying outliers (Semczyszyn, 2010).
These principles are used in understanding human/computer interface, visual analytics, as well as in imaging.For example, gray scale is often used in imaging when finer comparisons must be made within an image.Using colour to correspond with differences in the tissue can make areas seem more dissimilar, making recognition more difficult as we saw in the case of MRI, or creating kind boundaries where there are none (Tufte, 1990).
In the Minard image above, different colours represent direction of the flow of soldiers, length represents distance, and width represents numbers.The correspondence in this case is symbolic yet not arbitrary, these correspondences were chosen for the ease with which they present the encoded information.The same is true of pie charts, bar graphs, and other familiar kinds of graphical representations.These kinds of representations can allow us to see new relations between the data imaged.The data is presented in ways that make salient certain of its gross features.Our visual organization and pattern recognition are also what make Radiologic signs effective, but the kind of conventional relationship they involve does not explain all image interpretation.Sometimes when we look at medical images we see patterns, but sometimes what we see are bodies, specifically because the technologies have developed in order to take advantage of preexisting visual knowledge of anatomy and pathology by representing tissue in a way that allows for recognition.Because we see objects we are able, with training, to use imaging technologies as instruments to extend our perception; to see bone as bruised, white matter as lesioned, or brains as misshaped by tumours.It is only by seeing the bone or brain as bone or brain, and being to recognize and navigate the anatomical features that imaging becomes a useful diagnostic tool.

SEEING BODIES IN CONTEXT
So what can I say about cases with contested subjects, as in foetal sonography where there seem to be irreconcilable differences in image interpretation?One of the strengths of the approach I offer is that it provides a way of discussing the imaged features of the body separate from a context of social interpretation.Of course technologies are built around our desire to see certain things, and the desire to visualize the foetus is a deeply political desire.Nevertheless, obstetric sonograms represent the womb and the foetal body as being thus-and-so.A sonogram can depict the foetal body as malformed, as ossified, as developing or not in virtue of depicting tissue types in relation to each other.What it means that the body is malformed, or that it is developing as it is, or is there at all needs to be dealt with separately.Denying that a moving foetal body is represented in the image, or obscuring the visibility of the body as a body under layers of social interpretation is problematic.The body is visible to people familiar with the appearance of developing foetuses in slices and sometimes also to non-expert viewers because the technology is constructed around that visibility.Being able to talk about the variety of objects that we see in medical imaging and being able to ground an account of imaging in vision does not detract from a discussion of their social context or meaning.It provides new tools to enhance both analysis and critique of the images.

Figure 6 :
Figure 6: Charles Minard's 1869 chart showing the losses in men, their movements, and the temperature of Napoleon's 1812 Russian campaign.Wikimedia Commons