Increasing and Decreasing Perceived Bias by Distorting the Quality of News Website Design

News website design has previously been shown to impact perceived credibility, and one of its core dimensions and measures, bias. This paper demonstrates that by adapting the quality of the visual presentation of webpages from nine of the most popular news websites, to reflect high quality and low quality news agencies, we can predicatively increase or decrease perceived bias in the news articles they contain. This effect was common across the websites of traditional print, news magazine, and international news agencies, and across articles with different levels of bias. The distortions focused on the visual quality of a websites’ design, including the amount, size, and prominence of advertising, news article meta data, supporting material, gaudy calls to action, and the percentage of the webpage dedicated to the news article. Higher quality visual experiences reflecting quality news agencies were shown to reduce bias, while those with a low quality visual experience reflecting less professional news agencies increased bias. Significant differences were also found between low and high quality designs showing the same news articles. This paper reports results on one part of a large study on the impact of visual appearance and design on the perception of bias in online news.


INTRODUCTION
Real, or perceived bias is a serious concern in the dissemination of news online. It is also a core dimensions and measures of credibility . Bias is also a key concern of reputable journalists, editors, and news agencies, who take their responsibility of impartially reporting news seriously. This is exacerbated with increased claims and counterclaims of bias in the news. There is also a backlash against some mainstream news providers by some segments of the population and concerns about accuracy and the influences of extreme partisan actors by another segment of the population. Rightly or wrongly, it is unsurprising that bias is one of the most common accusations leveled at the purveyors of news (Efron, 1971;Groeling, 2013;Stevenson and Greene, 1980). News websites are increasingly the main public interface of news agencies (Mitchell, 2015). Traditionally there was a visual distinction between high quality news organizations, typified by broadsheet newspapers, and the low-quality publications, often represented by tabloids or fringe publications (Cooke, 2005;Richardson and Stanyer, 2011). This distinction persists online. Apart from the news they report, the quality of the journalism and imagery, quality news agencies have a distinctive air of professionalism in their websites. This is conveyed through a professional design overlaying complex underlying technologies. This is indicative of organizations with large budgets and a capable technical staff.
The Internet has given rise to a new category of online only and predominantly low-quality news agencies. These news websites, often daubed content farms, rely almost completely on the online advertising model. Articles are written, in response to social media trends, to attract rather than inform users. In many instances content is syndicated, aggregated or even harvested from other publications (Bakker, 2012;Napoli, 2014). Content and headlines are often salacious in nature, yet lacking any real or informative news. Large volumes of such news articles can now be produced by algorithm, language checked by low pay content checkers (Napoli, 2013). The visual distinction between websites of high quality news agencies and those of tabloids is now also evident between high quality news agencies and clickbait content farms masquerading as news agencies.
With falling sales and ad revenue, quality news organizations have been under increasing pressure to monetize their online presence. Many have found it necessary to give control over increasingly large proportions of each webpage to outside organizations, decreasing the proportion of screen space that is focused on reporting news. This vicious circle has resulted in a race to the bottom. As ad revenue per hit decreased, increasingly large proportions of each webpage have become revenue generating. They have also come under pressure to provide social feeds, and loud, gaudy or prominent calls to action to share content online or connect on social media. These have contributed to a lowquality user experience. This shortsighted approach to attract users has ultimately proved damaging.
Ironically, in an attempt to return to a high-quality visual experience, a backlash has resulted in record numbers of users installing ad-block software to reduce the impact of advertising and other targeted selling or promotions of paid for content (Pujol et al., 2015). This has resulted in quality news agencies losing out on even more ad revenue. Increasing proportions of users are also limiting their social media connections and interaction with news websites and what they share online. This is the result of increasing awareness of privacy online.
Many quality news organizations have begun to recognize the failings of the online advertising business model with a heavy reliance on social media that leads itself towards click-bait style headlines and content to drive readership. Visually this has resulted in news webpages cluttered with advertising and gaudy or loud calls to action. These damage the visual presentation, user experience and consequently their brand and reputation. Many news agencies have therefore begun to revert to the high-quality journalism business model by instituting paywalls (Kumar et al., 2012;Sjøvaag, 2016). While initially there was some backlash for paying for what was once free, appealing to fairness and quality can reduce negative feelings (Cook and Attari, 2012).
This research investigates whether it is possible to deliberately increase or decrease the perception of bias in a news article by distorting the visual quality of the webpage. It is conducted with the supposition that a high quality visual experience will decrease perceived bias while low quality visual experience will increase it. It will test this by distorting nine of the most popular mainstream news websites, three each from print, news magazine and international news agencies 1 . This paper is the first from a large experiment on the impact of visual appearance and design on the perception of bias in online news.

MOTIVATION
Although some news organizations have begun moving away from the online advertising model due to its unsustainability, they and others, may not be aware of the overall impact the resulting low quality visual experience may have on the content. In the overarching domain of credibility, several studies have demonstrated the impact of a websites' aesthetic treatment on the credibility of its content (Allport and Pendley, 2010;Jung et al., 2017;Robins and Holmes, 2008). In a study with 2,684 participants which aimed to find out how users evaluate the credibility of websites, Fogg et al. found that 30.2% rely on bias when they were 1 www.theguardian.com, www.telegraph.co.uk, www.reuters.com www.independent.co.uk, www.economist.com, www.spectator.co.uk, www.newstatesman.com, www.aljazeera.com, and www.bbc.com evaluating the credibility of news websites (Fogg et al., 2003). In a reflective question after the main experiment in this research, 217 participants out of 405 selected bias as a key concern when evaluating the credibility of news online. The multiple-choice question had 25 of the most common measures of credibility as options. Modern news websites are among the most adaptive and personalized services (Thurman and Schifferes, 2012). They are also heavily integrated with external advertising, which is also often heavily personalized to individual users due to cross website tracking and social media integration (Bleier and Eisenbeiss, 2015). Consequently, large proportions of many news webpages are given over to third parties with motivations other than the balanced presentation of news. Thus, the webpages of many high-quality news organizations have begun to resemble those of lower quality, content farm or clickbait news agencies. The resulting visual experience may be impacting the perception of bias in the news article.

RELATED WORK
Research in the overarching domain of credibility has shown that increased levels of aesthetic treatment can positively impact perceived credibility (Robins and Holmes, 2008). There has also been significant work undertaken to understand the impact of structural features of website design on credibility with the broad consensus being that, at a minimum, they have a large influence on judgement (Chung et al., 2012;Flanagin and Metzger, 2007;Rains and Karmikel, 2009;Rieh and Hilligoss, 2008).
Bias has been investigated at almost every stage of the news production, dissemination, and consumption cycle. Historically much of bias research focused on the identification of bias within the source, such as the facial expressions of news anchors, stereotyping of language with respect to race and partisan agenda setting of national newspapers (Friedman et al., 1980;Gorham, 2006;Larcinese et al., 2011;Lowry and Shidler, 1998). The focus of researches then evolved into the study of and comparison of bias across different mediums (Groseclose and Milyo, 2005;Lee, 1991). Recently, research has tended to focus online with an emphasis on identifying content bias (Balahur et al., 2009;Budak et al., 2016;Earl et al., 2004;Ma and Yoshikawa, 2009). Presentation bias, closely associated with this work, and which has been looked at in traditional mediums is also now being investigated in the context of search engine results (Bar-Ilan et al., 2009;Groeling, 2013). However, despite some recent work (Spillane et al., 2017a(Spillane et al., , 2017b, there is a lacuna in the literature as to how the presentation of news online may impact bias. Understanding how the visual presentation of content affects perceived bias is also important. Unfortunately, no models, theories or frameworks were found in the literature explaining how individuals form opinions or judgements of bias online or otherwise. It is therefore necessary to look to the overarching domain of credibility. In comparison, there are ten human-centric models, theories, frameworks or schematics which attempt to explain how individuals form judgements of credibility online (Fogg, 2003;Fogg and Tseng, 1999;Fritch and Cromwell, 2001;Hilligoss and Rieh, 2008;Lucassen et al., 2013;Metzger, 2007;Metzger and Flanagin, 2015;Rieh, 2002;Sundar, 2008;Wathen and Burkell, 2001). Many of these, such as Sundar's MAIN Model or Metzger's Dual Processing Model of Website Credibility Assessment have a strong focus on the use of heuristics and the visual aspect of the website rather than the textual content to form judgements of credibility. In fact, eight out of ten of these directly reference one or more Dual Models of Information Processing such as the Elaboration Likelihood Model, the Heuristic Systematic Model or the Controlled and Automatic Human Information Processing Models (Chaiken, 1980;Petty and Cacioppo, 1986;Schneider and Shiffrin, 1977;Shiffrin and Schneider, 1977). These models maintain that when a user is not motivated by task or importance, they adopt less cognitively taxing peripheral or heuristic routes to judgements of credibility. This research contends that as credibility judgements are predominantly made from the visual characteristics of a websites design, then perceived bias, a core dimension and measure of credibility, may also be influenced by a website's technical features and design characteristics, the combination of which contributes to the aesthetics of a website.

Bias
As per Lowry, it is only possible to measure bias within the context of a definition (Lowry, 1973). Consequently, the definition put forward for this research is: "Deliberate or accidental slant by the journalist, editor or publication to distort reality." This was designed to focus the user on the articles contained in each webpage without specifying the type of news production bias they might be exposed to. It was also chosen as many of the existing definitions in the literature are either too vague, such as that by Stevenson et al. who conceptualized bias as: "imbalance or inequality of coverage" (Stevenson et al., 1973), or too general such as McQuail's: "a consistent tendency to depart from the straight path of objective truth by deviating either to left or right" (McQuail, 1992).

High Visual Quality and Low Visual Quality
High visual quality news websites are defined as those of mainstream news organizations, many of whom have or had broadsheet traditional print publications. Traditionally these were known as broadsheets media organizations, however in recent years many have begun publishing in alternative formats such as Berliner or compact, or have even become online only. Consequently, many have adopted the term 'quality press'. They investigate and commission original journalism, claim to adhere to high standards and act as a check and balance on society. Typical examples include The Guardian or The New York Times. Many of the websites used in this research fall into this category or are from high quality news magazines or news agencies.
Low visual quality websites are defined as those who primarily syndicate, aggregate, or harvest content from other sources on the web (Bakker, 2012;Napoli, 2014). In many instances, content is automatically generated on whichever topics are currently trending on social media or traditionally get regular traffic (Napoli, 2013). The sole purpose of such websites to generate traffic to make money from advertising. They do not report any public duty or adhere to high standards of journalism. Such websites are typically classified as clickbait or content farms.

HYPOTHESIS
1.1 H 0 Perceived bias will not be increased due to a reduction in the quality of visual presentation.
This hypothesis is based on the supposition that by reducing the quality of the visual experience, perceived bias within the article will be increased when each is compared to its respective control.
2.1 H 0 Perceived bias will not be decreased due to an improvement in the quality of visual presentation.
This hypothesis is based on the supposition that by improving the visual experience that the perception of bias in each article will be reduced when each is compared to its respective controls.
3.1 H 0 There will be no difference in the perception of bias between high-quality and low-quality visual presentations of the same news article.
The quality of visual presentation has previously been shown to impact credibility and some of its individual measures such as trust. This hypothesis aims to ascertain if the same is true for bias.

STUDY DESIGN
This research was conducted as part of a larger experiment investigating the impact of the design of news websites on the perception of bias. The experiment had several research threads, each with a different focus, and are presented separately. The first stage required participants to give background information to build a sample profile. The second stage was the experiment proper, where they rated the perceived bias in each distorted webpage/article combination. The final stage comprised reflective questions about the experiment, bias, and news website design.
For the duration of the experiment, each webpage was paired with a single news article. This was directly inserted into the HTML so that the website's CSS rendered it exactly like an original article. Four distorted versions of each webpage/article were used in the experiment.
Distortion 0, or D0, is a basic version of each news article using only <h1></h1> and <p></p> tags for spacing with no CSS or other design applied. The D0 bias ratings are not used to satisfy any of the hypothesis, but only to provide a ground truth bias rating for each article. D1, the control, is an exact copy of the original news webpage with only the branding and logos removed and replaced with a generic NewsCom moniker. D2 is a distorted version of each webpage designed to improve the quality of the presentation and decrease perceived bias. D3 is a distorted version of each webpage designed to decrease the quality of the visual presentation and increase perceived bias.
There are two independent variables in this study. The first is the webpage/article, the second is the distortion. Combined these make a (distortion) x (webpage/article) combination. Perceived bias is the dependent variable.

METHODOLOGY
The experiment was set up as a 9x4 within-subject, incomplete counterbalanced measures design. Incomplete counterbalancing was achieved by arranging the webpages and distortions, D0-D3, in a reduced form Latin square. 9 news webpages, from the aforementioned websites, were selected for inclusion in the experiment. The original article in each webpage was removed and replaced with one of 9 articles modified to increase or decrease perceived bias, see Table 1. The manipulations of D2 and D3 are detailed in Table 2. Participants were randomly assigned to one of 9 diagonal paths through the Latin square that intersected with each distortion, webpage or article, thus ensuring that each participant experienced each distorted and webpage/article combination once. To reduce carryover effects and task fatigue, once assigned to a path, the webpage/article combinations the participant would encounter were displayed in random order, meaning the impact of carryover and task fatigue would be equally distributed. Two attention questions, to determine diligence were also added to each participant's path.

Article Selection, Biasing, and Assignment
Nine existing news articles from mainstream publications were selected in May 2017. Articles were selected to be un-emotive by ignoring topics such as race, immigration etc. Each was shortened to circa 200 words. The two main classes of bias, epistemological bias and framing bias, described by Recasens et al. were added to each article by modifying the text to create a designed range from relatively unbiased to extremely biased content (Recasens et al., 2013). To evaluate the article modification process and see if each of the articles matched their designed bias ranking, a pre-test evaluation with 24 participants was undertaken, see columns 2 and 3 of Table 1.
In the pre-test, participants independently rated the amount of bias in each article from 0 to 100. The results show a strong correlation with the designed bias rating for each article. The only inconsistency was the article designed to be the 4 th most biased (60), was instead considered the second most biased (74.56) by the pre-test group. This process was undertaken to ascertain whether any impact of the distortions was common across all levels of biased content. The news articles were then assigned to websites based on their pre-test ratings scores in Table 1, with each website category being assigned a high, mid, and low-bias article. While there is inconsistency between the pre-test results and the D0 results from the crowdsourced experiment, the purpose was to create a range of biased content, which was successful.

Selection of Websites and Webpages
To ascertain if the impact of the distortions extended across the domain, nine webpages from the aforementioned news websites were selected. Existing news websites were chosen rather than trying to recreate their level of professional design from scratch. They were selected: 1) To represent the three main categories, traditional print, news magazines, and international news agencies. Tabloid websites were not used due to possible strong feelings about the category influencing results 2) Such that all the websites were British-based other than Al Jazeera, which has a large London office. 3) As participants were recruited from the US. English language websites were chosen. US websites were not used to help negate possible strong feelings about certain news agencies in case they were recognized. 4) To ensure good representation, three websites from each category were included. 5) Lastly, based on their popularity.
While it is practically impossible to ensure that the crowdsourced participants would not recognize or have experience of the websites in question, it was felt that selecting websites which, for the most part, do not have a focus on the US, would reduce their likelihood of being recognized. And if so, still reduce the impact of strong feelings compared to selecting websites such as the conservative Fox News website or the liberal Huffington Post. To select the individual webpages from each website for inclusion in the experiment, a search was undertaken to find the most recent article and webpage relating to the same or similar topic as the article that would be inserted. Thus, many characteristics of each webpage matched the topic, e.g. related content links, bread crumb trails etc. and would more closely reflect a realistic scenario. As this was part of a larger study in the domain it was not possible to include the websites of low visual quality news agencies.

Distortions
To understand which elements should be selected for distortion, and the form these distortions should take, a design review was undertaken following the format of Zhang and von Dran, of the nine websites used in this experiment and those of other news organizations (Zhang and von Dran, 2000). Twenty-six features and characteristics of news webpages which can convey quality to the viewer were identified. Using a factor analysis, these were then assigned to eleven categories. Features were defined as elements of a webpage used to provide information or a service, e.g. search facility. Characteristics were defined as the abstract influences on the design such as the color scheme or how busy the design was. The underlying means of affordance of each of these eleven categories was then identified. Sundar maintains that the credibility of a website, of which professionalism is a common measure, is conveyed by the underlying affordances in the technology behind the design which are conveyed via cues to the user (Sundar, 2008). There are two ways they convey cues. Firstly by their sheer presence and secondly by assembling information, related to the message or the topic, which he previously coined 'Information Scent', that is pertinent to the user (Sundar et al., 2007). Lineage / Information Scent was also used to categorize one of the groupings of identified features and characteristics in the factor analysis.
The D2 and D3 rows in Table 2 depict the eleven categories identified in the factor analysis. Nine of these were distorted to reduce or improve the visual quality of the webpages. Two categories, Name/Branding and Color, were not used to distort the websites. All names, branding and logos were replaced with the generic moniker, NewsCom with a plain circle logo in in each webpage's colors. Changes to each webpages color scheme to increase and decrease quality were not implemented due to being too subjective. It would also likely result in too great a change to the design to make any findings useful. Figure 1 depicts the D1, D2 and D3 versions of the same BBC webpage used in this experiment. In each case, it displays the same news article within the same fundamental design.
D0: Ground truth. This is a plain text version of each article to establish a bias rating. The article uses <h1></h1> and <p></p> tags only.
D1: The control. This is an original copy of the webpage with branding removed and replaced with NewsCom. The assigned news article was inserted into the underlying code of each webpage.

D2
: High visual quality. The following were changes were implemented; Limited advertising of high quality products; increase in the proportion of the screen dedicated to the news article, author profile, complete with picture and bio with related experience; a graph relating to the content on each page; Information Scent in the forms of links to related articles or sources.
D3: Low visual quality. It includes increasing the size, amount of, prominence, and gaudiness of advertising while reducing the quality of the products or services offered. This included services such as dating websites, hair growth products and anti-ageing treatments. Advertisements were placed in positions for maximum visibility displacing news content, navigation or other useful services, interrupting the visual experience. The percentage of the screen dedicated to sponsored content was significantly increased.
Ads were also inserted into the middle of the article to interrupt the reading experience. All links to supporting content, similar articles, or any indication of a history or experience of covering the same topic, so called information scent (Sundar et al., 2007), were removed. Author name, credentials, photo and any other profile information were removed. Each news article was also anonymized. All meta data, including time stamps, article corrections, syndication information was removed. Social media interaction facilities, previously restrained in the D1, were moved to the top of the news article, made larger and a demanding call to action on the user added. The percentage of each webpage focused on delivery of the news was reduced. Each webpage went through the same process. E.g. the same low quality ads were added in the same position to each D3. Once each distortion was created, a static image of each was created for use in the experiment.

Crowdsourcing and Participant Profile
Participants were recruited on Prolific Academic 2 and paid £1.25. This was limited to US nationals, over 18, whose first language was English. The experiment was run over four days in late August 2017. 508 submissions were recorded. 47 were rejected for failing Spike questions. 32 were rejected for having wrong completion codes or for completing in less than 2 minutes, indicating automated bots. 10 were not included in the final analysis due to the participants rating >4 distorted webpage/article combinations from the larger experiment as having 0 bias. Finally, 14 were not included in the analysis due to having missing data, or the requirement to balance the dataset for ANOVA. In total 405 submissions were evaluated. 51.4% Male, 47.2% Female, 1.4% Other. Mean age 36.48. 34% were <30. Education and Occupation revealed a highly educated, mostly professional sample.

Statistical Power Analysis, Validity and VAS
A post-hoc test to compute achieved power with the 405 submissions revealed a >.95 actual power of detecting an effect size of 0.24. Four measures were also taken to increase validity. 1) Upon beginning the study, participants undertook two instruction tasks explaining how to use the experiment interface. 2) Two spike tasks were added to measure attention. 3) Participants were provided a second opportunity to comparatively re-evaluate their ratings (Spillane et al., 2017b). 4) No emotive topics were used. A Visual Analogue Scale (VAS) from 0-100 was used to measure perceived bias.

Figure 1 Three versions of the BBC website. D1 is the original, D2 is the version designed to improve the quality of the design and reduce the perception of Bias and D3
is the distortion designed to reduce the quality of the design and increase the perception of bias.

Participant Instructions
As per Lowry, it is only possible to measure bias within the context of a definition (Lowry, 1973). The instructions and the definition of bias directed the participant to read the news article contained in each webpage and assess the amount of bias in it. This was reiterated multiple times to ensure that the focus of each participant was on each news article. Participants first experience of the purpose of the experiment and the instructions was in the advertisement. "In this study, you will be asked to rate, from 0 to 100, how biased different online news articles are. 0 being unbiased and 100 being extremely biased. Bias is defined as 'Deliberate or accidental slant by the journalist, editor or publication to distort reality.'" Participants were once again presented with this in a pop-up dialog during the first instruction task, and again after they answered the user profile questions.

STATISTICAL ANALYSIS
To derive the impact of the distortions, nine oneway repeated measures ANOVAs using the Bonferroni correction were conducted as simple main effects in SPSS 24. While simple main effects would usually not be explored without a significant two-way interaction in a two-way repeated measures ANOVA, the focus of this research is on comparing the distortions, D2 and D3 to their respective D1 controls and to each other. Alternatively, one could perform multiple paired samples t-tests, however the increased validity of the one-way ANOVA due to use of Bonferroni is preferred. The data assumptions for two-way ANOVAs are valid for one-way ANOVAs. Results are shown in Table 3. As the focus is on investigating the impact of distortions, simple main effects of websites were not undertaken.

Simple Mean / One Way Repeated Measures ANOVAs
The experiment design passed the first two assumptions for ANOVAs, namely that the dependent variable was a continuous interval type data and the independent variable should consist of two or more independent groups. The data collected was analyzed for the remaining three ANOVA data assumptions. A visual inspection showed a small number of outliers in the boxplots. It was decided to continue the analysis with the original data as the results would not be materially affected as there was little difference in the findings of the original and transformed ANOVAs. A Sharpio-Wilk test (p = >.05) for normality on the studentized residuals revealed that some of the data was not normally distributed.
It was again decided to continue with the test as the one-way ANOVA is considered robust to deviations from normality when populations are >50 (405). Due to the central limit theorem, one-way ANOVAs can still provide valid results even when the distribution of the data is very non-normal. Table 3 depicts the results of the nine individual ANOVAS for each website. D1 for each website is the control. D0 is included in the analysis to provide ground truth and a greater overview. Mauchly's Test of Sphericity (MTS) (p >.05) indicated that the assumption of sphericity had been met for each of the ANOVAS. Therefore, the Sphericity assumed significance values are reported without correction. Results for each are shown in Table 3.
For brevity, only the results of the websites with statistically significant interactions (Guardian, Telegraph, BBC and Reuters) are discussed here in detail. However, Mean, Standard Error, F values and p values for all interactions are shown in Table 3.
110. An analysis of the pairwise comparisons showed an increase in perceived bias from D2, 53.884 ± 31.577 to D3, 75.156 ±18.619 (95% CI 5.207 to 37.415), p = .004. The D2, designed to have a high quality visual experience, is considered significantly less biased than its visually diminished D3.
117. An analysis of the pairwise comparisons showed an increase in perceived bias from D2, 35.773 ±28.600 to D3, 58.420 ±31.731 (95% CI 4.719 to 40.658), p = .007. A further inspection also showed a decrease in perceived bias from D0, 54.267 ±27.908 to D2, 35.773 ±28.600 (95% CI 2.005 to 35.062), p = .020. While D0 was included in the analysis, it is not used to satisfy any of the hypothesis. The visually improved D2 was also considered significantly less biased than its visually diminished D3.
074. An analysis of the pairwise comparisons showed an increase in perceived bias from D2, 23.24 ±22.091 to D3, 39.44 ±23.261 (95% CI 1.797 to 30.603), p = .020. The D2, distorted to improve its appearance, was rated as being significantly less biased than its visually diminished D3.
Reuters: The difference between D2 and D3 mean bias ratings was statistically significant F(3, 132) = 2.690, p = .049, partial η2 = .058 An analysis of the pairwise comparisons showed an increase in perceived bias from D2, 18.42 ±19.262 to D3, 32.47 ±25.964 (95% CI .071 to 27.914), p = .046. Like the Guardian, Telegraph and BBC, the visually improved D2 version of Reuters was considered significantly less biased than its diminished D3.

RESULTS
The results in Table 3 and in Figure 2, clearly demonstrate the impact of design quality on the perception of bias. While D0 results are included in the analysis and as ground truth, they are not used to satisfy the hypothesis.

H 0 Perceived bias will not be increased due to a reduction in the quality of visual presentation.
The answer to the first hypothesis can be determined by inspecting Table 3 and Figure 2 to ascertain if perceived bias increased from D1 to D3. Eight of the nine distorted webpage/article combinations showed some increase in perceived bias between D1 and D3.
While no results were significant, three distorted webpage/article combinations, Guardian p = .107, Telegraph, p = .116 and the BBC p = .097 showed marked increases. Only Al Jazeera demonstrated a decrease in perceived bias. It should be noted however, that Al Jazeera demonstrated an increase in perceived bias when D3 is compared to D2 showing that the effect was at least partially in place when compared to the improved version of the website. Despite the trends shown in these results we cannot reject the null hypothesis.
However, it is possible to say that there is a correlation between a reduction in visual quality and increase in perceived bias. It should also be noted that if the level of reduction for D3 had been more, it is likely one or more significant results would have been found.

H 0 Perceived bias will not be decreased due to an improvement in the quality of visual presentation.
The second hypothesis can be tested through a comparison of the D1 and D2 distortions in Table 3 and Figure 2. While none of the results of the pairwise comparisons of the individual one-way ANOVAs were significant, the trend depicted in Figure 2 is obvious. Every website showed a distinct decrease in perceived bias when the design was distorted to improve the visual presentation of the news. It should be noted that improving the visual quality of the design of international news organizations that spend millions on their websites is a much more difficult task than reducing the visual quality of the design. It should also be noted that it is virtually impossible to create a news article which a large population of users will consider unbiased. An inspection of Figure 2 demonstrates this point.
The lower the rate of perceived bias in each article, as rated in D1, the lower the decrease in perceived bias in D2. That is, there is only scope for a marginal reduction in bias when there is less bias in the original article.
This is most evident in the New Statesman which was displaying the least biased article, as rated by participants in the pre-test (see Table 1), and in the experiment, and showed the smallest decrease. This indicates that there is a minimum baseline which even unbiased articles cannot descend beneath. This article also had a large amount of data in the text which may indicate participants relied on some form of numbers or quantitativness heuristic. The existence of such has been demonstrated previously, when the presence of quantitative data in a passage of text was found to increase the likelihood of a user to judge information by the peripheral route and which can increase perceived credibility (Yalch and Elmore-Yalch, 1984).
Conversely, the greater the perceived bias in the D1 version of each article, e.g. Guardian, Telegraph and Al Jazeera, the greater the reduction in their respective D2s. This likely accounts for the smaller effect on the results of D1 v D2 compared to D1 v D3. The increase in perceived bias is also much greater than the decrease since it is much easier to make an existing professional website of an international news organization look bad than it is to improve them. However, while there is a definite and common decrease in perceived bias from D1 to D2 as demonstrated in Figure 2 it is not possible to reject the null hypothesis. 3.1 H 0 There will be no difference in the perception of bias between high-quality and low-quality versions of the same news website.
A comparison of the results of D2 to D3 for each website provides the answer to the third hypothesis. Figure 2 shows a definite increase in perceived bias from D2 to D3. This is common across each website category and across articles containing different levels of bias. More importantly an inspection of Table 3 shows four websites with significant differences in perceived bias between their D2 and D3 distortions. Guardian p = .004, Telegraph p = .007, BBC p = .020 and Reuters p = .046. As stated already, it must be noted that it was difficult to improve upon the quality of the D2 webpage/article combinations. At the same time, it would be easily possible to further degrade the quality of the visual experience in D3. Based on this, the null hypothesis can be rejected and the alternative accepted.

DISCUSSION
This experiment demonstrates that perceived bias in a news article, can be decreased or increased through improving or diminishing the visual quality of the aesthetics and design. While there were no significant differences between the D2s and D3s to their respective D1 controls, the trends are evident in Figure 2. When the visual quality of the design is decreased, perceived bias increases. Conversely, when the visual quality increases perceived bias decreases. Furthermore, the results demonstrate significant differences in perceived bias between four of the low-quality D2 and high-quality D3 webpage/article combinations, see Table 3 and Figure 2. Therefore, depending on whether a news article is presented with a high or low quality visual presentation could make a significant difference to how biased it is perceived by its readers. It is also likely that participants associate low quality visual presentations as being more likely to be purveyors of biased news. This is a serious issue as Fico has demonstrated that as the perception of bias in an article increases, the perception of credibility of the news organization behind it decreases (Fico et al., 2004). The distortion techniques, such as the addition of extremely low quality advertisements and large gaudy calls to action, adopted in this experiment to decrease the quality of the visual presentation, are all in common usage on content farms, arguably the lowest tier of online news 'agencies'. However, many reputable news agencies have, out of necessity and unintentionally, begun to emulate the visual style of such websites. Increasingly large proportions of the screen are now monetized and under the control of third parties. In many cases these are displaying personalized ads and social feeds which can drastically diminish the visual quality and thus the experience of the user. What is worse, because of the quantity and nature of such personalization, reputable news agencies have little or no idea of the visual experience of each user. Consequently, users may be judging news articles from reputable news agencies as more biased than they are, and thus also judging the organization as less credible. This could have serious long term repercussions for some agencies.

CONCLUSIONS
This experiment demonstrated that the perceived level of bias within the same news article can be increased or decreased through distorting the visual quality of its presentation. Low-quality visual representations of the same webpage had the effect of increasing perceived bias in the article while highquality visual representations decreased it. Significant differences were found between the low quality and high quality distortions. It is likely that a major influence was the proportion of each webpage focused on presenting the actual text of the news article as this is the most obvious difference between the high and low quality distortions. Increasing the proportion of each webpage under the control of outside agencies, such as advertisers, was also likely responsible for increasing perceived bias.