Assessing bias in online surveys using alternative survey modes

Spencer, Neil H.; Syrdal, Dag S.; Coates, Matthew L; Huws, Ursula

doi:10.13169/workorgalaboglob.16.1.0034

ScienceOpen: research and publishing network

For Publishers

For Researchers

Blog
About

Search
Advanced search

659

views

recommends

Call for Papers: Hierarchies of domesticity – spatial and social boundaries. Deadline for submissions is 30^th September, 2024. Full details can be read here.

Articles to be no longer than 6,000 words (excluding footnotes and bibliography) and submitted in two forms: an anonymised version in which all references to the authors’ institution and publications are omitted; and a full version including the authors’ titles and institutional affiliations. For complete instructions on style, formatting, etc., please consult: https://www.plutojournals.com/wp-content/uploads/WOLG-Instructions-for-Authors2023.pdf

scite_

Work Organisation, Labour & Globalisation

Volume 16, Issue 1

Record: found
Abstract: found
Article: found

Is Open Access

Assessing bias in online surveys using alternative survey modes

Published

research-article

Author(s): Neil H. Spencer , Dag S. Syrdal , Matthew Coates , Ursula Huws

Publication date Pub: 14 May 2022

Journal: Work Organisation, Labour & Globalisation

Publisher: Pluto Journals

Keywords: bias, offline surveys, online surveys, platform work, survey mode

Bookmark

Abstract

Due to concerns that respondents to online surveys are different from populations of interest, parallel offline surveys can be undertaken and results compared. In this article we create a set of principles to compare results from online surveys with those from surveys using other survey modes. Rather than just comparing estimates and confidence intervals from the different modes, these principles consider biases that each survey mode introduces and whether the results obtained are compatible with each other, given these different biases. Using the example of a survey of platform work, we demonstrate that this approach can be used effectively and be applied to a variety of social science studies that use online surveys.

Content

Author and article information

Contributors

Neil H. Spencer:

Bio :

Neil H. Spencer is Professor of Applied Statistics at Hertfordshire Business School, University of Hertfordshire, Hatfield, UK

Dag S. Syrdal:

Bio :

Dag S. Syrdal is Statistical Consulting Associate at Hertfordshire Business School, University of Hertfordshire, Hatfield, UK

Matthew Coates:

Bio :

Matthew Coates is Statistical Consulting Associate at Hertfordshire Business School, University of Hertfordshire, Hatfield, UK

Ursula Huws:

Bio :

Ursula Huws is Director of Analytica Social and Economic Research, London, UK

Journal

Journal ID (doi): 10.13169/workorgalaboglob

Title: Work Organisation, Labour & Globalisation

Abbreviated Title: WOLG

Publisher: Pluto Journals

ISSN (Electronic): 1745-6428

ISSN (Print): 1745-641X

Publication date Pub: 14 May 2022

Publication date Collection: 2022

Volume: 16

Issue: 1

Pages: 34-51

Article

DOI: 10.13169/workorgalaboglob.16.1.0034

SO-VID: 6effc1a7-b304-4959-bf1f-398e213ab825

License:

All content is freely available without charge to users or their institutions. Users are allowed to read, download, copy, distribute, print, search, or link to the full texts of the articles in this journal without asking prior permission of the publisher or the author. Articles published in the journal are distributed under a http://creativecommons.org/licenses/by/4.0/.

History

Page count

Pages: 18

ScienceOpen disciplines: Sociology,Labor law,Political science,Labor & Demographic economics,Political economics

Keywords: offline surveys,survey mode,platform work,bias,online surveys

References

E. ArmanoA. BoveA. Murgia (eds) (2017) Mapping Precariousness, Labour Insecurity and Uncertain Livelihoods: Subjectivities and Resistance, London: Routledge.
K. BahiaS. Suardi (2019) Connected Society: The State of Mobile Internet Connectivity 2019, London: GSMA.
P. P. BiemerE. D. de LeeuwS. EckmanB. EdwardsF. KreuterL. E. LybergN. C. TuckerB. T. West (eds) (2017) Total Survey Error in Practice, New York: John Wiley & Sons.
M. CallegaroK. L. ManfredaV. Vehovar (2015) Web Survey Methodology, London: SAGE.
E. D. de Leeuw (2008) ‘Choosing the method of data collection’ in E. D. de LeeuwJ. J. HoxD. A. Dillman (eds) International Handbook of Survey Methodology, Hove: Psychology Press:113–135.
E. D. de Leeuw (2018) ‘Mixed-mode: past, present and future’, Survey Research Methods, 12 (2):75–89.
E. de LeeuwN. Berzelak (2016) ‘Survey mode or survey modes?’ in C. WolfD. JoyeT. W. SmithY.-C. Fu (eds) The Sage Handbook of Survey Methodology. London: Sage Publications.
S. DenchR. IphofenU. Huws (2004) An EU Code of Ethics for Socio-Economic Research, Brighton: Institute for Employment Studies.
X. DengR. D. GalliersK. D. Joshi (2016) ‘Crowdworking – a new digital divide? IS design and research implications’, Association for Information Systems Research Papers, 148. https://aisel.aisnet.org/ecis2016_rp/148
V. De Stefano (2016) The Rise of the ‘Just-in-Time Workforce’: On-Demand Work, Crowdwork and Labour Protection in the ‘Gig-Economy’, Geneva: International Labour Office.
D. J. C. DodouF. de Winter (2014) ‘Social desirability is the same in offline, online, and paper surveys: A meta-analysis’. Computers in Human Behavior, 36:487–495.
Eurofound (2015) New Forms of Employment, Luxembourg: Publications Office of the European Union.
A. Felstead (2021) ‘Are online job quality quizzes of any value? Selecting questions, maximising quiz completions and estimating biases’, Employee Relations: The International Journal, 43 (3):724–741.
F. Flanagan (2019) ‘Theorising the gig economy and home-based service work’, Journal of Industrial Relations, 61 (1):57–78.
F. J. Fowler Jr (2014) Survey Research Methods, 5th ed., Thousand Oaks, CA: SAGE.
P. GalvinA. BiermanS. Schieman (2021) ‘Über-alienated: Powerless and alone in the gig economy’, Work and Occupations, published online 22 June 2021.
A. Gooch (2015) ‘Measurements of cognitive skill by survey mode: Marginal differences and scaling similarities’, Research and Politics, July–September:1–11.
A. GreenM. de HoyosS.-A. BarnesB. BaldaufH. Behle (2014) Exploratory Research on Internet-Enabled Work Exchanges and Employability: Analysis and Synthesis of Qualitative Evidence on Crowdsourcing for Work, Funding and Volunteers, European Commission Joint Research Centre Institute for Prospective Technological Studies, JRC Scientific and Policy Reports, Luxembourg: Publications Office of the European Union.
U. Huws (2016) A Review on the Future of Work: Online Labour Exchanges, or ‘Crowdsourcing’: Implications for Occupational Safety and Health, Bilbao: European Agency for Safety and Health at Work.
U. HuwsN. H. SpencerM. CoatesK. Holts (2019) The Platformisation of Work in Europe: Results from Research in 13 European Countries, FEPS – Foundation for European Progressive Studies, UNI Europa, Hatfield: Hertfordshire Business School, University of Hertfordshire.
K. F. Jesnes Braesemann (2019) Measuring Online Labour: A Subcategory of Platform Work, Nordic Future of Work, Brief 2, Oslo: Fafo Institute for Labour and Social Research.
H. JohnstonC. Land-Kazlauskas (2018). Organizing On-Demand: Representation, Voice, and Collective Bargaining in the Gig Economy, Geneva: International Labour Office.
L. F. A. KatzB. Krueger (2016) ‘The rise and nature of alternative work arrangements in the United States, 1995–2015’, NBER Working Paper No. 22667, Cambridge, MA: National Bureau of Economic Research.
J. A. D. KrosnickF. Alwin (1987) ‘An evaluation of a cognitive theory of response-order effects in survey measurement’, Public Opinion Quarterly, 51 (2) 201–219.
I. Mandl (2016) ‘Working conditions in crowd employment’ in W. WobbeE. BovaC. Dragomirescu-Gaina (eds) The Digital Economy and the Single Market, Brussels: Foundation for European Progressive Studies:111–140.
K. MellahiL. Harris (2016) ‘Response rates in business and management research: An overview of current practice and suggestions for future direction’, British Journal of Management, 27 (2):426–437.
C. A. MoserG. Kalton (1971) Survey Methods in Social Investigation, 2nd ed., London: Heinemann Educational Books.
R. O’FarrellP. Montagnier (2020) ‘Measuring digital platform-mediated workers’, New Technology, Work and Employment, 35 (1):130–144.
J. Prassl (2018) Humans as a Service: The Promise and Perils of Work in the Gig Economy, Oxford: Oxford University Press.
N. Rangaswamy (2019) ‘A note on informal economy and ICT’, Electronic Journal of Information Systems in Developing Countries, 85 (3).
J. RossL. IraniM. S. SilbermanA. ZaldivarB. Tomlinson (2010) ‘Who are the crowdworkers? Shifting demographics in Mechanical Turk’, Proceedings of the 28th International Conference on Human Factors in Computing Systems, CHI 2010, Extended Abstracts Volume: 2863–2872.
F. SarracinoC. A. F. RiilloM. Mikucka (2017) ‘Comparability of web and telephone surveys for the measurement of subjective well-being’, Survey Research Methods, 11 (2):141–169.
R. Tourangeau (2018) ‘Data collection mode’ in D. L. VannetteJ. A. Krosnick (eds) The Palgrave Handbook of Survey Research, Cham: Palgrave Macmillan.
P. TubaroC. Le LudecA. A. Casilli (2020) ‘Counting “micro-workers”: Societal and methodological challenges around new forms of labour’, Work Organisation, Labour & Globalisation, 14 (1):67–82.
S. VallasJ. B. Schor (2020) ‘What do platforms do? Understanding the gig economy’, Annual Review of Sociology, 46:273–294.
N. van Doorn (2017) ‘Platform labor: On the gendered and racialized exploitation of low-income service work in the “on-demand” economy’, Information, Communication & Society, 20 (6):898–914.
C. C. WilliamsF. Schneider (2016) Measuring the Global Shadow Economy: The Prevalence of Informal Work and Labour, Cheltenham: Edward Elgar.
A. J. WoodM. GrahamV. LehdonvirtaI. Hjorth (2019) ‘Good gig, bad gig: Autonomy and algorithmic control in the global economy’, Work, Employment & Society, 33 (1):56–75.
J. ZumbrunA. L. Sussman (2015), ‘Proof of a “gig economy” revolution is hard to find’, Wall Street Journal, 26 July.

Comments

Comment on this article

[1] E. ArmanoA. BoveA. Murgia (eds) (2017) Mapping Precariousness, Labour Insecurity and Uncertain Livelihoods: Subjectivities and Resistance, London: Routledge.

[2] K. BahiaS. Suardi (2019) Connected Society: The State of Mobile Internet Connectivity 2019, London: GSMA.

[3] P. P. BiemerE. D. de LeeuwS. EckmanB. EdwardsF. KreuterL. E. LybergN. C. TuckerB. T. West (eds) (2017) Total Survey Error in Practice, New York: John Wiley & Sons.

[4] M. CallegaroK. L. ManfredaV. Vehovar (2015) Web Survey Methodology, London: SAGE.

[5] E. D. de Leeuw (2008) ‘Choosing the method of data collection’ in E. D. de LeeuwJ. J. HoxD. A. Dillman (eds) International Handbook of Survey Methodology, Hove: Psychology Press:113–135.

[6] E. D. de Leeuw (2018) ‘Mixed-mode: past, present and future’, Survey Research Methods, 12 (2):75–89.

[7] E. de LeeuwN. Berzelak (2016) ‘Survey mode or survey modes?’ in C. WolfD. JoyeT. W. SmithY.-C. Fu (eds) The Sage Handbook of Survey Methodology. London: Sage Publications.

[8] S. DenchR. IphofenU. Huws (2004) An EU Code of Ethics for Socio-Economic Research, Brighton: Institute for Employment Studies.

[9] X. DengR. D. GalliersK. D. Joshi (2016) ‘Crowdworking – a new digital divide? IS design and research implications’, Association for Information Systems Research Papers, 148. https://aisel.aisnet.org/ecis2016_rp/148

[10] V. De Stefano (2016) The Rise of the ‘Just-in-Time Workforce’: On-Demand Work, Crowdwork and Labour Protection in the ‘Gig-Economy’, Geneva: International Labour Office.

[11] D. J. C. DodouF. de Winter (2014) ‘Social desirability is the same in offline, online, and paper surveys: A meta-analysis’. Computers in Human Behavior, 36:487–495.

[12] Eurofound (2015) New Forms of Employment, Luxembourg: Publications Office of the European Union.

[13] A. Felstead (2021) ‘Are online job quality quizzes of any value? Selecting questions, maximising quiz completions and estimating biases’, Employee Relations: The International Journal, 43 (3):724–741.

[14] F. Flanagan (2019) ‘Theorising the gig economy and home-based service work’, Journal of Industrial Relations, 61 (1):57–78.

[15] F. J. Fowler Jr (2014) Survey Research Methods, 5th ed., Thousand Oaks, CA: SAGE.

[16] P. GalvinA. BiermanS. Schieman (2021) ‘Über-alienated: Powerless and alone in the gig economy’, Work and Occupations, published online 22 June 2021.

[17] A. Gooch (2015) ‘Measurements of cognitive skill by survey mode: Marginal differences and scaling similarities’, Research and Politics, July–September:1–11.

[18] A. GreenM. de HoyosS.-A. BarnesB. BaldaufH. Behle (2014) Exploratory Research on Internet-Enabled Work Exchanges and Employability: Analysis and Synthesis of Qualitative Evidence on Crowdsourcing for Work, Funding and Volunteers, European Commission Joint Research Centre Institute for Prospective Technological Studies, JRC Scientific and Policy Reports, Luxembourg: Publications Office of the European Union.

[19] U. Huws (2016) A Review on the Future of Work: Online Labour Exchanges, or ‘Crowdsourcing’: Implications for Occupational Safety and Health, Bilbao: European Agency for Safety and Health at Work.

[20] U. HuwsN. H. SpencerM. CoatesK. Holts (2019) The Platformisation of Work in Europe: Results from Research in 13 European Countries, FEPS – Foundation for European Progressive Studies, UNI Europa, Hatfield: Hertfordshire Business School, University of Hertfordshire.

[21] K. F. Jesnes Braesemann (2019) Measuring Online Labour: A Subcategory of Platform Work, Nordic Future of Work, Brief 2, Oslo: Fafo Institute for Labour and Social Research.

[22] H. JohnstonC. Land-Kazlauskas (2018). Organizing On-Demand: Representation, Voice, and Collective Bargaining in the Gig Economy, Geneva: International Labour Office.

[23] L. F. A. KatzB. Krueger (2016) ‘The rise and nature of alternative work arrangements in the United States, 1995–2015’, NBER Working Paper No. 22667, Cambridge, MA: National Bureau of Economic Research.

[24] J. A. D. KrosnickF. Alwin (1987) ‘An evaluation of a cognitive theory of response-order effects in survey measurement’, Public Opinion Quarterly, 51 (2) 201–219.

[25] I. Mandl (2016) ‘Working conditions in crowd employment’ in W. WobbeE. BovaC. Dragomirescu-Gaina (eds) The Digital Economy and the Single Market, Brussels: Foundation for European Progressive Studies:111–140.

[26] K. MellahiL. Harris (2016) ‘Response rates in business and management research: An overview of current practice and suggestions for future direction’, British Journal of Management, 27 (2):426–437.

[27] C. A. MoserG. Kalton (1971) Survey Methods in Social Investigation, 2nd ed., London: Heinemann Educational Books.

[28] R. O’FarrellP. Montagnier (2020) ‘Measuring digital platform-mediated workers’, New Technology, Work and Employment, 35 (1):130–144.

[29] J. Prassl (2018) Humans as a Service: The Promise and Perils of Work in the Gig Economy, Oxford: Oxford University Press.

[30] N. Rangaswamy (2019) ‘A note on informal economy and ICT’, Electronic Journal of Information Systems in Developing Countries, 85 (3).

[31] J. RossL. IraniM. S. SilbermanA. ZaldivarB. Tomlinson (2010) ‘Who are the crowdworkers? Shifting demographics in Mechanical Turk’, Proceedings of the 28th International Conference on Human Factors in Computing Systems, CHI 2010, Extended Abstracts Volume: 2863–2872.

[32] F. SarracinoC. A. F. RiilloM. Mikucka (2017) ‘Comparability of web and telephone surveys for the measurement of subjective well-being’, Survey Research Methods, 11 (2):141–169.

[33] R. Tourangeau (2018) ‘Data collection mode’ in D. L. VannetteJ. A. Krosnick (eds) The Palgrave Handbook of Survey Research, Cham: Palgrave Macmillan.

[34] P. TubaroC. Le LudecA. A. Casilli (2020) ‘Counting “micro-workers”: Societal and methodological challenges around new forms of labour’, Work Organisation, Labour & Globalisation, 14 (1):67–82.

[35] S. VallasJ. B. Schor (2020) ‘What do platforms do? Understanding the gig economy’, Annual Review of Sociology, 46:273–294.

[36] N. van Doorn (2017) ‘Platform labor: On the gendered and racialized exploitation of low-income service work in the “on-demand” economy’, Information, Communication & Society, 20 (6):898–914.

[37] C. C. WilliamsF. Schneider (2016) Measuring the Global Shadow Economy: The Prevalence of Informal Work and Labour, Cheltenham: Edward Elgar.

[38] A. J. WoodM. GrahamV. LehdonvirtaI. Hjorth (2019) ‘Good gig, bad gig: Autonomy and algorithmic control in the global economy’, Work, Employment & Society, 33 (1):56–75.

[39] J. ZumbrunA. L. Sussman (2015), ‘Proof of a “gig economy” revolution is hard to find’, Wall Street Journal, 26 July.