Validating The IS-Impact Model At Queensland University Of Technology (QUT): Part A

The overall research aims to develop a standardised instrument to measure the impacts resulting from contemporary Information Systems (IS). The research adopts the IS-Impact measurement model, introduced by Gable et al, (2008), as its theoretical foundation, and applies the extension strategy described by Berthon et al. (2002); extending both theory and the context, where the new context is the Human Resource (HR) system. The research will be conducted in two phases, the exploratory phase and the specification phase. The purpose of this paper is to present the findings of the exploratory phase. 134 respondents from a major Australian University were involved in this phase. The findings have supported most of the existing IS-Impact model’s credibility. However, some textual data may suggest new measures for the IS-Impact model, while the low response rate or the averting of some may suggest the elimination of some measures from the model.


INTRODUCTION
In a demanding business environment, organisations have invested heavily on Enterprise Systems (ES).However, economical factors and increasing competition influence organisations to cut down costs.These pressures make organisations keen to measure the success of the Information Systems (IS) and their impacts on both the organisation and the individuals to justify their value and contribution to the productivity, quality, and competitiveness of organisations (Gable et al, 2003).Nonetheless, for several reasons, the impacts resulting from ES are arguably difficult to measure -e.g.ES include a wide range of different applications (e.g.financials, human resource, facility management, sales and distribution, and manufacturing) (Markus and Tanis, 2000) that span the organisation, and a diversity of capabilities and functionality (Sedera, 2005) supporting many users ranging from top management to data entry operators.Further, ES implementations impose logical structures (Rabaa'i and Gammack, 2008;Rabaa'i, 2009) that can require massive structural and cultural changes (Al-Mashari et. al., 2003).
Evaluating the impacts of Information Technology (IT) is one of the critical issues in IS literature (Kim and Kim, 1999), as the impacts of IT are often indirect and influenced by human, organisational, and environmental factors (Petter et al, 2008).Yet, it is argued "if information systems research is to make a contribution to the world of practice, a well-defined outcome measure (or measures) is essential" (DeLone and McLean, 1992: 61).Studies concerned with evaluating information systems success has started since the late 1970's (Delone andMclean, 1992, 2003; Gable et al. 2003).However, "there is little consensus among practitioners or researchers on how best to measure the impact of IS in organisations" Gable et al. (2008: 378).As a result, academics as well as practitioners are still struggling with the question of which constructs best signify IS success."The problem lies in the ambiguity of the concept and the multiplicity of IS success constructs pervading the research" (Rai et al, 2002: 50).Given the size of ES investments and uncertain related benefits (e.g Knowles et al., 2000;Sedera et al., 2001) there is need for an economical and valid approach to the measurement of their impacts (Gable et al., 2003;Sedera et. al., 2001; Shang and Seddon, 2000).Because ES are more complex than traditional standalone IS, success measurement models used for other typical information systems evaluation may not be adequate for ES (Ifinedo, 2006).Gable, Sedera, and Chan (2008) report a relatively recent effort to validate their 'IS-Impact' measurement model against contemporary IS -SAP Financials and Oracle Financials.This study adopts the IS-Impact model (Gable et al. 2008) as the primary commencing theory-base.Akin to analytic theory 1  (Gregor, 2006), the IS-Impact model is conceptualised as a formative, multidimensional index, wherein the dimensions have a causal relationship with the overarching measure -IS-Impact.The study too employs the extension strategy as described by Berthon et al. (2002)  2 , extending both theory and the context; where the new context is different ES application; namely, the Human Resource (HR) system implemented at Queensland University of Technology (QUT).
The overall research will be conducted in two phases, the Exploratory Phase and the Specification Phase.The purpose of this paper is to present the findings from Phase 1: Exploratory Phase.There are two key objectives of the overall research.Firstly, the research will identify relevant new or confirm existing measures when extending the IS-Impact model into a new context (i.e. the HR System) and further test its validity, reliability and generalisability.Secondly, by using new sets of data, the research will statistically enhance the robustness and minimise limitations of the IS-Impact model, which will be adapted, extended and tested in relation to the HR system to yield a more generalisable IS-Impact measurement model (possibly a contingency model) for different system circumstances.
The following section briefly discusses the theoretical foundation of the proposed study.Then the approach and methodology are presented.This is followed by the results of the exploratory phase.The paper concluded with a discussion and implications of the findings and the conclusion.

THEORITICAL FOUNDATION
Through the review of the literature, it is noticed that a myriad of measures and dimensions for IS Success or IS Impact exist.However, structured and robust models, that capture the whole IS Success/Impact scenario are scarce (DeLone and McLean, 1992;Petter et al, 2008).Gable et al, (2008), in reference to Gregor's (2006) analytic theory, suggest a reference model or a theoretical framework should have the characteristics of a strong analytic theory, that meets the criteria in terms of utility, intuitiveness, mutual exclusivity, completeness, and where relevant, appropriate hierarchy.Gable et al. (2008) also suggested that, beyond those quality of analytic theory, a framework of IS Success/Impact should maximally reflect the full range of IS impact and accommodate the views of the multiple internal stakeholder groups.
DeLone and McLean (1992) reviewed 180 studies, both empirical and conceptual, and identified over 100 measures used to evaluate IS success.The DeLone and McLean (1992) IS Success model (D&M Model) is most widely cited (Heo and Han, 2003) (see also DeLone and McLean (2003)).Based on the work of Shannon and Weaver (1963) and Mason (1978), DeLone and McLean (1992) proposed an IS success model that synthesized and harmonized previously disparate measures reported.The D&M model consists of six major IS success constructs: System Quality, Information Quality, Use, User Satisfaction, Individual Impact, and Organisational Impact.Though the D&M model was a conceptual model, it formed the basis of much IS success research, with many researchers testing parts of the model empirically with case studies and surveys.In fact, 285 articles from the top IS journals referred to and made used of the model between the year 1993 to the mid 2002 (Delone and Mclean, 2003).Seddon (1997) was among the first to test the model (Delone and Mclean, 2003).Seddon listed several contributions from the D&M model, including: (1) it combines previous research, (2) it provides a scheme for classifying the different measures of IS success models that have been proposed in the literature into six dimensions.(3) It suggests a model of temporal and causal 1 The first of Gregor's (2006) five types of theory in IS, analytic theories, "analyse 'what is' as opposed to explaining causality or attempting predictive generalizations … they describe or classify specific dimensions or characteristics of individuals, groups, situations or events by summarising the commonalities found in discrete observations" (2006: 612). 2 According to Berthon et al., (2002) there are three research strategies, namely: replication, extension, and generation.
The importance of such extension strategy (theory and context extension) is that it discovers whether theories that explain a phenomenon in one context can effectively explain it in another context.It also tests whether a method that works in one context can efficiently work in another context.Gable et al. (2008) stated that one of the limitations of their study was that the study has been only conducted in the Australian public sector.In their opinion this limitation affects the generalisability of the model and they recommended that further extension studies be conducted to generalize the model.
interdependencies between the identified categories.( 4) it has been considered an appropriate base for further empirical and theoretical research, and ( 5) it has met general acceptance in the IS community (Seddon, 1997).The D&M model has received criticism from several researchers, including Seddon (1997), who criticised the model in two points: (1) the model combined both causal and process relationship explanations, which is confusing; and (2) The Use dimension is ambiguous and is not appropriate for causal relationship explanations.
A relatively recent model to measure IS success or impact is the IS-Impact Measurement Model introduced by Gable et al. (2008) (see also Gable et al. (2003) and Sedera and Gable (2004)).Gable et al. (2008) define the IS-impact of an Information System (IS) as "a measure at a point in time, of the stream of net benefits from the IS, to date and anticipated, as perceived by all key-user groups".According to Gable et al. (2003), the driver for the study is the lack of reliable standardized and empirically validated measurement model for IS success.According to Gable et al, (2008: 389-390) "Individual Impact is a measure of the extent to which (the IS) has influenced the capabilities and effectiveness, on behalf of the organization, of keyusers, Organizational Impact is a measure of the extent to which (the IS) has promoted improvement in organizational results and capabilities Information Quality is a measure of the quality of (the IS) outputs: namely, the quality of the information the system produces in reports and on-screen, and System Quality is a measure of the performance of (the IS) from a technical and design perspective".
In attention to proliferation of overlapping measures, as a consequence of success (and antecedent) rather than a construct (see Figure 2).
The complex, multi-dimensional nature of IS success is represented by four constructs.The four-dimensional IS-Impact measurement model consists of two halves; the "impact" half includes Organizational-Impact and Individual-Impact constructs, "The IS-Impact Model is a holistic index representing the stream of net benefits; the 'impact' half measuring net benefits to date, while the 'quality' half, forms our" best" proxy measure of probable future impacts, with 'impacts' being the common denominator" (Gable et al., 2008:381).
As mentioned earlier, the researcher has adopted the IS-Impact model as the theoretical foundation of this research.The IS-Impact model, by design, is intended to be robust, simple and generalisable, to yield results that are comparable across time, stakeholders, different systems and system contexts.The model and measurement approach employ perceptual measures and offer an instrument that is relevant to all key stakeholder groups, thereby enabling the combination or comparison of stakeholder perceptions 4 .

Queensland University of Technology (QUT) Profile
Located in Brisbane, Australia, Queensland University of Technology (QUT) traces its origin back to 1849, with the establishment of Brisbane School of Arts.Through the years, the institution evolved several times, eventually becoming Queensland University of Technology in January 1989.Its original goal is "To strengthen its distinctive national and international reputation by combining academic strength with practical engagement with the world of the professions, industry, government, and the broader community".This goal has inspired the University's dedication to the education of students, research in a broad range of disciplines, and service to the state's citizens.QUT is focused on being a university for the real world, delivering relevant and practical courses leading to excellent graduate outcomes.The University continues to refine its winning balance of theory and practical experience with the addition of the latest technology in their teaching and has a world-class record of success in industry-linked research.
QUT is part of a three-campus system and now is home to several national research centres and research institutes supported by government and philanthropic bodies.Since 1990, QUT is using ALESCO/Talent2 Human Resource system to manage and control its HR activities and payroll.

Research Design
The  The Identification-survey (1 st survey) aims to generate a comprehensive inventory of 'impact statements' of the system.Respondents are prompted to describe (both closed and open questions), and anchor their responses in, the target Human Resource (HR) system.These statements are decomposed into their component impact-citations (content analysis), and then mapped into the IS-Impact model.The Specification-survey (the 2 nd survey) aims to further specify and test the a-priori model employing data gathered (primarily 7-point Likert scales) with an instrument that operationalises the dimensions, sub-dimensions and measures deriving from the identification-survey.The Confirmation-survey (the 3 rd survey) aims to further validate the study model and instrument deriving from the exploratory-phase, and to further illustrate the mutual exclusivity and additivity of the measures and dimensions in the Model using confirmatory data analysis techniques and new data.To complete the research cycle proposed by Mackenzie et al. (1979), construct validation tests similar to the Specification-Survey are conducted on the Confirmation-Survey data.
The Exploratory Phase will entail the conduct of the I-Survey.The nature of the, I-Survey is exploratory-qualitative.An instrument was developed with an open question aimed to capture IS impact statements from the respondents.The relevant population of the I-Survey was the set of individuals that uses ALESCO (the HR) system from all employment cohorts.This will ensure the multiple perspectives of the responses.Access to participants was identified from a contact list of the systems' users provided to the researcher by the HR Director at QUT.The I-Survey was disseminated by email.
The I-Survey was divided into two sections.Section one; collects demographic information about the participants, including: gender, age, business title, department, duration of employment with their current role, and description of their current jobs; this will help in identifying and distinguishing key-user groups of the systems.Section two contains only one question: "what do you consider have been the impact of ALESCO (the human resource system) in your division/department?" this question will help in gathering salient IS-Impact statements from the participants.

Description Name of the Organisation
Queensland University of Technology (QUT)

Research Method
A qualitative survey distributed by email

Instrument Type
Open-ended Qualitative Question in a survey instrument: What do you consider have been the impact of ALESCO (human resource) system in your division/department?

Sample Selection
All the users of ALESCO system

FINDINGS AND DISCUSSION
A number of observations were made based on the findings from the Exploratory Phase.These implications will be discussed in this section.Some textual data may suggest new measures for the IS-Impact model, while the low response rate or the averting of some may suggest the elimination of some measures from the model.The key-user groups of ALESCO system and implications for the Specification Phase will be discussed.

Key-User Groups
Key-user groups according to Gable et al. (2008: 386-387), "would include the main groups of direct users of the IS -those users who access the system directly, or who use its direct outputs".Based on Anthony's (1965) framework, for employment cohort classification in management science, Shang and Seddon (2000;2002), Singletary et al. (2003), and according to the respondents' job title, their job role and involvement with the ALESCO (HR) system; the researcher classifies key-user groups of ALESCO (HR) system into four levels, including: ( The Strategic level Users, focuses on deciding organisational-wide objectives and allocates necessary resources to achieve the objectives.The Strategic level is involved in complex, irregular decision making and relies on predictive information for long term organisational goals. (2) The Management Level Users, information requirements are focused on assuring that the resources, both human and financial, are used effectively and efficiently to accomplish goals stated at the strategic level.The characteristics of information required by the management level are different to those required at the Strategic level.Managers tend to prefer integrated, procedural information that is for a precise task.(3) The Operational Level Users, employees are involved in highly structured and specific tasks that are routine and transactional.The operational level tends to deal with real time data and interact with the system in daily-basis.(4) The Technical Level Users, employees are those who maintain the system, writing scripts and dealing with different technical issues associated with the ALESCO (HR) system-with 17, 37, 66, and 14 respondents, respectively.The classification made Table 2 summarises the respondents' demographics'.

Citation Mapping
The citation mapping is the process of mapping the citations (impact statements) from the respondents to the existing dimensions and measures of the IS-Impact model.The I-Survey has generated textual responses from the respondents reflecting their observation of the impact of ALESCO on their departments/divisions and their individual work.The researcher then decomposed these textual responses into short sentences, and refers to them as 'impact statements' yielding 411 impact statements (average of approximately three (3) impact statements per respondent).Decomposition of the text was straightforward, simply involving the extraction of contiguous phrases, without modification.These impact statements were then mapped into relevant IS-Impact measure.Each statement must explicitly carry a single piece of information about the impact of ALESCO.The citation decomposition and mapping process was done using Nvivo (ver.8), a qualitative data analysis tool.All of the citations were closely related and mapped directly into the four dimensions of the IS-Impact model.The researcher acknowledges that the sample size for the I-Survey is small, but the findings are valuable and can provide the missing elements of IS-Impact model for the Human Resource systems.However, eliminating, modifying or adding measures to a pre-rigorously validated model, like the IS-Impact, requires further evidence and validation from the empirical data of these measures in the Specification and the Confirmation Phases.

CONCLUSION AND FUTURE RESEARCH
The overall research adopts the IS-Impact measurement model introduced by Gable et al, (2008) to develop a standardised instrument for measuring IS-Impact by employing the extension strategy as described by Berthon et al. (2002); extending both theory and the context, where the new context is the ALESCO (HR) system implemented at (QUT).The overall research will be conducted in two phases, the exploratory phase and the confirmation phase.
Figure 1 depicts the IS-Impact Measurement Model.The IS-Impact model, which is based in DeLone and McLean's work, overcomes many concerns with past IS Success models.Gable et al.(2003) pointed out that the IS-Impact Model deviates from the traditional DeLone and McLean model in the following ways: (1) it depicts a measurement model and does not purport a causal/process model of success, (2) it omits the use construct, (3) satisfaction is treated as an overall measure of success, rather than as a construct of success, (4) new measures were added to reflect the contemporary IS context and organisational characteristics, and (5) it includes additional measures to probe a more holistic organisational impacts construct.The IS-Impact model has been extensively validated statistically and uses mainly perceptual measures.According toGable et al. (2008), the model was developed in two phases: the exploratory phase and the confirmatory phase.Two surveys were conducted in the exploratory phase where the purpose of the first one is identify success measures and the purpose of the second is to test what is called a priory model.In the confirmatory phase, the model was tested for reliability and validity using different data set.

Figure 1 :
Figure 1: The IS-Impact Measurement.From Gable et al., (2008) Firstly, in the exploratory phase, the model was empirically tested using survey data gathered from 456 respondents representing twenty-seven Queensland public sector organizations that implemented SAP R/3 in the late 1990s.The a priori model was initially based on the DeLone and McLean's model with exclusion of the "Use" construct.After testing the priory model for construct validity, user satisfaction construct were taken out because it loaded along with the System Quality.As a result, the model consists of four constructs: Information Quality (IQ), System Quality (SQ), Individual Impact (II), and Organisational Impact (OI).The model was next tested at Queensland University of Technology (QUT) through confirmatory factor analysis of 157 survey responses regarding their ORACLE Financials systems.Analysis of the secondround, confirmatory, 'weights' survey data (model testing), demonstrated the discriminate validity of the four constructs, as well as their convergence on the single higher-order phenomenon -IS success.Criterion validity testing further demonstrated the additivity of the four constructs of success, and the completeness of the resultant over-arching, second-order measure of IS success.Figure 2 demonstrates the 37 measures of the IS-Impact measurement model.

Figure 2 :
Figure 2: The 37 Measures of the IS-Impact Model.FromGable et al, (2008: 390) study employs a longitudinal, multi-method research design, extending the research cycle proposed byMacKenzie and House (1979),McGrath (1979), andGable et al. (2008) for developing and validating a measurement model.The research design (Figure3) entails two main phases and three surveys: (1) an exploratory-phase, to develop the hypothesised measurement model, and (2) a confirmatory-phase, to test the hypothesised measurement model against new data gathered.The exploratory phase adheres with the two-step approach of Burton-Jones and Straub (2006) for operationalising constructs and identifying measures, the related aim being to adequately account for the context of contemporary IS, and to ensure model completeness and an appropriate and complete choice of measures and dimensions.The exploratory phase consists of two surveys, an identification-survey followed by a specification survey.The identification-survey, akin to the 'function' phase of the Burton-Jones et al. (2006) approach, is intended to identify the salient dimensions and measures of the study IS-Impact conceptual model; these dimensions and measures will later become the basis of an a-priori model to be operationalised in the specification-survey.As mentioned earlier, this study will report on the I-survey of the Exploratory Phase.Figure3depicts the instrument development.

3
Gable et al. (2008) argue that User satisfaction has been possibly the most extensively employed single measure for IS evaluation [as cited in Gable et al., (2008): DeLone and McLean, 1992, Doll and Torkzadeh, 1988a, Etezadi-Amoli and Farhoomand, 1991, Gatian, 1994, Igbaria and Tan, 1997, Lucas, 1975].Several widely cited studies developed Gable et al. (2008)s that measure satisfaction[Bailey and Pearson, 1983, Baroudi and Orlikowski, 1988, Doll and  Torkzadeh, 1988a].Early satisfaction constructs in IS success evaluation (e.g., user information satisfaction-Bailey and Pearson 1983) have been found to mix measures of multiple success constructs (e.g.quality and impact) rather than measuring a distinct satisfaction construct[Gable, 1996].Rai et al (2002), state that user satisfaction has been measured indirectly through Information-Quality, System-Quality and other variables in prior studies.Additionally,[Sedera and Tan, 2005] demonstrated -through content analysis of 192 satisfaction-related items from 16 Satisfaction instruments -that 98% (189) of the measures readily map into existing measures pertaining to: System-Quality, Information-Quality, Individual-Impact and Organizational-Impact; with only 2% of the items (3 items) appearing to measure Satisfaction explicitly.4Note, this research, in attention to revalidate the model in new context, will employ the 37 measures of the IS impact model based on the a-priori model developed byGable et al. (2008).
Today, QUT employed approximately 5,000 employees (Full time equivalent).QUT's enrolment is approximately 40,000 students who study in the university's nine faculties-Built Environment and Engineering, Business, Science and Technology, Creative Industries, Law, Humanities, Education, Health, as well as QUT International College.QUT's annual budget reached more than AU$ 500 million in 200.QUT was among the first universities in Australia to Implement ES.

Table 1 :
Summary of the exploratory phase context

Table 3
summarises impact statements for each of the four Gable et al.'s (2008) dimensions of IS Impact, by key user group and in total.

Table 3 :
Mapping of impact statements according to key-user groups and total.Finally, Strategic Users were inclined to give impact statements that relate to the 'Organisational Impact'(21, 5%), Management Users expressed concerns about 'Individual Impact' (34, 8%), and Operational Users responses were more related to 'Information Quality' (86, 21%) than other impact dimensions, while Technical Users provided impacts that are allied to 'System Quality' (45, 11%).The findings of the I-Survey confirm the existence of patterns between ALESCO key-user groups and the IS-Impact dimensions.Strategic Users respondents tend to be more informed about the Organisational Impact and Information Quality, Management Users place more emphasis on Individual Impact, Operational Users were highly correlated to Information Quality and Individual Impact and Technical Users were more informed about the System Quality (see Table3).These findings highly support whatGable et al. (2008: 387)suggest "It is noted that the largest number of citations by Strategic users pertain to the Organizational Impact and Information Quality dimensions, the largest number by Operational users pertain to System Quality and Information Quality, and the largest number by Technical-users pertain to System Quality.This may suggest the relative closeness of these employment cohorts to the respective dimensions in their overall evaluation of IS-Impact.",with only one exception; Operational Users for ALESCO system were more comfortable and better informed to address issues highly correlated to the Individual Impact not the System Quality which oppose Gable et al's (2008) findings.Table 4 presents the counts and distributions of the impact statements according to the measures of the IS-Impact model.The averting of some measures like: Sophistication (0% of total System Quality statements) and Uniqueness (0% of total Information Quality statements) as well as the low response rate for Cost Reduction (2% of total Organisational Impact statements) and Customisation (1% of total System Quality statements) may suggest the elimination of these measures. A number of statements were not mapped into any of the IS-Success model dimensions, including: Data Security and Privacy (17 statements), Training Programs (24 statements), quality of the training programs (18 statements), the IT support provided to ALESCO users by the IS personnel (14 statements), the knowledge and experience of the IS personnel (12 statements), The interest of the IS people to solve problems (9 statement), The response time for new functionalities (8 statements), and the relationship with the IS personnel (6 statements).
A large portion of citations (158, 38%) pertain to 'Information Quality' (IQ), followed by 'System Quality' (SQ), 'Individual Impact' (II), and 'Organisational Impact' (OI) -with (98, 24%), (90, 22%), and (65, 16%) impact statements, respectively.Secondly, table 3 also demonstrates the instantiation of dimensions of the IS-Impact model in relation to four employment cohorts, for ALESCO system, at QUT. (175, 43%) of the impact statements came from Operational Users, followed by (97, 24%) from Management Users, (86, 21%) from Technical Users, and (53, 13%) from Strategic Users.In the 'Individual Impact' dimension, respondents emphasised Individual productivity (36, 40% of total Individual Impact statements), followed by Decision Effectiveness (23, 26%).For the 'Organisational Impact' dimension, (27, 42% of total Organisational Impact statements) are related to Business process change, and (8, 12%) related to Staff Requirements and e-Business equally.It is also noted that (1, 2%) respondent consider Cost reduction as a crucial impact of ALESO in this sample.Inthe'System Quality' dimension, respondents highlighted Ease of Use (15, 15% of total System Quality statements), followed by Flexibility (14, 14%).It is also noted though (0, 0%) and (1, 1%) of the respondents raised concerns about Sophistication and Customisation respectively.Forthe 'Information Quality' dimension, a large portion of the respondents (32, 20% of total Information Quality statements) acknowledged Content Accuracy as an essential impact of ALESCO, and followed by (26, 16%), (24, 15%), and (23, 15%) of the respondents emphasised Importance, Availability, and Timeliness respectively.However, (0, 0%) of the respondents regards Uniqueness as a significant impact of ALESCO in this sample.The results of the I-Survey support the existing IS-Impact model's credibility.Yet, the results of this phase are worth investigating in the next research phase, the Specification Phase, for three main reasons, including: 

Table 4 :
The distribution of the impact statements to all measures in the IS-Impact model.