Longitudinal Impact

Overview

Researchers in CeRDI are currently implementing a programmatic approach to measuring the impact of technology on improved decision making and on supporting practice change. To achieve this impact assessment goal the researchers are applying a longitudinal research design across a range of CeRDI programmatic themes, with a specific focus on the areas of:

• Natural Environment
• Agriculture
• Hazard Planning and Resilience
• Health and Wellbeing
• Heritage and Culture, &
• Regional Planning

This research work has emerged in response to the contemporary global movement to maximise, through technology, access to information by a greater number of people. In parallel to this movement is a distinct and recurrent call, at the national and international level, to prove the value case for making data available (Keserĕ & Kin-sing Chan, 2015; Research Development Corporation 2015; Uhlir et al., 2010). The longitudinal research program seeks to provide answers to important, widely-asked questions about the economic, social, environmental and industry benefits of open access data. The ultimate aim of the research is to provide insights to inform government and industry about the benefits of funding and making available, open access data.

The research approach

Measuring the impact of technology on decision making, and on practice change, is complex. Current research is limited in approach, offering few insights about the motives of people to use technology, and the extent to which technology informs their decision making. Too often research focuses on quantitative measures such as level of use of a portal, tick box assessments around data quality and analytics relating to a range of site access features. There is limited work available which explores why individuals seek data, how they access it and, importantly, what impact it has on building evidence, shaping decisions and informing practice. Without this type of insight the ability to accurately measure the impact of technology into the longer term is limited. To address this, research staff at CeRDI have developed and implemented a structured mixed method approach for measuring the impact of open access data, technology and innovation; the Multi- Bounded System Design (MBSD) research model.

The foundational research approach for the MBSD comprises case study research, using qualitative and quantitative evidence gathering methods, with each case examined classified as a bounded system. A case study approach allows for the exploration of the narrative relating to adoption of innovation, as well as the collection of more standard quantifiable data relevant to uptake.

The goal in this type of research is to understand the boundaries and the complexity of behaviour patterns and approaches in relation to a particular issue(s), intervention(s) and/or innovation(s) (Stake, 1994; 1995; Yin, 1984; 1994; 2003; 2009; 2014). This approach increases capacity for external generalisability of results (due to a deceased likelihood of findings being assessed as unique to a single case) and provides for greater analytical benefits from replicable and/or comparative findings (Yin, 2003: 54). By undertaking research within and across each of the identified bounded systems, the MBSD will enable the identification of consistent trends and emerging impact at the programmatic level and, through cross program analysis, at the broader level of technological innovation in general. These insights mean that approaches can be altered or consolidated across a broad range of bounded systems, dependent on identified critical success factors for innovation (Corcoran et al., 2004).

A single bounded system within the context of this longitudinal research project refers to a single CeRDI research theme. These research themes, and the portals within them, can be accessed via this website (research themes: www.cerdi.edu.au, portals: www.cerdi.edu.au/FedUniSpatial). Research undertaken simultaneously across a range of research themes creates a multi-bounded system, as illustrated in the following figure:

The capacity for replication and validation of findings across multiple bounded systems addresses perceived criticisms that social research has low levels of research rigour in a number of ways. Validation of findings occurs through replication of research across a range of bounded systems (research themes) and through the development of a strong systemic procedure which ensures consistency and uniformity in data collection.

The application of a mixed method research approach strengthens data validity through triangulation. This strong qualitative study design builds validity through source and methods data triangulation.  Data is drawn from a range of sources (end users, government representatives, community groups, organisations and managers) and the same question is tested trough a range of methods (survey, analytics, interviews and organisational documentation).

All CeRDI Study Designs for the MBSD, as a fundamental element of study design planning, will involve, at a minimum these two triangulation approaches.

Additionally, to maximise validity, a systematic research procedure is implemented across all of the CeRDI research themes when determining inclusion in the longitudinal research (through consistent selection criteria) and the type of information gathered by the research process (through uniform research questions).

1. Selection criteria have been established which assist in determining which cases are included in the MBSD research approach. These criteria are available for review through contacting CeRDI  directly.
   
   The development and application of selection criteria ensures that decisions to include or exclude a system from the research process is determined by a fixed framework. This maximises the capacity for comparability across bounded systems and minimises the potential for error in data analysis.
    
2. Research questions: Overarching foundational research questions assists in guiding data collection across all bounded systems involved in the research by ensuring that the same data insights will be sought across each of the bounded systems, providing a mechanism to maximise comparability of knowledge discoveries.
   
   The five overarching foundational research questions have been developed to build consistency across CeRDI bounded systems research are:
    
   1. What strategies in implementation of the technological innovation being researched within an identified programmatic theme have worked most effectively to support and facilitate data access to the maximum numbers of potential end users?
   2. What approaches applied during the research relationship for the individual programmatic theme were found to maximise stakeholder input to the knowledge building process?
   3. What have been the key impacts, in the identified programmatic area, for a range of key stakeholders, including business, industry, community and researchers?
   4. How has data that has been captured within a programmatic theme, been used to build evidence based knowledge repositories to maximise access and minimise research duplication over the longer term?
   5. In what ways does open data access, as provided through the tools and mechanisms of eResearch, impact on decision making and facilitate practice change within a programmatic theme domain?

The capacity to apply the same questions across a range of bounded systems allows a consistent pool of new knowledge to be built, particularly as the same information will be collected yearly for all bounded systems (research themes) across a four year data collection timeframe.

There timeframes for the four waves of research are:

1. Wave 1 research occurred across 2015 – 2017. This timeframe was extended across 2 years as the initial process of establishing a uniform methodology, identifying themes and ensuring uniformity in selection required additional planning and implementation times.
2. Wave 2 will occur across late 2017 to mid 2018.
3. Wave 3 will occur across late 2018 to mid 2019.
4. Wave 4 will occur across late 2019 to md 2020.
    

Wave 1 research will be completed during 2017 although, based on the extensive work that has been done to date, a number of consistent findings relating to the impact of technology on enhanced decision making for practice change have already emerged. These findings show that:

Theme 1: Positive shifts in usage trends are facilitated by high levels of ease of accessibility and confidence in data quality. Upward trends in frequency of use and repeat visitation rates are dependent on these factors and the way in which they enable the resource to be viewed as a viable and sustainable mechanism for information access.

Theme 2: The dominant characteristic of online technology, particularly in relation to access and data reliability, is that it is emerging as instrumental in facilitating the breakdown in knowledge silos. This characteristic, if developed to maximum potential, can shift the dominant paradigm from one of knowledge controlled by individual information providers to one of knowledge sharing and increased accessibility at the local, state and national level for all end users.

Theme 3: Positive experiences for end users in accessing online technology can, over time, shift how they interact with the resource – changing the relationship from one in which the end user accesses portals infrequently to a more complex interaction which includes end user information input, advice seeking and participation in the process of building/enhancing knowledge.

Theme 4: The provision of diverse clusters of complex information through the mechanism of multiple data sets increases, in an upward trajectory, the relevance and applicability of data and consequently provides end users with a resource to strengthen planning and facilitate informed decision making.

Theme 5: Single access, multiple data set web portals enhance capacity across community and industry in terms of the provision of:

• Timely, informed and accurate responses to those seeking information/answers to queries;
• Improved mechanisms for making informed decisions - maximising the potential for good outcomes and positive developments across a field of study; and,
• Increased potential for collaboration and connection through shared local and national data sets.

Theme 6: The provision of multiple data sets with a single point of access, and the facility of interoperability, establishes a unique opportunity to collate, cross reference and consolidate data that has historically been hidden. This shift establishes a new foundation in accessing research ready data sets and a new capacity for achieving research discoveries. 

Theme 7: Technological innovation allows for information access and sector engagement capacities that are not available through more traditional forms of information sharing. This is a notable advance however the introduction of new technology needs to be carefully managed. Before adding more features or datasets it is important, for maximum success capacity, to ensure there is programmatic support to maintain and enhance the usability and capacity of the resource.

Theme 8: Sustainability of technological innovations beyond seed funding requires targeted marketing and a capacity to provide a solid body of evidence on the successes of the innovation and the benefit it provides to the community and the industry for whom the technology has the greatest relevance.

Where to from here?

The longitudinal research program will continue in 4 waves until conclusion in 2020. Wave 1 research data collection has concluded and a series of reports will be available on the CeRDI website, within each of the research themes, by late 2017. As identified previously, Wave 2 data collection will commence in August 2017, Wave 3 in 2018 and Wave 4 in 2019 with an overarching report capturing final results across all waves and all research themes to be completed by 2020.

While this longitudinal research study will not be concluded for a number of years, the intent, supported by the findings of Wave 1 research, is that this will be a significant body of research with the capacity to build the evidence on impact of open access and technological innovation on knowledge building and on decision making. It is an approach underpinned by a strong framework for ensuring data validity, generalisability and research replication across the eResearch environment.

These factors make it unique.

An example of CeRDI's longitudinal impact research

An example of CeRDI's longitudinal impact research can be taken from the Visualising Victoria's Groundwater project.

This ease of access and exploration of Victoria’s groundwater data means that VVG can now be readily adopted by water users, resource managers, landowners and conservation groups to inform their decisions about managing consumptive water use and environmental water flows.

CeRDI impact research has identified that:

• A strength of VVG is that it is an interoperable resource with a single point of access allowing users to readily explore groundwater issues or the questions over time;
• VVG has led to productivity and efficiency gains through decreased time in data sourcing and processing, and in responding to inquiries;
• Users saw benefits in consolidating, combining and comparing data from multiple sources to identify data duplication, data gaps and/or data inaccuracies and emerging issues;
• The VVG portal had increased user capacity to manage the knowledge themselves; and
• The VVG portal is seen as a de facto information provider:
   
  • Since establishment, the VVG has become a central resource in terms of knowledge management and the provision of trusted advice on groundwater issues;
  • Ease of access and an open access policy plays an important role in building end user confidence in seeking information relevant to their needs;
  • VVG, through the provision of support, advice, and web services is addressing a service need not currently addressed in more traditional areas of groundwater information provision.

The provision of multiple datasets from disparate sources within a single portal has changed practices in the Victorian groundwater industry.