A key aspect of the EVIDENT project is its five use cases consisting of a series of experiments, quasi-experiments and a serious game which explore the impact of behavioural biases on energy use (see a description of each of these here). An important aspect of designing these use cases is piloting, where researchers work with members of the general public to check the usability, acceptability and accessibility of the proposed use case design. This is an important step as it allows researchers to ensure that use cases reflect what matters to participants and ensure that the research captures meaningful data, enhancing external validity. This is particularly important when exploring energy behaviour, as challenges often emerge in engaging individuals in these experiments [1], [2]. This is particularly problematic in the context of energy behaviour, where consequences are often small, delayed and uncertain, and where the immediate impact of behaviour is invisible further limiting uptake of behaviour change initiatives. To increase engagement in EVIDENT, we must ensure that use cases accurately reflect the real-world contexts ad situations that participants encounter [3], are easy to understand and easy to access for vulnerable populations [3], and digital natives [4] alike. User inclusion in design and piloting is one means through which this can be achieved. Through including representative samples in pilot testing, particularly those who are vulnerable or traditionally excluded from research, the acceptability and usability of EVIDENT use cases can be enhanced. This approach can aid both acceptability and validity by ensuring cultural narratives, expectations and social rules are considered and included in the development of experiments. This blog post provides a brief overview of the efforts undertaken by EVIDENT to include users in co-design through iterative pilot analyses to explore usability, acceptability and accessibility. Additional detail on these efforts can be found in [5] and [6].
Use Case 4: The EVIDENT Serious Game
The EVIDENT serious game is an online life simulation game in which a player is tasked with maintaining a virtual home, while making effective energy decisions. As described in a previous blog post (here), the player must navigate tricky energy decisions while determining how best to maintain their avatars comfort. Within the game, users find a household appliance has broken and must decide whether they would like to repair or replace this appliance. This decision point is presented as a choice experiment, through which participants’ willingness to pay for a repair is determined. The EVIDENT serious game has two aims 1) to determine the impact of financial literacy, demographic factors, and behavioural intention on decisions to repair or replace household appliances across resident types; and 2) to teach users how to make more effective repair/replace decisions, when considering both the financial and environmental impacts. However, for EVIDENT researchers to be able to answer these questions through the serious game, we need to ensure that the game reflects real-world scenarios participants would encounter; and is understandable and fun for participants. As such, a series of pilot workshops were conducted to ensure that the EVIDENT game effectively addressed these factors.
What did we do?
Across Autumn of 2022 the usability, acceptability and external validity of EVIDENT serious game was examined through a series of six pilot usability workshops. These workshops also explored how easy and fun the serious game was to play, alongside exploring the factors which might impact repair or replace decisions in natural settings. These workshops took place alongside iterative rapid-prototyping of the serious game, with updates made in response to workshop feedback. An outline of the structure of these workshops is presented in Figure 1 below. Each workshop consisted of a brief introduction, and opportunity to play the serious game, and group discussion. For both in-person and online games, co-located game play occurred with participants encouraged to share their thoughts as they played. Importantly, workshop participants (total = 44) reflected a variety of age groups (18-30=4; 31-40=6; 51-60=2; 61-70=3; 70+=3), employment types (n=16; part-time employed=2; retired=6; student=3), income levels (<€50,000 = 13; €50-75,000=4; €76-100,000=2; €101-200,000=1; >€200,000=1) and residence types (home-owner=13; renter =10; landlord=1).
Serious Game Workshop Procedure
What did we learn?
Results from the workshops suggested the positive impact of this approach on overall serious game usability and acceptability. While initial workshop results highlighted challenges with player engagement (i.e. understanding the display, use of prompts, player motivation etc.), and usability (i.e. understanding how to play, ease of use, confidence, etc.), each of these scores increased across successive workshops as attendee recommendations were adopted. As a result of this iterative inclusive approach to the EVIDENT serious game development the overall usability and acceptability of the game was enhanced. This can also be seen in the difference between initial (42.7) and final System Usability Scale Results (72.5) highlighting the positive effects achieved.
Key themes arising from focus group discussions highlighted that while participants felt that the serious game was a novel and engaging means to address energy decision making, several key aspects required attention to enhance acceptability and usability. Specifically, clearer detail on how to play the game and its aims were needed.
To address the feedback arising from the pilot study several changes to the serious game were made. Firstly, the pre-game tutorial was re-developed allow players the opportunity to practice prior to playing. Secondly, the amount of feedback given to players on in-game actions was increased, with visual (i.e. item is shaded in red to direct attention to it), and written prompts (i.e. pop-up text reminding players to take certain actions). To provide participants more information on the impact of their actions on their energy, financial and comfort gauges, additional feedback was provided. Specifically, text appears to provide feedback on the action, and the impact on the gauges shown (i.e. ‘Careful increasing the heat too high, look how much your energy use has risen’). Through this, greater participant understanding of the motivations and aims of the game is achieved.
Example in-game feedback and direction
Use Case 5: EVIDENT Discrete Choice Experiment to Investigate Energy Demand Curves
Consumers often tend to make ineffective energy investment decisions, tending to prefer the least expensive option, even if this option is not most beneficial for them [7], or will cost more [8]. A key factor impacting the decision to choose less expensive now, versus less expensive over time is temporal discounting. Temporal discounting refers to individuals’ preference for rewards which are immediate and certain, versus those which are delayed and uncertain. While lots of research has demonstrated the impact of temporal discounting on energy decision making, analysis of the impact of financial literacy, environmental literacy, and behavioural biases on discounting is outstanding. As such, use case 5 explores the impact of these factors on the decision to invest in home energy appliances through a series of choice experiments. Through this, greater understanding of the factors impacting decision making can be attained, and programmes developed to effectively support consumers. To ensure that this quasi-experiment effectively meets our aims, it is important that the choices included mimic real-world choices. Without this, our results may be biased or might miss important variables. When developing a choice experiment, researchers must choose the attributes (i.e. aspects of the items being chosen between such as energy rating, cost, energy use etc.) and levels (i.e. the differences between attributes across choice options, i.e. including an A, B and C rated appliance levels versus A, D and F) to be included. To ensure that the choice experiment can effectively answer its research question, it is important to gather feedback on which attributes should be included alongside review of the literature. As a result, to determine the attributes and levels for the EVIDENT choice experiment, a pilot usability analysis was conducted.
What did we do?
To explore the usability, acceptability and feasibility of this quasi-experiment, and to determine the final attributes and levels to be included in the choice experiment, a pilot study was conducted. A key additional focus of the pilot was to check that the response effort for participants was as low as possible, to reduce risk of attrition. Pilot testing took place in March and April 2022, with participants invited to complete an initial version of the quasi-experiment using the Qualtrics platform. Following completion, participants were provided a second survey which sought to gather their feedback on the use cases’ usability and acceptability. To ensure that a wide range of views were gathered, efforts made to specifically recruit specific socio-economic groups who have been historically underrepresented in these types of studies. Specifically, invitations to participate were circulated to key stakeholder groups, including groups representing university students, senior citizens, community groups, and low-income households. In total, 52 individuals completed the pilot survey. Respondents were primarily aged 41-50 (30.4%), from Ireland (69.6%), and had household incomes of less than €50,000 (56.5%), with disposable incomes of under €500 per month (30.4%).
What did we learn?
Pilot study results highlighted several important aspects of the quasi-experiment requiring additional attention. Firstly, we found that the survey took quite a long time to complete (average duration = 17 minutes 43 seconds), suggesting a need to reduce its length. Additionally, by looking at where participants chose to leave the survey without completing it we could identify areas which may need to be changed to reduce this attrition. Data obtained from the pilot also provided important information on the choices offered to participants, with technical appliance technical features, cost and delivery speed highly favoured. Regarding attributes for inclusion in the choice experiment, responses obtained highlighted the importance of operating cost and energy discount conditions, with positive feedback on warranty and operating cost timeframes also. In sum, pilot data suggested that the choices presented were appropriate, and provided participants sufficient information to facilitate informed choices. Positively, the quasi-experiment was found to be highly acceptable, with respondents agreeing that the survey was easy to understand and was not boring or repetitive.
Next steps
The design of each of the use cases outlined above were updated in response to the feedback obtained through each of the pilots described above. Each use case is now open for participants, and we look forward to analysing and sharing results shortly.
How you can get involved!
If you are interested in participating in EVIDENT, there are lots of ways you can get involved.
References
[1] A. Alberini, “How effective are energy-efficiency incentive programs? Evidence from Italian homeowners,” Energy Economics, p. 10, 2015.
[2] P. Bertoldi, “Overview of the European Union policies to promote more sustainable behaviours in energy end-users,” in Energy and Behaviour, Elsevier, 2020, pp. 451–477. doi: 10.1016/B978-0-12-818567-4.00018-1.
[3] C. Boomsma, R. Hafner, S. Pahl, R. V. Jones, and A. Fuertes, “Should we play games where energy is concerned? Perceptions of serious gaming as a technology to motivate energy behaviour change among social housing residents,” Sustainability (Switzerland), vol. 10, no. 6, 2018, doi: 10.3390/su10061729.
[4] E. Knol and P. W. De Vries, “EnerCities, a Serious Game to Stimulate Sustainability and Energy Conservation: Preliminary Results,” eLearning Papers, vol. 25, 2011.
[5] E. Delemere and P. Liston, “Engaging Serious Games for Energy Efficiency,” in HCI International 2022 – Late Breaking Papers. Interaction in New Media, Learning and Games, G. Meiselwitz, A. Moallem, P. Zaphiris, A. Ioannou, R. A. Sottilare, J. Schwarz, and X. Fang, Eds., in Lecture Notes in Computer Science. Cham: Springer Nature Switzerland, 2022, pp. 567–580. doi: 10.1007/978-3-031-22131-6_42.
[6] E. Delemere and P. Liston, “Exploring the Use of Behavioural Techniques in Serious Games for Energy Efficiency: a Systematic Review and Content Analysis,” Behav. Soc. Iss., Oct. 2022, doi: 10.1007/s42822-022-00103-4.
[7] H. Allcott and D. Taubinsky, “Evaluating Behaviorally Motivated Policy: Experimental Evidence from the Lightbulb Market,” American Economic Review, vol. 105, no. 8, pp. 2501–2538, Aug. 2015, doi: 10.1257/aer.20131564.
[8] D. Zha, G. Yang, W. Wang, Q. Wang, and D. Zhou, “Appliance energy labels and consumer heterogeneity: A latent class approach based on a discrete choice experiment in China,” Energy Economics, vol. 90, p. 104839, Aug. 2020, doi: 10.1016/j.eneco.2020.104839.
This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No