For the past seven months I was part of the research team on a large-scale project that incorporated two research methods repeated over the course of seven waves of participants. The first method was a hybrid of contextual inquiries and usability testing where researchers went to participants’ homes and watched them interact with hardware and software for a new in-home device. The second method, and the subject I’m discussing today, was a digital diary study, of which I was the lead moderator.
Diary studies will vary from research project to research project but in the context of our work participants were given devices to use in their home and each day, over the course of about a week, they would complete activities with the device and answer questions using an online questionnaire or digital diary tool. The diary portion helped us collect valuable quantitative and qualitative feedback, enabled us to interact with participants in different geographical areas, and helped us to discover how behaviors and attitudes changed over time.
Repeating this over seven months with seven waves of participants allowed us to also iterate on how we administered the diary portion (as well as the in-homes sessions) of this study. That iteration has produced some key “Do’s and Don’ts” of diary studies. Use these to help ensure your research is the most effective it can be.
Use the right tool - Try to pick a tool that matches your users’ tech proficiency. Revelation and Dscout are robust enterprise tools which require participants to create accounts but a free and familiar-to-most tool like Google Forms or Typeform may be all that you need.
Schedule a 30-minute on-boarding meeting with participants - This give you a chance to meet your participants face-to-face, build rapport, and assist with any technical setup necessary. This should also create more buy-in from your participants. Use the great online meeting tool Zoom to connect with your participants and show them how to use your selected diary tool.
BONUS: During this meeting you can also convey the importance of the diary study they are participating in. Let them know that their feedback and insights will be used to improve the product/service/experience you’re studying. This can also help with participant fatigue (more on that later).
Set proper expectations - Let participants know what will be required of them and how much time they should commit to completing the work. Clearly defining the activities and diary schedule is crucial for two reasons. This will aid your participants in completing the activities in the timeframe you’ve designed and should result in more accurate and thoughtful feedback.
Ask a mix of quantitative and qualitative questions - With a mix you will uncover the “why” behind the hard numbers. Because your ultimate aim is to find the answers to your research objectives it’s important to get a mix of data points. Your quantitative data will help you understand success/failure rates and satisfaction ratings. Your qualitative data will help you better understand what is causing those success/failure rates and satisfaction ratings.
Ask questions that get at the impact value versus just satisfaction rating - This will ensure you are uncovering the impact of the product/service/experience you’re researching and help provide actionable findings. Using scales that show the ultimate impact on participants will provide more valuable insights. Consider scales similar to the following:
1 = I had a significant problem and was unable to complete the setup successfully
2
3 = I had a few problems but was still able to complete the setup successfully
4
5 = I had no problems and was able to complete the setup successfully
OR
1 = The instructions were not clear, and I was unable to continue
2
3 = The instructions were not clear, but I was still able to continue
4
5 = The instructions were clear, and I was able to continue
Provide regular updates to clients or stakeholders regarding the progress of participants - This gives everyone a sense of how your participants are advancing through the study. Additionally, by including some initial findings in your progress report, clients and stakeholders can feel they are a part of the research process as well.
Create a spreadsheet to collect all your quant and qual data - It makes for easier analysis. As data comes in, quickly scan it and input it into your data results spreadsheet. This will help with analysis because everything is in one place and you’ve already reviewed it once.
Our data results spreadsheet from the final wave of testing.
Assume your participants will dedicate as much time to the study as you think or hope they will - Despite your best efforts to clearly communicate expectations things will come up and participants may show signs they’re giving less effort (simple one-word answers, skipping questions, lagging behind, etc.). Reach out to these participants and see if they can expand their thoughts or encourage them to complete late activities when they can.
Ask too many questions - Participant fatigue is real. As your participants progress through your diary study help ensure they continue to devote quality time and maximum effort by asking the appropriate number of questions. Even consider reducing the number of questions per activity as participants get closer to the end of the study.
Devote time and energy to areas outside the original scope - Keep your study on track and more manageable by concentrating on the pertinent questions to help with your research goals. It can be tempting to add questions that help inform other research questions you may have but stay focused on your original questions. You don’t want your data set to grow too large and don’t forget about participant fatigue!
Be afraid to ask clarifying questions - Ask follow up questions to better understand what they really meant. Participant’s answers may be vague or unclear and may not help you understand the “why” you’re after. Respectfully ask participants to clarify or expand their thoughts on the answer they’ve provided - most are happy to oblige.
Do it all alone - Depending on the number of participants and the number of questions, there can be a lot of data points so don’t be afraid to get support from others. Divide and conquer or ask a colleague to focus on a specific aspect of your study. Google Docs’ ability to do real-time collaboration helps tremendously here.
Overcomplicate the implementation of your activities/questions - Keep the schedule simple so participants can focus on completing the activities and providing quality answers. It is very likely that your participants will have different schedules so give them the freedom to complete activities when it best suits them. We had the most success when directing participants to complete the day’s activities when it fit into their schedule.
These ‘Do’s and Don’ts’ came from our desire to continually improve our process. Hopefully you can utilize them in your next diary study. And, as every study is different, you’ll likely discover other ways to run more effective and insightful research.
⬡ ⬡ ⬡
Do you have experience running diary studies? Please share your ‘Do’s and Don’ts’ by leaving a comment below. Thanks!