Handshake /
Diary study
With the UX Research team’s creation of student archetypes at Handshake, we now have an even stronger basis to learn more about our students journeys in the job search. A diary study is something that I had been asked to prepare for since I started at Handshake, in an effort to generate an insightful source of truth about our student users. Leading this study allowed our team and larger organization to collect longitudinal, qualitative data from 15 different students at the same time, without in-person moderation.
The objective of this study was to gain a deep understanding of the authentic student experience during the job search process, encompassing their successes, failures, challenges and emotional journey. The insights gained from this study have helped in developing strategies and initiatives to more effectively engage and assist students in their career development and embed new opportunities into over 5 different teams product roadmaps.
MY ROLE
Lead researcher
Strategy planning
Tool research
Recruitment
Ongoing synthesis
Newsletter creation
Executive reporting
METHODS
Longitudinal /
diary study
TIMELINE
10 weeks
COLLABORATORS
Product analytics
Spoilers
As the first study of it’s kind successfully completed at Handshake, it has had a large impact in the entire design and product organization understanding our students better, helping shape the roadmap for the next 2 years by either validating existing projects or providing evidence to create new ones, and an educational resource for Handshake employees not as involved on the student side of our marketplace.
The insights from the diary study have impacted all five Content and Community department roadmaps, (Notifications, Feed, Profile, AI, and Content) and what teams dedicate resources to in 2024 and beyond.
I. BACKGROUND
What is a diary study?
A diary study is a contextual, qualitative, longitudinal research methodology used to capture user behaviors, activities, and experiences. Diary studies take place over time and offer insight into people’s day-to-day lives. They are researcher prompted and user generated—meaning that participants follow our instructions and show their experience in their
own words.
Research objectives
What are the primary tasks of different students (and archetypes) on our platform, and how do they differ?
How does this translate to their primary obstacles and difficulties that we can help solve?
How do students adapt their job search strategies? What factors drive these adaptations?
How does our platform experience impact retention and engagement of different students (and their archetypes)?
At what point do they use other platforms to be more successful?
How do priorities and criteria for opportunities evolve?
What are the “micro-moments” and interactions that affect big decision making like applying for a job and interviewing?
How can Handshake better meet different student needs when they need us most?
How do students leverage networking and professional connections during their job search?
What emotional experiences do students associate with different stages of the job search? How do they affect their overall process and outcomes?
Timeline
I created a 10 week timeline for this project and meticulously managed it, recruiting a diverse range of participants, onboarding them onto Recollective, and oversaw each upload which added up to over 250+ entries to synthesize. Throughout each week after launch, I made sure to synthesize the collected data, identifying patterns and themes to draw meaningful conclusions to share out on a bi-weekly basis in the form of a newsletter. This careful execution ensured an efficient and successful study.
II. RECRUITMENT
Process
As this was a high priority and expensive project, we wanted to ensure we pulled the most qualified and responsive participants as we could. Understanding students have busy schedules and we may experience dropoff, our process was built around minimizing the potential for poor recruits. This included…
Recruiting ~15 students that had access to our beta Feed program, with the aim of including a diverse pool of primary archetypes, race, gender, school year, field of study, and other criteria (ie, Core/non-Core segments, likelihood to be a Group leader).
Selected participants would be contacted, and interviewed by me to go over all study details before agreeing to participate.
I would have participants set up on Recollective at least one week before study launch.
During recruitment, participants will be sent a screener survey pre- and post-study to evaluate their demographic information, current job search activities and feelings towards them, social media habits/usage, and general confidence entering this new school year.
III. METHODOLOGY
Tools
All participants would be instructed to use Recollective, an industry leading research tool for conducting qualitative studies. With Recollective, we can design engaging studies using a combination of activities, journals, real-time chat rooms and discussion forums – as well as interactive question types and media upload (image, video). In addition, it allows moderators to communicate naturally with participants through the dashboard, which they can access on both desktop/mobile web.
Diary types
In the study, participants were required to upload to two diaries every Monday and Friday (Timed Diary) of the 10-week period, with the ability of uploading to an optional diary (Open Diary) any day of the week, as often as they’d like. The categories they’d be replying to were segmented into buckets of topics such as:
Preparation
Learning about different companies and what matters most to students in the job search, saving job opportunities, seeking advice and tactics, sharing jobs, content exploration
Applications
Positions students apply to or intend on applying to, experiences tailoring resumes and cover letters, results from previously submitted apps
Networking
Individuals students are connecting with and why, platforms used for networking, what networking events look like, follow up actions taken from networking
Interviewing
Companies students hear back from or turn them away, actual interview experiences and reflections, outcomes of interviews
Other / Open Diary
Anything that doesn’t fit in the above topics, or spontaneous entries
Logging types
With the above format in mind, we were primarily using interval-contingent (Timed Diary) and event-contingent (Open Diary) logging techniques to capture as much as we can without overburdening participants.
This means that students would be on a set schedule to regularly upload, with an optional, shorter diary to upload to when significant events happened outside their regular interval. If they uploaded to the Open Diary, they would also receive additional “points” leading to an extra incentive.
IV. INCENTIVES AND ENGAGEMENT
Point systems
Aside from base compensation, all participants were eligible and incentivized to keep active engagement throughout the 10 week study by using a point system that they could track.
After every 2 weeks, three participants with the most points would be awarded an additional monetary incentive.
To maintain engagement and avoid feeling isolation, I allowed participants to engage anonymously in bi-weekly discussion forums as another form of qualitative data collection. They would receive additional points for participation.
Many participants said that this was one of their favorite parts of the study!
Group discussion forums
Cheat sheet
Within the dashboard, participants had access to a cheat sheet that highlights a study overview, FAQs, and example responses as a standing document for them to reference. This allowed them to find quick answers to any questions, and served as an anchor to stay on track throughout the 10 weeks.
V. STAKEHOLDER ENGAGEMENT
Staying in the loop
In this type of project with a longer duration, I wanted keep stakeholders engaged and involved in this diary study throughout its course, knowing it would touch on several in-flight initiatives and would influence both short term and longer term roadmap planning.
A few ways I did this was through…
A dedicated Slack channel
I created a slack channel for the #student-diary-study that collected over 70 team members, voluntarily joining in an effort to stay up to date with this fast moving and insight-rich project.
In the channel, I would share out excerpts of quotes, clips, and images from the study which prompted active discussion and reactions over the course of 10 weeks.
Bi-weekly newsletters
Every two weeks, I would partner with a member of the product analytics team to combine what I observed in the last sprint re: qualitative insights. Using student ID’s, we were then able to pair that qualitative information with quantitative actions students were actually taking on Handshake, allowing us to better understand gaps and opportunities we could be better serving. This combination of data told a compelling story to turn our insights into a story that all stakeholders were interested in following along with.
Readouts
After reaching the halfway point of the study (5 weeks in), I presented remotely on Zoom to key stakeholders all the large insights we had collected so far around the student job search - this went so well and prompted such rich discussion that I was invited to speak and present again in person at our headquarters in San Francisco to a larger audience in a company onsite.
Visibility into the tool(s)
For those who wanted to dig into insights themselves, I granted viewer access to stakeholders who want to more closely evaluate journal entries from participants on Recollective.
VI. SYNTHESIS
Process
As a part of my synthesis process, I used Figma to organize individual boards for each of my diary study categories. From there, I created organized columns per participant and the corresponding date to be able to follow along each student’s weekly journey, tag insights by process/pain point/opportunity/emotion, and stay on top of a continuously growing journey map as the final deliverable to this work.
Journey maps
VI. DELIVERABLES
After completion of the diary study and synthesizing the entries from that 10 week period, I shared out the four maps that visually represent the experiences, actions, emotions, and opportunities highlighted in the student job search.
With these maps drawing out the student experience, our team and entire company can make more informed decisions on where to enhance different parts of our product to improve user sentiment and MAU - specifically in the first two maps of Preparation and Application for the near term.
Opportunities in these maps (scraped for confidentiality) have helped inform roadmaps for 2024, and the UXR team conducted additional workshops and executive presentations with different squads to dive into more areas of innovation.
Workshops
In the workshops conducted using these journey maps, we brought in different product managers, engineering leads, and designers to discuss areas of innovation to impact the future of Handshake. After effective collaboration, several participants learned more about how to ground their roadmapping and decisions in user needs. We repeated this process with every squad across the Content and Community team, which also led into collaboration with Marketing, Employer squads, and C-Suite.
VII. IMPACT
Impact
As the first study of it’s kind successfully completed at Handshake, it has had a large impact in the entire design and product organization understanding our students better, helping shape the roadmap for the next 2 years by either validating existing projects or providing evidence to create new ones, and an educational resource for Handshake employees not as involved on the student side of our marketplace.
The insights from the diary study have impacted all five Content and Community department roadmaps, (Notifications, Feed, Profile, AI, and Content) and what teams dedicate resources to in 2024 and beyond.
VII. TESTIMONIALS
Quotes from teammates
“Sean led the execution of this project seamlessly. We were able to collect so many insights that will carry Handshake for many years to come! CONGRATS SEAN!!! Thank your for all your hard work, weekends and evening work. This was not a small project to take on as a single researcher. It required staying up-to-date and great collaboration! We are so thankful for all you have done!!!”
“Thanks Sean and team, this study has some of the best qualitative insights I’ve seen at Handshake, keep up the good work”