For three months in 2017, Sarah Betts of Olark conducted a gender experiment while providing customer service on live chat. Sarah measured responses to three different chat agent avatars - Samuel (male), Sarah (female), and Sam (neutral). This presentation summarizes her experiment results and shows how customers interacted with agents of different gender types.
At Olark we <3 experiments. So much so we have an entire Jira filter for them, and a template for setting them up.
I wanted take at some support issues, especially around how folks perform with CSAT, Escalations, and the like
I enlisted some folks from my Olark team to help
Let’s meet my co-conspirators!
Stop--as a manager, how are you going to view these? I might want to work with Samuel on his communication and empathy. Sarah and Sam may need some help skill building to reduce load on the escalation team
5 years on frontline support, concern about burnout levels for unsupported teams
Work in tech
Flashy articles aren’t enough, wanted numbers
NickO’s monthlong experiment was inspiring
Because I wanted to know how my perceived gender affected my interactions with customers
Also, seeing this every day gets quite old
Sexism is very real and flashy stories aren’t enough
But with data collection comes responsibility
The flip side of that is with data, we have the power to rethink how we do things and make a change
So let’s backtrack to how I set this up
Because experiments are FUN and give you powerful information
Any experiment starts first with a hypothesis.
If you’re thinking an experiment starts with data, no
Why did I choose these? What we’ve seen is Women are less respected, less valued. Treated poorly. I wanted to see if I could quantify that. And talked to our data ops, Ben Schultz who helped me find what we can collect and think through the experiment
Then make a plan. Time, who is involved, resourcing, etc
Trolls SUCK, Facebook example, proposition with apology
Explain !block feature & prob density
Again though, as a manager, I might worry about Sarah’s quick banhammer
Data doesn’t play fair because life is real, yo
Story of “I’ll have my team take a look, thanks!”
Let’s talk about customer satisfaction ratings
Overall, all 3 reps scored very well. 4.52-4.8/5
Stop--as a manager, how are you going to view these? I might want to work with Samuel on his communication and empathy. Sarah and Sam may need some help skill building to reduce load on the escalation team
So let’s have the responsibility talk
We’re data obsessed.
We think about responsibility with customer data, privacy acts, COPA, GDPR
But it goes further. We USE data all the time, HOW?
Determine what you want to know, then design the experiment not the other way round
With piles of data at our fingertips, that just appears through our tools, do we know what the data really measures? If you measure ticket escalation, are you measuring knowledge? OR your customers willingness to listen to your team member?
Because of this, I stopped short of measuring quite a few things
No financial, cultural appropriation (personal comfort) or other channels
My anecdata-ey takeaways
In the coming month, I’ll be sharing some resources on how you can set up a similar experiment. I think we have a responsibility to collect data on issues we see and share that. With enough data, support can be a leader in fair practices, impacting the broader world of tech