Imagine you have plenty of smart, well-trained customer support agents to keep your ticket queues low and your reply times fast, but you have no control over when they work. You know they will work a certain amount of time each week, but you have no idea when that will be.
How do you handle this ambiguity and lack of control? As a manager, how do you keep yourself from jumping in and answering tickets when more and more tickets flow into the queue?
At Magoosh, we face this challenge — all the people and no control over when they work — but with any good challenge, a creative solution is waiting to be unearthed. Instead of looking for a solution among other customer support teams, we adapted an idea popularized by Uber and Lyft: surge pricing!
Over the past year, we experimented with increasing pay when we were inundated with questions as a way to nudge our test prep experts to jump in and answer questions. I’d like to share the results of this experiment with you. I hope you can learn from our experience. Maybe it will inspire you to try something similar with your teams.
The Background
“Wait, Kevin, why don’t you have control over when your agents work?” You might ask. “Can’t you easily avoid this challenge?”
Yes, we could avoid this scenario. We could hire a full-time staff, or even a larger part-time staff, and schedule their time to handle ticket levels. But that’s not how we’ve approached support at Magoosh up to this point.
We hire teachers and tutors with expertise across multiple tests as independent contractors. Our “customer support agents” work independently to answer questions about exams: from coordinate geometry on the GRE, to critical reasoning on the GMAT, to dual reading passages on the SAT, to speaking strategies on the TOEFL. The people with this type of expertise tend to be PhD students, classroom teachers, private tutors, or recent graduates, all of whom have fluctuating and dynamic schedules.
As independent contractors, they set their own schedules, finding time to answer questions in between their other commitments. This flexibility is great for them. They don’t have to tell us when they will work or notify us of changes — they just work when they can.
This is great for us too. We don’t have to keep a schedule or approve schedule changes. No one has to worry about shifts or coverage in the traditional sense. All our hires are experts and already know how to teach. We can hire the best people from across the country.
But it’s not all sugar plums and gumdrops. We’ve traded control for uncertainty, and as such, we’re left to experiment and find solutions to certain challenges — will all the questions get answered today?
Hypothesis
Surge pricing works for companies like Uber and Lyft, so we wanted to find out if we could make it work in customer support.
The process was straightforward enough: Email everyone on the team at the beginning of the day and tell them that they’ll be paid 25% more for any work they do during a specific time period.
When we see that our demand (tickets) is rising and our supply (people who can answer questions) is decreasing, we’d send out an email to the team. A situation similar to this contrived example.
If surge pay worked, we’d see this happen:
Our supply rushes to meet demand and our students begin to receive answers to their questions. Demand begins to drop off as our tutors answer questions, and the incentive of surge pay is no longer needed.
How it Worked
When we “turned on” surge pay, we sent an email to our team, letting them know that surge pay is in effect for a certain amount of time, usually the rest of the day.
Here’s an example of an email sent to the team:
Since June 2015, we’ve had surge pay 26 times. Each of these surge pay experiments was different, but not by design. Sometimes they were for a full day, sometimes they were across multiple days, and sometimes they would only last an afternoon.
This variation was not for pure experimental purposes. Due to consistent high demand across multiple days, we’d have surge pay across multiple days. Or sometimes we waited to see what the morning would be like, to see if people would jump in and answer questions, and then later, send out the surge pay email. Other times we’d check the queue in the morning and know that we needed to flip the switch.
Results
The graphs below show nearly all the surge pay days in 2015. Each vertical line represents the day before surge pay. I’ve decided to mark these days because it gives a better picture of what we were looking at when we decided to start surge pay. To understand the results of surge pay, we need to look at what happened the day after the line.
Let’s look at some of these days to evaluate the success of surge pay. On 6/23/2015, we had roughly 280 tickets created — our demand — but only around 200 tickets updated — our supply. On 6/24/2015, I emailed the team to begin surge pay, and by the end of the day, supply surged to around 300 tickets. In this case, we can tentatively say that surge pay worked. The steep increase in tickets updated from 6/23 to 6/24 would indicated that we were able to encourage people to jump in and answer questions.
Now let’s look at another day. On 6/25/2015, demand was only slightly above our supply. On 6/26, I sent out the email to start surge pay, and both supply and demand dropped off with supply never overtaking demand. Surge pay didn’t seem to change anything this time.
In August we saw more mixed results. For the first time, we had back-to-back surge pay days. If we look at 8/10 – 8/12, we see a big spike at the beginning, but then a drop off in our supply and demand. Ticket updates dropped below new tickets created. This might indicate that surge pay is only effective in short bursts. Perhaps after the third day of surge pay, the team was less motivated to jump in.
This brings up some points that we need to consider when looking at the results of surge pay. We can’t conclusively say that surge pay worked or not. At this stage, we can only have hunches and draw tentative conclusions because there is already so much variation in how much people work. There’s no way to know if surge pay influenced people to work or if those people already planned to work. Take, for example, the steep increases in ticket updates on days without surge pay (6/15, 6/21, 6/27, 8/30, 9/8, 9/24, 9/26, and 9/27). When we look at a surge pay day, it’s hard to know for sure if this increase is similar to one of these days or due to an increase in pay. Finally, we are implementing surge pay on days where demand is high. Supply might naturally surge during these periods without us doing anything. Ideally, we would A/B test surge pay and no surge pay at the same time, but that’s not fair to our test prep experts nor our students waiting for answers.
September offers us mixed results with surge pay. Let’s look at the dates 9/9 – 9/13. On 9/9 there was definitely a gap in supply and demand before we sent out the email about surge pay. By the next day, supply overtook demand as we would expect.
Over the next 3 days, both supply and demand fell, but supply stayed above demand, until 9/12 when supply dropped below demand again. It’s unclear why this happened. Maybe the effects of surge pay are not effective over a long period of time. But if we look at the last day of surge pay (9/13) we see a spike in both supply and demand. During this time, supply does overtake demand.
What’s interesting is that we see a similar pattern when we look at the string of surge pays days later on from 9/17 – 9/20. Supply overtakes demand, both drop off in the coming days, supply drops below demand towards the end, and on the final day, it surges. This is a fascinating pattern and we don’t have a good explanation to explain it.
Learnings
At this point, we are not ready to draw any strong conclusions about surge pay. As you can see from the results, there are a lot of variables. The natural fluctuations in people working and ticket numbers makes it hard to draw clear correlations with surge pay.
Even so, the preliminary results are positive enough for us to keep trying it. From the perspective of handling a backlog of tickets, surge pay has helped. It does bring down ticket numbers on days of high tickets. We are also expanding our definition of surge pay. It doesn’t have to be based on a percent increase in pay. We are now experimenting with rewards around number of tickets answered in a week, number of hours worked in a week, and first to answer 10 questions.
Photo attributions:
1 – Photo of Uber app courtesy of GongTo / Shutterstock.com.
Leave a Reply