Friday, June 28, 2013

How eBay Uses Data and Analytics to Get Closer to Its (Massive) Customer Base

MIT Sloan Management Review

Big Idea: Data & AnalyticsInterview June 25, 2013  Reading Time: 11 min 

Online auction site eBay uses data about the behavior of its millions of customers to drive analytics at every level of the organization, and get closer to its customers.

You can find just about anything on eBay: A vintage BMW, a Lear jet, a half-million-dollar yacht. Or perhaps a domain name, industrial equipment, software and services from the likes of IBM, a food safari in San Francisco. Or even a previously undiscovered species, such as Coelopleurus exquisitus, a heretofore-unknown sea urchin sold on eBay.
The e-commerce giant has localized operations in over 30 countries, with 100 million registered users. The latest number of sellers listed by eBay in 2009 is well in excess of 1.5 million (it’s hard to tell the exact number of sellers, given buyers are often sellers and vice versa).
From all that activity stems a lot of data and, eventually, information — which eBay is capitalizing upon through the use of data analytics research. The results: eBay is much closer to its tremendous customer base than ever before, and it is able to iterate faster on fulfilling customer requirements.
In a conversation with MIT Sloan Management Review contributing editor Renee Boucher Ferguson, Neel Sundaresan, senior director of research at eBay discusses how the company uses data and analytics at every level to continuously evolve eBay’s numerous sites and services for buyers and sellers.
Can you talk briefly about how eBay uses analytics?
Analytics at eBay is used at every level and scale. A/B tests are common in understanding user response to site or feature changes, and policy changes. These tests can get complex, as the site has many complementary and competing features and policies. So one has to be systematic in ensuring that the experiments are clean, and also in reading the results of the experiments and in attributing measures of success to the changes. Then, if the result reveals a positive or a negative response, the algorithm or systems designers can take that information and update the models or design better algorithms, features or systems and the policy makers can revisit the policies. Data from these experiments can come in various forms — user behavior data, transactional data, and customer service data.
What would you say are the biggest technical issues that you’re facing with data today?
Everybody talks about big data. The first aspect of this is building or implementing hardware and software systems that can handle data at a large scale, and can make them available and respond at speed and scale. The second aspect is tagging our software system to collect the right kind of data. The belief is that more data means more information. As researchers and data scientists we see that, while this is true — that adding more data brings in more features to consider and that it addresses some of the sparsity issue — it is also possible that more data can introduce more noise. Cutting the data the right way is key to good science.
One of the biggest tasks in this effort is data cleaning. For example, let’s say all these websites collect data, but the data is ridden with bots. They might be web crawlers from search engines like Google or Bing, or they might be some other agents that somebody has let loose to discover information from websites. But that makes it difficult to separate clean data from dirty data on our side.
A significant part of the skill required is the ability to look at data and cut it the right way.
What are the biggest management issues that you’re facing with data?
A funny thing about data is that the more you collect it, the more you want it. And you want it in shapes and forms that you did not think about before, because now it’s possible. So data is growing faster than ever before.
As you make analytics-driven decisions, you want to learn more. That means you have to tag more pages of the site, and you have to track more of the data, and you have to produce better reports. Suddenly you’re growing your data much faster than you thought you would be growing. The challenge of managing scale is primary here.
I believe, as a scientist, that no data should be thrown away, meaning you should always keep data around for better data science. Just keeping large amounts of data, managing them, is another challenge.
The third challenge is new kinds of data. For example, if you go back even six, seven years, most of the data was text data. Suddenly, with the massive adoption of smartphone devices, there’s a huge explosion of image and video data, location data, and other sensor data. Being able to deal with new kinds of data and understand these new kinds of data — understand what they mean — is a challenge. As personal devices get smarter, as we augment ourselves with more and more devices, be it the smartphone or watch or eye-wear, new kinds of data [are] going to be everywhere.
Analytics is starting to look quite different from what it looked like a while ago. Suddenly we are seeing new forms of data, and we need to be prepared to process this data really well and in a near-real-time manner.
What do you see as the biggest management opportunities connected with analytics, to change or enhance the way that you operate?
I think the biggest change you see is that everybody in the organization — whether they are a technical person, a researcher or an engineer, whether they’re a product manager, a businessperson, a usual contributor or a manager — everybody has to be data driven.
Now, not everybody has to look at data, but everybody has to understand data at some level. And that’s a skill that most people were not trained for at school, or even in their previous jobs. And suddenly everyone has to know some basic statistics. They have to have some basic understanding of data.
A lot of data is coming from the behavior of millions of users on our site. So, being able to understand and kind of get your head around that data and the analysis is really important. You can think of it as an attitude change in all grades of people.
How do organizations go about implementing those skills sets for non-data analysts? Is it training?
Yes, a lot of it is training, either through courses with hands-on work and some of it is just learning on the job. For example, if you’re a manager, you have to understand the graphs that your people produce, and you have to know what it means to say, “you know, it’s sort of statistically significant or within the noise.” You need to know what that means. Otherwise, you will make decisions that are not data-driven, which won’t be correct in this new world. The depth of the skills might be different for an engineer or a technical manager or a product manager. But everyone needs to understand and be able to make data-driven decisions at some level.
Are you using data analytics to compete in new ways or to compete more effectively than you did, say, a couple of years ago?
In areas where we didn’t even think we could use analytics, we can use it now, just because it’s available. And when it’s available, you find it useful. When it’s useful, you want more of it available. So, just pretty much everything seems to have turned into this data-driven space of doing things.
I’ll give you an example: Business decisions that were made through qualitative data analysis and surveys are now augmented or replaced by readings from real use of the product from large numbers of users. This provides both immediacy and scale to the analysis and decision.
eBay is unique in that we have a large amount of data, and we also have a large amount of people using the system. We have 100 million signed users, which makes things interesting.
Let me give you an example from a paper we wrote last year. Economists often ask questions about consumer response to policy changes. They study the response by conducting either lab experiments or field experiments. The former is in an artificial setting, and the latter is at a limited scale. Let’s say the question is, “How do people internalize shipping costs in online commerce?” One way to do field experiment is, they would buy 100 Pok√©mon cards or 100 DVDs, and then they would sell 50 with free shipping at some price and 50 at a different price with $5 shipping. And then they would collect all the data and slice and dice it, run regression on it, and draw their conclusions.
What we found, when looking at this data at scale, was that our sellers and our buyers are already running field experiments for us. So, instead of asking, “Is free shipping a good policy or not?” — our sellers, especially our power sellers, are already running these experiments at scale. These are naturally occurring field experiments at web-scale! So we can look at the data and answer these questions — even before we create a policy or run an experiment.
And that’s a huge, huge change in how we look at data and make decisions.
Many of our power sellers are very smart about their business, and they know what to do. When they run these experiments, they see what happens. And they probably are correcting themselves when something happens. But at the same time, the data tells us the story that we want to hear.
What are the implications of that understanding within your industry? And given your huge user base, are there wider implications?
I think it has wider implications because a lot of platforms, not just eBay, have users that are powerful, and there’s constantly experiments occurring on the side. So, as an online site, you don’t have to actually go and experiment. For example, in the advertising space, advertisers had to advertise on Google or Bing. They stop advertising, then they advertise again.
Now, the reason why they might stop advertising is because they run out of budget or because they want to study the effectiveness of their own advertising. So, now that not only tells advertisers about the effectiveness of advertising, but also informs the people that have access to the data — in this case Google or Bing — the effectiveness of the existence or non-existence of an advertiser in their Internet advertising space.
Suddenly you have new kinds of data that you could not imagine before. So, it’s only your cleverness in understanding the data or analyzing the data that can inform you better and faster. You understand your customers better.
Often you run surveys and the customers tell you something. But with surveys, you have the “squeaky wheel problem” — the ones who complain are the ones who complain a lot, while a lot of unhappy people may never say anything. They may just walk away from the site or suffer quietly, but at the same time are getting burned.
When you look at user behavior in a systematic way, you can do often see the discrepancy between what they say and what they do. Some behaviors that cannot be captured in surveys are better seen in data. This helps us understand the friction points when they use our system.
That’s how data brings us closer to our customer than ever before. When you use analytics, you can go back to the customer and understand them better. You can create tools or deploy a product, see how they use it, and correct course — you can iterate much faster.
Where do you think the greatest opportunity for impact is in the future utilizing data analytics and big data analytics?
I believe data is everywhere, beyond the commercial context I discuss. You see that in the health space, certainly — with access to a lot more data, you can work with simple algorithms and you can answer questions at scale, at speed, much more than you could before. Local weather to climate changes, allergies to pandemics, education, food, calamities, politics, medicine — every aspect our daily lives, our economy and livelihood starts to look different when we are driven by data. Data science is not just for organizations and scientists; it is for consumers and individuals as well. As business, social, political and personal decisions are driven by data and information, we can address problems in a more systematic and transparent way.
What are the challenges that lie ahead?
I think the challenges are that we are collecting a lot more data. There are questions about privacy and security and abuse that can happen with access to data in the hands of those who shouldn’t have access to it. While these are important issues, I will leave these issues out of this conversation. I’m looking only at the positive side and good use of data. The big challenges are, how quickly can you scale to be able to handle this data? And can the tools that come with the data scale at the same pace? While these are indeed challenges, we are lucky to live in the golden age of data; we should use it to benefit the good.

The Network Secrets of Great Change Agents

by Julie Battilana and Tiziana Casciaro
Artwork: Jessica Snow, Louis II, 2010, acrylic on paper, 13.5" x 11.5"

Change is hard, especially in a large organization. Numerous studies have shown that employees tend instinctively to oppose change initiatives because they disrupt established power structures and ways of getting things done. However, some leaders do succeed—often spectacularly—at transforming their workplaces. What makes them able to exert this sort of influence when the vast majority can’t? So many organizations are contemplating turnarounds, restructurings, and strategic shifts these days that it’s essential to understand what successful change agents do differently. We set out to gain that insight by focusing on organizations in which size, complexity, and tradition make it exceptionally difficult to achieve reform.
There is perhaps no better example than the UK’s National Health Service. Established in 1946, the NHS is an enormous, government-run institution that employs more than a million people in hundreds of units and divisions with deeply rooted, bureaucratic, hierarchical systems. Yet, like other organizations, the NHS has many times attempted to improve the quality, reliability, effectiveness, and value of its services. A recent effort spawned hundreds of initiatives. For each one, a clinical manager—that is, a manager with a background in health care, such as a doctor or a nurse—was responsible for implementation in his or her workplace.
In tracking 68 of these initiatives for one year after their inception, we discovered some striking predictors of change agents’ success. The short story is that their personal networks—their relationships with colleagues—were critical. More specifically, we found that:
1. Change agents who were central in the organization’s informal network had a clear advantage, regardless of their position in the formal hierarchy.
2. People who bridged disconnected groups and individuals were more effective at implementing dramatic reforms, while those with cohesive networks were better at instituting minor changes.
3. Being close to “fence-sitters,” who were ambivalent about a change, was always beneficial. But close relationships with resisters were a double-edged sword: Such ties helped change agents push through minor initiatives but hindered major change attempts.
We’ve seen evidence of these phenomena at work in a variety of organizations and industries, from law firms and consultancies to manufacturers and software companies. These three network “secrets” can be useful for any manager, in any position, trying to effect change in his or her organization.
You Can’t Do It Without the Network
Formal authority is, of course, an important source of influence. Previous research has shown how difficult it is for people at the bottom of a typical organization chart—complete with multiple functional groups, hierarchical levels, and prescribed reporting lines—to drive change. But most scholars and practitioners now also recognize the importance of the informal influence that can come from organizational networks. The exhibit “Two Types of Workplace Relationships” shows both types of relationships among the employees in a unit of a large company. In any group, formal structure and informal networks coexist, each influencing how people get their jobs done. But when it comes to change agents, our study shows that network centrality is critical to success, whether you’re a middle manager or a high-ranking boss.
Consider John, one of the NHS change agents we studied. He wanted to set up a nurse-led preoperative assessment service that would free up time for the doctors who previously led the assessments, reduce cancelled operations (and costs), and improve patient care. Although John was a senior doctor, near the top of the hospital’s formal hierarchy, he had joined the organization less than a year earlier and was not yet well connected internally. As he started talking to other doctors and to nurses about the change, he encountered a lot of resistance. He was about to give up when Carol, a well-respected nurse, offered to help. She had much less seniority than John, but many colleagues relied on her advice about navigating hospital politics. She knew many of the people whose support John needed, and she eventually converted them to the change.
Another example comes from Gustaf, an equity partner at a U.S. law firm, and Penny, his associate. Gustaf was trying to create a client-file transfer system to ensure continuity in client service during lawyers’ absences. But his seniority was no help in getting other lawyers to support the initiative; they balked at the added coordination the system required. That all changed when Penny took on the project. Because colleagues frequently sought her out for advice and respected her judgment, making her central to the company’s informal network, she quickly succeeded in persuading people to adopt the new system. She reached out to stakeholders individually, with both substantive and personal arguments. Because they liked her and saw her as knowledgeable and authentic, they listened to her.
It’s no shock that centrally positioned people like Carol and Penny make successful change agents; we know that informal connections give people access to information, knowledge, opportunities, and personal support, and thus the ability to mobilize others. But we were surprised in our research by how little formal authority mattered relative to network centrality; among the middle and senior managers we studied, high rank did not improve the odds that their changes would be adopted. That’s not to say hierarchy isn’t important—in most organizations it is. But our findings indicate that people at any level who wish to exert influence as change agents should be central to the organization’s informal network.
The Shape of Your Network Matters
Network position matters. But so does network type. In a cohesive network, the people you are connected to are connected to one another. This can be advantageous because social cohesion leads to high levels of trust and support. Information and ideas are corroborated through multiple channels, maximizing understanding, so it’s easier to coordinate the group. And people are more likely to be consistent in their words and deeds since they know that discrepancies will be spotted. In a bridging network, by contrast, you are connected to people who aren’t connected to one another. There are benefits to that, too, because you get access to novel information and knowledge instead of hearing the same things over and over again. You control when and how you pass information along. And you can adapt your message for different people in the network because they’re unlikely to talk to one another.
Which type of network is better for implementing change? The answer is an academic’s favorite: It depends. It depends on how much the change causes the organization to diverge from its institutional norms or traditional ways of getting work done, and how much resistance it generates as a result.
Consider, for instance, an NHS attempt to transfer some responsibility for patient discharge from doctors to nurses. This is adivergent change: It violates the deeply entrenched role division that gives doctors full authority over such decisions. In the legal profession, a divergent change might be to use a measure other than billable hours to determine compensation. In academia, it might involve the elimination of tenure. Such changes require dramatic shifts in values and practices that have been taken for granted. A nondivergent change builds on rather than disrupts existing norms and practices. Many of the NHS initiatives we studied were nondivergent in that they aimed to give even more power to doctors—for example, by putting them in charge of new quality-control systems.
A cohesive network works well when the change is not particularly divergent. Most people in the change agent’s network will trust his or her intentions. Those who are harder to convince will be pressured by others in the network to cooperate and will probably give in because the change is not too disruptive. But for more-dramatic transformations, a bridging network works better—first, because unconnected resisters are less likely to form a coalition; and second, because the change agent can vary the timing and framing of messages for different contacts, highlighting issues that speak to individuals’ needs and goals.
Consider, for instance, an NHS nurse who implemented the change in discharge decision authority, described above, in her hospital. She explained how her connections to managers, other nurses, and doctors helped her tailor and time her appeals for each constituency:
“I first met with the management of the hospital to secure their support. I insisted that nurse-led discharge would help us reduce waiting times for patients, which was one of the key targets that the government had set. I then focused on nurses. I wanted them to understand how important it was to increase their voice in the hospital and to demonstrate how they could contribute to the organizational agenda. Once I had their full support, I turned to doctors. I expected that they would stamp their feet and dig their heels in. To overcome their resistance, I insisted that the new discharge process would reduce their workload, thereby enabling them to focus on complex cases and ensure quicker patient turnover.”
By contrast, another nurse, who led the same initiative at her hospital, admitted that she was handicapped by her cohesive network: Instead of supporting her, the key stakeholders she knew quickly joined forces against the effort. She never overcame their resistance.
The cases of two NHS managers, both of whom had to convince colleagues of the merits of a new computerized booking system (a nondivergent change), are also telling. Martin, who had a cohesive network, succeeded in just a few months because his contacts trusted him and one another, even if they were initially reluctant to make the switch. But Robert, whose bridging network meant that his key contacts weren’t connected to one another, struggled for more than six months to build support.
We’ve observed these patterns in other organizations and industries. Sanjay, the CTO of a software company, wanted his R&D department to embrace open innovation and collaborate with outside groups rather than work strictly in-house, as it had always done. Since joining the company four years earlier, Sanjay had developed relationships with people in various siloed departments. His bridging network allowed him to tailor his proposal to each audience. For the CFO, he emphasized lower product development costs; for the VP of sales, the ability to reduce development time and adapt more quickly to client needs; for the marketing director, the resources that could flow into his department; for his own team, a chance to outsource some R&D and focus only on the most enriching projects.
Change agents must be sure that the shape of their networks suits the type of change they want to pursue. If there’s a mismatch, they can enlist people with not just the right skills and competencies but also the right kind of network to act on their behalf. We have seen executives use this approach very successfully by appointing a change initiative “cochair” whose relationships offer a better fit.
Keep Fence-Sitters Close and Beware of Resisters
We know from past research that identifying influential people who can convert others is crucial for successful change. Organizations generally include three types of people who can enable or block an initiative: endorsers, who are positive about the change; resisters, who take a purely negative view; and fence-sitters, who see both potential benefits and potential drawbacks.
Which of these people should change agents be close to—that is, share a personal relationship built on mutual trust, liking, and a sense of social obligation? Should they follow the old adage “Keep your friends close and your enemies closer”? Or focus, as politicians often do, on the swing voters, assuming that the resisters are a lost cause? These questions are important; change initiatives deplete both energy and time, so you have to choose your battles.
Again, our research indicates that the answers often depend on the type of change. We found that being close to endorsers has no impact on the success of either divergent or nondivergent change. Of course, identifying champions and enlisting their help is absolutely crucial to your success. But deepening your relationships with them will not make them more engaged and effective. If people like a new idea, they will help enable it whether they are close to you or not. Several NHS change agents we interviewed were surprised to see doctors and nurses they hardly knew become advocates purely because they believed in the initiative.
With fence-sitters, the opposite is true. Being personally close to them can tip their influence in your favor no matter the type of change—they see not only drawbacks but also benefits, and they will be reluctant to disappoint a friend.
As for resisters, there is no universal rule; again, it depends on how divergent the change is and the intensity of the opposition to it. Because resistance is not always overt or even conscious, change agents must watch closely and infer people’s attitudes. For nondivergent initiatives, close relationships with resisters present an opportunity—their sense of social obligation may cause them to rethink the issue. But in the case of divergent change, resisters typically perceive a significant threat and are much less susceptible to social pressure. It’s also important to note that the relationship works both ways: Change agents might be reluctant to pursue an initiative that’s opposed by people they trust. They might decide that the emotional cost is too high.
An NHS clinical manager who failed in her effort to transfer responsibility for a rehabilitation unit from a physician to a physiotherapist—a divergent change—described her feelings this way: “Some of my colleagues with whom I had worked for a long time continued to oppose the project. Mary, whom I’ve known forever, thought that it was not a good idea. It was a bit hard on me.”
By contrast, a doctor who launched the same initiative in her organization did not try to convert resisters but instead focused on fence-sitters. This strategy was effective. As one of her initially ambivalent colleagues explained, “She came to me early on and asked me to support her. I know her well, and I like her. I could not be one of the people who would prevent her from succeeding.”
Similarly, John, a member of the operating committee of a boutique investment bank, initiated a rebalancing of traditional end-of-year compensation with a deferred component that linked pay to longer-term performance—a particularly divergent change in small banks that rely on annual bonus schemes to attract talent. His close relationships with several fence-sitters enabled him to turn them into proponents. He also heard out the resisters in his network. But having concluded that the change was needed, he maintained his focus by keeping them at a distance until the new system had the green light.
The important point is to be mindful of your relationships with influencers. Being close to endorsers certainly won’t hurt, but it won’t make them more engaged, either. Fence-sitters can always help, so make time to take them out to lunch, express an authentic interest in their opinions, and find similarities with them in order to build goodwill and common purpose. Handle resisters with care: If you’re pursuing a disruptive initiative, you probably won’t change their mind—but they might change yours. By all means, hear them out in order to understand their opposition; the change you’re pursuing may in fact be wrongheaded. But if you’re still convinced of its importance, keep resisters at arm’s length.
All three of our findings underscore the importance of networks in influencing change. First, formal authority may give you the illusion of power, but informal networks always matter, whether you are the boss or a middle manager. Second, think about what kind of network you have—or your appointed change agent has—and make sure it matches the type of change you’re after. A bridging network helps drive divergent change; a cohesive network is preferable for nondivergent change. Third, always identify and cultivate fence-sitters, but handle resisters on a case-by-case basis. We saw clear evidence that these three network factors dramatically improved NHS managers’ odds of successfully implementing all kinds of reforms. We believe they can do the same for change agents in a wide variety of organizations.
Julie Battilana is an associate professor of organizational behavior at Harvard Business School. Tiziana Casciaro is an associate professor of organizational behavior at the University of Toronto’s Rotman School of Management.

Discover Your Personal Narrative

by Dorie Clark  |   9:00 AM June 28, 2013
I recently had coffee with a senior partner at a large consulting firm. He'd just had a "milestone birthday" and now hoped to shift into roles that felt more meaningful to him — speaking, writing, teaching, and becoming a thought leader. He had great contacts; newspaper columns and teaching positions could be his for the asking. The only problem, he told me, was he didn't know what he wanted to say.
Should he focus on the industry practice areas where he'd made his name? Global leadership, since he had so much international experience? Education or healthcare, topics of great personal interest to him? He had no idea where to begin.
"Message development" is a process I'm certainly familiar with. As a former presidential campaign spokesperson and political consultant, I've worked with innumerable candidates to hash out their visions for America and their policy stance on the issues of the day. But I don't think a top-down process is generally the best way for executives — or candidates, for that matter — to determine what they really stand for.
Oftentimes, we're too close to our own experience to be able to distill the common strand — the narrative thread that's implicitly guiding us. That was the case for Chris Guillebeau, an eclectic entrepreneur who has written books including The Art of Non-Conformity and The $100 Startup. "Is there a larger narrative [to my life]? Yes, but it took me a while to find it," he told me in a recent interview. "The larger narrative stems from the central mission: 'You don't have to live your life the way others expect'...[but] it took some time to get specific on what this looked like. In the beginning I floundered a lot."
John Hagel, the co-author of The Power of Pull and co-chairman of Deloitte's Center for the Edge, agrees. Even among corporations, he told me in a recent interview, "Narratives can't be handed over to the PR department; they emerge from shared experiences. The first step for businesses is saying, 'What's our narrative?' Because even if you don't have a conscious one, you've been living one."
We've all, as individuals and as corporations, been living an implicit narrative. But articulating it, as my consultant friend found, can be devilishly hard. There is a pathway to discovery, however. One strategy I developed in the course of writing my book, Reinventing You, is for executives to block out time to write down their "war stories" — the anecdotes that best capture their experience, successes, failures, and views of the world. Whether it's insights about how to build a team or launch a new product, those recollections often contain the kernels of what matters most to them.
Sure, your personal brand and your message can be focus-grouped and wordsmithed by others. But the best place to look, at least initially, is at the stories you tell, to yourself and about yourself. You'll start to see patterns and themes — if most of your most meaningful experiences are centered on global leadership, or if the "moral" of most of your stories is about the need for better executive communication, then you're on your way to finding the essence of your brand.
After a recent lecture at Harvard Business School, a 20-something student came up to me. "I want to start blogging," she said, "but I'm not sure what I should write about. What should my topic be? What if I change my mind and decide I want a different brand later on?" My advice to her — just like to the senior consulting partner — was to get started, try it out, write things down, and iterate (an approach definitively articulated by Len Schlesinger, Charlie Kiefer, and Paul Brown in their bookJust Start.) You'll only find your voice, and your authentic brand, by seeing what stories matter to you most.