Facebook and Microsoft shared how they build their software. Some excellent insight into modern software practices.
The most obvious approach might be to imagine the future you want and build it. Unfortunately, that doesn’t work that well because technology co-evolves with people. It’s a two step—technology pushes people to move forward and then people move past technology and it has to catch up. The way we see the future is constantly evolving and the path you take to get there matters.
Often, however, we’re working on products that have no analog for comparison in research and whose merits are difficult to gauge in the abstract or at small scale. To keep improving, we must constantly test different versions of Facebook with real people to even have a chance at creating the best possible experience.
As Mike Moran said in his book: “If you have to kiss a lot of frogs to find a prince, find more frogs and kiss them faster and faster.”
Experimenting at large scale is fundamental for improving Bing. Last June, we published a blog in this Search Quality Insights series titled Experimentation and Continuous Improvement at Bing, which covered a specific type of experiments known as interleaving. In this blog, Dr. Ronny Kohavi describes our broader online experimentation efforts at large scale and includes compelling examples that illustrate the power of these efforts, e.g., he shows how a controlled experiment at large scale of a relatively small feature change can lead to many millions of dollars in revenue. Ronny’s blog is a brief summary of a research paper titled Online Controlled Experiments at Large Scale which will he will present next week at the international conference on Knowledge Discovery and Data Mining (KDD 2013). The paper has already received positive feedback by well-known experts in this field, and we’re sharing their comments in this blog with their permission.
Here are the links to the articles:
Large scale experimentation at Bing
Building and testing at Facebook
Alan Tatourian 