In this post I share the Free Exchange column from the September 6, 2014, edition of The Economist. It addresses one of my favorite things: Change.

DESTROYING the old to make way for the new is the essence of market economies. Karl Marx thought it one of the nastier qualities of capitalism whereas Joseph Schumpeter, an Austrian economist, cast “creative destruction” in more positive light, as the only route to sustained growth. In the 1990s Clayton Christensen, a professor at Harvard Business School, gave the notion a modern sheen with his theory of “disruptive innovation”. The term is now everywhere. Uber is said to be disrupting the taxi business, Cronuts are disrupting breakfast and Twitter is disrupting communication.

A disruptive innovation, in Mr Christensen’s work, is a very specific thing: a new technology that is inferior in certain respects to existing ones, but has other desirable attributes. He cites eight-inch floppy disks, which could store more data than smaller ones, but were nonetheless supplanted because they were too big and expensive for desktop computers. By the same token, publishers of music and newspapers were wrong-footed by the advent of online distribution, which was initially of lower quality. So was digital photography, but it nonetheless ended up displacing film.

Most studies that examine the size of a firm and its capacity to innovate fail to detect a relationship between the two. Yet that in itself is odd. Established firms ought to enjoy big advantages over would-be disrupters: skilled employees, infrastructure at the ready and the opportunity to share costs among products. At worst, incumbents should be as capable as new entrants of succeeding in nascent markets. Yet research by Rebecca Henderson of Harvard Business School finds that the money old firms in fast-evolving industries devote to research brings much lower returns than the research budgets of their younger rivals.*

Divining the reason for this poor performance is a challenge. Firms with straitened finances might be unable to invest adequately in a new business without cannibalising the old. That, in turn, might lead them either to decide not to invest in the new market, or to invest less than they should in both the old and the new. Yet the adjustment is often most difficult for firms that are wildly profitable in their established lines of business.

Profitable firms might underinvest in a competing technology for fear of hastening the end of their existing, successful business. Rational managers should be able to see that cannibalising their own sales and surviving is preferable to sticking to their knitting and falling prey to competitors. But bosses may not think much about the long term, or may be reluctant to write off sunk costs.

Yet even short-sighted or embarrassed managers should react when the threat from upstart technologies becomes clear. They do not, economists reckon, because of organisational rigidities. Ms Henderson suggests firms can be seen as giant information-processing organisations that evolve a structure and personnel to fit their respective business. Information on sales or production is efficiently filtered to decision-makers, who can then direct new research. When new technologies are no more than tweaks to old ones, this set-up is a competitive advantage. When innovations are more radical, however, the old networks are a hindrance.

In other research with Sarah Kaplan of the Wharton School of Business, Ms Henderson considers why older firms struggle to pursue new technologies. Many of them have systems in place to detect and respond to changing market conditions or new technologies. But they have also built up a system of incentives to ensure employees meet existing goals. Systems designed to encourage consistency and efficiency in the production of established goods or services might be a powerful deterrent to experimentation or creative thinking about new markets, regardless of what the corporate memos say.

One still might expect more adaptability given a serious enough threat, argues Ms Henderson in another paper along with Timothy Bresnahan of Stanford University and Shane Greenstein of Northwestern University. Established firms, they point out, can always set up loosely affiliated “entrepreneurial” divisions with the freedom to build a new business from the ground up. Yet even this will often prove difficult, they argue, thanks to assets like a firm’s reputation that cannot help but be shared between the old business and its internal rival.

IBM, for example, was initially a mainframe computing company that set up an internal unit devoted to developing personal computers. The PC business did well at first, thanks in part to IBM’s longtime reputation for quality. Yet this became a problem when the mainframe buyers became big consumers of PCs—an embryonic business in which kinks were still being ironed out. Customers began to interpret quality problems in the PC business as quality problems within IBM as a whole, undermining the existing mainframe business. When IBM resolved internal tensions by reabsorbing the PC unit, it in effect conceded the PC market to others.

Innovate or buy

Disruption need not be a death sentence, however. IBM remains a big, profitable firm. Work by Matt Marx of MIT, Joshua Gans of the University of Toronto and David Hsu of the Wharton School suggests that survival often comes down to what they call “co-operative commercialisation”. Once it becomes clear that start-ups have an edge in a new technology, incumbents can respond by striking deals with or purchasing their new rivals. The authors focus on the business of speech-recognition, but there are lots of other examples: Facebook, for instance, has gobbled up one competitor after another. To most, “If you can’t beat them, join them” has a more appealing ring than “Innovate or die”.

The future is here, happening every day. And, the robots are coming. Are you ready?