The Long Road To Multitouch: How a mix of new ideas shaped the way we communicate with computers


When Apple unveiled the multitouch interface with the iPhone launch in 2007, it seemed as though computing had been reinvented overnight. But in reality, it was an innovation three decades in the making. 

 Illustration: Ania Augustynowicz. Photos: Jeff Han: © Robert Leslie /. Courtesy of TED / G. S. Hurst: © University of Kentucky Libraries Special Collections Research Center / Ken Kocienda: © Imran Chaudhri
Hurst Han Kocienda
Hurst Han Kocienda
Hurst Han Kocienda

If you ask most people to pinpoint the origin of the multitouch interface—one of the most significant revolutions in how we interact with our computers—one now-legendary event will usually come to mind: Steve Jobs’ dramatic demo of the iPhone in January 2007. In 65 riveting minutes of high-tech showmanship, many of us saw for the first time a whole suite of interactions that would become second nature to billions of people around the world: pinch-and-zoom, inertial scrolling, virtual keyboards, “springboard” home screens. 

But as tempting as it is to attribute technological revolutions like multitouch to a single visionary, the story of the multitouch interface is actually a case study in how major breakthroughs are inevitably collective affairs—the aggregate work of many different inventors solving distinct problems that can often seem completely unrelated. 

Steven Jobs presents the iPhone during the MacWorld Conference & Expo in San Francisco, 2007.
© Blake Patterson, CC BY 2.0, via Wikimedia Commons

Sometimes our true software heroes are hidden because one single innovator has failed to get the recognition they deserve, because they didn’t become a wealthy entrepreneur or a celebrated industry visionary. But sometimes, the heroes are hidden because there are simply too many of them lurking behind a single innovation to do them all justice.

The multitouch interface is now a routine part of daily life for billions of people around the planet, used to execute countless tasks: checking email, playing games, composing music. But it began with one very urgent task: keeping airplanes from crashing into each other.

From the air traffic control to the physics lab

Sometime in the early 1960s, a British engineer at the Royal Radar Establishment named E. A. Johnson began tinkering with ideas for a new interface for air traffic controllers to manage flight paths in and out of UK airports—in addition to tracking planes for the purposes of air defense. The commercial aviation age was dawning, and the airspace around large metropolitan areas was becoming more challenging to manage. New jet planes were traveling at much higher speeds, making it even more essential that flight controllers could make quick, real-time decisions on the job.

The touch display. Illustration: Ania Augustynowicz. Photos: © Crown copyright reserved / MRATHS collection. Released under OGL v3.0

Air traffic controllers were already used to tracking flight paths via radar screens; Johnson’s insight was that the screen itself could be used to help control those paths and not just visualize them. He built a working prototype of what is now called a “capacitive” touchscreen, with some core features that are still used today on multitouch devices, more than 50 years later. Because glass cannot conduct electricity, the screen is coated with a grid of transparent conductive material, like indium tin oxide. A minuscule amount of electricity flows through the grid at all times, but when a finger makes contact with a point on the screen, a tiny amount of electricity flows through the skin rather than the grid, which enables the device to detect the location of the finger on the screen.

The RAF plotters and civilian scientists monitoring the radar screens in the experimental facilities at Malvern. This is how reports from radar detections were handled and passed to fighter command and control. The touchscreen was invented to speed up this process. © Crown copyright reserved / MRATHS collection. Released under OGL v3.0

“The touch display,” Johnson explained in a paper he published in 1965, “can be considered as a means by which a computer presents a list of those 'choices' which are available to a controller at any given instant and as the means by which the controller indicates his choice back to the computer.” Johnson had primarily focused on its use in the aviation context, but he seems to have grasped that his innovation had broader applications. The device, he noted, “has the potential to provide a very efficient coupling between man and machine.”

The touch display invented by E. A. Johnson. © Crown copyright reserved / MRATHS collection. Released under OGL v3.0
George Samuel Hurst © University of Kentucky Libraries Special Collections Research Center

British air traffic controllers actually integrated Johnson’s device into their workflow in the late 1960s, but the technology failed to advance much beyond that limited application. Like many stories from the history of innovation, the next big breakthrough that would lay the groundwork for the multitouch interface would come out of a fortuitous accident. A few years after Johnson’s original paper was published, a physicist at the University of Kentucky named Samuel Hurst was working with a piece of equipment known as a Van de Graaff accelerator which was commonly used to study charged particles. Hurst and his colleagues had to rely on sluggish, analog strip chart recorders to collect data from their experiments, which slowed down their work. One day, Hurst hit upon the idea of using electrically conductive paper to automatically record x and y coordinates from the experiments. It made the process of data analysis much faster, but in building the device, Hurst began thinking about how the same technology could be applied to the x and y coordinates of a computer monitor.

Before long, Hurst had left the Van de Graff accelerator behind and founded a company called Elographics producing touchscreen interfaces for computers, which he initially ran out of his basement. Hurst correctly envisioned that a touch-based interface would be a major advance in making computers more accessible. “You could just look at a screen, poke your finger, and get an answer,” Hurst, who died in 2011, told a reporter many years later. “Anybody can poke a finger!”

“You could just look at a screen, poke your finger, and get an answer.”
Computer poking device. Illustration: Ania Augustynowicz. Photos: © Oak Ridge National Laboratory, courtesy AIP Emilio Segrè Visual Archives, Physics Today Collection / Portrait © University of Kentucky Libraries Special Collections Research Center

Hurst eventually sold his company—it continues to operate to this day under the name Elotouch—and returned to his career as an academic researcher. But over the next decade or two, the idea that he had stumbled across in the lab began to spread around the world, as touchscreens slowly became a familiar experience, though the range of interaction was limited to simple, single-touch events, like typing a PIN into an ATM. A few new computing experiences—the Apple Newton or the Palm Pilot—explored the idea of direct contact with the screen using a stylus, but the vast majority of computers in circulation relied exclusively on mouse and keyboard-based interfaces.

The Apple Newton, Palm Pilot, and other PDA devices

The Apple Newton is now widely considered one of the most notorious flops in Apple’s history. Still, with hindsight, the device anticipated a number of computing developments that would become mainstream a decade later. Introduced in 1993, the Newton was a handheld device that lacked a physical keyboard—the primary input mechanism was handwriting using a special stylus and touch-sensitive screen. Unfortunately, the Newton’s handwriting recognition software proved to be famously unreliable (even the newspaper cartoon Doonsebury mocked it) and the product line was ultimately retired in the late 1990s. Shortly after that, Palm Computing had more success with their handheld device called the Palm Pilot that shared many features with the Newton but was less reliant on handwriting recognition.

The electrifying demo

All that began to change in the early 2000s, when a number of academic researchers, tech startups, and R&D labs inside of Silicon Valley companies began experimenting with direct onscreen manipulation, using multiple fingers simultaneously. “The [people behind] the original graphic user interface systems developed at Xerox-PARC that were then brought over Apple—they were all about this idea of ‘direct manipulation,’” says Ken Kocienda, the former Principal Engineer for iPhone Software at Apple. “Unlike a command line where you are typing a command and the files are these invisible things in the computer, there's this little picture of an icon which represents an app or a file. And if you double click it or you drag it to the trash or something like that, they thought of that as direct manipulation.” But in truth, those interfaces all relied on indirect manipulation: you changed the pixels on the screen by touching some other device: a mouse or a keyboard or a trackpad. The screen itself was off limits.

The first public hint of the revolution to come would arrive via the stage of the TED Conference in 2006, courtesy of a professor at New York University's Courant Institute of Mathematical Sciences named Jeff Han. Han’s research group at NYU had developed a prototype that featured some of the key interactions that would later become ubiquitous: dragging icons by touching your finger to the screen and sliding it across the surface, pinching or expanding two fingers to zoom in or out on a photo. “Someone sent me a video of Jeff showing off aspects of the interface literally a week before TED2006,” TED curator Chris Anderson recalls. “It only had a few thousand views. I found it electrifying and just reached out to him right away to beg him to drop everything and come to Monterey.”

“I found it electrifying and just reached out to him right away to beg him to drop everything and come to Monterey.”
Electrifying demo. Illustration: Ania Augustynowicz. Photos: © Robert Leslie / Courtesy of TED

“I’m about to show you some stuff that is just about ready to come out of the lab,” Han announced at the beginning of the talk. “I really think this is going to change the way we interact with machines from now on.” His demo device was a large screen arranged like a drafting table in front of him. During the talk, he gave a quick survey of a handful of use cases: arranging photographs, navigating maps, and a few screen-saver-like animations that he manipulated with his hands. But the real star of the show was not the content on the screen, but the way Han’s interface allowed him to engage with it.

“The real star of the show was not the content on the screen, but the way Han’s interface allowed him to engage with it.”

Watching the talk now, it seems almost quaint to hear how amazed the crowd is by gestures that are now so second nature to us. “There’s a moment two minutes in when the audience suddenly got that the future of computer interfaces would change forever,” Chris Anderson recalls. “He was showing photographs on a [virtual] lightbox. He used two fingers to dramatically stretch one photo to fill the screen. You can hear the audience gasping in unison.”

“For both Jeff and most of the audience, the conversation later was about large-screen applications of the kind that started showing up on CNN a year or two later. I didn’t hear anyone say: wait, you could transform mobile phones and computing using this.”
Talk “The radical promise of the multi-touch interface” by Jeff Han at an official TED conference in 2006. © Robert Leslie / Courtesy of TED

A few months after the conference, TED put the talk online as one of the first videos ever shared on its website. (It has since been viewed more than five million times.) Han went on to create a company called Perceptive Pixel that was mostly focused on large-screen applications of multitouch—most famously in the CNN “Magic Wall” used in election coverage. (He later sold his company to Microsoft, where he worked on their Surface Hub computer.) But for all the excitement Han’s demo sparked, the true potential of the new interface was not yet apparent. “For both Jeff and most of the audience, the conversation later was about large screen applications of the kind that started showing up on CNN a year or two later,” Anderson says now. “I didn’t hear anyone say: wait, you could transform mobile phones and computing using this.

Multi-Touch Collaboration Wall

CNN’s “Magic Wall” debuted in 2008 as part of the network’s coverage of the Iowa caucus, the first election in the quest for each party’s presidential nomination. A CNN producer had seen a demo of Han’s technology at a military trade convention and realized that the device could be programmed to show real-time updates and maps of election results. CNN’s John King navigated around the wall using his fingers as the input device while explaining the developing results in counties across the state. The Magic Wall was an immediate sensation, a vast improvement over the whiteboard and marker approach that NBC’s Tim Russert had used to calculate real-time results during the contested 2000 presidential election.

All fingertips on board

The multitouch interface turns out to be a terrific example of a phenomenon that has long been observed in the history of innovation: simultaneous discovery. A mix of new ideas or technologies or scientific breakthroughs coalesce, and all of a sudden there’s a whole cohort of people independently chasing the same idea. (Edison gets all the credit for inventing the lightbulb, but there were more than a dozen inventors around the world working on incandescent light in the 1870s and 1880s, while just a decade or two before, no one was working on the problem.) While Han was working on his multitouch prototype, a startup called Fingerworks that was also experimenting with multitouch systems had been quietly acquired by Apple to help develop a top-secret project called Purple that would ultimately lead to the iPhone.

The Touchstream keyboard by Fingerworks. Its design was a major influence on the iPhone. It was discontinued when Apple bought Fingerworks in 2005. ©Aslak Raanes, CC BY 2.0, via Wikimedia Commons
Project Purple

Project Purple was launched shortly after Apple’s breakthrough success with the iPod in the early 2000s. It was originally inspired by a tablet computer Microsoft had been touting that employed a stylus as an input device. Much like the original Apple Newton, Project Purple’s goal was to develop a tablet that relied exclusively on touch. (Then-CEO Steve Jobs was notoriously opposed to stylus input—during the iPhone launch he joked, “‘Who wants a stylus? You have to get ‘em, put ‘em away. You lose them. Yuck.”) Initial Purple prototypes were all large screen format, but existing battery and screen technology at the time made it difficult to make a device light enough to hold comfortably in your hand. The team ultimately realized that a smaller screen could be supported by a much lighter battery, and the project shifted its focus to making a phone. Eventually, the tablet form factor reappeared with the launch of the iPad in 2010.

Ken Kocienda had joined Apple a few years before the Purple project got underway, initially working on Apple’s Safari web browser, which launched in 2003. He remembers seeing an early demo of the Purple interface, courtesy of legendary Apple designer Bas Ording, showcasing what is now called “inertial scrolling” and the “rubber band effect,” where you flick the screen to scroll quickly through a long list, while the list itself bounces to signal that you’ve reached the end of it. “It was just one of those moments where you say: computers are going to work like this,” Kocienda recalls.

Bas Ording

15 years at Apple, hundreds of patents, the brains behind multiple interactions and animations we immediately recognize from iOS and Mac interfaces, and no Wikipedia entry—Bas Ording is yet another little-known expert whose work was crucial to developing gesture-based interfaces and creating the iPhone as we know it. The inventor of the “rubber band effect” studied Interaction Design in the Netherlands and quickly landed his first job at Apple. Restless experimentation and the ability to instantly recognize interactions that simply “felt good” to the user earned Ording “wizard” status among Apple’s leading designers. Reportedly, the “rubber band effect” kick-started the iPhone’s development.

Simultaneous discovery. Illustration: Ania Augustynowicz. Photos: Jeff Han: © Robert Leslie / Courtesy of TED / G. S. Hurst: © University of Kentucky Libraries Special Collections Research Center / Ken Kocienda: © Imran Chaudhri, Courtesy of Ken Kocienda / iPad & business card: © Courtesy of Ken Kocienda

But as promising as the Purple interface was, the software suffered from a potentially fatal flaw: it was impossible to use a virtual keyboard on a phone-sized screen. Remember, this was the era where Blackberry was the dominant device for mobile communications; people were pecking out texts and email messages on tiny physical keyboards. The radical proposition behind Purple was that you could replace all the real estate on the device devoted to the keyboard, and make it all screen, giving you enough pixels to show web pages or maps or high-resolution photos. If you needed to type, a virtual keyboard would pop up, and you’d enter text by tapping on the screen itself.

All that sounded enticing in theory, but in practice it was a disaster. Given the size of the phone, if you wanted a full keyboard with all twenty-six letters of the alphabet, the virtual keys had to be extremely small—too small, it turned out, for the human finger to type accurately. You’d try to hit the target of the letter “k,” but the pad of your finger would also cover a few adjacent keys at the same time, “j” or “m” or “o.”

Initially, Apple had assigned a small team to work on the software keyboard for Purple, with other engineers and designers working on additional elements of the interface. But week after week, the Apple executive overseeing Purple, Scott Forstall, would visit with the keyboard team and try out their latest iteration, attempting each time to type out his name using the virtual keyboard. Week after week, Forstall’s typing would result in gibberish.

Scott Forstall

A member of the exclusive circle of Steve Jobs’ most-trusted people, he went from Stanford University to NeXT to Apple, and now Broadway. Scott Forstall’s career moved in quantum leaps. As an avid student of human-computer interaction, guru Terry Winograd, and as a brilliant programmer, Forstall became fascinated by human-centered software design. At Apple, he supervised the creation of user interfaces for the Macintosh line, the development of Safari, and the entire iOS that powers Apple’s mobile devices. When Jobs decided that the future belongs to finger-driven interfaces, Forstall was tasked with making sure that hardware, software, and UX teams translated that vision into a user-friendly device. What does one do after leading the development team for the most iconic mobile phone in history? Forestall moved on to produce award-winning musicals.

The erratic keyboard was particularly alarming because of a notorious touchscreen-related fiasco in Apple’s history: the Newton device, which had been launched with a revolutionary new handwriting-detection interface that turned out to be comically unreliable. If Apple was going to take on Blackberry, it needed a way to input text that didn’t require tiny fingers or a steep learning curve. An unreliable keyboard posed an existential threat to the multitouch interface itself.

“I think that what happened was that Scott Forstall went away to Steve Jobs and got the buy-off to say: okay, we're just gonna have an all hands on deck,” Kocienda recalls. “So one day our manager called the whole Purple team out in the hallway and said: stop what you're doing. Stop working on Safari and Notes, Photos and Calendar, everything. You're all keyboard engineers now.”

Keyboard constellations

It was Ken Kocienda who ultimately cracked the virtual keyboard puzzle. He built five distinct solutions over the weeks that followed the hallway meeting, all of which failed for different reasons. But eventually, during a walk around Apple’s legendary “Infinite Loop” campus, an idea came into his head. “I remember thinking: I don’t know how, but I'm going to have to have a dictionary,” he says now. “There's going have to be some little agent looking skeptically at the input that's coming in saying: no, no, no, no—that's not what you meant. This is what you meant.”

“There's going have to be some little agent looking skeptically at the input that's coming in saying: no, no, no, no—that's not what you meant. This is what you meant.”

The brilliance of Kocienda’s idea lay in the fact that it revolved around geometry as much as it did around text. He had realized that each time a user typed a word on the virtual keyboard, the computer could construct a distinct shape—Kocienda called it a constellation—that mapped the various places on the screen where the user had touched the keyboard.

Courtesy of Ken Kocienda

“For each word in the dictionary,” Kocienda explains, “there’s an ideal constellation that is drawn by taking the center point from every key on the keyboard.” For his demo, Kocienda converted the dictionary words into a list of these “ideal constellations,” each one a unique shape based on the arrangement of letters on the keyboard. While the user pecked away on the virtual keys, behind the scenes the software would map that less-than-ideal constellation, and compare it to all the ideal versions in the dictionary. “And so when you typed in three letters, [the software] would look at the points and say, okay, well: what three-letter word looks the most like this constellation?”

Ideal constellations. Illustration: Ania Augustynowicz. Photos: © Imran Chaudhri

Several weeks after the hallway meeting, Apple gathered the Purple team in a conference room to hold what they called a “keyboard derby,” where Scott Forstall would try out all the various demos that had been developed by the group. When Forstall finally made it to Kocienda’s demo, he pecked away at the virtual keyboard. Behind the scenes, the software registered the following keystrokes: yui  r as r [space] r nm r yui r [space] nm r as r nm r qwe. But after translating those nonsensical letters back into the “ideal constellations,” the text on the screen read: “Scott is my name.”

Kocienda won the keyboard derby—and the product that would become the iPhone was off to the races.

“Virtuous collaboration”

It would be convenient—and make for more legible storylines—if every major innovation, in software or any other field, had a single heroic inventor that we could point to. But the reality is that most important new ideas are collaborative, multi-layered affairs. In his book, Kocienda calls this “virtuous collaboration.” Multitouch began with a mechanical breakthrough—using the conductive properties of the human finger to interact with pixels on a screen—but then it required new ideas in interface design to imagine all the ways our fingers could manipulate those pixels in real-time: the magical effects of pinch and zoom, or inertial scrolling. And then it took another brilliant insight to solve a bottleneck that kept the entire project from going mainstream: the hidden geometry behind Kocienda’s keyboard. It also took a variety of organizational structures to make the idea mainstream as well: government agencies like the Royal Radar Establishment; academic institutions like the University of Kentucky and NYU; and multinational corporations like Apple.

When Steve Jobs strode onstage in January of 2007 and first demonstrated the “magical” interface of the iPhone, it seemed like one of those rare moments in tech history where the whole field leaps forward in a flash, and indeed the interface he shared that day did, in fact, live up to its initial hype, leading to what is arguably the most profitable single consumer product ever created in the history of capitalism. But the magic itself behind the user experience was 50 years in the making.

The most important new ideas are collaborative, multi-layered affairs.

Steven Johnson is the bestselling author of 13 books, including Where Ideas Come From. He’s the host of the PBS/BBC series Extra Life and How We Got to Now. He regularly contributes to The New York Times Magazine and has written for Wired, The Guardian, and The Wall Street Journal. His TED Talks on the history of innovation have been viewed more than ten million times.

Don't miss a good story
Don't miss a good story
Don't miss a good story
Don't miss a good story
Don't miss a good story
Don't miss a good story

Sign up to uncover the stories of Hidden Heroes with us