Pierre Buttin
In 1977, David Mills, an eccentric engineer and computer scientist, took a job at comsat, a satellite corporation headquartered in Washington, D.C. Mills was an inveterate tinkerer: he’d once built a hearing aid for a girlfriend’s uncle, and had consulted for Ford on how paper-tape computers might be put into cars. Now, at comsat, Mills became involved in the arpanet, the computer network that would become the precursor to the Internet. A handful of researchers were already using the network to connect their distant computers and trade information. But the fidelity of that exchanged data was threatened by a distinct deficiency: the machines did not share a single, reliable synchronized time.
Over decades, Mills had gained wide-ranging expertise in mathematics, engineering, and computer science. In the early seventies, as a lecturer at the University of Edinburgh, he’d written programs that decoded shortwave radio and telegraph signals. Later, largely for fun, he’d studied how the clocks in a power grid could wander several seconds in the course of a hot summer’s day. (The extent of their shifts depended not just on the temperature but on whether the grid used coal or hydropower.) Now he concentrated on the problem of keeping time across a far-flung computer network. Clock time, Mills learned, is the result of an unending search for consensus. Even the times told by the world’s most precise government-maintained “master clocks” are composites of the readings of several atomic clocks. The master clocks, in turn, are averaged to help create international civil time, known as Coördinated Universal Time and initialized as U.T.C.
To solve the problem of time synchronization on the arpanet, Mills built what programmers call a protocol—a collection of rules and procedures that creates a lingua franca for disparate devices. The arpanet was experimental and capricious: electronics failed regularly, and technological misbehavior was common. His protocol sought to detect and correct for those misdeeds, creating a consensus about the time through an ingenious system of suspicion. Mills prided himself on puckish nomenclature, and so his clock-synchronizing system distinguished reliable “truechimers” from misleading “falsetickers.” An operating system named Fuzzball, which he designed, facilitated the early work. Mills called his creation the Network Time Protocol, and N.T.P. soon became a key component of the nascent Internet. Programmers followed its instructions when they wrote timekeeping code for their computers. By 1988, Mills had refined N.T.P. to the point where it could synchronize the clocks of connected computers that had been telling vastly differing times to within tens of milliseconds—a fraction of a blink of an eye. “I always thought that was sort of black magic,” Vint Cerf, a pioneer of Internet infrastructure, told me.
Today, we take global time synchronization for granted. It is critical to the Internet, and therefore to civilization. Vital systems—power grids, financial markets, telecommunications networks—rely on it to keep records and sort cause from effect. N.T.P. works in partnership with satellite systems, such as the Global Positioning System (G.P.S.), and other technologies to synchronize time on our many online devices. The time kept by precise and closely aligned atomic clocks, for instance, can be broadcast via G.P.S. to numerous receivers, including those in cell towers; those receivers can be attached to N.T.P. servers that then distribute the time across devices linked together by the Internet, almost all of which run N.T.P. (Atomic clocks can also directly feed the time to N.T.P. servers.) The protocol operates on billions of devices, coördinating the time on every continent. Society has never been more synchronized.
For decades, Mills was the person who decided how N.T.P. should work (though he disputes the suggestion that he acted with total sovereignty). Quirky, prickly, authoritative, and sometimes opaque—“He does not suffer fools gladly,” one longtime collaborator said—he has served as the Internet’s Father Time. But his tenure is coming to an end. Mills was born with glaucoma. When he was a child, a surgeon was able to save some of the vision in his left eye, and he has always worked using very large computer displays. Around a decade ago, his vision began to fail, and he is now completely blind. Examining computer code and writing out explanations and corrections have become maddeningly tedious. Drawing diagrams or composing complex mathematical equations is nearly impossible.
A couple of years ago, I visited Mills in his unassuming house in the Delaware suburbs. He and his wife, Beverly, have lived there since 1986, when Mills became a professor at the University of Delaware, a position he held for twenty-two years until his retirement. While we sat in his kitchen, our conversation was regularly interrupted by an automated voice announcing the time from the next room. The oven and microwave clocks were out of synch. Mills, who has a snow-white beard and wore a charcoal fisherman sweater, tracks the time for himself using a speaking wristwatch, which connects by radio signals to a master clock in Colorado.
He led me upstairs to his office, slowly making his way through the house by feeling for a series of memorized “navigation points.” At his desk, where a cat lay atop some crackling ham-radio equipment, Mills sat down at his computer. He used the keyboard to pull up a research paper he was working on, with suggestions for improvements to N.T.P. (He asks his wife and daughter to proofread what he types.) As he used the arrow keys to scroll, the computer spoke aloud. “This memo explores new security and protocol enhancements,” a voice said. “Blank. Table of contents. Blank. One. Two. Two point. . . . Three. Three. Four. Four point one. . . .” Soon, he got lost. “I do what I can using the voice that you hear,” Mills said. “But I observe myself and comment on the following: man was made to do English composition by eyeball.”
Technology doesn’t stand still. The Internet continues to grow in both scale and complexity; even as its infrastructure ages, our world depends upon its functioning to an ever-increasing degree. The continued evolution of the Internet’s time-synchronization system is essential. And yet Mills’s inability to swiftly contribute to N.T.P. has sapped his authority over it. In his absence, only a few people appear to be both capable and willing to oversee the critical yet overlooked software. A contest for influence over how clocks are kept in synch across the Internet has begun.
Mills was born in 1938 in Oakland, California, eleven years after the development of the first quartz clock and nine years before the construction of the first transistor. He took a steam-powered train to a school for the visually impaired, in San Mateo, and marvelled at the engineers who ran it. In his teens, he became a model-railroad and ham-radio enthusiast, communicating with friends and patching Navy Seabees at the South Pole through to their wives. His father, an engineer and salesman, co-founded National Oil Seal, a company that manufactured equipment to prevent leakage within machinery. (“You might not know what it is, but there are at least two of them in the engine of your car,” his father told him, of the seals.) His mother trained as a pianist at the Toronto Conservatory of Music before staying home to raise him and his two younger brothers.
The family moved around, and Mills’s teachers didn’t always accommodate his visual impairment. Mills recalls an eleventh-grade teacher telling him, “You’re never going to get to college”—a remark that was “like waving a flag in front of a bull,” he said. In 1971, Mills earned a Ph.D. in computer and communication sciences at the University of Michigan; after a two-year stint lecturing in Edinburgh, he moved with his wife and two children to the University of Maryland, which denied him tenure after five years. “It was the best thing that ever happened to me,” Mills said. He started work at comsat, where he had access to funding from the Department of Defense, some of which was earmarked for the arpanet. “It was a sandbox,” he later told an interviewer. “We just were told, ‘Do good deeds.’ But the good deeds were things like develop electronic mail, and protocols.” Part of the allure of the time-synchronization work, he told me, was that he was just about the only one doing it. He had his own “little fief.”
In N.T.P., Mills built a system that allowed for endless tinkering, and he found joy in optimization. “The actual use of the time information was not of central interest,” he recalled. The fledgling Internet had few clocks to synchronize. But during the nineteen-eighties the network grew quickly, and by the nineties the widespread adoption of personal computers required the Internet to incorporate millions more devices than its first designers had envisioned. Coders created versions of N.T.P. that worked on Unix and Windows machines. Others wrote “reference implementations” of N.T.P.—open-source codebases that exemplified how the protocol should be run, and which were freely available for users to adapt. Government agencies, including the National Institute of Standards and Technology (nist) and the U.S. Naval Observatory, started distributing the time kept by their master clocks using N.T.P.
A loose community of people across the world set up their own servers to provide time through the protocol. In 2000, N.T.P. servers fielded eighteen billion time-synchronization requests from several million computers—and in the following few years, as broadband proliferated, requests to the busiest N.T.P. servers increased tenfold. The time servers had once been “well lit in the US and Europe but dark elsewhere in South America, Africa and the Pacific Rim,” Mills wrote, in a 2003 paper. “Today, the Sun never sets or even gets close to the horizon on NTP.” Programmers began to treat the protocol like an assumption—it seemed natural to them that synchronized time was dependably and easily available. Mills’s little fief was everywhere.
N.T.P. works by telling computers to send tiny, time-stamped messages to time-checking devices superior to them in a hierarchy. The hierarchy’s uppermost layer consists of servers that are closely connected to highly accurate clocks kept in tight synchronization with Coördinated Universal Time. The time then trickles, from strata to strata, to the machines at the bottom of the hierarchy, such as ordinary laptops. The protocol tracks the instants that elapse as a time-checking message is sent, received, returned, and received again by its original sender. All the while, a collection of algorithms—the “popcorn spike suppressor,” the “huff-n’-puff filter”—sifts through the data, singling out falsetickers and truechimers and instructing the clocks on how to adjust their times based on what the time-stamped messages tell them.
Perfection is an elusive dream in engineering. Mills received a stream of complaints about N.T.P.’s hiccups, and also suggestions for possible patches and improvements. Unprecedented problems arose. Some hardware designers who’d built N.T.P. into their devices turned out to have configured it improperly; the devices sent suffocating swarms of messages to individual servers in places such as Wisconsin and Denmark. Some coders thought N.T.P. was to blame. “I got really angry, because my name was getting mentioned as sort of ‘I’m the single person choking the Internet,’ ” Mills said. He and others quickly wrote countermeasures, including a “kiss-o’-death” packet capable of quieting needlessly frequent time requesters. Meanwhile, some N.T.P. contributors began to bristle at Mills’s intransigent tendencies. Several objected to the kookiness of his code. “Be advised I dearly love the English patois and will on occasion torque the language to my personal whimsy,” Mills wrote, in response to a complaint.
In a 2000 essay, the software developer Eric S. Raymond describes the “benevolent dictator” model of open-source coding. In it, suggestions for improvements flow from a community of volunteers toward a single decider. Mills, receiving these suggestions, could be warm and enthusiastic but also obstinate and dismissive. “I am considered by most to be somewhat [of] a carmudgeon [sic] and I apologize in advance for my shortcomings,” he wrote, in an e-mail to an N.T.P. mailing list, in 2005. At the same time, he argued, “there are really important principles behind the design of this monkey.”
Poul-Henning Kamp, a developer who operated the briefly beleaguered Danish N.T.P. server, told me that, though Mills welcomed contributions to the more obscure elements of N.T.P., he grew harder to convince as collaborators “edged closer to the ‘crown jewels’ ”—the protocol’s core time-deciding algorithms. When Kamp first began engaging in the small N.T.P. community, in the late nineties, Mills struck him as a “jolly old elf, with a lot of wisdom to share and many interesting tales.” Still, Kamp understood the exhaustion and exasperation others felt. “You had to climb a very steep mountain to get anywhere at all with Dave,” he said. It wasn’t enough to come up with a patch that fixed a problem; Mills wanted “a mathematical analysis proving that it would do so in all cases.” Few other benevolent dictators expected this level of exactitude from their volunteers. As a result, almost no one else gained full insight into how N.T.P. worked. “I had a reason for doing this instead of that, but I wasn’t able to communicate in language that others in the crew would appreciate,” Mills recalled.
Aprogrammer named Harlan Stenn has been Mills’s most devoted collaborator. Born in 1956 and raised in a Chicago suburb, Stenn fell in love with computers in high school because, he said, “I got very clear yes-or-no information about whether something was going to do the right thing, and to me that was very peaceful and enjoyable.” His contributions to N.T.P. began in the early nineties when he volunteered to make it easier for a systems administrator to tweak the software across an array of different devices. After Stenn’s ideas succeeded, Mills asked him to look at a backlog of bug reports.
By the mid-nineties, Mills entrusted Stenn with releasing new versions of the code. Stenn had started working at an insurance company, but his responsibilities there largely involved being around in case a system failed. He began devoting most of his working hours to N.T.P. The two often clashed, yet, a decade later, Stenn got Mills’s permission to take full control of N.T.P.’s reference implementation. Although they have collaborated for decades, they have never met in person; instead, Stenn has spent untold hours consulting Mills’s documentation and discussing the code with Mills over the phone. Stenn sometimes struggles to understand some of N.T.P.’s most complex functions, and told me that he considers Mills “a super genius.” Although most of the few dozen contributors to N.T.P. have come and gone over the decades, Stenn has remained. In 2010, after he concluded a term at Netflix, he took a two-year sabbatical from contract work and devoted it entirely to maintaining N.T.P.; he has not taken another full-time job since.
Technology companies worth billions rely on open-source code, including N.T.P., and the maintenance of that code is often handled by a small group of individuals toiling away without pay. In “Working in Public,” a book that explores the code-maintenance economy, the writer Nadia Eghbal notes how, even though “more people use open source code than ever before, its developers failed to capture the economic value they created.” Because the Internet is ever-changing, there’s always something to fix, and so “maintaining code for general public use quickly becomes an unpaid job you can’t quit.” Stenn told me that he has received requests for free fixes from companies that charge customers for services that depend on N.T.P.; meanwhile, he has struggled to secure funding for his efforts. In 2011, he founded Network Time Foundation—in a Millsian twist, the name omits the “the”—which he hoped to make into a hub for those working on various time-synchronization systems. It gathers, through donations, roughly three hundred thousand dollars a year, which Stenn uses to pay part-time staffers, as well as his Internet, electricity, and grocery bills. He is sixty-six years old, and told me that he has burned through his retirement savings.
Bugs aren’t the only troubles that beset N.T.P. In 2011, N.T.P. servers at the U.S. Naval Observatory received a deluge of time requests, the majority of which came from what appeared to be Chinese agencies; the servers were forced to go offline for nine and a half hours. In 2014, trolls abused an N.T.P. command function to launch a so-called distributed denial-of-service (DDoS) attack on a number of gaming platforms, temporarily shutting down their services. N.T.P. soon became a popular vector for such attacks; the same N.T.P. exploit was later used against a client of the cybersecurity provider and content-delivery network Cloudflare, in what was among the largest DDoS attacks in Internet history. These events helped spur competition. One open-source group, called NTPsec, whose current technical lead is Eric S. Raymond, cut out vast swaths of N.T.P.’s code, reasoning that fewer lines meant fewer vulnerabilities; another competitor, Chrony, was written from scratch, and prioritized synchronizing clocks quickly with less data, while omitting certain oft-unused modes of operation. It is still being maintained by Miroslav Lichvar, a Czech developer whom Stenn described as a “wizard.”
There have been more elemental concerns, too. The speed of the Earth’s rotation is affected by a variety of atmospheric and geologic factors, including the behavior of the planet’s inner layers; the reshaping of its crust, such as through the growth of mountains or bodies of magma; and the friction of the ocean’s tides against the seafloor. The aggregate effect of these forces has historically been to slow the rate at which the planet spins. In prior ages, days were, on average, shorter: seventy million years ago, a year consisted of three hundred and seventy-two days of twenty-three and a half hours each. The length of a day also fluctuates by a minuscule amount from month to month, season to season, and year to year. This erraticism has created a problem for timekeepers ever since it was first confirmed, in the early twentieth century. A further wrinkle developed in the following decades, when the world’s master clocks began keeping time by following the unvarying behavior of certain highly controlled atoms. The drifting duration of the day wanders out of synch with the steady marching of the atomic second, and this risks opening a rift between the time as told by atoms and the time as told by astronomy.
To address the problem, the world’s leading timekeepers began sprinkling in individual “leap” seconds in 1972 whenever the Earth’s slowing rotation put clocks just shy of a second out of synch with the time kept by atomic clocks; this practice realigns the clocks with the stars. So far, there have been thirty-seven leap seconds. When Mills wrote N.T.P., he accounted for them by allowing the protocol to count certain seconds twice. But some programmers, perhaps unaware of the existence of leap seconds, neglected to account for them. In 2005, engineers at Google realized that some of their systems were vulnerable to leap-second malfunctions. “They go, ‘I see time goes backward. That’s not supposed to happen. I will crash,’ ” Jamie Wilkinson, one of Google’s site-reliability engineers, told me; he explained that the programs would choose to shut down rather than risk corrupting user data.
Google began updating its internal version of N.T.P. But its programmers took a different approach to the issue: instead of counting a leap second twice, as N.T.P. does, Google’s systems effectively redefine the second for a certain period of time, during which they add a handful of milliseconds to each second, spreading out the addition of time rather than concentrating it. Google first utilized this approach, which it calls the leap smear, in 2008, in its internal systems. But in 2016 it introduced the leap smear to the rest of the Internet, via its N.T.P. servers.
Google considers the leap smear to be a useful and clever solution to a strange problem. And yet the simultaneous existence of two leaping techniques may sow confusion. If a device attempts to ascertain the time by consulting two sources, only one of which smears the second, it will struggle to distinguish a truechimer from a falseticker, and may settle on a minutely incorrect time. Nevertheless, other companies serving time to the public, including Amazon and Facebook, have begun offering smeared time as well.
“They should all be sent to bed without dinner,” Judah Levine, a physicist who has worked at nist since 1969, told me, referring to the tech companies. Levine has maintained nist’s own N.T.P. service for decades, and he and a select few of his colleagues are responsible for reconciling its master clocks with Universal Coördinated Time, often within billionths of a second. If the companies disliked leap seconds, Levine said, they should have lobbied for change through the intergovernmental bodies that have the power to abolish them. (Levine himself would prefer a world without leap seconds, and the U.S. government, with his help, has proposed eliminating them for the sake of computer systems; astronomers and other governments have objected. The next opportunity for formal discussion will come in 2023, and the matter is already actively being debated.) “There’s a process,” Levine said. “And if they don't like it they don’t get to say, ‘Well, I’m not going to play anymore. I’m going to do what I’m going to do, and tough luck on you.’ ” As one of the world’s preëminent observers of nanoscopic slices of seconds, Levine argues that there could be legal implications if computers disagreed about when a message was sent, or about when a stock trade occurred.
Meanwhile, for unknown reasons, the Earth’s rotation has been going against trend in recent years: instead of slowing, it has been accelerating. This year had the shortest day observed since people started tracking the length of the day, about a half century ago. If this development continues, then the world may need to remove a second rather than add one—a task that the architects of modern time synchronization have yet to attempt.
Coders sometimes joke, morbidly, about the “bus factor.” How many people need to get hit by a bus before a given project is endangered? It’s difficult to determine the bus factor for N.T.P., and time synchronization more broadly, especially now that companies such as Google have developed their own N.T.P.-inspired proprietary code. But it seems reasonable to say that N.T.P.’s bus factor is rather small. Eghbal, in “Working in Public,” describes code as “a living form of knowledge”; in the case of N.T.P., very few people possess that knowledge. Stenn said that for a long time he was reluctant to change his N.T.P. code too drastically, because significant alterations could untether it from Mills’s recollections; Mills is eighty-four, and his memory is resilient yet fitful. Sitting in his kitchen, Mills told me, “My brain is fried by age.”
There has not been a fully new version of N.T.P. since its fourth iteration, which was published in 2010. Now a subgroup of about two dozen contributors within the Internet Engineering Task Force (I.E.T.F.), the Internet’s principal standards body, is working toward version five. It may prove to be a dramatic departure from what currently exists. When the I.E.T.F. was created, decades ago, it was inspired by an early working group chaired by Mills. But his role in and influence over the N.T.P. community has waned. In the fall of 2020, Mills e-mailed the paper I’d seen him working on to the subgroup and received no substantive response for months. He wasn’t surprised. “I’m considered a bit of an old fogey,” Mills said. “When I was a professor, and I was in my twenties, my students thought of me as one of them. In the next twenty years, I was their father, and I was evil. And in the next twenty years, I was an old geezer and could be ignored.” Stenn has attempted to act as Mills’s emissary in the group, but has occasionally caused offense by doing so. In one instance, Stenn voted against a proposal on Mills’s behalf before Mills had had a chance to review it, and a fellow-member decried his “attempt to seize the mantle of Prof. Mills’ authority.” The member went on, “You are not acting as a faithful messenger and I will not accept your words as his.”
The subgroup working on version five of N.T.P. makes decisions by consensus. But Stenn, like Mills, is neither a skilled diplomat nor a committed one. “My goal isn’t to build consensus,” Stenn told me. “My goal is to have the best quality timekeeping stuff we can have.” Even as Stenn insists that he is not Mills’s intellectual equal, he believes that decades of experience have given him unique insight into the details and structure of the protocol, and how people actually use (and abuse) it. Some members, however, question Stenn’s authority, including whether the code he has devoted decades to maintaining should be considered the sole reference implementation of N.T.P. Many want to see their ideas reflected in the group’s output. A widely held view is that N.T.P. should become more modular and flexible so that it can be improved more frequently. A reverence for Mills bumps against a skepticism of whether what he built was truly optimal. “I think in the past there was a lot of ‘magic math’ in the algorithms, claiming ‘don’t ever change this, or the world will fall apart’ ;-),” one member has written.
It is difficult to know in what ways N.T.P. might be helped or hurt by having a larger community devoted to it. It has rarely, if ever, fostered such a community in the way that other open-source systems have. Unconvinced of version five’s trajectory, Stenn has largely chosen to continue focussing on version four. Meanwhile, a leading draft of the next version is being written by Miroslav Lichvar, the “wizard” Czech coder behind Chrony, the alternative to N.T.P. Lichvar is a somewhat mysterious presence in the N.T.P. community. One observer said, of him, “He’s extremely quiet, and nobody knows him.”
Lichvar got involved in N.T.P. around fifteen years ago, when he took over maintaining the open-source software firm Red Hat’s N.T.P. package. He soon sent ideas and patches to Mills, only to have them rejected. “That was frustrating,” he told me, over e-mail, after declining an initial interview request. “It seemed they were happy with what they had and were not interested in making major changes.” His years developing the Chrony code confirmed his admiration for Mills’s work, but also convinced him that alternative timekeeping algorithms can work effectively. He believes that N.T.P. should operate without specifying which time-synchronization algorithms must be used; to him, they are not crown jewels. “Chrony has different algorithms, which I think work better in most conditions, but I wouldn’t want to force them on anyone,” he said. “There certainly could be even better algorithms implemented.” Multiple people, including Stenn, told me that Lichvar is one of the most respected people in the I.E.T.F. working group, but Lichvar said that he doesn’t think of the group as having leaders. “It’s just a bunch of random people who usually have a hard time reaching a consensus,” he wrote.
Should Internet time synchronization run on rigorously tested and battle-worn but whimsical and arguably bloated code that someone may still struggle to fully understand, even after devoting decades to it? Or should it be based on a nimbler, less pedantic standard designed by people who can’t agree on what’s best? There won’t be one answer: no open-source author has enforcement power over what implementations companies and system administrators choose to deploy. (According to Stenn, much of the Internet still utilizes version three of N.T.P., which was published in 1992.) Finding consensus can be difficult for both clocks and people. In the meantime, our society continues to develop new uses for synchronized time, which demand further technical innovations. The Precision Time Protocol, which was first published in the early two-thousands, was designed primarily for automated robotic manufacturing; it offers especially close time synchronization for hardwired networks, and is now routinely used to synchronize financial trading. In 2019, after years of effort, the I.E.T.F. released a standard for Network Time Security, a mechanism which adds capabilities atop N.T.P. in an attempt to make it more secure. (Time underlies much of the Internet’s cryptography infrastructure.) The expanding Internet of Things will only contribute to the ever-growing need for synchronization. Sharon Goldberg, a computer scientist at Boston University who worked on the Network Time Security effort, told me that she thinks time synchronization should have a cryptocurrency-like buzz around it (ideally with less controversy)—coders who contribute to it, she said, should feel proud enough to declare, “Everyone uses the software, it’s in everything, and I wrote it!” It’s striking how few people know Mills’s name, given how many know the pseudonym of whoever created Bitcoin.
Nowadays, Mills listens to books and papers on the big bang, and on Egyptian, Greek, Roman, English, Jewish, and Soviet histories; he spends time with his children, listens to TV alongside his wife, and plays with his radios. Every few months, Stenn asks him to explain some aspect of the current version of N.T.P. Mills is disgruntled about aging: each time he walks he feels on the verge of stumbling, and he told me on a recent phone call that he is concerned about his “geezer quotient.” He has more difficulty conceiving of and describing mathematical ideas, and his writing’s friskiness has faded. “I’m too busy trying to tease the meaning out of a sentence,” he said.
Despite this, he has been spending a couple of hours a day composing a paper that outlines a redesign of the “bare bones” of N.T.P. His aim is to revise and improve its basic model, and to simplify it. Today’s programming culture, he feels, is surprisingly expansionist: coders want “to add features, improve security, and do things that I thought would be out of the scope of what the protocol is expected to do.” Mills is unsure if he’ll be able to find someone to edit the paper; he fears that Stenn is too weighed down by “real-world issues” to aid him in his reconsideration of N.T.P.’s “foundational principles.” He doesn’t expect the paper to be read by many in the N.T.P. community, either. “I think they will ignore this one, too,” he said. I asked him why, then, did he keep working on it. “Because it’s there,” he said. “I like to improve what I do.”
No comments:
Post a Comment