Waymo Is Millions Of Miles Ahead In Robot Car Tests; Does It Need A Billion More?
Sometime this year Waymo, Alphabet Inc.’s prized driverless car bet, starts a first of its kind revenue-generating robo-taxi service in Phoenix.
Ahead of that the unit is maintaining a steady cadence of news underscoring how mature the former Google Self-Driving Car project is – including how big a lead it has over rivals in test miles.
Waymo this week said its test fleet has logged 5 million miles driving in autonomous mode on public roads. That’s more than double the 2 million miles Uber reached in December (though both companies are now capable or racking up a million test miles about every three months, based on reporting by Forbes’ Biz Carson). Waymo’s tally in computer-simulated tests, where it’s running 10,000 virtual vehicles through scenarios 24 hours a day, has passed 5 billion miles, and it also tries to stump robot drivers at a private test facility in rural California.
“In raw miles, Waymo is by far the leader,” said Grayson Brulte, a Beverly Hills-based driverless industry consultant. “They’re like Jesse Owens or Carl Lewis – running a 100-meter dash around everybody.”
But as many miles as it’s logged in the real world, Waymo may still be far from what it or any other company needs to do that. A 2016 study by RAND Corp. determined that demonstrating the reliability of autonomous vehicle tech to handle anything that could happen on public roads, in terms of reducing traffic fatalities and injuries, could require hundreds of millions or even hundreds of billions of test miles.
“They do have a meaningful lead – nobody else comes close to the millions of miles Waymo has driven on roads over the past decade,” said Nidhi Kalra, a San Francisco-based RAND scientist who was the lead author of the 2016 report. “It means they are finding the rarer and trickier situations and learning more and more. There’s just no true substitute for this.”
All those Waymo test miles racked since 2009 – predominantly in places with lots of sun and little snow or inclement weather – mean its fleet of robotic minivans have contended with more roadway circumstances than a human driver might confront in multiple lifetimes. Each mile is also uploaded to the cloud and shared across its fleet in a never-ending learning process for the artificial intelligence behind the wheel that Waymo says is key to building a better driver.
“We’ve now test driven in 25 U.S. cities, gaining experience in different weather conditions and terrains: from the snowy streets of Michigan to the steep hills of San Francisco, to the desert conditions of Greater Phoenix,” Waymo said in a recent blog post. “And because the lessons we learn from one vehicle can be shared with the entire fleet, every new mile counts even more.”
That helps explain why it does so well in annual tallies by California’s Department of Motor Vehicles that summarize how often a manufacturer’s test vehicles disengage from autonomous mode and hand over control to a human technician. Waymo said last month that its California test fleet had just 0.18 disengagements per thousand miles last year, compared with 0.80 disengagements per thousand miles for General Motors Cruise, the second-best performer in the DMV data.
Nothing is as critical as logging miles to verify the technology with statistical comparisons to human beings, said Chris Gerdes, a professor of mechanical engineering at Stanford University and Director of its Center for Automotive Research.
“Humans tend to have roughly one fatal crash every 100 million vehicle miles traveled, so 5 million is still too small to compare on that basis,” he told Forbes. Yet if road tests focus on particularly difficult driving scenarios, “you can get a handle on these from much less than 5 million miles, assuming those miles are well distributed – not all on a couple of freeways, for instance,” he said.
Continually adding real-world test miles is critical for developing statistical models that can be used in computer simulation, Gerdes said. “These additional miles provide insight into how likely certain situations are and what sort of variability exists. This enables increasingly refined models that can point out potentially troublesome or critical situations for simulation.”
At this point, Waymo’s simulated miles may be more meaningful, owing to the greater complexity of “critical situations” it can create, he said. “I would imagine their simulator is highly refined and enables a lot of sensitivity testing in critical situations. The quality and comprehensiveness of a simulator is, to me anyway, more impressive than the number of miles simulated.”
RAND’s Kalra isn’t certain how to assess the value of Waymo’s massive amount of simulated driving data for two reasons. “We don’t know what those miles are like or how well they represent the real world, and we have no idea how well the vehicle actually did in simulation,” she said.
Given that Waymo’s robot chauffeur service will soon be offered to the public for the first time, we’ll soon find out exactly how much it’s learned from all those miles and the company has truly built a better driver.