Uber Self-Driving Car Research: What the Company had to Offer Aurora?

In our AV report, we talked about the 30 autonomous vehicle companies researching self-driving technologies. And most of these companies are competing for providing a single service, i.e. Driverless taxi service. 

Waymo, GM, Aptiv, are some well-known companies that are building their self-driving vehicles for providing driverless taxi service. In fact, Waymo even successfully launched its driverless taxi services in limited areas.

Uber, being one of the largest ride-hailing companies, has a major stake in the mobility business. Given the competition and opportunities at stake, it becomes crucial for Uber to invest in Self-driving vehicles. 

“If we are not tied for first, then the person who is in first, or the entity that’s in first, then rolls out a ride-sharing network that is far cheaper or far higher quality than Uber’s, then Uber is no longer a thing,” Travis Kalanick had told Business Insider in a 2016 interview.

The Curious Case of Uber ATG

To achieve self-driving solutions, in 2016, Uber started research on autonomous vehicles and invested a huge amount in its self-driving project, better known as Uber ATG. Uber spent a lot of money on self-driving research and related acquisitions such as it bought Otto for 1% of its valuation which was worth $678 million at that time.

In April 2019, Bloomberg reported that Uber has spent over a billion dollars on self-driving. Uber spent $457 million in 2018 on self-driving technologies and the amount grew $73 million in 2019, i.e. to a total of $530 million. By the time they sold the business, its ATG unit had already cost them $303 million in 2020 alone. 

By combining all the figures, we have a total investment of around $1.8 billion.

Despite spending billions of dollars, Uber wasn’t making progress and was quite behind its competitors. 

One of the prime reasons is the infamous Arizona Crash. After that Uber had to halt all the on-road activity for all four locations including San Francisco. And when they started again, the competitors were already far ahead then Uber. 

The truth of the day is Uber is still not making a profit and the company has been facing huge losses. In 2019, even with $14.1 billion in earnings, the company suffered a loss of $8.6 billion which was almost 3x more than in 2018. 

The pressure from investors and the shortage of funds led Uber to sell its self-driving unit, Uber ATG, to Aurora for $4 billion or say a 26% stake in Aurora along with a board seat for its CEO, Dara Khosrowshahi.  

The interesting part is Uber further made an additional investment of $400 million in Aurora. This acquisition increased Aurora’s valuation to $10 billion. 

Aurora already has a name in the autonomous vehicle industry along with other startups like Zoox, AutoX, Argo.ai, etc. And with Uber’s ATG, it would surely get an edge in self-driving.

Now the question is – What did Uber have to offer Aurora?

That’s the question we tried to find answers to which gave us an insight into Uber’s work in self-driving tech.  

To achieve this, we examined Uber’s patent portfolio and research activities in autonomous vehicles to find what the company was up to. 

What are the technologies that Uber has been researching?

In recent years, Uber has increased the budget for its R&D activities and was intensely researching self-driving technologies like image processing, pattern recognition, navigation systems, autopilot, lidar, etc. 

The proof lies in the company’s patent portfolio that gives a glimpse of its research in different technologies: Uber has 844 patent families and a total of 2497 patents in its portfolio. 

These patent families can be classified into 6 research areas on which Uber was focusing. 

The above chart clearly indicated the scope of self-driving research for the company. 

Uber had started researching autonomous driving back in 2015 when it created a subsidiary Advanced Technologies Groups (Uber ATG). Major players like Softbank, Toyota, and Denso had previously invested in Uber ATG to give it a boost. 

Further, Uber spent around two billion dollars on its ATG unit. Uber was certainly not the only one to spend this much on self-driving,  but if they are successful in developing a self-driving vehicle in collaboration with Aurora, it certainly would be the most beneficial one since they are the biggest ride-hailing company in the world.

Which Self Driving Technologies Uber was researching on?

We wanted to go a tad deeper and figure which were the core self-driving research areas of Uber, so we looked up its patents and research publications. Below we have listed all the core technologies Uber was focusing intensely on for making its self-driving ride-hailing cab service a reality. 

Artificial Intelligence

Uber has a dedicated department for AI — UberAI, which is important for Uber’s future growth as it underpins many technologies and services on the platform. 

According to Ghahramani, these are some of the areas in which Uber AI is working.  

“On the research side, what we’re trying to do is we invest in areas where we think Uber needs to be at the cutting edge. Those areas of research include reinforcement learning, which is an area of research around systems that make sequential decisions to optimize something in the real world, obviously relevant to Uber; deep learning, a technology that’s used very widely; probabilistic modeling; and evolutionary algorithms— algorithms that can optimize their behavior through generations of improvement.” Zoubin Ghahramani, Chief Scientist at Uber. 

UberAI is divided into two different teams, i.e. AI Engagements and AI Platform. 

The AI Engagements team leverages various ML/AI models for forecasting to make data-driven decisions at scale, which include:

Marketplace Forecasting: Predicts user supply and demand to direct driver-partners to high-demand areas before they arise, thereby increasing their trip count and earnings.

Hardware Capacity Planning: Predicts hardware capacity requirements to avoid under-provisioning which may lead to service outages or over-provisioning which can be costly to pay for expensive underutilized infrastructure.

Marketing: Makes estimates on marginal effectiveness of different media channels while controlling for trends, seasonality, and other dynamics (e.g., competition or pricing).

Setting Financial Goals: Predicts future values of time-dependent data such as sales, weekly trip counts, available driver levels, app traffic, economic conditions, etc.

Its AI Platform team, on the other hand,  researches technologies like signal processing, computer vision, and Natural language processing to focus on user’s pain points such as crash detection, improved GPS, automatic driver license approval, etc. 

Some of these research areas would come in handy to Aurora for its self-driving initiative. However, two areas we feel Aurora will benefit from and Uber AI majorly focused are on Neural Networks and Machine Learning. Let’s look into both of them. 

Neural Networks

Neural Networks are an important aspect of artificial intelligence. Uber patents and research publications show that the company is heavily investing in neural networks to solve important problems of self-driving vehicles.

Patent Insights and Publications

Uber neural networks patents can be categorized into training, architecture, neural networks models, and artificial life. Further, Training and Architecture neural networks are more research-focused as per the patent filings. 

In fact, there are tens of research publications that hinted at Uber’s desire to utilize neural networks in self-driving improvements. 

• For example, one publication talks about how Uber’s deep neural network helps identify unknown objects. It’s a challenging task for a self-driving vehicle to segment known and unknown objects. But to make it happen, the ride-hailing giant proposed the Open-Set Instance Segmentation (OSIS) network.

Below is Uber’s note on how the OSIS network works: 

The proposed deep convolutional neural network encodes points into a category agnostic embedding space in which they can be clustered into instances. 

As a result, the method is able to perceptually group points into instances, irrespective of whether they belong to a known or unknown class. We validate our method on two large-scale self-driving datasets and achieve state-of-the-art performance in the open-set setting.

• Another publication talks about the usage of graph neural networks for joint interaction and trajectory prediction for Autonomous Driving. 

In this research paper, Uber proposed a graph-based model for multi-agent trajectory and interaction prediction, which explicitly models discrete interaction types using programmatically generated weak labels and typed edge models. 

The main advantages of this approach are: i)  gaining a boost in performance without additional labeling costs when compared to the baseline, and ii) the model can effectively capture the multi-modal behavior of interacting agents while learning semantically meaningful interaction modes.

Uber further was researching training neural networks based on different techniques such as safe mutation based on output gradients (US10699195B2), learning methods using an evolutionary program (US20200151576A1), and training artificial neural networks using evolution-based strategies and novelty search  (WO2019116353A1). 

Machine Learning

Machine Learning is crucial for autonomous vehicles, as certain driving systems are improved by using ML-based models. Machine learning relies hugely on data to improve car function day by day. The deal is higher the data, the faster the improvement. And in terms of data, Uber was and is way behind its counterparts. 

Thus, researching ML was a top priority to get their hands on more data. However, like other companies, Uber hid the information about how the system works in its self-driving cars until March 2020. 

On March 4, 2020, Uber revealed the machine learning technology behind its self-driving cars. Machine learning models were an essential part of Uber’s self-driving cars as they handled tasks such as processing sensor input, identifying objects, and predicting where those objects might go. 

Uber created a five-step life cycle, from data ingestion to model serving, for the training and deployment of ML models in its self-driving vehicles (illustrated below for reference).

Source – Uber

Uber had also developed a tool named VerCd to provide versioning and continuous delivery of all machine learning code as well as artifacts for Uber ATG’s self-driving vehicle software. 

The ML model life cycle process and tools like VerCD helps researchers manage various complex ML models as well as iterate them in a faster manner. 

Patent Insights

Now while this information was released in 2020, the research on ML was ongoing for quite a long time. 

While Uber shared the technology behind its driverless car and its machine learning-based models in the last few months, a lot of information could have been figured if anyone looked into the ride-hailing company’s published research papers and patents. 

Most ML models predict different things for a car like detecting and prioritizing objects, coordinates, safety incidents, speed limit awareness (EP3655727A1), etc. While other models talk about space finding for parking (US20190094858A1), user awareness (US20180157984A1), AV management system (US10248121B2), etc. All these activities could have been figured if anyone kept track of its patenting activity. 

Side Note: Uber selling off its Self-driving business is an exceptional case and chances are your competitors might never leave the domain. Would you like to keep a track of their R&D activity? Just click below and we will walk you through our approach to competitive monitoring. Looking forward to talking to you !

LiDAR

LiDAR is a basic necessity for a self-driving car (at least for most of the autonomous vehicle companies that use LiDAR while Tesla chooses to use RADAR and Camera combination to compensate for the LiDAR technology). 

Uber has a history with LiDAR which resulted in Uber paying $244 million and an 18-month sentence for Anthony Levandowski for Waymo trade secrets theft.  But that doesn’t mean Uber hasn’t done any research on their side. 

Uber has been researching LiDAR technology, as suggested by its research publications and intellectual property data.

Patent Insights and Publications

Uber has filed multiple patent applications for LiDAR, some of which are granted as well. 

Besides patents, Uber’s LiDAR-related publications since 2018 also hint at its research in this area. In 2020 alone, Uber published 3 LiDAR-related publications which throw light on their research on real data simulation using LiDAR data and physical adversarial examples. 

One of their recent publications on LiDAR provides a realistic LIDAR simulation (LIDARsim) by leveraging the real data which Uber collects by driving around several cities with their self-driving vehicles. A scene will be selected from a large catalog of 3D static maps and 3D driving objects and a self-driving car will be placed there “virtually”. The company further says they developed a novel simulator that captures both the power of physics-based and learning-based simulation.

Uber had said that the LiDARsim sensor simulator has a very small domain gap that provides the ability to test more confidently the full autonomy stack. Uber uses perception algorithms’ ability to detect unknown objects in the scene with LiDARsim. The company also uses LiDARsim to better understand how the autonomy system performs under safety-critical scenarios in a closed-loop setting that would be difficult to test without realistic sensor simulation.

Some other inventions by the company are basic yet crucial such as recognizing and classifying objects, for which Uber is using LIDAR sensors as suggested by its patent (US10310087B2). 

Also, one of the few granted patents (US10677925B2) on LiDAR addresses the shortcomings of Fixed-beam LiDAR systems and provides solutions to overcome those shortcomings. 

In 2020, Uber’s two more patent applications were published on USPTO related to LIDAR, one of them (US20200041622A1) talks about systems and methods, and devices to find the LiDAR cross-talk and the mitigation process, while the other one (US20200033449A1) discusses measuring distance using LiDAR sensors. 

V2X Communication

The V2X market is still in its nascent stage but it is a crucial communication source for self-driving and it’s not a secret that every self-driving player is researching it. 

Patent Insights and Publications

V2X communication is not a big part of Uber’s portfolio but it does have few patents that discuss the technology in one way or another such as EP3679710A1, US20190222986A1, etc. It would be fair to say that Uber wasn’t heavily investing their research in this technology like the other technologies.

However, Uber knows that V2X is important, as it assists in the safety feature of a self-driving car. With its history of the infamous road accident, Uber wanted to make sure that it won’t repeat. 

Rather than patents, Uber has a few research publications on V2X communication. 

In one publication, Uber hinted at using V2V communication for joint perception and prediction. The company explored the use of vehicle-to-vehicle (V2V) communication to improve the perception and motion forecasting performance of self-driving vehicles. 

By intelligently aggregating the information received from multiple nearby vehicles, Uber can observe the same scene from different viewpoints. It allows a self-driving vehicle to see through occlusions and detect vehicles at a long range, where the observations are very sparse or non-existent.

Uber’s IP and RD efforts alone do not paint the entire picture of the company’s work in Self-driving. The company had also acquired few self-driving startups to get an edge over other players in the domain. Let’s have a look at these acquisitions. 

Uber Self-Driving Acquisitions

Uber’s self-driving research team got some help from its acquisitions. Uber made a total of 12 acquisitions in which 3 were directly related to self-driving solutions. 

Otto

Uber’s first and the most famous acquisition of Otto in 2016, was perhaps the biggest move by the company to expand its service beyond ride-hailing. Unfortunately, the acquisition of Otto was a stroke of bad luck for Uber as the company had to fight against Waymo regarding trade secret theft by Otto’s co-founders Anthony Levandowski. 

Uber even had to shut down the Otto project to focus more on self-driving cars and their main operation. 

Otto was researching hugely on LiDAR technology so even though Uber closed its operation, Otto’s technology could still be in use to improve Uber’s self-driving vehicles or rather say Aurora’s AVs. 

Geometric Intelligence

Uber acquired GI in December 2016 and had also built a dedicated AI lab for self-driving research. Geometric Intelligence pioneered machine learning through innovative, patent-pending techniques that learn more efficiently from fewer data. 

“It extends to teaching a self-driven machine to safely and autonomously navigate the world, whether a car on the roads or an aircraft through busy airspace or new types of robotic devices.”, 

wrote Jeff Holden, Uber’s chief product officer, in a blog post, announcing the creation of Uber AI Labs.

The acquisition helped Uber in AI more broadly, with a focus on basic research that’s likely to impact across a range of potential uses, including things like route management. 

Mighty AI

Uber acquired Mighty AI, a startup that develops training data for computer vision models, in June 2019.

“The team at Mighty AI has built technology to the label at scale using the latest AI and user experience techniques,” Jon Thomason, vice president of software engineering for Uber’s Advanced Technologies Group, said in a statement. 

Uber acquired Mighty AI’s intellectual property, tooling, tech talent, and labeling community, and a little more than 40 employees from Mighty AI joined Uber at its Seattle engineering office, operating as a team within Uber’s self-driving division. 

When things don’t go to plan – What was the company’s vision for self-driving?

Uber had planned to make self-driving ride-sharing a reality as soon as possible. 

It could have served as its gateway to profitability, but it would definitely be a huge challenge to achieve, despite having Aurora. 

Uber was planning to launch its self-driving cars in the cities where weather, demand, roads, and other conditions were most favorable. The newly devised unconventional strategy was designed to help Uber cut down costs and show investors a clear path to profitability.

“The vision for growth is absolutely there. But growth where it makes sense.” 

Dara Khosroshahi, Uber’s CEO.

The aforementioned strategy hinted that Uber didn’t want to solve every self-driving problem in every city and thus competing with the likes of Waymo, Tesla, and Lyft. Rather, it would launch its self-driving services to markets where it best suits in terms of technological feasibility, user safety, and cost-effectiveness.

As per Eric Meyhofer, the Head of Uber ATG, the company has to pass through three stages: developing, piloting, and commercialization. But as fate would have it,  Uber’s ATG unit was in the development stage by the time it sold its self-driving unit.

Things don’t go often as planned and Uber ATG being sold to Aurora way before its pilot is a good example of the same. 

Concluding Notes

It would be fair to say that Uber was still quite behind other self-driving players like Waymo, Baidu, and Tesla when it comes to data and other technologies such as LiDAR. 

Waymo’s vehicles have driven more than 20 million miles as per their safety report, and Tesla well over 2 billion miles. Baidu, on the other hand, as of December 2020 had driven 4.35 Million Miles with zero accidents and its growth poses a challenge for most of those working on the technology before Baidu.

By the time Uber had joined the race after the Arizona crash, the progress gap between Uber and other companies became huge. Thus, the financial struggle was inevitable which resulted in the company selling its self-driving unit.

However, this should bring some relief to the company as the sale of its self-driving research will reduce its expenditures hugely and the research will still get to live the day through Aurora. 

Now only time could tell how Uber’s strategy to outsource its self-driving research would work out in making its self-driving cab service a reality. Knowing Aurora, I personally feel they might pull off what Uber has been wished for. 

What’s the reasoning behind the strong intuition? I know a little more about the startups than you do. Want to level up? Read about Aurora along with other Top Autonomous Vehicle Startups here.

Authored by: Vipin Singh, Market Intelligence