Making Learning Abundant
Learning is the only scalable way to achieve goals in a large and ever-changing world. Currently, there are two key inefficiencies that prevent learning from being widespread and continual.
First, our learning algorithms are too computationally expensive; to learn complex tasks, they require computational resources that are only available to a handful of organizations. As a result, these systems only learn tasks whose value is apparent to these organizations.
Second, our algorithms lack a mechanism to learn from raw experience. Instead, they outsource the bulk of the learning to humans and require access to highly curated datasets to generalize well.
To teach a new task to an existing AI system, humans first learn to perform that task. The human data is then recorded in diverse settings. The knowledge in the dataset is distilled into the AI system using supervised learning. It’s only after the system has learned to mimic humans that it learns on its own.
Even when the system does learn from its own experience, it does so in simulated environments instead of the real world. The inaccuracies in the simulations prevent the use of powerful learning methods—optimizing too much in imperfect simulations can degrade the performance in the real world.
Creating simulations is also an expensive endeavor that requires task-specific R&D. Often, the first version of the simulation is inadequate, and getting to a simulation that is useful requires multiple iterations.
The consequence of using this iterative and human-in-the-loop process is that the cost of learning a new task is high.
When the task to learn is valuable enough, such as coding, then these costly dependencies are not a deal breaker; but if an individual task is not particularly valuable—perhaps it is performed infrequently, or its value is temporary—then there are no economic incentives to teach it to the system. The bad economics of learning uncommon tasks is likely the reason that the frontier AI systems have jagged intelligence—only tasks that are common enough and economically valuable enough are worth learning well. A cartoonish depiction of the subset of tasks that can be automated with the current paradigm is shown in Figure 1.
If we lived in a world in which a handful of tasks produced the bulk of the value, then the current paradigm would be sufficient and making learning more efficient would have diminishing returns. But if we lived in a world in which the set of useful tasks had a long tail, as implied by the big world hypothesis, then the current paradigm would leave a lot of value on the table (see Figure 2 for a cartoonish depiction).
To unlock the value in the long tail, we have to make learning abundant by developing learning algorithms that are orders of magnitude more efficient and that can learn autonomously from raw experience without relying on humans. These algorithms would make it possible to have hundreds of millions of unique agents that continually learn in their unique environments, understand the preferences of their users, and strive to achieve their individual goals.
A glimpse at Oak Lab’s approach to computationally efficient learning from experience
To make learning more computationally efficient, we are developing algorithms that learn in real time, in a sample-efficient way, without ever storing or replaying data. Additionally, they learn sparse distributed representations, which can be implemented as event-driven networks. By combining the ideas of batch-size-one learning and event-driven networks, our systems can potentially learn using multiple orders of magnitude less computation than current algorithms do.
To make learning more autonomous, we are developing algorithms that learn to assign credit, as opposed to assigning credit for every error to every parameter with a non-zero gradient. By learning to assign credit, our algorithms decide which associations in the data stream are worth learning, and which associations are temporary artifacts that do not generalize well to the future. Smarter credit assignment makes it possible to learn from a raw sensory stream without the need for curating datasets.
If our mission and approach to making learning abundant appeal to you, consider joining us.