Evolving Fully Automated Machine Learning via Life-Long Knowledge Anchors
Abstract : Automated Machine Learning (AutoML) has achieved remarkable progress on various tasks, which is attributed to its minimal involvement of manual feature and model designs. However, existing AutoML pipelines only touch parts of the full machine learning pipeline, e.g., Neural Architecture Search or optimizer selection.This leaves potentially important components such as data cleaning and model ensemble out of the optimization, and still results in considerable human involvement and suboptimal performance.The main challenges lie in the huge search space assembling all possibilities over all components, as well as the generalization ability over different tasks like image, text, and tabular etc.In this paper, we present a rst-of-its-kind fully AutoML pipeline, to comprehensively automate data preprocessing, feature engineering, model generation/selection/training and ensemble for an arbitrary dataset and evaluation metric.Our innovation lies in the comprehensive scope of a learning pipeline, with a novel life-long knowledge anchor design to fundamentally accelerate the search over the full search space.Such knowledge anchors record detailed information of pipelines and integrates them with an evolutionary algorithm for joint optimization across components. Experiments demonstrate that the result pipeline achieves state-of-the-art performance on multiple datasets and modalities.
? We use a uniformly random choice among the following three transformations:
? (i) add or remove an instruction; instructions are added at a random position and have a random op and random arguments; to prevent programs from growing unnecessarily, instruction removal is twice as likely as addition;
? (ii) completely randomize all instructions in a component function by randomizing all their ops and arguments; or
? (iii) modify a randomly chosen argument of a randomly selected existing instruction
? We discuss the formal definition of AutoML.The definition is not only general enough to include all existing AutoML problems, but also specific enough to clarify what is the goal of AutoML. Such definition is helpful for setting future research target in the AutoML area.
? We propose a general framework for existing AutoML approaches. This framework is not only helpful for setting up taxonomies of existing works, but also gives insights of the problems existing approaches want to solve. Such framework can act as a guidance for developing new approaches.
? We suggest four promising future research directions in the field of AutoML in terms of the problem setting, techniques, applications and theory. For each, we provide a thorough analysis of its disadvantages in the current work and propose future research directions.
? In this paper, we proposed an ambitious goal for AutoML: the automatic discovery of whole ML algorithms from basic operations with minimal restrictions on form.
? The objective was to reduce human bias in the search space, in the hope that this will eventually lead to new ML concepts.
? As a start, we demonstrated the potential of this research direction by constructing a novel framework that represents an ML algorithm as a computer program comprised of three component functions (Setup, Predict, Learn).
? Starting from empty component functions and using only basic mathematical operations, we evolved neural networks, gradient descent, multiplicative interactions, weight averaging, normalized gradients, and the like.
? We use the term configuration to denote all factors but the model parameters x (which are usually obtained from model training) that influence the performance of a learning tool. Examples of configurations are, the hypothesis class of a model, the features utilized by the model, hyper-parameters that control the training procedure, and the architecture of a neural network.
? When original features are not informative enough, we may want to construct more features to enhance the learning performance.
? Basically, the policy in RL acts as the optimizer, and its actual performance in the environment is measured by the evaluator. However, unlike previous methods, the feedbacks (i.e., reward and state) do not need to be immediately returned once an action is taken. They can be returned after performing a sequence of actions.
|