Researchers from Apple and EPFL Introduce the Boolformer Model: The First Transformer Architecture Trained to Perform End-to-End Symbolic Regression of Boolean Functions

The optimism that deep neural networks, particularly those based on the Transformer design, will speed up scientific discovery stems from their contributions to previously intractable problems in computer vision and language modeling. However, they still need help to handle more complex logical problems. The combinatorial structure of the input space in these tasks makes it more difficult to collect representative data than in typical vision or language tests. As a result, the deep learning community has focused heavily on reasoning tasks, including both explicit reasoning in the logical domain (such as arithmetic and algebra tasks, algorithmic CLRS tasks, or LEGO) and implicit reasoning in other modalities (such as Pointer Value Retrieval and Clevr for vision models, or LogiQA and GSM8K for language models). Tasks that can be solved with Boolean modeling rely heavily on reasoning, especially in biology and medicine. Since these efforts continue to be difficult for standard Transformer structures, it is only natural to investigate whether they may be managed more efficiently with alternative methods, such as making better use of the Boolean nature of the task.

A research team from Apple and EPFL introduces the Boolformer model that offers a ground-breaking approach to problems in symbolic logic. It’s the first machine-learning method to infer condensed Boolean formulas only from input-output samples. To emphasize, Boolformer is demonstrated to generalize consistently to functions and data that are more sophisticated than those encountered during training. This defining feature of advanced comprehension has so far eluded other state-of-the-art models.

You might think of a Boolean formula as a symbolic statement of the Boolean function in terms of the three basic logical gates (AND, OR, and NOT), and that’s exactly what the Boolformer is supposed to do. This problem is formulated as a sequence prediction problem, with synthetically created functions serving as training examples and their truth tables providing input for the work. One can gain generalizability and interpretability by switching to this setting and gaining control over the data production process. Researchers from Apple and EPFL demonstrate the method’s startling effectiveness on a range of logical problems in both theoretical and practical contexts, and they explain the path forward for further development and use cases. 

Contributions

• Researchers show that the Boolformer can predict a compact formula when given the whole truth table of an unseen function by training on synthetic datasets for symbolic regression of Boolean formulas.
• By supplying false truth tables with flipped bits and irrelevant variables, they demonstrate that Boolformer can handle noisy and missing data. 
• They test Boolformer on several binary classification tasks pulled from the PMLB database and find that it produces competitive results against traditional machine learning methods like Random Forests while still allowing for interpretation.
• They use Boolformer to model gene regulatory networks (GRNs), a well-studied topic in biology. They also use a recently released benchmark to demonstrate the model’s ability to compete with state-of-the-art approaches while providing inference times that are many times faster.

Visit https://github.com/sdascoli/boolformer to get the code and models. The boolformer pip package makes installation and uses a breeze.

Learned formulae reveal the model’s inner workings in full detail, allowing for interpretation. This is a huge improvement as opposed to traditional neural networks, which are notoriously opaque. Safe AI deployment will depend on the system’s interpretability. Experiments show that when applied to real-world binary classification situations, Boolformer’s predicted accuracy is on par with or even better than traditional machine learning methods like random forests and logistic regression. Nonetheless, Boolformer, in contrast to these methods, also offers clear and convincing justifications for its forecasts.

Constraints that point to new areas for research

• The model’s effectiveness on high-dimensional functions and big datasets is constrained by the quadratic cost of self-attention, which caps the number of input points at one thousand.
• The model’s capacity to anticipate compact formulas and express complex procedures like parity functions is constrained by the fact that the XOR gate is not explicitly included in the logical tasks on which it is trained. This restriction exists because the expression simplification step in the generation process requires rewriting the XOR gate in terms of AND, OR, and NOT. Adapting the production of simplified formulas comprising XOR gates and operators with higher parity is left as a future effort by the research team.
• Additionally, the model only handles single-output functions, with multi-output functions being predicted independently component-wise, and (ii) gates with a fan-out of one, limiting the simplicity of the projected formulas.

In conclusion, the Boolformer is a major step in making machine learning more accessible, logical, and scientific. Its combination of high performance, solid generalization, and clear reasoning indicates a shift in artificial intelligence toward more reliable and helpful systems.

Check out the Paper and Github. All Credit For This Research Goes To the Researchers on This Project. Also, don’t forget to join our 30k+ ML SubReddit, 40k+ Facebook Community, Discord Channel, and Email Newsletter, where we share the latest AI research news, cool AI projects, and more.

If you like our work, you will love our newsletter..