Liquid AI Redesigns Neural Networks: Introducing Liquid Foundation Models

Liquid AI has introduced its first Liquid Foundation Models (LFMs) series. A new class of generative AI designed to combine high performance with efficiency. These models aim to push the boundaries of what AI can achieve at every scale, delivering top-tier results with smaller memory requirements and faster processing times.

What Makes LFMs Different?

Built from first principles, LFMs take inspiration from engineering disciplines like dynamical systems, signal processing, and numerical linear algebra. This design approach allows them to offer more control and better efficiency than traditional models. LFMs are versatile enough to handle different types of sequential data, whether it’s video, audio, text, time series, or signals.

Liquid AI’s focus is not just on large-scale models but on building adaptable systems that can maintain performance across different environments. Their name, “Liquid,” reflects the dynamic and adaptive nature of these models.

Key Models in the LFM Series

1. LFM-1.3B: Designed for resource-constrained environments with minimal memory usage.

2. LFM-3.1B: Optimized for edge deployment, allowing it to run efficiently on local devices.

3. LFM-40.3B Mixture of Experts (MoE): Built to handle complex tasks with a highly efficient architecture that balances workload dynamically.

These models aim to demonstrate that performance isn’t just about scale but also about innovation in design.

Performance and Availability

Liquid AI reports that the LFM-1B model has set a new benchmark in its category, outperforming other 1B models evaluated using Eleuther AI’s lm-evaluation-harness v0.4. This marks a significant achievement, as it’s the first time a non-GPT architecture has surpassed transformer-based models at this scale.

The LFMs are now available on several platforms:

• Liquid Playground
• Lambda (Chat UI and API)
• Perplexity Labs

Additionally, they will soon be accessible through Cerebras Inference. The models are being optimized for various hardware, including NVIDIA, AMD, Qualcomm, Cerebras, and Apple.

The LFM-3B model offers impressive performance for its size, setting a new benchmark among models with 3 billion parameters, including transformers, hybrids, and RNNs. It even surpasses many older models in the 7B and 13B parameter range. Additionally, LFM-3B performs at a comparable level to Phi-3.5-mini across several benchmarks, despite being 18.4% smaller. This makes it a top choice for mobile and edge text-based applications, where efficiency and performance are both crucial.

*Scores reported by the developers. All the other scores were calculated with the same evaluation harness we used for our own models.
 

The LFM-40B model strikes an optimal balance between size and output quality. It utilizes 12B active parameters during operation, delivering performance on par with larger models. Thanks to its Mixture of Experts (MoE) architecture, LFM-40B achieves higher throughput while remaining compatible with more affordable hardware, making it both powerful and cost-efficient for deployment.

*Scores reported by the developers. All the other scores were calculated with the same evaluation harness we used for our own models.
 

LFMs Prioritize Memory Efficiency

LFMs are designed to be more memory-efficient than traditional transformer models, especially when handling long inputs. In transformer-based LLMs, the KV cache expands linearly with the sequence length, consuming more memory. LFMs, however, use optimized input compression techniques, allowing them to process longer sequences without requiring additional hardware. This efficiency gives LFMs an edge over other models in the 3B parameter range by maintaining a smaller memory footprint, even for complex tasks.

Image Source: Liquid. (Total inference memory footprint of different language models vs. the input+generation length).

LFMs Make the Most of Their Context Length

A new standard is being set for models of this size with the introduction of 32k token LFMs in this preview release. Their ability to manage such long contexts ensures they perform optimally even with extended inputs. According to the RULER benchmark (Hsieh et al., 2024), a context length is considered effective if it achieves a score higher than 85.6, a threshold that LFMs meet with ease. The table below highlights how LFMs stack up against other models at various context lengths.

The efficient context window of LFMs makes it possible to perform long-context tasks on edge devices for the first time. This advancement opens doors for developers to build new applications, such as document analysis, text summarization, and more meaningful interactions with context-aware chatbots. It also enhances Retrieval-Augmented Generation (RAG) by improving the model's ability to handle extended input.

Looking ahead, the focus is on scaling LFMs further across model size, compute time, and context length. In addition to language-based LFMs, Liquid AI is working on models tailored for different data types, domains, and applications, with more releases planned in the coming months.

Advancing the Pareto Frontier of Large AI Models

Liquid AI has made significant strides in optimizing its pre-training and post-training processes, along with the infrastructure supporting its models. This effort focuses on five essential criteria to ensure the models excel:

1. Knowledge Capacity

This aspect highlights the range and depth of information that the models can handle across various domains and tasks, regardless of their size. Liquid AI achieves this by utilizing a diverse pre-training dataset and implementing advanced model architectures. They have also introduced new strategies for pre-training, mid-training, and post-training, allowing LFMs to compete effectively with larger models in knowledge-based tasks.

2. Multi-Step Reasoning

This means being able to break down a problem and think logically about it. Liquid AI has emphasized this type of thinking during important training stages, which helps improve the analytical abilities of their smaller model architectures. This way, even compact models can perform strong reasoning tasks effectively.

3. Long Context Recall

The maximum input size of a model isn't the same as how effectively it can recall information from that input. Liquid AI specifically trained LFMs to enhance their recall performance and in-context learning abilities throughout the entire range of input. This means they can better remember and use information from longer inputs.

 4. Inference Efficiency

Transformer-based models often use a lot of memory when handling long inputs, making them less suitable for use on edge devices. In contrast, LFMs maintain almost constant inference time and memory usage. This means that as the input length increases, it doesn't significantly slow down generation speed or require much more memory.

5. Training Efficiency

Training models similar to GPT typically require a lot of computing power. However, LFMs are designed to be efficient when training on long-context data, making the process more manageable and less resource-intensive.

Rethinking Model Architecture for Enhanced Performance

Drawing from extensive research on creating effective and efficient learning systems, Liquid AI has created a new design framework for foundation models that considers various modalities and hardware needs. The aim is to go beyond traditional Generative Pre-trained Transformers in developing these models.

With LFMs, Liquid AI is putting into action new principles and methods that their team has developed over the past few months to guide the design of these models.

LFMs Use Structured Operators

LFMs are made up of structured computational units, which are the core building blocks of the architecture within a new design framework. These Liquid systems and how they are put together help increase knowledge capacity and reasoning skills. They also make training more efficient, reduce memory costs during inference, and improve performance when working with different types of data, such as video, audio, text, time series, and signals.

LFMs Architecture: A Controlled Approach

The design of LFMs plays a key role in shaping their scaling, inference, alignment, and analysis strategies. This allows for a detailed examination of LFMs’ dynamics using established signal processing techniques, providing insights into their behavior from the outputs they generate to their internal operations.

LFMs Are Adaptive: A Foundation for AI at Every Scale

LFMs can automatically adjust their architectures to suit specific platforms, such as Apple, Qualcomm, Cerebras, and AMD. This adaptability allows them to meet various parameter requirements and optimize inference cache size, making them versatile for AI applications at any scale.

Defining the Liquid Design Space

Liquid's design space is shaped by the featurization and footprint of architectures and their core operators. Featurization involves transforming input data, such as text, audio, images, and video, into a structured set of features or vectors. These features help modulate computations within the model in an adaptive way. For instance, audio and time series data typically require less featurization due to their lower information density compared to language and multimodal data.

Another crucial aspect is the computational complexity of the operators. By effectively navigating and utilizing the design space of structured adaptive operators, we can enhance performance while maintaining controlled computational requirements.

At their foundation, LFMs consist of computational units that act as adaptive linear operators, with their actions shaped by the inputs they receive. The LFM design framework integrates various existing computational units in deep learning, offering a systematic method for exploring different architectures. Our analysis specifically enhances model building by focusing on three important areas:

1. Token-mixing structure – how the operator mixes embeddings within the input sequence.
2. Channel-mixing structure – how it combines different channel dimensions.
3. Featurization – which modulates computation based on the input context.

Image Source: Liquid.

Invitation to Early Adoption of LFMs

Liquid AI is currently in the early stages of developing its Liquid Foundation Models (LFMs) and invites collaboration to explore the strengths and weaknesses of these innovative systems.

Strengths of Language LFMs:

• General and Expert Knowledge: LFMs demonstrate a robust understanding of various topics, catering to both general inquiries and specialized knowledge.
• Mathematics and Logical Reasoning: These models are equipped to tackle mathematical problems and logical reasoning tasks effectively.
• Efficient Long-Context Processing: LFMs excel in handling tasks that require processing long contexts efficiently.
• Multilingual Capabilities: While primarily designed for English, they also support multiple languages, including Spanish, French, German, Chinese, Arabic, Japanese, and Korean.

Limitations of Language LFMs:

• Zero-Shot Code Tasks
• Precise Numerical Calculations
• Time-Sensitive Information
• Counting r's in the word "Strawberry"
• Limited Human Preference Optimization

Liquid AI is committed to an open-science approach, contributing to the advancement of the AI field by publishing findings and methodologies through scientific and technical reports. They plan to release relevant data and models generated from their research to the broader AI community. However, due to the significant time and resources invested in developing these architectures, they are not currently open-sourcing their models, allowing continued progress and a competitive edge in the AI landscape.

For enterprises seeking to explore the latest advancements in AI, engagement with Liquid AI is encouraged. The company is in the early stages of its journey, actively innovating in foundation model development and deployment. They welcome feedback and insights from enthusiastic users, inviting participation in efforts to enhance the capabilities of their models.