
Databricks Auto Loader for Incremental Delta Lake Ingestion

Key Takeaways
Databricks Auto Loader is the default choice for incremental file ingestion into Delta Lake, if teams plan Unity Catalog file events and automatic restarts early.
- The checkpoint records every processed file, so each file loads once and the duplicate rows that cron scripts create disappear.
- Databricks recommends file notification mode with managed file events for most workloads. Directory listing stays the default and suits proofs of concept, small volumes and Azure Blob Storage.
- Run managed file events streams at least once every seven days.
- Scheduled batches use trigger availableNow and continuous streams an explicit processingTime interval; only the trigger changes, and Trigger.Once is deprecated. After a gap longer than seven days, a file events stream falls back to a full directory listing.
- In the default addNewColumns mode, a new column stops the stream with an UnknownFieldException. The stream continues with the new schema after a restart, so production jobs need automatic retries.
- Supplying a full schema switches the default evolution mode to none, which can silently turn off automatic evolution. Schema hints on key columns keep it on.
- Each stream needs its own permanent checkpoint, because the checkpoint is the stream's identity.
Plan Unity Catalog file events, retries and a permanent checkpoint before the first production deployment.
Cron-driven ingestion scripts usually fail in two ways. They reload files that were already processed, which creates duplicate rows, and they break the moment an upstream system adds or renames a column. Databricks Auto Loader handles both. It picks up only new files from cloud storage, records each processed file in a checkpoint, evolves the table schema as columns appear and writes the results to Delta Lake tables.
Auto Loader is the right default for incremental file ingestion in 2026. There's a catch, though. Two things need planning early, namely the Unity Catalog setup that file events depend on and the automatic restarts that schema changes trigger, and this guide covers both along with detection modes, triggers, schema behavior and Lakeflow pipelines, using lessons from a production log pipeline.
What is Databricks Auto Loader?
In code you'll see it as cloudFiles. Databricks Auto Loader is a Structured Streaming source that incrementally processes new data files as they arrive in cloud object storage and writes them to Delta Lake, and on its first run it can also load the files already sitting in the input path. Databricks' "What is Auto Loader?" documentation (2026) says it scales to near real-time ingestion of millions of files per hour and can process billions of files to migrate or backfill a table.
Eight formats are supported: JSON, CSV, XML, Parquet, Avro, ORC, text and binary files. That covers most bucket contents. Files can load from Amazon S3, Azure Data Lake Storage (ADLS), Google Cloud Storage or Azure Blob Storage, though the legacy WASB driver for Blob Storage is deprecated and best left alone in new work.
State lives in the checkpoint. Inside the checkpoint location, a RocksDB key-value store holds metadata about every discovered file, which is how Auto Loader processes each file exactly once and resumes from the right place after a failure. Nobody has to maintain a list of loaded files.

How Auto Loader Moves Files into Delta Lake
Every trigger runs the same five steps, whether the stream runs continuously or as a scheduled batch.
- First it finds new files in the input path, either by listing the directory or by reading the cloud provider's file notifications.
- Each file is checked against the checkpoint, and anything already recorded is skipped.
- The files are read with the schema stored in the schema location. On the first run that schema is inferred, and new columns are added as they appear.
- Only files that arrived since the last trigger get processed. Historical data is never reloaded.
- Rows land in a Delta Lake table, which adds ACID transactions and schema enforcement.

Why Replace Custom Ingestion Scripts with Auto Loader?
Custom ingestion scripts pile up bookkeeping code. A typical script has to find new files, skip files it already loaded, cope with schema changes and record which file each row came from. Then every new source or schema change brings its own edge cases, and another failure that someone has to fix by hand. Auto Loader replaces that code.
The fit is strongest when:
- Files arrive at irregular intervals, some every 30 seconds and others once a day.
- Semi-structured sources such as JSON gain new fields over time.
- File counts run into thousands per day, spread across nested folders, or a large historical backlog needs loading.
- Auditors or downstream teams need every row traced to its source file.
- The team wants to stop maintaining ingestion cron jobs and error-handling wrappers.
For a one-time load of a few files, a streaming source buys little. Our view is that a plain batch read is simpler there.

Which Auto Loader File Detection Mode Should You Use?
Databricks recommends file notification mode with file events for most workloads and advises teams already on directory listing to migrate, even though directory listing is still the default because it needs no permissions beyond access to the data. Listing is still the quickest way to start.
Earlier advice, including previous versions of this guide, was to start with directory listing and switch once volume grew. That advice has aged. With file events enabled on a Unity Catalog external location, the Auto Loader stream itself needs no additional permissions, which removes most of the reason to wait.
| Detection mode | How it finds new files | Setup needed | Main caveat | Best for |
|---|---|---|---|---|
| Directory listing | Lists the input path on each trigger | Access to the storage path | Less scalable than file events | Proofs of concept, small volumes, Azure Blob Storage |
| Managed file events | Reads a cache of storage events that Databricks keeps for the external location | Unity Catalog external location with file events enabled, Databricks Runtime 14.3 LTS or above | Stream should run at least once every seven days | Most production workloads |
| Classic file notifications | Auto Loader creates its own notification and queue resources | Elevated cloud permissions | Resources are not torn down automatically | Existing streams not yet moved to file events |
Directory listing mode
Zero extra setup is the appeal here. Auto Loader lists the input path to find files it hasn't seen and checks them against the checkpoint, which fits proofs of concept, low file volumes and sources in Azure Blob Storage, where file events are unsupported. Scale is where listing gives ground. Databricks describes file notification mode with file events as more performant and scalable than listing.
Managed file events
This is the mode Databricks recommends. A Databricks service listens to the cloud provider's storage events for a Unity Catalog external location and caches the file metadata, and Auto Loader reads that cache when the stream sets cloudFiles.useManagedFileEvents to true. Production streams on S3, ADLS or Google Cloud Storage in a Unity Catalog workspace, on Databricks Runtime 14.3 LTS or above, are where it fits best.
The main caveat is timing. Databricks' Auto Loader file notification documentation (2026) recommends running file-events streams at least once every seven days; after a longer gap the stored read position becomes invalid and Auto Loader falls back to a full directory listing to resync. A monthly availableNow job on file events would pay for that full listing on every run, so weekly is the practical minimum schedule.
.option("cloudFiles.useManagedFileEvents", "true")Classic file notification mode
Setting cloudFiles.useNotifications to true on a stream makes Auto Loader create its own event subscriptions and queues in the cloud account. It suits existing streams that haven't moved to file events yet. Like file events, it avoids repeated listing and lets Auto Loader ingest millions of files an hour.
The catch is housekeeping. Creating those resources automatically requires elevated cloud permissions, and Auto Loader doesn't tear them down when a stream is retired, so someone has to track the queues and subscriptions and delete them by hand.

Should Auto Loader Run as a Stream or an availableNow Batch?
Freshness decides it. When downstream users need new data within minutes, run a continuous stream; for files that land every few hours, or tables where a daily refresh is enough, an availableNow batch does the job. Both use the same spark.readStream code, and only the trigger changes.
Continuous streaming load
df = (spark.readStream
.format("cloudFiles")
.option("cloudFiles.format", "json")
.option("cloudFiles.schemaLocation", "<path-to-schema-location>")
.option("cloudFiles.useManagedFileEvents", "true")
.load("<path-to-source-data>")
.selectExpr("*", "_metadata.file_path AS source_file"))
(df.writeStream
.option("checkpointLocation", "<path-to-checkpoint>")
.trigger(processingTime="1 minute")
.toTable("<target-delta-table>"))The schema location stores the inferred schema and its history. The checkpoint location records processed files and lets a restarted stream continue from there, while the source_file column carries each row's file path into the Delta table. No file events on the source path? Drop the useManagedFileEvents line.
The trigger line matters more than it looks. Without one, Structured Streaming defaults to a processingTime trigger, so the interval is worth setting explicitly to control cost and latency. One minute is only a starting point to tune.
Scheduled availableNow batch
(df.writeStream
.option("checkpointLocation", "<path-to-checkpoint>")
.trigger(availableNow=True)
.toTable("<target-delta-table>"))An availableNow run processes every file that arrived since the last run and then stops. Databricks' "Configure Structured Streaming trigger intervals" documentation (2026) states that Trigger.Once is deprecated in Databricks Runtime 11.3 LTS and above, and Databricks recommends Trigger.AvailableNow for incremental batch work.
A batch like this runs as a task in a scheduled Lakeflow Jobs pipeline, the scheduler formerly called Databricks Workflows. For files that arrive unpredictably, a file arrival trigger is cheaper than a fixed schedule because the cluster only runs when there are new files to process.
How Does Auto Loader Work in Lakeflow Pipelines?
Less bookkeeping falls on the team here. Auto Loader becomes the source of a streaming table, Databricks manages the schema location and other checkpoint information automatically, and Databricks recommends this setup for incremental ingestion. Lakeflow pipelines is the updated name for the product formerly known as Delta Live Tables (DLT). Existing DLT code keeps working with no migration.
New Python code imports the pipelines module from pyspark in place of import dlt:
from pyspark import pipelines as dp
@dp.table
def raw_logs():
return (spark.readStream
.format("cloudFiles")
.option("cloudFiles.format", "json")
.load("<path-to-source-data>")
.selectExpr("*", "_metadata.file_path AS source_file"))SQL pipelines get the same behavior with CREATE OR REFRESH STREAMING TABLE and STREAM read_files(...). Managed checkpoints are, in our view, the main reason to build new ingestion work as a pipeline, because the team has one fewer storage path to protect.
How Does Auto Loader Schema Evolution Behave in Production?
This part catches teams out. In the default addNewColumns mode, a new column stops the stream with an UnknownFieldException. Auto Loader adds the column to the stored schema before the stream fails, and after a restart the stream carries on with the new schema, which is why production streams need a job that retries automatically on failure.
Inference happens when the stream is first defined. Auto Loader samples up to 50 GB or 1,000 files, whichever limit is crossed first, and stores the result under cloudFiles.schemaLocation, and JSON and CSV columns come back as strings unless cloudFiles.inferColumnTypes is true. Schema hints pin the types of specific columns using SQL schema syntax. They can also add a column that isn't in the data yet.
There are five schema evolution modes:
- addNewColumns, the default when no schema is supplied, adds new columns and then stops the stream.
- addNewColumnsWithTypeWidening does the same and also widens supported types, such as int to long. It is in Public Preview on Databricks Runtime 16.4 and above.
- rescue never evolves the schema and never fails on a schema change. Unexpected values go to the rescued data column.
- failOnNewColumns stops the stream until someone updates the schema or removes the offending file.
- none ignores new columns and rescues nothing unless rescuedDataColumn is set.
Supplying a full schema switches the default mode to none, and addNewColumns isn't allowed with a provided schema. That's a trap. Teams that pass a full schema just to be safe can lose automatic evolution without noticing, so for drifting sources the safer move is schema hints on the few columns whose types actually matter, because hints leave evolution switched on.
What Are the Trade-offs of Databricks Auto Loader?
Most ingestion bookkeeping goes away, but responsibility moves into a handful of settings the team has to understand. The checkpoint and the schema location cause most of the trouble.
Where Auto Loader saves work
- The checkpoint tracks every discovered file, which removes the need for a custom state table.
- Large file counts and backfills of billions of files need no custom code.
- New columns reach the table through schema evolution without manual DDL.
- The _metadata column gives each row its file path, name, size and modification time.
- On Databricks Runtime 18.0 and above, cloudFiles.maxFilesPerTrigger is configured automatically and no longer needs manual tuning.
What Auto Loader asks of the team
- The checkpoint is the stream's identity. Changing the checkpoint location abandons the old stream and starts a new one.
- A file overwritten in place is processed once by default, and Auto Loader cannot guarantee which intermediate version it read. Setting cloudFiles.allowOverwrites to true processes the latest version.
- input_file_name() is deprecated from Databricks Runtime 13.3 LTS, unavailable on Unity Catalog and no longer supported on Databricks Runtime 17.3 LTS and above or in Databricks SQL. Pipelines that still call it need to switch to the _metadata column before upgrading.
- File events need Unity Catalog objects, and classic notifications need elevated cloud permissions.
- Engineers used to batch scripts may find streaming failures harder to debug, because investigation means reading stream logs and the Spark UI.
- Source files stay in place by default. On Databricks Runtime 16.4 and above, cloudFiles.cleanSource can delete or move processed files after a retention period that defaults to 30 days.
Ingesting Nested, Drifting Log Files with Auto Loader
The log pipeline comes from an earlier production project that used Auto Loader to ingest system logs from hundreds of remote sources into Delta Lake. Several teams built business metrics on those logs, and the metrics had to be accurate and current.

What the log pipeline had to handle
- Files sat in nested folders four or more levels deep, and every level had to be scanned.
- File names were not standardized, and naming conventions sometimes changed without notice.
- Columns were added, removed or renamed without warning.
- No duplicate rows were allowed.
- Data had to land on the day it was generated, sometimes several times a day.
- Every row had to be traceable to its original file for audits.
How Auto Loader handled nested folders and schema drift
- A wildcard prefix such as /*/*/*/*/ in the load path made Auto Loader scan every folder level. Prefix glob patterns go in the path itself, while suffix patterns such as a file extension go in the pathGlobFilter option.
- Every row carried the path of the file it came from. The project used input_file_name(), which was common at the time; a new build would use _metadata.file_path. The column was used constantly during audits and debugging.
- A filter processed only files with data from the current date, which kept stale and duplicate data out.
- Schema changes went through Auto Loader's schema evolution. When a new column appeared, the stream stopped with an UnknownFieldException and the job's retry setting restarted it with the updated schema.
- Output went to Delta tables with checkpoints in a dedicated path. The job ran as a scheduled batch with trigger(once=True), and the current equivalent is trigger(availableNow=True).
The results matched the requirements. Logs reached Delta Lake on the day they were generated, checkpointing kept duplicate rows out of the tables and every row could be traced to its source file. The pipeline handled thousands of files per day without a custom cron script, and other teams got a deduplicated source for their log-based metrics even as sources and schemas kept changing.

Auto Loader Best Practices for Production Pipelines
These practices come from the Databricks documentation and the log pipeline in this article.
- Set cloudFiles.schemaLocation on every stream outside Lakeflow pipelines, and add schema hints for columns whose types must not change.
- Use managed file events for new production streams on S3, ADLS or Google Cloud Storage, and run them at least once every seven days.
- Use trigger(availableNow=True) for scheduled batches and an explicit processingTime interval for continuous streams.
- Add _metadata.file_path as a column from the first deployment.
- Run streams in a job with automatic retries, because a new column stops the stream by design.
- Give each stream its own checkpoint location and treat it as permanent.
- Filter unwanted records early in the read, and plan source file retention with cloudFiles.cleanSource.
Frequently asked questions
Does existing Delta Live Tables code still run in Lakeflow pipelines?
No migration is required. Lakeflow pipelines is the updated name for Delta Live Tables, code that uses import dlt keeps working, and the import can be replaced with from pyspark import pipelines as dp.
Can Auto Loader load files that are already in the bucket?
Yes. The cloudFiles.includeExistingFiles option defaults to true, so the first run loads everything already in the path along with new arrivals. Changing it later has no effect on a running stream, because the option is evaluated only when a stream starts for the first time and is ignored on every restart after that.
Can Auto Loader read from Unity Catalog volumes?
Yes. Auto Loader accepts Unity Catalog volume paths that start with /Volumes/, alongside S3, ADLS, Google Cloud Storage and Azure Blob Storage paths, and the same cloudFiles options, checkpoints and schema location settings apply to volume paths too.
What happens to a value with the wrong data type?
Nothing is lost if a rescued data column is configured. Auto Loader keeps the row and moves the mismatched value into that column, which holds a JSON blob with the rescued columns and the source file path of the record. The rest of the row stays usable while the team inspects those values later.
Getting Help with a Databricks Auto Loader Pipeline
For any pipeline where files land incrementally in cloud storage and need deduplicated, schema-aware loading into Delta Lake, Auto Loader should be the starting point in 2026. Managed file events make sense where Unity Catalog is in place, and directory listing where it isn't. For scheduled loads pick availableNow, and for near real-time data a continuous stream.
The main qualification is operational. Every production stream needs a permanent checkpoint, a job that restarts after schema changes and, with file events, a run at least once a week. If your team is designing a new ingestion pipeline or replacing custom scripts, talk to Arbisoft's Databricks consulting team.
...Loading Related Blogs









