Lakeflow Jobs: How We Automated Our Employee Satisfaction Analytics

Iqra SarwarLast Updated on October 27, 2026

10-11 Min Read TimeAdd as preferred on Google

Key Takeaways

Lakeflow Jobs automated one team's employee satisfaction reporting, giving HR daily updates and cutting compute costs 42% in internal measurements, though pipeline design and several limitations needed care.

• Manual reporting left HR with data that was T+3 days at best and cost the team 2 hours every Monday, which prompted the move to Lakeflow Jobs.
• The pipeline runs ingestion, silver and gold tasks in dependency order using depends_on, and adapts to daily pulse, quarterly and ad-hoc survey data without separate workflows.
• Parameters such as survey_type, date_range and department_filter let HR run specific reports through "Run Now" with no code changes.
• Giving each task its own compute matches resources to workload: ingestion is light, while the silver and gold tasks need more memory and compute.
• After 4 months, the team measured 42% lower compute costs and 75% faster issue detection. These are internal figures that the authors say other teams should not expect to match.
• Lessons and limits: one giant job was hard to debug, a 10-minute health rule caused false Monday alarms, and parameter validation and dev-to-prod promotion remain manual.

Start with one simple workflow for the most frequent reporting need, add parameters, then expand gradually.

We used to run reports for our HR team by hand. Every Monday morning, our team would waste 2 hours pulling employee satisfaction data, cleaning it up, and generating dashboards that were already outdated by the time anyone looked at them. HR would get impatient, and we would start the week already behind.

This post walks through how we automated employee satisfaction reporting with Lakeflow Jobs: the pipeline design, parameters, per-task compute, alerting, the results we measured and the limitations we still hit. It is written for data engineers and data leads who are considering Lakeflow Jobs for orchestration.

The Breaking Point

The final straw came when our CEO asked for ad-hoc reports during an all-hands meeting. There I was, frantically querying databases while everyone waited. I had to admit that our "real-time" employee satisfaction tracking was actually T+3 days at best. That embarrassing moment finally pushed us to properly implement Lakeflow Jobs.

What Exactly Is Lakeflow Jobs?

Lakeflow Jobs is an orchestration layer that lets you automate and schedule data pipelines, and it runs inside the Databricks Lakehouse architecture. Databricks' December 2025 blog post "From Airflow to Lakeflow: Data-First Orchestration" describes Lakeflow Jobs as Databricks' native orchestrator, formerly known as Databricks Workflows. We built this pipeline before the rename, and the features described here are the same in Lakeflow Jobs. Strip away the marketing talk, and it comes down to:

• A task scheduler that can run notebooks, SQL, and Python jobs
• A dependency manager that ensures tasks run in the correct order
• A resource optimizer that starts and stops compute as needed
• A monitoring system that alerts you when things break

What I appreciate most is how it handles compute resources. Before, our clusters kept running even when no jobs needed them. With Lakeflow Jobs, we use job clusters that spin up just for the task and terminate when done. Our finance team noticed the savings immediately.

Our Employee Satisfaction Data Pipeline

Our employee mood tracking system has three main data sources:

1. A daily "mood pulse" where employees record how they're feeling based on different questions (1-5 scale)
2. Quarterly satisfaction surveys with detailed feedback
3. Ad-hoc surveys around specific initiatives or events

Previously, I manually combined these sources into reports. Now, our Databricks Workflow combines them automatically, the kind of pipeline our Databricks data engineering services team builds for clients.

Setting Up the Pipeline

Here's how I built our employee satisfaction workflow:

1. Created a new workflow called "satisfaction_survey_main"
2. Added the first task to ingest data from various sources.
   This task extracts raw employee mood pulse data by consolidating sources, including historical survey records and Workstream database entries. It captures key fields such as timestamps, employee IDs, departments, and mood scores.
   • The historical survey data was fetched once during initial ingestion.
   • Ongoing ingestion continues from the Workstream database.
   • Runs on our existing cluster (we didn't need a job cluster for this small task)
3. Created a silver-level processing task that depends on the ingestion task
   • This task cleans and standardizes the mood data
   • Maps employee IDs to departments and teams
   • Handles inconsistencies (like when someone accidentally submits multiple mood responses)
   • Augments with metadata (like company events or holidays that might affect mood)
4. Added a gold-level task that takes the silver data and creates analytics-ready datasets
   • Calculates aggregations like average mood by department, team, and date
   • Identifies trend changes and statistical anomalies
   • Joins with ad-hoc survey results when available
   • Produces the final tables that power our dashboards

The interesting part was connecting these jobs together. In the workflow definition, we use a "depends_on" field to specify that the silver task can only run after the ingestion completes, and the gold task only runs after the silver task finishes.

Managing Different Data Sources

One challenge was handling our different data frequencies:

• Mood pulse data comes in daily
• Quarterly surveys arrive, well, quarterly
• Ad-hoc surveys appear unpredictably

Rather than creating separate workflows, we built intelligence into our tasks:

1. The ingestion task always runs and gets the latest mood pulse data
2. It checks for new survey data (both quarterly and ad-hoc)
3. If found, it processes those too; if not, it just processes the mood data
4. The silver and gold tasks adapt their processing based on what data came in.

This approach means our workflow runs the same way every day, but handles different incoming data appropriately.

Parameters for Flexibility

After running the workflow for a few weeks, our HR team started asking for specific reports. Instead of creating multiple workflows, I added parameters:

• survey_type: toggles between pulse, annual, or exit surveys
• date_range: lets us run historical analyses for specific time periods
• department_filter: allows reports focused on specific parts of the company
• sentiment_analysis: enables NLP processing on free-text comments
• anonymity_threshold: the minimum number of responses a group needs before its results appear in a report, so small teams cannot be singled out

Now, when HR needs something specific, they just click "Run Now" and change the parameters. No code changes needed, no manual data pulls. In my notebooks, I access the parameters like this: dbutils.widgets.get("date_range").

Handling Different Processing Needs

One technical challenge was that different parts of our pipeline needed different resources:

1. Data ingestion is lightweight and runs fine on our shared cluster
2. The silver task needs more memory for complex joins and data cleaning
3. Our gold task needs heavy compute for statistical anomaly detection

In Lakeflow Jobs, each task gets its own compute, so we specified a different cluster for each task in our workflow configuration. The silver and gold tasks run as "run_job_task" types. Databricks' Run Job task documentation (as of October 2026) says this task triggers other jobs configured in the workspace, and our silver and gold tasks use it to call separate jobs that carry their own high-memory or high-compute cluster configurations. Databricks does not support circular Run Job dependencies or nesting more than three Run Job tasks, which matters if you split workflows into linked jobs the way we recommend later in this post.

We also matched compute to each job to control cost. Ingestion runs on our existing shared cluster, and serverless compute is the option we recommend for lightweight tasks like it. Databricks' serverless compute for workflows documentation says jobs run without configuring and deploying infrastructure. Startup time depends on the performance mode: standard mode uses less compute to reduce cost but can have a startup latency of 4 to 6 minutes, and Performance optimized mode starts and runs faster. The cost impact of our compute choices is covered in The Real Results below.

Daily Alerting for HR

Beyond scheduled reports, we added alerting to our data observability strategy so the pipeline detects concerning patterns in employee sentiment, such as sudden drops in mood scores:

1. Created a monitoring task that runs after the gold task
2. It looks for:
   • Teams with declining satisfaction trends over two weeks
   • Departments with scores 20% below the company average
   • Sudden drops in individual employee mood scores
3. When triggers are detected, the workflow:
   • Creates an alert in our HR dashboard
   • Sends notifications to relevant managers
   • Generates a preliminary analysis suggesting potential factors

In our experience, the biggest gain was speed. Because the monitoring task runs right after each daily gold run, HR sees an alert within hours of the data arriving instead of waiting for the Monday report.

Technical Setup Details

For those interested in the actual technical implementation, our workflow configuration uses:

• Parameter passing between tasks to maintain context
• Bundle deployment for version control of our workflow (Databricks documentation now calls these Declarative Automation Bundles)
• Health rules that alert us if the job runs longer than 30 minutes
• Queue management to handle backlogged executions properly

The configuration specifies notification settings, health metrics, cluster IDs, notebook paths, job dependencies, and parameters all in a structured format that can be version controlled.

Monitoring Our Workflow

The basic Databricks monitoring worked well enough, but we wanted more specific insights. We added:

1. Custom metrics tracking:
   • Number of employees reporting daily (participation rate)
   • Processing time for each task
   • Data quality scores (completeness, consistency)
2. A monitoring notebook that captures operational metrics:
   • It pulls run data from the Databricks Jobs API
   • Records duration, status, and row counts for each run
   • Identifies trends in processing time or failure rates
3. A separate dashboard specifically for data pipeline health

This has helped with capacity planning. For example, we noticed our Monday processing took longer because weekend data arrived in batches. We adjusted our cluster sizing accordingly.

The Real Results

These are our own internal measurements after 4 months of running the automated workflow:

1. HR gets daily updates instead of weekly ones
2. Our compute costs dropped by 42%, based on our comparison of the pipeline's compute spend on job clusters against the always-on clusters we used before
3. We detect satisfaction issues 75% faster than before, measured as the time from a drop in mood scores to HR seeing it under the daily run plus alerts, compared with the old weekly Monday report cycle
4. Department managers are more engaged with the data (because it's current)

These figures come from our data volumes and cluster setup, so other teams will see different numbers. When our CEO now asks for employee satisfaction stats during meetings, we just point to the dashboard that refreshes daily instead of frantically querying data.

Pain Points & Lessons Learned

To be honest, it wasn't all smooth sailing. Here are some hard-won lessons:

1. Start with simple flows: My first version tried to process everything in one massive job. Debugging was a nightmare. Break complex workflows into multiple linked workflows.
2. Watch your cluster configs: Our initial runs would sometimes fail because:
   • Memory settings were too low for our data volume
   • Autoscaling wasn't configured properly
   • Driver node would run out of memory while the workers sat idle
3. Be careful with delta tables: We initially had locking issues when multiple workflows tried to write to the same delta tables. Switched to a more granular approach with specific write paths.
4. Set sensible timeouts: Our first health rule triggered alerts if jobs ran longer than 10 minutes. This caused false alarms every Monday when processing was heavier.
5. Document everything: I created a wiki page explaining the workflow, parameters, and expected outputs. This saved me from endless questions when I went on vacation.

What Still Bothers Us

Lakeflow Jobs isn't perfect. These are our ongoing frustrations, with update notes as of October 2026:

• Limited parameter validation (it's easy for users to enter invalid values)
• We had to build our own data quality checks for this pipeline. Databricks now offers data quality monitoring for data assets in Unity Catalog, and today we would evaluate that built-in option first.
• The UI locks get annoying when you need to make quick changes
• Bundles (now called Declarative Automation Bundles in Databricks documentation) give an infrastructure-as-code way to configure jobs, but in our experience promoting between dev, test and prod still takes manual work.
• The jobs UI can clone a job, creating an identical copy except for the job ID. For us the friction is that the separate jobs our silver and gold Run Job tasks call are their own jobs, so we have to copy and re-link them one by one.

Despite these issues, it's still better than our old manual process or trying to maintain Airflow ourselves.

Conclusion

If you're ready to escape spreadsheet hell and manual data pulls:

1. Map out your current reporting process
2. Identify logical breakpoints between tasks
3. Determine dependencies between those tasks
4. Start with a simple workflow that handles your most frequent data needs
5. Add parameters to make it flexible
6. Gradually expand to handle more complex cases

Don't try to automate everything on day one. Get one piece working well, then expand. We are still learning and improving, and would love to hear what's working for others.

If your team is building a similar pipeline on Lakeflow Jobs and wants a second pair of hands, Arbisoft's Databricks services team can help with design, compute setup and alerting.