Replacing a Fragile Cron Pipeline with Temporal Workflows

A daily data-processing pipeline had gradually outgrown its original cron-based design.

The job was responsible for coordinating work across many tenants and several external data providers. For each tenant, it needed to import fresh operational data and then run the business rules that produced reports and analytics.

When everything succeeded, cron looked sufficient.

The problem appeared when something failed.

A scheduled process could tell us that a run had started and eventually failed, but it did not naturally answer the operational questions that mattered:

Which tenants had completed successfully?
Which provider was still running?
Had the import finished before the rule calculations started?
Was a failed step safe to retry?
Should the whole batch restart, or only one tenant?
What would happen if the worker or service restarted halfway through the process?

Recovering from failures required manual investigation and application-specific coordination logic.

The main reason for introducing Temporal was therefore not scheduling.

It was durable and visible workflow state.

What the pipeline was doing

The daily process had two main phases.

For each tenant:

Import the latest operational data from its source system.
Run the daily rule calculations after the import had completed.

Different tenants used different external source-system integrations, but the business sequence remained the same:

Import fresh data
        |
        v
Confirm import completion
        |
        v
Run daily rule calculations
        |
        v
Produce updated analytics and reports

The ordering mattered.

Rule calculations could not start before the tenant’s new data was available. At the same time, one slow or failing tenant should not prevent unrelated tenants from making progress.

That created two different concurrency requirements:

work for each tenant had to remain sequential;
independent tenants could be processed in parallel.

Cron could start the batch, but the application itself had to implement all of that coordination.

Why the scheduled runner became fragile

The original runner had accumulated responsibilities over time.

It needed to:

discover which tenants should be processed;
group them by provider;
start imports;
wait for completion;
run calculations;
handle missing or invalid tenant configuration;
retry selected failures;
limit parallel work;
report progress and errors;
avoid duplicating completed work after a restart.

This is a common transition point in backend systems.

A cron expression is still perfectly good at saying:

Start this process every morning.

It is much less useful at describing:

This process contains many independent units of work, each with several ordered stages, retries, partial failures and durable progress.

The scheduling mechanism was not broken. The process had simply become a workflow.

The Temporal design

The new implementation separated batch orchestration from per-tenant processing.

At the top level, a batch workflow coordinated the daily run.

Daily batch workflow
        |
        +-- Ordered provider phase
        |      |
        |      +-- Tenant 1 workflow
        |      +-- Tenant 2 workflow
        |
        +-- Independent provider groups
               |
               +-- run concurrently where safe
               +-- each starts tenant workflows

Each tenant had its own pipeline workflow:

Tenant pipeline
        |
        v
Import activity
        |
        v
Validate import result
        |
        v
Daily rule-check workflow
        |
        v
Complete or surface failure

The batch workflow was responsible for orchestration.

Activities and child workflows performed the external work.

This distinction was important because Temporal workflow code must remain deterministic. Database access, HTTP calls and other side effects belonged in activities, while the workflow described the order, dependencies and retry behaviour.

Preserve sequence where the business requires it

The most important business constraint was simple:

Do not calculate rules for a tenant until its latest import has completed successfully.

That sequence was encoded directly in the tenant workflow.

The workflow first awaited the import result. Only after a valid completion did it start the daily rule-check stage.

This made the dependency explicit rather than relying on timing assumptions or loosely coordinated scheduled jobs.

At the batch level, tenants remained independent.

A failure in one tenant pipeline could be recorded and surfaced without invalidating successful work for every other tenant.

The general design principle was:

Sequence work within one business unit; parallelize across independent business units.

In this case, the independent business unit was a tenant.

Keep provider phases separate

Tenants were processed through several provider-specific integrations.

The import behaviour, dependencies and failure modes differed between providers, so the workflow did not treat every tenant as one undifferentiated queue.

Provider groups were executed according to their real dependencies:

providers that depended on an earlier phase were processed sequentially;
independent provider groups could run concurrently;
tenant work inside each active group still followed the same per-tenant pipeline.

This gave us clearer operational visibility without introducing parallelism where the business process did not allow it.

Ordered provider phase
        |
        v
Dependent provider phase

Independent provider A ----+
                           +--> concurrent where safe
Independent provider B ----+

The workflow structure therefore expressed both kinds of relationship: ordering where it mattered and concurrency where the work was independent.

Failure isolation with child workflows

Per-tenant child workflows became a useful isolation boundary.

Each workflow had its own:

execution history;
current state;
retry behaviour;
failure details;
completion result.

This was much easier to reason about than one large scheduled process with a growing in-memory list of partially completed operations.

If one tenant was misconfigured or no longer existed in a downstream service, that case could be handled inside its own execution rather than crashing the entire batch.

The parent awaited all child workflows. A child failure was caught and converted into a structured result instead of being allowed to terminate the parent immediately.

The final batch result could therefore distinguish between outcomes such as:

COMPLETED
SKIPPED
NOT_FOUND
FAILED

After all children had finished, the workflow produced a final summary showing how many tenants were processed, how many succeeded and which ones failed.

Child workflow failures remained visible. Retryable failures were handled according to the configured retry policy; after retries were exhausted, the parent converted the failure into a structured result while the recurring schedule continued to start future daily runs.

That made partial success a first-class result rather than an accidental side effect.

Visible state changed operations

The largest practical improvement was not a code abstraction.

It was visibility.

With the previous runner, investigating a failed batch meant reading logs and reconstructing the process:

Did the import start?
Did it finish?
Did the calculation stage begin?
Was the service restarted?
Was this tenant already processed?

With Temporal, the workflow history showed the execution state directly.

For a tenant pipeline, we could see:

Import started
Import completed
Daily rule check started
Activity retry scheduled
Daily rule check completed
Workflow completed

Or, when something failed:

Import started
Import failed
Retry 1 scheduled
Import failed
Retry 2 scheduled
Workflow failed

This did not remove the need for logs and monitoring, but it removed much of the guesswork around orchestration state.

A failed workflow could be investigated as a specific execution rather than as one event inside a large daily log stream.

Concurrency needed an explicit limit

Once tenant pipelines became independent, it was technically possible to start many of them at once.

That did not mean it was a good idea.

Every workflow eventually consumed real resources:

database connections;
CPU;
memory;
external API capacity;
downstream calculation capacity.

Maximum parallelism would have made the workflow graph look fast while making the dependent systems less predictable.

We therefore limited concurrent tenant processing to three within the active batch execution.

Waiting tenants
        |
        v
Per-batch concurrency gate: 3
   |       |       |
   v       v       v
Pipeline Pipeline Pipeline
   |
   v
Next waiting tenant

The exact number was not a property of Temporal. It was an operational choice based on the capacity of the surrounding system.

Because scheduled overlap was buffered rather than allowed concurrently, two ordinary daily schedule runs could not multiply this per-batch limit. A direct workflow start outside the Schedule would need the same operational care because it could bypass the Schedule’s overlap protection.

This made concurrency visible and intentional rather than an accidental result of thread-pool configuration.

An early design became too complicated

The first implementation was not the final one.

At one stage, orchestration relied on a larger set of promises and workflow signals. The design attempted to coordinate too much state inside the parent workflow.

That complexity produced two problems.

First, the flow was difficult to understand. It was not always obvious which component owned completion state.

Second, an early version triggered Temporal’s workflow deadlock detection while coordinating asynchronous work.

The right response was not to weaken the detector or increase a timeout.

The design was simplified:

unnecessary signals were removed;
the parent workflow awaited child-workflow results directly;
tenant state stayed inside tenant workflows;
external work remained in activities;
the batch workflow focused on coordination and aggregation.

This produced a workflow graph that more closely matched the actual business process.

The lesson was useful beyond Temporal:

Durable orchestration becomes easier when ownership boundaries are explicit.

Signals are valuable when an external actor genuinely needs to change a running workflow. They are less useful when they are being used only to recreate ordinary parent-child completion.

Retry does not mean “repeat everything”

Temporal makes retries convenient, but retries still need business meaning.

An import activity could be retried for a temporary provider or network failure.

That did not imply that the whole daily batch should restart.

Similarly, a tenant that had already completed successfully should not be processed again merely because another tenant failed.

The workflow structure made retry scope explicit:

Temporary API error
        |
        v
Retry import activity
        |
        v
Continue same tenant workflow

Instead of:

One tenant fails
        |
        v
Restart entire daily batch

Activities also needed to be safe when repeated.

Where possible, imports and calculation triggers used stable identifiers and existing state so that a retry would not silently create duplicate work.

Temporal provided durable retry mechanics, but the application still had to define idempotent behaviour.

Scheduling became the smallest part

The daily process was eventually started through a Temporal Schedule configured in UTC.

The Schedule used a buffered overlap policy. If the next daily trigger arrived while the previous batch was still running, a new scheduled run was buffered instead of starting concurrently.

The recurring schedule was not paused when an individual batch contained failed tenants. Retryable failures were handled at the workflow level, exhausted failures remained visible in the final summary, and future daily runs continued normally.

The Schedule’s catch-up behaviour applied only to missed triggers; it was not used to recover failed tenant work. Failed processing remained a workflow-level concern handled through retries, while the Schedule remained responsible for the normal daily cadence.

The schedule replaced the old cron trigger, but that was the least interesting part of the migration.

The real value was what happened after the trigger:

the batch had durable execution state;
tenant pipelines were visible independently;
retries happened at the correct scope;
failures no longer erased the progress of successful tenants;
concurrency was controlled deliberately;
worker restarts did not require reconstructing the batch from application memory.

The Schedule answered when to start.

The workflows answered how the process should survive reality.

Operational result

The new design made the daily pipeline easier to operate and recover.

Instead of one opaque scheduled execution, we had:

one visible batch workflow;
ordered and concurrent provider groups based on real dependencies;
one child workflow per tenant;
ordered import and calculation stages;
child failures converted into structured results;
explicit retry behaviour;
a per-batch concurrency limit of three;
buffered scheduled overlap;
a final success-and-failure summary;
durable workflow histories.

When a tenant failed, we could identify the exact stage and execution.

When a service restarted, the orchestration state was not lost.

When one provider had a problem, successful work for other providers and tenants remained visible.

The result was not that failures disappeared.

The result was that failures became understandable and recoverable.

What I would improve next time

The migration worked, but I would make several decisions earlier in a future implementation.

Define workflow boundaries before writing workflow code

The cleanest boundary was one workflow per tenant, with the batch workflow responsible only for coordination.

Reaching that design earlier would have avoided some of the initial signal and promise complexity.

Keep result handling explicit

Each child workflow result, including exhausted failures, should be converted into a structured outcome before the parent aggregates the batch.

COMPLETED
SKIPPED
NOT_FOUND
FAILED

This keeps one child failure from obscuring the outcome of every other tenant and makes the final operational summary straightforward.

Design idempotency with each activity

Retries are safest when every activity has a documented answer to:

What happens if this exact call runs twice?

That question should be part of activity design, not something considered only after the first failure.

Treat concurrency as configuration

The limit of three was an operational setting, not a permanent architectural constant.

Keeping it configurable made it possible to tune throughput without redesigning the workflows.

The main lesson

Cron was not the real problem.

The problem was using a simple scheduler as the foundation for a process that had become distributed, multi-stage and failure-prone.

Temporal helped because it made execution state durable and visible.

But adopting the platform was only part of the work. The application still needed clear answers to:

what one workflow represents;
which operations must remain sequential;
which units can run in parallel;
where retries are safe;
how failures are isolated;
how much concurrency the system can support;
which state belongs in the parent and which belongs in child workflows.

The strongest outcome was not that the daily job used Temporal.

It was that the implementation finally represented the business process directly:

import data for each tenant, confirm the result, run the calculations, and preserve enough state to recover when something goes wrong.