On Friday afternoon a customer signs up for a new mobile plan and asks to keep their number. The CRM captures the order, billing creates the account, and a port-in request goes out to the losing carrier. At 2:07 a.m. on Saturday, the old carrier finally confirms the port. A provisioning adapter picks up the confirmation, and the call to the network activation system times out because a nightly batch job has locked the inventory database.
Nobody notices. The adapter logged an error and moved on. On Monday the port window has lapsed, the customer’s new line was never activated, and the first invoice has already been generated.
If you work on order-to-activation at a telco, MVNO, or ISP, you have probably seen a version of this. The interesting question is not why one call failed. Calls fail all the time. The question is why a single failed call can leave an order stranded across five systems with no one in charge of finishing it.
Why activation orders get stuck
A service activation order is not a transaction. It is a long-running business process that crosses systems with very different behavior.
The CRM is usually synchronous and fast. Billing may accept real-time account creation but run rating and invoicing in batches. Inventory and number management are often older platforms with their own locking and maintenance windows. Network provisioning goes through an activation layer that talks to HLR/HSS, policy, and eSIM systems. Number portability depends on another carrier and sometimes a national clearinghouse, which can take hours or days.
When these steps are wired together as point-to-point calls or independent message consumers, the state of the order ends up scattered. CRM thinks the order is “in progress.” Billing thinks the account is active. The network has no record of the subscriber. No single component knows that step four failed after step three succeeded, so nothing knows billing should be paused.
Three patterns make this worse:
- Long waits held in memory or cron. Porting windows are often tracked with a database flag and a polling job. If the job misses an event or the service restarts mid-wait, the order silently stalls.
- Retries without limits or context. An adapter that retries forever hammers a struggling legacy system. One that gives up after three tries drops the order into a log file.
- No compensation path. Few teams design the “undo” steps (suspend billing, release the reserved number, cancel the SIM profile) as first-class parts of the flow.
Why the usual workarounds fall short
Most operators already have mitigations. They help, but they don’t change the structure.
Fallout queues. A dedicated team fixes stuck orders by hand. This keeps customers whole, but it scales with order volume, depends on tribal knowledge, and grows with every product launch.
Reconciliation jobs. A nightly job compares billing against the network and flags mismatches. It catches “billed but not activated” eventually, but often after the customer has noticed.
A homegrown state machine. Teams build status columns in a relational table and workers that advance them. This works until you need timers, parallel branches, partial retries, and versioning of in-flight orders. Then the homegrown engine becomes its own legacy system.
Heavy COTS orchestrators. Commercial order management suites can model this well, but they are often slow to change and still need custom integration with systems they don’t own.
The common gap is that the process itself is not durable. The order’s progress, its pending waits, and its recovery logic live in places that can be lost.
The general approach: make the order a durable workflow
The vendor-neutral fix rests on a few principles.
Model each order as one durable workflow instance. The workflow owns the sequence: validate, reserve number, create billing account, submit port, wait for confirmation, activate network, enable billing, notify customer. Its state is persisted after every step, so a crash or redeploy resumes from the last completed step.
Use the saga pattern for partial failure. Every step that changes another system has a matching compensation: release the number, suspend the billing account, cancel the eSIM profile. If activation fails after billing succeeded, the workflow runs compensations or routes to a defined remediation path. “Billed but not activated” becomes a handled state, not an accident.
Use durable timers for porting windows and partner SLAs. A 48-hour wait should be a timer stored with the workflow, not a thread or a cron query.
Treat partner responses as external events. The port confirmation, a partner callback, and a fallout agent’s approval are events delivered to a specific workflow instance. The workflow waits for whichever comes first: the event or the timeout.
Integrate legacy systems through events. Publishing order events to a broker and letting adapters consume them at a pace the legacy system can handle decouples orchestration from maintenance windows.
Apply resiliency policies at the edges. Timeouts, bounded retries with backoff, and circuit breakers protect fragile systems and give the workflow a clear failure signal.
How to implement it with open source
Dapr, the CNCF graduated Distributed Application Runtime, maps well to these principles. It runs as a sidecar next to each service on Kubernetes, VMs, or self-hosted environments, and its building blocks can be adopted one at a time.
Dapr Workflow lets you write the order process as code in .NET, Java, Python, Go, or JavaScript. Workflow history is event-sourced into a state store, so after a crash the workflow replays and continues without re-executing completed activities. It supports chaining, fan-out/fan-in, child workflows, durable timers, external events, and compensation. Orchestrator code must be deterministic, so all calls to CRM, billing, or the network belong in activities.
Here is a minimal sketch of the porting wait from the opening scene:
import dapr.ext.workflow as wf
from datetime import timedelta
wfr = wf.WorkflowRuntime()
retry = wf.RetryPolicy(first_retry_interval=timedelta(seconds=2), max_number_of_attempts=5, backoff_coefficient=2, max_retry_interval=timedelta(minutes=1))
@wfr.workflow(name=”port_in_order”)
def port_in_order(ctx: wf.DaprWorkflowContext, order: dict):
yield ctx.call_activity(submit_port_request, input=order, retry_policy=retry)
confirmed = ctx.wait_for_external_event(“port_confirmed”)
timeout = ctx.create_timer(timedelta(hours=48))
winner = yield wf.when_any([confirmed, timeout])
if winner == timeout:
yield ctx.call_activity(open_fallout_case, input=order)
return {“status”: “port_timeout”}
yield ctx.call_activity(activate_network, input=order, retry_policy=retry)
If the confirmation arrives at 2 a.m. and activation fails, the retry policy absorbs transient errors. A lock that lasts longer than the retry window falls through to compensation or a fallout case, which is the point. If retries are exhausted, the workflow still holds the order’s state and can run compensation or raise a fallout case with full context.
Pub/sub connects the workflow to legacy and batch systems. Dapr supports brokers such as Kafka, RabbitMQ, and Redis, uses CloudEvents, delivers at least once, and supports dead-letter topics. Because delivery is at least once, make adapters idempotent, keyed on the order ID. The state management building block also offers a transactional outbox for services that must update data and publish an event together.
Resiliency policies are declared in YAML (kind: Resiliency): timeouts, constant or exponential retries, and circuit breakers. You can tighten them for a fragile inventory platform without touching application code. Separately, a large retail bank in Asia throttles incoming traffic to a legacy core system using Dapr without modifying the core.
Observability comes from Prometheus metrics on every sidecar and OpenTelemetry tracing across service invocation and pub/sub.
As one data point, a communications-platform company saved 1,000+ development hours by adopting Dapr. Measure against your own baseline.
You can run all of this yourself. If you’d rather not operate the workflow backend, commercial options exist. Diagrid, founded by the creators of Dapr, is one provider offering enterprise Dapr support and Catalyst, a durable workflow platform that can be hosted, run in your own cloud, or deployed on-premises. Test any vendor against your own failure scenarios before committing.
A practical roadmap
Pick one order type with real fallout pain, usually mobile port-in or broadband activation:
- Map the real flow. List every system the order touches, what each call changes, and worst-case durations, including partner waits.
- Define compensations. For each state-changing step, write down its undo action and owner.
- Name the external events. Port confirmations, partner callbacks, and manual approvals become events delivered by order ID.
- Set timers from SLAs. Replace polling jobs with durable timers tied to contractual or regulatory windows.
- Wrap legacy systems behind events. Use idempotent, rate-aware pub/sub adapters.
- Add resiliency policies. Start with conservative timeouts and bounded retries, then add circuit breakers.
- Route exhausted retries to humans with context. The fallout queue shrinks, and every case arrives with its workflow history.
- Run in shadow mode. Execute the new workflow alongside the existing process for a subset of orders and compare outcomes.
Before going live, check:
- ☐ Every activity is idempotent and keyed on order ID
- ☐ Orchestrator code contains no direct I/O
- ☐ Billing activation follows network activation, or has a tested compensation
- ☐ Each timer has a defined escalation action
- ☐ Dead-letter topics are monitored
- ☐ Workflow versioning is planned for orders in flight across deployments
Conclusion
Orders get stuck because no component owns them end to end. Each system does its part, and the gaps between them are where customers lose service and still receive a bill. Treating each order as a durable workflow, with explicit compensations, timers for slow external parties, and event-driven links to legacy systems, puts that ownership in one place.
You don’t need to rebuild your OSS/BSS stack. Start with one painful order type and prove that the 2 a.m. failure becomes a handled case instead of a Monday escalation. Whether you build on open-source Dapr directly or use a managed durable workflow platform, the principles are the same and worth testing against your own fallout data.
