Business Process Automation: Beyond the Fragile Script

When to automate and when not to: the decision that gets skipped

Not every process should be automated. The practical rule: automate processes that run frequently (at least weekly), have predictable and repeatable steps, and whose manual error cost is significant. Processes that change business rules frequently, require constant human judgment, or run less than once a month have low automation ROI — the maintenance cost can exceed the manual cost.

n8n: workflow automation without complex code

n8n is the open-source alternative to Zapier and Make for enterprises that need full control over their data and integrations. Unlike SaaS tools, n8n can be deployed on-premises or in your own cloud, supports executable JavaScript/Python code inside workflows, and allows integrations with internal systems that don't have public APIs.

Apache Airflow: orchestrating complex data pipelines

Airflow is the right tool when workflows have complex inter-task dependencies, need historical data backfill, or integrate multiple data sources with transformations. A DAG (Directed Acyclic Graph) in Airflow defines the workflow as Python code, with explicit dependencies and full parametrization.

from datetime import datetime, timedelta
from airflow.decorators import dag, task
from airflow.providers.postgres.hooks.postgres import PostgresHook

@dag(
    schedule_interval='@daily',
    start_date=datetime(2025, 1, 1),
    catchup=False,
    default_args={
        'retries': 3,
        'retry_delay': timedelta(minutes=5),
        'retry_exponential_backoff': True,
    }
)
def daily_sales_report():

    @task()
    def extract_daily_sales():
        hook = PostgresHook(postgres_conn_id='erp_production')
        records = hook.get_records(
            "SELECT product_id, sum(total) FROM sales WHERE date = CURRENT_DATE GROUP BY 1"
        )
        return [{'product_id': r[0], 'total': float(r[1])} for r in records]

    @task()
    def calculate_metrics(sales: list):
        total = sum(v['total'] for v in sales)
        top_products = sorted(sales, key=lambda x: x['total'], reverse=True)[:5]
        return {'day_total': total, 'top_5': top_products}

    @task()
    def save_to_warehouse(metrics: dict):
        hook = PostgresHook(postgres_conn_id='analytics_warehouse')
        hook.run(
            "INSERT INTO daily_reports (date, total, metrics_json) VALUES (CURRENT_DATE, %s, %s)",
            parameters=(metrics['day_total'], str(metrics))
        )

    sales = extract_daily_sales()
    metrics = calculate_metrics(sales)
    save_to_warehouse(metrics)

dag = daily_sales_report()

Idempotency: the principle that saves automations

An idempotent operation produces the same result regardless of how many times it's executed with the same inputs. In enterprise automations, idempotency is fundamental: tasks retry on failures, cron jobs can run twice, and queue messages can be delivered as duplicates.

import hashlib

def process_order(order_id: str, items: list) -> None:
    payload_hash = hashlib.sha256(
        f"{order_id}:{sorted(items)}".encode()
    ).hexdigest()

    cursor.execute(
        "SELECT id FROM processed_orders WHERE idempotency_key = %s",
        (payload_hash,)
    )
    if cursor.fetchone():
        logger.info(f"Order {order_id} already processed, skipping")
        return

    result = _apply_discounts_and_invoice(order_id, items)
    cursor.execute(
        "INSERT INTO processed_orders (idempotency_key, order_id, result, processed_at) VALUES (%s, %s, %s, NOW())",
        (payload_hash, order_id, str(result))
    )

Error handling in automated pipelines

• Transient errors (network timeout, external service momentarily down): retry with exponential backoff. The pipeline should recover automatically.
• Data errors (invalid fields, unexpected format): move to dead letter queue or error table, alert the team, continue processing other records. The pipeline should not block on a bad record.
• Business logic errors (business rule violated): stop the pipeline, alert immediately, and don't retry until a human reviews. These errors require intervention.

Observability for automatic workflows

• Success/error rate per workflow and per individual task
• Execution latency (P50, P95) — to detect gradual degradation
• Processed record volume vs. expected — to detect unexpected silence
• Pending message queue depth — to detect backpressure
• SLA alerts: if the sales report isn't generated before 08:00, automatic alert