Webhook Error Handling & Retries

Webhooks are at-least-once: networks fail, servers time out, and Truedy may re-deliver the same event. Your receiver must stay correct even when it sees the same event twice. This guide covers:

The correct processing model
How to pick a dedupe key
Response semantics (what status codes to return when)
Full working implementations in Node.js and Python

The correct processing model

receive → verify → dedupe check → apply side effects → record dedupe key → return 204

Never do it the other way around (apply side effects first, then record the dedupe key) — a crash between those two steps causes duplicate processing.

Verify the signature

Reject immediately if the HMAC or timestamp check fails. Return 401. See Securing Webhooks.

Parse event and extract a dedupe key

Pull the event type from envelope.event and the dedupe key from envelope.data. See Dedupe key rules below.

Check idempotency

Look up event_type + ":" + dedupe_key in your database or cache. If already processed, return 204 immediately — no further action.

Apply side effects in a transaction

Update your DB, call downstream APIs, fire queued jobs — whatever your business logic requires.

Record the dedupe key

Inside the same transaction as step 4, insert the dedupe record so it’s atomic with your side effects.

Return 204 and return fast

Return 204 No Content immediately. If step 4 involves slow operations, enqueue them in a background job and return 204 before they complete.

Dedupe keys

Truedy sends a JSON envelope:

{
  "event": "call.ended",
  "timestamp": "2026-03-18T14:32:00.000Z",
  "data": { ... }
}

Pick the best available dedupe identifier from data:

Priority	Key to use	Event families
1st	`data.id` or `data.eventId`	Any event that includes it
2nd	`data.call.callId` (or `data.call_id` / `data.callId`)	Call events
2nd	`data.batch_id` (or `data.batchId`)	Batch/campaign events
2nd	`data.voice_id` (or `data.voiceId`)	Voice training events
Fallback	Composite key	If none of the above exist

Composite key fallback:

event_type + ":" + iso_timestamp + ":" + phone_number_or_agent_id

Support both camelCase and snake_case variants of identifier fields — Truedy may deliver either depending on the event source. Check data.callId ?? data.call_id ?? data.call?.callId.

Response status codes

Scenario	Return
Signature valid, event already processed (duplicate)	`204 No Content`
Signature valid, event processed successfully	`204 No Content`
Signature invalid or timestamp expired	`401 Unauthorized`
Payload malformed or missing required fields	`400 Bad Request`
Your DB is down or a transient error occurred	`500 Internal Server Error` (triggers re-delivery)
Permanent business logic failure (e.g. contact not found)	`204` — record as “handled with error” to avoid infinite retries

Complete working implementations

import express, { Request, Response } from 'express'
import { Pool } from 'pg'
import crypto from 'crypto'

const app = express()
const db = new Pool({ connectionString: process.env.DATABASE_URL })
const WEBHOOK_SECRET = process.env.TRUEDY_WEBHOOK_SECRET!

// Use raw body middleware to preserve bytes for HMAC verification
app.post('/webhooks/truedy', express.raw({ type: 'application/json' }), async (req: Request, res: Response) => {
  const rawBody = req.body as Buffer
  const timestamp = req.headers['x-truedy-timestamp'] as string ?? ''
  const signature = req.headers['x-truedy-signature'] as string ?? ''

  // Step 1: Verify signature
  if (!verifySignature(rawBody, timestamp, signature)) {
    return res.status(401).json({ error: 'Invalid signature' })
  }

  let envelope: Record<string, unknown>
  try {
    envelope = JSON.parse(rawBody.toString('utf8'))
  } catch {
    return res.status(400).json({ error: 'Invalid JSON' })
  }

  const eventType = envelope.event as string
  const dedupeKey = extractDedupeKey(envelope)

  if (!dedupeKey) {
    console.error('Missing dedupe key for event:', eventType, JSON.stringify(envelope).slice(0, 200))
    return res.status(400).json({ error: 'Missing dedupe key' })
  }

  const recordKey = `${eventType}:${dedupeKey}`

  // Step 2: Idempotency check + side effects in a single transaction
  const client = await db.connect()
  try {
    await client.query('BEGIN')

    const existing = await client.query(
      'SELECT 1 FROM webhook_events WHERE dedupe_key = $1',
      [recordKey]
    )
    if (existing.rowCount && existing.rowCount > 0) {
      await client.query('ROLLBACK')
      return res.status(204).send() // Already processed — safe to ignore
    }

    // Apply your business side effects here
    await handleEvent(client, eventType, envelope)

    // Record dedupe key atomically with side effects
    await client.query(
      'INSERT INTO webhook_events (dedupe_key, event_type, processed_at) VALUES ($1, $2, NOW())',
      [recordKey, eventType]
    )

    await client.query('COMMIT')
  } catch (err) {
    await client.query('ROLLBACK')
    console.error('Webhook processing failed:', err)
    return res.status(500).json({ error: 'Processing failed' }) // Truedy will retry
  } finally {
    client.release()
  }

  res.status(204).send()
})

function verifySignature(rawBody: Buffer, timestamp: string, signature: string): boolean {
  const now = Math.floor(Date.now() / 1000)
  const ts = parseInt(timestamp, 10)
  if (isNaN(ts) || now - ts > 300 || ts > now + 60) return false

  const message = `${timestamp}.${rawBody.toString('utf8')}`
  const expected = crypto.createHmac('sha256', WEBHOOK_SECRET).update(message).digest('hex')

  if (expected.length !== signature.length) return false
  return crypto.timingSafeEqual(Buffer.from(expected, 'hex'), Buffer.from(signature, 'hex'))
}

function extractDedupeKey(envelope: Record<string, unknown>): string | null {
  const data = envelope.data as Record<string, unknown> | undefined
  if (!data) return null

  // Check data.id / data.eventId first
  if (typeof data.id === 'string') return data.id
  if (typeof data.eventId === 'string') return data.eventId

  // Call events
  const call = data.call as Record<string, unknown> | undefined
  const callId = call?.callId ?? call?.call_id ?? data.call_id ?? data.callId
  if (typeof callId === 'string') return callId

  // Batch events
  const batchId = data.batch_id ?? data.batchId
  if (typeof batchId === 'string') return batchId as string

  // Voice events
  const voiceId = data.voice_id ?? data.voiceId
  if (typeof voiceId === 'string') return voiceId as string

  return null
}

async function handleEvent(client: unknown, eventType: string, envelope: Record<string, unknown>) {
  const data = envelope.data as Record<string, unknown>
  switch (eventType) {
    case 'call.ended': {
      const call = data.call as Record<string, unknown> | undefined
      console.log('Call ended:', call?.callId, '— duration:', call?.duration, 's')
      // e.g. update your CRM, store transcript
      break
    }
    case 'batch.completed': {
      console.log('Batch completed:', data.batch_id ?? data.batchId)
      // e.g. send summary email to campaign owner
      break
    }
    default:
      console.log('Unhandled event type:', eventType)
  }
}

import hashlib
import hmac
import json
import os
import time
from contextlib import asynccontextmanager
from typing import Any

import asyncpg
from fastapi import FastAPI, Header, HTTPException, Request
from fastapi.responses import Response

WEBHOOK_SECRET = os.environ["TRUEDY_WEBHOOK_SECRET"]
DATABASE_URL = os.environ["DATABASE_URL"]

app = FastAPI()
db_pool: asyncpg.Pool | None = None


@asynccontextmanager
async def lifespan(application: FastAPI):
    global db_pool
    db_pool = await asyncpg.create_pool(DATABASE_URL)
    yield
    await db_pool.close()

app = FastAPI(lifespan=lifespan)


def verify_signature(raw_body: bytes, timestamp: str, signature: str) -> bool:
    """Verify HMAC-SHA256 signature with replay protection."""
    try:
        ts = int(timestamp)
    except (TypeError, ValueError):
        return False
    now = int(time.time())
    if now - ts > 300 or ts > now + 60:
        return False
    message = f"{timestamp}.{raw_body.decode('utf-8')}"
    expected = hmac.new(
        WEBHOOK_SECRET.encode("utf-8"),
        message.encode("utf-8"),
        hashlib.sha256,
    ).hexdigest()
    return hmac.compare_digest(expected, signature)


def extract_dedupe_key(data: dict[str, Any]) -> str | None:
    """Extract the best available dedupe identifier from event data."""
    if data.get("id"):
        return str(data["id"])
    if data.get("eventId"):
        return str(data["eventId"])
    # Call events
    call = data.get("call", {})
    call_id = call.get("callId") or call.get("call_id") or data.get("call_id") or data.get("callId")
    if call_id:
        return str(call_id)
    # Batch events
    batch_id = data.get("batch_id") or data.get("batchId")
    if batch_id:
        return str(batch_id)
    # Voice events
    voice_id = data.get("voice_id") or data.get("voiceId")
    if voice_id:
        return str(voice_id)
    return None


async def handle_event(conn: asyncpg.Connection, event_type: str, data: dict[str, Any]) -> None:
    """Apply business-side effects for a given event type."""
    if event_type == "call.ended":
        call = data.get("call", {})
        print(f"Call ended: {call.get('callId')} duration={call.get('duration')}s")
        # e.g. UPDATE calls SET status='ended', duration=... WHERE call_id=...
    elif event_type == "batch.completed":
        print(f"Batch completed: {data.get('batch_id') or data.get('batchId')}")
        # e.g. send completion notification
    else:
        print(f"Unhandled event: {event_type}")


@app.post("/webhooks/truedy")
async def truedy_webhook(
    request: Request,
    x_truedy_timestamp: str = Header(""),
    x_truedy_signature: str = Header(""),
):
    raw_body = await request.body()

    # Step 1: Verify signature
    if not verify_signature(raw_body, x_truedy_timestamp, x_truedy_signature):
        raise HTTPException(status_code=401, detail="Invalid signature")

    try:
        envelope = json.loads(raw_body)
    except json.JSONDecodeError:
        raise HTTPException(status_code=400, detail="Invalid JSON")

    event_type: str = envelope.get("event", "")
    data: dict[str, Any] = envelope.get("data", {})
    dedupe_key = extract_dedupe_key(data)

    if not dedupe_key:
        raise HTTPException(status_code=400, detail="Missing dedupe key")

    record_key = f"{event_type}:{dedupe_key}"

    # Step 2: Idempotency + side effects in one transaction
    async with db_pool.acquire() as conn:
        async with conn.transaction():
            existing = await conn.fetchval(
                "SELECT 1 FROM webhook_events WHERE dedupe_key = $1",
                record_key,
            )
            if existing:
                return Response(status_code=204)  # Already processed

            await handle_event(conn, event_type, data)

            await conn.execute(
                "INSERT INTO webhook_events (dedupe_key, event_type, processed_at) VALUES ($1, $2, NOW())",
                record_key, event_type,
            )

    return Response(status_code=204)

Database table for idempotency tracking

CREATE TABLE webhook_events (
  dedupe_key   TEXT PRIMARY KEY,
  event_type   TEXT NOT NULL,
  processed_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

-- Optional: purge old records after 30 days to keep the table small
CREATE INDEX ON webhook_events (processed_at);

Operational runbook

Signature verification fails

Log X-Truedy-Timestamp and event (if parseable). Do not log the full secret.
Return 401 and stop.
Check: are you reading the raw body before JSON parsing? Body parsers can normalize whitespace and break HMAC.

Payload parse error / schema mismatch

Log the parsing error and a redacted payload snapshot.
Return 400.

Transient processing failure (DB down, downstream API timeout)

Return 500 — Truedy will re-deliver the event.
Ensure your handler checks the dedupe table at the start so the retry doesn’t double-process.

Permanent business logic failure

Return 204 after recording the event as “failed but handled” so the retry loop doesn’t run forever.
Alert your on-call team separately.

Next steps

Securing Webhooks

Full HMAC verification code examples

Available Webhooks

Event catalogue and all payload shapes

Idempotency

Platform-wide idempotency patterns

Getting Started

How Truedy Works

Agents

Calls

Outbound Campaigns

Telephony

Tools

Knowledge

Embed & Web Calls

Webhooks

Platform

Webhook Error Handling & Retries

Webhook Error Handling & Retries

The correct processing model

Dedupe keys

Response status codes

Complete working implementations

Database table for idempotency tracking

Operational runbook

Signature verification fails

Payload parse error / schema mismatch

Transient processing failure (DB down, downstream API timeout)

Permanent business logic failure

Next steps

Securing Webhooks

Available Webhooks

Idempotency

​Webhook Error Handling & Retries

​The correct processing model

​Dedupe keys

​Response status codes

​Complete working implementations

​Database table for idempotency tracking

​Operational runbook

​Signature verification fails

​Payload parse error / schema mismatch

​Transient processing failure (DB down, downstream API timeout)

​Permanent business logic failure

​Next steps

Securing Webhooks

Available Webhooks

Idempotency

Webhook Error Handling & Retries

The correct processing model

Dedupe keys

Response status codes

Complete working implementations

Database table for idempotency tracking

Operational runbook

Signature verification fails

Payload parse error / schema mismatch

Transient processing failure (DB down, downstream API timeout)

Permanent business logic failure

Next steps