> ## Documentation Index > Fetch the complete documentation index at: https://docs.truedy.ai/llms.txt > Use this file to discover all available pages before exploring further. # Webhook Error Handling & Retries > Build idempotent receivers that handle duplicates, failures, and retries correctly # Webhook Error Handling & Retries Webhooks are **at-least-once**: networks fail, servers time out, and Truedy may re-deliver the same event. Your receiver must stay correct even when it sees the same event twice. This guide covers: * The correct processing model * How to pick a dedupe key * Response semantics (what status codes to return when) * Full working implementations in Node.js and Python *** ## The correct processing model ``` receive → verify → dedupe check → apply side effects → record dedupe key → return 204 ``` Never do it the other way around (apply side effects first, then record the dedupe key) — a crash between those two steps causes duplicate processing. Reject immediately if the HMAC or timestamp check fails. Return `401`. See [Securing Webhooks](/guides/securing-webhooks). Pull the event type from `envelope.event` and the dedupe key from `envelope.data`. See [Dedupe key rules](#dedupe-keys) below. Look up `event_type + ":" + dedupe_key` in your database or cache. If already processed, return `204` immediately — no further action. Update your DB, call downstream APIs, fire queued jobs — whatever your business logic requires. Inside the same transaction as step 4, insert the dedupe record so it's atomic with your side effects. Return `204 No Content` immediately. If step 4 involves slow operations, enqueue them in a background job and return `204` before they complete. *** ## Dedupe keys Truedy sends a JSON envelope: ```json theme={null} { "event": "call.ended", "timestamp": "2026-03-18T14:32:00.000Z", "data": { ... } } ``` Pick the best available dedupe identifier from `data`: | Priority | Key to use | Event families | | -------- | ------------------------------------------------------ | -------------------------- | | 1st | `data.id` or `data.eventId` | Any event that includes it | | 2nd | `data.call.callId` (or `data.call_id` / `data.callId`) | Call events | | 2nd | `data.batch_id` (or `data.batchId`) | Batch/campaign events | | 2nd | `data.voice_id` (or `data.voiceId`) | Voice training events | | Fallback | Composite key | If none of the above exist | Composite key fallback: ``` event_type + ":" + iso_timestamp + ":" + phone_number_or_agent_id ``` Support both camelCase and snake\_case variants of identifier fields — Truedy may deliver either depending on the event source. Check `data.callId ?? data.call_id ?? data.call?.callId`. *** ## Response status codes | Scenario | Return | | --------------------------------------------------------- | ---------------------------------------------------------------- | | Signature valid, event already processed (duplicate) | `204 No Content` | | Signature valid, event processed successfully | `204 No Content` | | Signature invalid or timestamp expired | `401 Unauthorized` | | Payload malformed or missing required fields | `400 Bad Request` | | Your DB is down or a transient error occurred | `500 Internal Server Error` (triggers re-delivery) | | Permanent business logic failure (e.g. contact not found) | `204` — record as "handled with error" to avoid infinite retries | *** ## Complete working implementations ```typescript Node.js / TypeScript (Express + PostgreSQL) theme={null} import express, { Request, Response } from 'express' import { Pool } from 'pg' import crypto from 'crypto' const app = express() const db = new Pool({ connectionString: process.env.DATABASE_URL }) const WEBHOOK_SECRET = process.env.TRUEDY_WEBHOOK_SECRET! // Use raw body middleware to preserve bytes for HMAC verification app.post('/webhooks/truedy', express.raw({ type: 'application/json' }), async (req: Request, res: Response) => { const rawBody = req.body as Buffer const timestamp = req.headers['x-truedy-timestamp'] as string ?? '' const signature = req.headers['x-truedy-signature'] as string ?? '' // Step 1: Verify signature if (!verifySignature(rawBody, timestamp, signature)) { return res.status(401).json({ error: 'Invalid signature' }) } let envelope: Record try { envelope = JSON.parse(rawBody.toString('utf8')) } catch { return res.status(400).json({ error: 'Invalid JSON' }) } const eventType = envelope.event as string const dedupeKey = extractDedupeKey(envelope) if (!dedupeKey) { console.error('Missing dedupe key for event:', eventType, JSON.stringify(envelope).slice(0, 200)) return res.status(400).json({ error: 'Missing dedupe key' }) } const recordKey = `${eventType}:${dedupeKey}` // Step 2: Idempotency check + side effects in a single transaction const client = await db.connect() try { await client.query('BEGIN') const existing = await client.query( 'SELECT 1 FROM webhook_events WHERE dedupe_key = $1', [recordKey] ) if (existing.rowCount && existing.rowCount > 0) { await client.query('ROLLBACK') return res.status(204).send() // Already processed — safe to ignore } // Apply your business side effects here await handleEvent(client, eventType, envelope) // Record dedupe key atomically with side effects await client.query( 'INSERT INTO webhook_events (dedupe_key, event_type, processed_at) VALUES ($1, $2, NOW())', [recordKey, eventType] ) await client.query('COMMIT') } catch (err) { await client.query('ROLLBACK') console.error('Webhook processing failed:', err) return res.status(500).json({ error: 'Processing failed' }) // Truedy will retry } finally { client.release() } res.status(204).send() }) function verifySignature(rawBody: Buffer, timestamp: string, signature: string): boolean { const now = Math.floor(Date.now() / 1000) const ts = parseInt(timestamp, 10) if (isNaN(ts) || now - ts > 300 || ts > now + 60) return false const message = `${timestamp}.${rawBody.toString('utf8')}` const expected = crypto.createHmac('sha256', WEBHOOK_SECRET).update(message).digest('hex') if (expected.length !== signature.length) return false return crypto.timingSafeEqual(Buffer.from(expected, 'hex'), Buffer.from(signature, 'hex')) } function extractDedupeKey(envelope: Record): string | null { const data = envelope.data as Record | undefined if (!data) return null // Check data.id / data.eventId first if (typeof data.id === 'string') return data.id if (typeof data.eventId === 'string') return data.eventId // Call events const call = data.call as Record | undefined const callId = call?.callId ?? call?.call_id ?? data.call_id ?? data.callId if (typeof callId === 'string') return callId // Batch events const batchId = data.batch_id ?? data.batchId if (typeof batchId === 'string') return batchId as string // Voice events const voiceId = data.voice_id ?? data.voiceId if (typeof voiceId === 'string') return voiceId as string return null } async function handleEvent(client: unknown, eventType: string, envelope: Record) { const data = envelope.data as Record switch (eventType) { case 'call.ended': { const call = data.call as Record | undefined console.log('Call ended:', call?.callId, '— duration:', call?.duration, 's') // e.g. update your CRM, store transcript break } case 'batch.completed': { console.log('Batch completed:', data.batch_id ?? data.batchId) // e.g. send summary email to campaign owner break } default: console.log('Unhandled event type:', eventType) } } ``` ```python Python (FastAPI + PostgreSQL) theme={null} import hashlib import hmac import json import os import time from contextlib import asynccontextmanager from typing import Any import asyncpg from fastapi import FastAPI, Header, HTTPException, Request from fastapi.responses import Response WEBHOOK_SECRET = os.environ["TRUEDY_WEBHOOK_SECRET"] DATABASE_URL = os.environ["DATABASE_URL"] app = FastAPI() db_pool: asyncpg.Pool | None = None @asynccontextmanager async def lifespan(application: FastAPI): global db_pool db_pool = await asyncpg.create_pool(DATABASE_URL) yield await db_pool.close() app = FastAPI(lifespan=lifespan) def verify_signature(raw_body: bytes, timestamp: str, signature: str) -> bool: """Verify HMAC-SHA256 signature with replay protection.""" try: ts = int(timestamp) except (TypeError, ValueError): return False now = int(time.time()) if now - ts > 300 or ts > now + 60: return False message = f"{timestamp}.{raw_body.decode('utf-8')}" expected = hmac.new( WEBHOOK_SECRET.encode("utf-8"), message.encode("utf-8"), hashlib.sha256, ).hexdigest() return hmac.compare_digest(expected, signature) def extract_dedupe_key(data: dict[str, Any]) -> str | None: """Extract the best available dedupe identifier from event data.""" if data.get("id"): return str(data["id"]) if data.get("eventId"): return str(data["eventId"]) # Call events call = data.get("call", {}) call_id = call.get("callId") or call.get("call_id") or data.get("call_id") or data.get("callId") if call_id: return str(call_id) # Batch events batch_id = data.get("batch_id") or data.get("batchId") if batch_id: return str(batch_id) # Voice events voice_id = data.get("voice_id") or data.get("voiceId") if voice_id: return str(voice_id) return None async def handle_event(conn: asyncpg.Connection, event_type: str, data: dict[str, Any]) -> None: """Apply business-side effects for a given event type.""" if event_type == "call.ended": call = data.get("call", {}) print(f"Call ended: {call.get('callId')} duration={call.get('duration')}s") # e.g. UPDATE calls SET status='ended', duration=... WHERE call_id=... elif event_type == "batch.completed": print(f"Batch completed: {data.get('batch_id') or data.get('batchId')}") # e.g. send completion notification else: print(f"Unhandled event: {event_type}") @app.post("/webhooks/truedy") async def truedy_webhook( request: Request, x_truedy_timestamp: str = Header(""), x_truedy_signature: str = Header(""), ): raw_body = await request.body() # Step 1: Verify signature if not verify_signature(raw_body, x_truedy_timestamp, x_truedy_signature): raise HTTPException(status_code=401, detail="Invalid signature") try: envelope = json.loads(raw_body) except json.JSONDecodeError: raise HTTPException(status_code=400, detail="Invalid JSON") event_type: str = envelope.get("event", "") data: dict[str, Any] = envelope.get("data", {}) dedupe_key = extract_dedupe_key(data) if not dedupe_key: raise HTTPException(status_code=400, detail="Missing dedupe key") record_key = f"{event_type}:{dedupe_key}" # Step 2: Idempotency + side effects in one transaction async with db_pool.acquire() as conn: async with conn.transaction(): existing = await conn.fetchval( "SELECT 1 FROM webhook_events WHERE dedupe_key = $1", record_key, ) if existing: return Response(status_code=204) # Already processed await handle_event(conn, event_type, data) await conn.execute( "INSERT INTO webhook_events (dedupe_key, event_type, processed_at) VALUES ($1, $2, NOW())", record_key, event_type, ) return Response(status_code=204) ``` ### Database table for idempotency tracking ```sql theme={null} CREATE TABLE webhook_events ( dedupe_key TEXT PRIMARY KEY, event_type TEXT NOT NULL, processed_at TIMESTAMPTZ NOT NULL DEFAULT NOW() ); -- Optional: purge old records after 30 days to keep the table small CREATE INDEX ON webhook_events (processed_at); ``` *** ## Operational runbook ### Signature verification fails * Log `X-Truedy-Timestamp` and `event` (if parseable). Do **not** log the full secret. * Return `401` and stop. * Check: are you reading the raw body *before* JSON parsing? Body parsers can normalize whitespace and break HMAC. ### Payload parse error / schema mismatch * Log the parsing error and a redacted payload snapshot. * Return `400`. ### Transient processing failure (DB down, downstream API timeout) * Return `500` — Truedy will re-deliver the event. * Ensure your handler checks the dedupe table at the start so the retry doesn't double-process. ### Permanent business logic failure * Return `204` after recording the event as "failed but handled" so the retry loop doesn't run forever. * Alert your on-call team separately. *** ## Next steps Full HMAC verification code examples Event catalogue and all payload shapes Platform-wide idempotency patterns