Skip to main content

Webhook Error Handling & Retries

Webhooks are at-least-once: networks fail, servers time out, and Truedy may re-deliver the same event. Your receiver must stay correct even when it sees the same event twice. This guide covers:
  • The correct processing model
  • How to pick a dedupe key
  • Response semantics (what status codes to return when)
  • Full working implementations in Node.js and Python

The correct processing model

receive → verify → dedupe check → apply side effects → record dedupe key → return 204
Never do it the other way around (apply side effects first, then record the dedupe key) — a crash between those two steps causes duplicate processing.
1

Verify the signature

Reject immediately if the HMAC or timestamp check fails. Return 401. See Securing Webhooks.
2

Parse event and extract a dedupe key

Pull the event type from envelope.event and the dedupe key from envelope.data. See Dedupe key rules below.
3

Check idempotency

Look up event_type + ":" + dedupe_key in your database or cache. If already processed, return 204 immediately — no further action.
4

Apply side effects in a transaction

Update your DB, call downstream APIs, fire queued jobs — whatever your business logic requires.
5

Record the dedupe key

Inside the same transaction as step 4, insert the dedupe record so it’s atomic with your side effects.
6

Return 204 and return fast

Return 204 No Content immediately. If step 4 involves slow operations, enqueue them in a background job and return 204 before they complete.

Dedupe keys

Truedy sends a JSON envelope:
{
  "event": "call.ended",
  "timestamp": "2026-03-18T14:32:00.000Z",
  "data": { ... }
}
Pick the best available dedupe identifier from data:
PriorityKey to useEvent families
1stdata.id or data.eventIdAny event that includes it
2nddata.call.callId (or data.call_id / data.callId)Call events
2nddata.batch_id (or data.batchId)Batch/campaign events
2nddata.voice_id (or data.voiceId)Voice training events
FallbackComposite keyIf none of the above exist
Composite key fallback:
event_type + ":" + iso_timestamp + ":" + phone_number_or_agent_id
Support both camelCase and snake_case variants of identifier fields — Truedy may deliver either depending on the event source. Check data.callId ?? data.call_id ?? data.call?.callId.

Response status codes

ScenarioReturn
Signature valid, event already processed (duplicate)204 No Content
Signature valid, event processed successfully204 No Content
Signature invalid or timestamp expired401 Unauthorized
Payload malformed or missing required fields400 Bad Request
Your DB is down or a transient error occurred500 Internal Server Error (triggers re-delivery)
Permanent business logic failure (e.g. contact not found)204 — record as “handled with error” to avoid infinite retries

Complete working implementations

import express, { Request, Response } from 'express'
import { Pool } from 'pg'
import crypto from 'crypto'

const app = express()
const db = new Pool({ connectionString: process.env.DATABASE_URL })
const WEBHOOK_SECRET = process.env.TRUEDY_WEBHOOK_SECRET!

// Use raw body middleware to preserve bytes for HMAC verification
app.post('/webhooks/truedy', express.raw({ type: 'application/json' }), async (req: Request, res: Response) => {
  const rawBody = req.body as Buffer
  const timestamp = req.headers['x-truedy-timestamp'] as string ?? ''
  const signature = req.headers['x-truedy-signature'] as string ?? ''

  // Step 1: Verify signature
  if (!verifySignature(rawBody, timestamp, signature)) {
    return res.status(401).json({ error: 'Invalid signature' })
  }

  let envelope: Record<string, unknown>
  try {
    envelope = JSON.parse(rawBody.toString('utf8'))
  } catch {
    return res.status(400).json({ error: 'Invalid JSON' })
  }

  const eventType = envelope.event as string
  const dedupeKey = extractDedupeKey(envelope)

  if (!dedupeKey) {
    console.error('Missing dedupe key for event:', eventType, JSON.stringify(envelope).slice(0, 200))
    return res.status(400).json({ error: 'Missing dedupe key' })
  }

  const recordKey = `${eventType}:${dedupeKey}`

  // Step 2: Idempotency check + side effects in a single transaction
  const client = await db.connect()
  try {
    await client.query('BEGIN')

    const existing = await client.query(
      'SELECT 1 FROM webhook_events WHERE dedupe_key = $1',
      [recordKey]
    )
    if (existing.rowCount && existing.rowCount > 0) {
      await client.query('ROLLBACK')
      return res.status(204).send() // Already processed — safe to ignore
    }

    // Apply your business side effects here
    await handleEvent(client, eventType, envelope)

    // Record dedupe key atomically with side effects
    await client.query(
      'INSERT INTO webhook_events (dedupe_key, event_type, processed_at) VALUES ($1, $2, NOW())',
      [recordKey, eventType]
    )

    await client.query('COMMIT')
  } catch (err) {
    await client.query('ROLLBACK')
    console.error('Webhook processing failed:', err)
    return res.status(500).json({ error: 'Processing failed' }) // Truedy will retry
  } finally {
    client.release()
  }

  res.status(204).send()
})

function verifySignature(rawBody: Buffer, timestamp: string, signature: string): boolean {
  const now = Math.floor(Date.now() / 1000)
  const ts = parseInt(timestamp, 10)
  if (isNaN(ts) || now - ts > 300 || ts > now + 60) return false

  const message = `${timestamp}.${rawBody.toString('utf8')}`
  const expected = crypto.createHmac('sha256', WEBHOOK_SECRET).update(message).digest('hex')

  if (expected.length !== signature.length) return false
  return crypto.timingSafeEqual(Buffer.from(expected, 'hex'), Buffer.from(signature, 'hex'))
}

function extractDedupeKey(envelope: Record<string, unknown>): string | null {
  const data = envelope.data as Record<string, unknown> | undefined
  if (!data) return null

  // Check data.id / data.eventId first
  if (typeof data.id === 'string') return data.id
  if (typeof data.eventId === 'string') return data.eventId

  // Call events
  const call = data.call as Record<string, unknown> | undefined
  const callId = call?.callId ?? call?.call_id ?? data.call_id ?? data.callId
  if (typeof callId === 'string') return callId

  // Batch events
  const batchId = data.batch_id ?? data.batchId
  if (typeof batchId === 'string') return batchId as string

  // Voice events
  const voiceId = data.voice_id ?? data.voiceId
  if (typeof voiceId === 'string') return voiceId as string

  return null
}

async function handleEvent(client: unknown, eventType: string, envelope: Record<string, unknown>) {
  const data = envelope.data as Record<string, unknown>
  switch (eventType) {
    case 'call.ended': {
      const call = data.call as Record<string, unknown> | undefined
      console.log('Call ended:', call?.callId, '— duration:', call?.duration, 's')
      // e.g. update your CRM, store transcript
      break
    }
    case 'batch.completed': {
      console.log('Batch completed:', data.batch_id ?? data.batchId)
      // e.g. send summary email to campaign owner
      break
    }
    default:
      console.log('Unhandled event type:', eventType)
  }
}

Database table for idempotency tracking

CREATE TABLE webhook_events (
  dedupe_key   TEXT PRIMARY KEY,
  event_type   TEXT NOT NULL,
  processed_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

-- Optional: purge old records after 30 days to keep the table small
CREATE INDEX ON webhook_events (processed_at);

Operational runbook

Signature verification fails

  • Log X-Truedy-Timestamp and event (if parseable). Do not log the full secret.
  • Return 401 and stop.
  • Check: are you reading the raw body before JSON parsing? Body parsers can normalize whitespace and break HMAC.

Payload parse error / schema mismatch

  • Log the parsing error and a redacted payload snapshot.
  • Return 400.

Transient processing failure (DB down, downstream API timeout)

  • Return 500 — Truedy will re-deliver the event.
  • Ensure your handler checks the dedupe table at the start so the retry doesn’t double-process.

Permanent business logic failure

  • Return 204 after recording the event as “failed but handled” so the retry loop doesn’t run forever.
  • Alert your on-call team separately.

Next steps

Securing Webhooks

Full HMAC verification code examples

Available Webhooks

Event catalogue and all payload shapes

Idempotency

Platform-wide idempotency patterns