Docs/Advanced/Webhooks

Webhooks

Receive real-time notifications for agent events, budget alerts, and provider status changes via webhook delivery.


Webhooks

Keystore can send real-time HTTP notifications to your endpoints when important events occur. Use webhooks to build alerting pipelines, trigger automations, or keep external systems in sync.

Supported Events

EventDescription
agent.createdA new agent was created in your organization
agent.deletedAn agent was permanently deleted
agent.budget_exceededAn agent's monthly spend has reached its budget cap
provider.circuit_openedA provider's circuit breaker tripped due to high error rates
provider.circuit_closedA provider recovered and the circuit breaker closed
abuse.detectedThe proxy detected potentially abusive behavior from an agent
usage.thresholdAn agent crossed a configurable usage threshold

Creating a Webhook Endpoint

Via API

bash
1
2
3
4
5
6
7
curl -X POST https://api.keystore.io/v1/webhooks \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://your-app.com/webhooks/keystore",
    "events": ["agent.budget_exceeded", "provider.circuit_opened", "abuse.detected"]
  }'

Response:

json
1
2
3
4
5
6
7
8
9
{
  "endpoint": {
    "id": "550e8400-e29b-41d4-a716-446655440000",
    "url": "https://your-app.com/webhooks/keystore",
    "events": ["agent.budget_exceeded", "provider.circuit_opened", "abuse.detected"],
    "status": "active",
    "secret": "whsec_a1b2c3d4e5f6..."
  }
}

Save the secret value -- it is used to verify webhook signatures and is only returned at creation time.

Via Dashboard

Navigate to Settings > Webhooks > Add Endpoint. Select the events you want to subscribe to and enter your endpoint URL. The signing secret is displayed once after creation.

Webhook Delivery

When an event is triggered, Keystore enqueues a delivery and sends an HTTP POST to your endpoint with the following headers:

http
1
2
3
4
POST /webhooks/keystore HTTP/1.1
Content-Type: application/json
X-Webhook-Signature: <HMAC-SHA256 signature>
X-Webhook-Event: agent.budget_exceeded

Payload Structure

json
1
2
3
4
5
6
7
8
9
10
11
{
  "event": "agent.budget_exceeded",
  "timestamp": "2026-03-06T12:00:00.000Z",
  "data": {
    "agentId": "550e8400-e29b-41d4-a716-446655440000",
    "agentName": "my-agent",
    "budgetCentsMonthly": 5000,
    "spentCents": 5012,
    "orgId": "org-id-here"
  }
}

The exact fields in data vary by event type, but event and timestamp are always present.

Verifying Signatures

Every webhook delivery is signed with your endpoint's secret using HMAC-SHA256. Verify the signature before processing the payload to ensure it came from Keystore.

typescript
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
import { createHmac } from "crypto";

function verifyWebhook(body: string, signature: string, secret: string): boolean {
  const expected = createHmac("sha256", secret)
    .update(body)
    .digest("hex");
  return signature === expected;
}

// In your webhook handler:
app.post("/webhooks/keystore", (req, res) => {
  const signature = req.headers["x-webhook-signature"];
  const event = req.headers["x-webhook-event"];
  const body = req.rawBody; // raw string, not parsed JSON

  if (!verifyWebhook(body, signature, process.env.WEBHOOK_SECRET)) {
    return res.status(401).send("Invalid signature");
  }

  const payload = JSON.parse(body);
  console.log(`Received ${event}:`, payload);

  res.status(200).send("OK");
});
python
1
2
3
4
5
6
7
8
import hmac
import hashlib

def verify_webhook(body: bytes, signature: str, secret: str) -> bool:
    expected = hmac.new(
        secret.encode(), body, hashlib.sha256
    ).hexdigest()
    return hmac.compare_digest(signature, expected)

Retry Policy

Failed deliveries are retried up to 3 times with exponential backoff:

AttemptDelay
1st retry10 seconds
2nd retry60 seconds
3rd retry300 seconds (5 minutes)

A delivery is considered failed if:

  • Your endpoint returns a non-2xx HTTP status code.
  • The HTTP request times out or encounters a network error.

After all retries are exhausted, the delivery is marked as failed permanently.

Delivery Statuses

StatusDescription
pendingQueued for delivery or waiting for retry
successDelivered successfully (2xx response)
failedAll retry attempts exhausted

Viewing Delivery Logs

Via API

bash
1
2
3
4
5
6
7
# List deliveries for an endpoint
curl https://api.keystore.io/v1/webhooks/<endpoint-id>/deliveries \
  -H "Authorization: Bearer $TOKEN"

# Filter by status
curl "https://api.keystore.io/v1/webhooks/<endpoint-id>/deliveries?status=failed" \
  -H "Authorization: Bearer $TOKEN"

Response:

json
1
2
3
4
5
6
7
8
9
10
11
12
13
14
{
  "deliveries": [
    {
      "id": "delivery-id",
      "event": "agent.budget_exceeded",
      "status": "success",
      "statusCode": "200",
      "attempts": 1,
      "lastAttemptAt": "2026-03-06T12:00:01.000Z",
      "createdAt": "2026-03-06T12:00:00.000Z"
    }
  ],
  "total": 1
}

Via Dashboard

Navigate to Settings > Webhooks > [Endpoint] > Deliveries to see a table of all deliveries with status, response code, and timestamps.

Manual Retry

If a delivery failed and the underlying issue has been fixed, you can manually re-enqueue it:

bash
1
2
curl -X POST https://api.keystore.io/v1/webhooks/deliveries/<delivery-id>/retry \
  -H "Authorization: Bearer $TOKEN"

This resets the attempt counter and re-queues the delivery for immediate processing.

Managing Endpoints

Update an Endpoint

Change the URL, subscribed events, or status:

bash
1
2
3
4
5
6
7
curl -X PATCH https://api.keystore.io/v1/webhooks/<endpoint-id> \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "events": ["agent.budget_exceeded", "agent.deleted"],
    "status": "active"
  }'

Setting status to "paused" temporarily stops deliveries without deleting the endpoint.

Delete an Endpoint

bash
1
2
curl -X DELETE https://api.keystore.io/v1/webhooks/<endpoint-id> \
  -H "Authorization: Bearer $TOKEN"

Deleting an endpoint also removes all associated delivery records.

Best Practices

  1. Always verify signatures. Never trust a webhook payload without checking the X-Webhook-Signature header against your secret.
  2. Respond quickly. Return a 200 status as soon as you have received and acknowledged the payload. Process the event asynchronously if it requires heavy work.
  3. Handle duplicates. In rare cases (network retries, infrastructure issues), you may receive the same event more than once. Use the delivery ID or event data to deduplicate.
  4. Monitor failed deliveries. Check the delivery logs regularly or set up alerts on repeated failures. Keystore does not retry beyond the 3-attempt policy.
  5. Use HTTPS endpoints. Webhook payloads may contain sensitive metadata. Always use HTTPS URLs.