How to Fix CircuitBreaker Webhook Failures?

What Does "CircuitBreaker Is Open" Mean?

Piano uses a circuit breaker mechanism to protect both your infrastructure and Piano's shared webhook delivery system. When Piano detects that recent webhook deliveries to your endpoint are failing frequently, it temporarily pauses sending to that endpoint. During this pause, Piano records placeholder responses containing "CircuitBreaker is open" in the response header and body.

The typical pause duration ranges from approximately 30 seconds up to 1 hour, depending on the volume and frequency of failures. After the pause, Piano will attempt delivery again. If failures continue, the circuit breaker will re-engage.

For information on webhook queuing, blocking, skipping, and resending, see the Webhooks documentation.

How Piano Determines Webhook Success vs. Failure

A webhook delivery is considered successful only when your endpoint returns:

• HTTP status 200 OK

• Within approximately 7 seconds

Any of the following are treated as a failure and can contribute to triggering the circuit breaker:

• Non-200 responses (e.g., 4xx, 5xx, 429)

• Timeouts (no 200 received within the timeout window)

• Connectivity errors (e.g., TLS/SSL handshake failures, DNS resolution failures, firewall or network blocks)

• Redirect responses such as 301 or 302 — Piano does not follow redirects

Common Triggers for Circuit Breaker Piano Subscriptions

The circuit breaker is most commonly triggered by:

• Endpoint latency: Responses exceeding the approximately 7-second timeout window.

• Burst overload: High volumes of webhooks that your endpoint cannot absorb (often resulting in 429 Too Many Requests).

• Persistent error responses: Repeated 4xx or 5xx status codes.

• Infrastructure issues: Network, TLS, DNS, or firewall problems preventing Piano from reaching your endpoint.

Retry Behavior Before and After Circuit Breaker

When a delivery fails, Piano retries the same webhook using an exponential backoff schedule:

A webhook can be retried up to 237 attempts. If failures persist and the circuit breaker engages during this process, retries are paused for the duration of the circuit breaker window and then resume. If the endpoint continues to fail long enough to reach the deactivation thresholds (see this Section), the endpoint will be disabled automatically.

Automatic Deactivation

If circuit breaker failures continue unresolved, Piano can automatically deactivate the webhook endpoint when either of the following conditions is met:

• The endpoint fails to return HTTP 200 for approximately 7 consecutive days, or

• The endpoint accumulates 237 failed attempts

These thresholds are fixed and cannot be customized per application or endpoint.

What Happens to Events During Downtime

Piano does not store unsent events for later replay. If your endpoint is disabled or blocked by the circuit breaker, events that occur during that period cannot be regenerated or resent from Piano later.

If you need to backfill missed events, use your own durable logging or queue on receipt, and/or recover state via Piano APIs (e.g., subscription or conversion reporting), depending on your use case.

Email Notifications

Piano sends webhook health notifications to the Notification Email configured in your Business settings. Update this address in Dashboard → Edit Business → Notification Email.

How to Fix Circuit Breaker Failures

Diagnose the Root Cause

Before attempting to recover, identify why the circuit breaker was triggered:

1. Navigate to Manage → Webhooks in the Piano Dashboard.

2. Filter by Failed status.

3. Open each failed item and review the last error (timeout, 4xx/5xx, TLS, redirect, "CircuitBreaker is open", etc.).

4. Cross-reference the failure timestamps with your own endpoint logs to identify the root cause (e.g., a deployment, outage, rate-limiting change, or infrastructure issue).

Fix Common Failure Codes

Once you have resolved the underlying issue, the circuit breaker will automatically close and delivery will resume on the next retry cycle. For information on how to manage queued, blocked, and skipped webhooks, see here.

Reactivate a Deactivated Endpoint

If the circuit breaker failures persisted long enough for the endpoint to be automatically deactivated:

1. Confirm the endpoint is healthy and consistently returns 200 within the timeout window.

2. In Manage → Webhooks, locate the endpoint and toggle it back on, depending on the UI state.

Monitor and Manage via API

For programmatic monitoring, you can use the following API endpoints:

• List webhook endpoints and statuses: /publisher/webhook/list

• Resend a failed webhook response: /publisher/webhook/response/resend

To clear a backlog programmatically, build a workflow that:

1. Calls /publisher/webhook/list to identify endpoints and responses in failing states.

2. Loops through failed responses and resends each one using /publisher/webhook/response/resend.

Endpoint Best Practices to Prevent Circuit Breaker Failures

• Respond immediately: Return 200 OK as soon as you have validated receipt, then enqueue the work for asynchronous processing. This prevents long processing times from causing timeouts.

• Handle bursts: Ensure your infrastructure can absorb parallel webhook deliveries. Enable keep-alive connections, set reasonable keep-alive timeouts (often 10–30 seconds), and provision adequate worker capacity.

• Use the final URL: Validate the webhook URL carefully before enabling; avoid redirects, typos, and unreachable hosts.

• Whitelist Piano's outbound IPs if you use IP filtering or firewalls.

• Implement structured logging on your endpoint (timestamp, status code, latency, request ID) so you can quickly diagnose issues when they arise.

• Set up monitoring that alerts if no webhook POST requests are received for a defined period.

• Consider temporarily disabling high-volume webhook types during large bulk operations to avoid bursts that could trigger the circuit breaker.

• If you suspect a platform-wide issue, check the Piano status page: https://status.piano.io/.

Last updated: June 26, 2026