Improving RPC reliability for Web3 dApps requires layered defenses: a multi-provider failover setup, request-level retry logic with exponential backoff, real-time monitoring of latency and error rates, infrastructure with isolated capacity. Teams running production dApps should target sub-200ms p99 latency on the primary endpoint and 99.9% endpoint availability across the failover chain.
If your dApp is on the public internet, it has already been paged at 3am because of an RPC issue. Or it will be. This guide is the framework for making that page never happen again.
The reliability patterns below come from running production RPC infrastructure across 24 networks for teams whose workloads range from low millions to enterprise-scale daily requests. Operational experience, not theory.
What does RPC reliability actually mean for a dApp?
RPC reliability is the combination of four measurable properties: availability (does the endpoint respond), latency (how fast), correctness (does it return the right data), consistency (does it return the same data on subsequent calls). A reliable RPC layer hits target thresholds on all four under load, during outages, across chain reorganizations.
Most teams optimize for availability and forget the other three. That works until a single slow endpoint cascades into a stuck UI, a frontend that shows stale balances, or a transaction that gets replayed because the read-after-write returned an outdated state.
The four reliability dimensions
| Dimension | What it measures | Production target |
|---|---|---|
| Availability | % of requests that get any response | 99.9% across the failover chain |
| Latency | p50, p95, p99 response time | p99 under 200ms for reads, under 500ms for writes |
| Correctness | Returned data matches chain state | 100% (assume any deviation is a hostile node) |
| Consistency | Repeated reads return the same data once finalized | Match commitment level (finalized for Solana, 12 confirmations for Ethereum mainnet) |
Set targets for all four. Track all four. Alert on all four.
How can I make my dApp’s RPC infrastructure more reliable?
Reliable dApp RPC infrastructure runs on five layers: request optimization to reduce load, multi-provider failover to survive single-endpoint outages, retry logic with exponential backoff to absorb transient errors, real-time monitoring to detect degradation before users notice, isolated infrastructure (dedicated nodes or enterprise-tier capacity) to remove noisy-neighbor effects.
Adding any one layer helps. Adding all five is how you stop getting paged.
The 5-layer reliability framework
┌──────────────────────────────────────────────────┐
│ Layer 5: Isolated infrastructure │
│ (dedicated nodes, enterprise capacity, ISO) │
├──────────────────────────────────────────────────┤
│ Layer 4: Real-time monitoring + alerting │
│ (success rate, p99 latency, error breakdown) │
├──────────────────────────────────────────────────┤
│ Layer 3: Retry logic │
│ (exponential backoff, jitter, idempotency) │
├──────────────────────────────────────────────────┤
│ Layer 2: Multi-provider failover │
│ (primary, secondary, tertiary endpoints) │
├──────────────────────────────────────────────────┤
│ Layer 1: Request optimization │
│ (batching, caching, correct method choice) │
└──────────────────────────────────────────────────┘Each layer compensates for the failure mode below it. Request optimization reduces the chance of hitting a rate limit. Failover compensates when one provider goes down. Retries compensate for transient network blips. Monitoring catches the failures you missed. Isolated infrastructure removes the shared-resource problem entirely.
The multi-provider failover pattern
A reliable RPC layer has two tiers of failover. Platform-level failover runs inside the provider, automatically routing your requests across globally distributed nodes when one region degrades. Application-level failover runs in your code, switching between providers when an entire provider has a catastrophic event no single vendor can prevent. Production dApps should have both.
BoltRPC provides globally distributed infrastructure with automatic failover at the platform level. Requests route across multiple nodes transparently, so most outage scenarios never reach your application. The application-level multi-provider pattern below is the defense-in-depth layer for the rare regional or cross-provider events.
Reference architecture
[ dApp ]
│
▼
[ RPC client wrapper ]
│
├─► [ BoltRPC primary ] ◄─ default route
│ (globally distributed, platform failover built-in)
└─► [ Backup provider ] ◄─ on BoltRPC catastrophic event
Each endpoint health-checked every 10 seconds.
On 3 consecutive failures, mark dead for 60 seconds.
Hyperscale or regulated teams sometimes add a third provider.
For most production dApps, two providers is the standard.Code example: viem failover wrapper
import { createPublicClient, http, fallback } from 'viem'
import { mainnet } from 'viem/chains'
const client = createPublicClient({
chain: mainnet,
transport: fallback([
// BoltRPC primary (globally distributed with platform-level failover)
http('https://eu.endpoints.matrixed.link/rpc/ethereum?auth=YOUR_KEY'),
// Backup provider for catastrophic-event defense-in-depth
http('https://backup-rpc.example.com/rpc/ethereum?auth=YOUR_KEY'),
], {
rank: {
interval: 60_000,
sampleCount: 10,
timeout: 1_000,
weights: { latency: 0.3, stability: 0.7 },
},
}),
})The rank block tells viem to continuously sample each endpoint and prefer the fastest stable one. This is automatic load balancing on top of failover.
Code example: ethers.js v6 FallbackProvider
import { FallbackProvider, JsonRpcProvider } from 'ethers'
// BoltRPC primary + secondary provider for defense-in-depth
const provider = new FallbackProvider([
{
provider: new JsonRpcProvider('https://eu.endpoints.matrixed.link/rpc/ethereum?auth=YOUR_KEY'),
priority: 1,
weight: 2,
stallTimeout: 1500,
},
{
provider: new JsonRpcProvider('https://backup-rpc.example.com/rpc/ethereum?auth=YOUR_KEY'),
priority: 2,
weight: 1,
stallTimeout: 1500,
},
])The stallTimeout is the time after which ethers gives up on a primary and tries the next. Set it tighter than the user’s perceived “slow” threshold. 1500ms is a sensible default for read operations.
Code example: web3.py multi-provider
from web3 import Web3, HTTPProvider
from web3.middleware import ExtraDataToPOAMiddleware
class FailoverWeb3:
def __init__(self, endpoints: list[str]):
self.providers = [Web3(HTTPProvider(url, request_kwargs={'timeout': 2}))
for url in endpoints]
def call(self, method: str, params: list):
for i, w3 in enumerate(self.providers):
try:
fn = getattr(w3.eth, method)
return fn(*params)
except Exception as e:
if i == len(self.providers) - 1:
raise
continue
# BoltRPC primary + secondary provider for defense-in-depth
w3 = FailoverWeb3([
'https://eu.endpoints.matrixed.link/rpc/ethereum?auth=YOUR_KEY',
'https://backup-rpc.example.com/rpc/ethereum?auth=YOUR_KEY',
])The timeout=2 is critical. Without a tight timeout, a hanging endpoint will block the request for the default 30 seconds and your dApp will appear frozen to the user.
For a deeper failover deep-dive, see our guide on RPC failover architecture.
Retry logic that does not make things worse
Retry logic should use exponential backoff with jitter, cap total wait time, never retry on errors that signal a deterministic failure. Naive retries on a rate-limit response (HTTP 429) amplify the problem and get your API key suspended. Naive retries on a malformed request waste time and confuse the upstream provider.
What to retry, what not to retry
| Response | Action |
|---|---|
| HTTP 429 (rate limit) | Retry with backoff. Read the Retry-After header if present. |
| HTTP 500, 502, 503, 504 | Retry with backoff. Provider is having a moment. |
| HTTP 400, 401, 403 | Do not retry. Your request is wrong or auth is bad. |
| Connection timeout | Retry with backoff up to 3 times, then fail to next provider. |
| Connection reset | Retry once immediately, then with backoff. |
| Valid JSON-RPC error code | Depends on error. -32603 (internal error) retry. -32600 (invalid request) do not. |
Code example: retry with exponential backoff in TypeScript
async function rpcWithRetry<T>(
fn: () => Promise<T>,
options = { maxAttempts: 3, baseDelay: 200, maxDelay: 5000 }
): Promise<T> {
let lastError: Error | null = null
for (let attempt = 0; attempt < options.maxAttempts; attempt++) {
try {
return await fn()
} catch (err: any) {
lastError = err
if (!isRetryable(err)) throw err
const delay = Math.min(
options.baseDelay * 2 ** attempt,
options.maxDelay,
)
const jitter = delay * Math.random() * 0.3
await new Promise((r) => setTimeout(r, delay + jitter))
}
}
throw lastError ?? new Error('Retry failed')
}
function isRetryable(err: any): boolean {
const status = err?.status ?? err?.response?.status
if (status === 429) return true
if (status >= 500 && status < 600) return true
if (err?.code === 'ECONNRESET' || err?.code === 'ETIMEDOUT') return true
return false
}Jitter is what stops a thundering herd of retries from hammering an already-struggling endpoint at exactly the same moment. Always add jitter.
For deeper coverage of error handling, see our guide on handling RPC errors in Web3.
Monitoring RPC reliability in production
Effective RPC monitoring tracks four metrics per endpoint and chain: success rate (target above 99.9%), latency distribution (p50, p95, p99), error breakdown by type, provider attribution (which endpoint served which request). Alerts fire on success rate drops below 99% over a 5-minute window or p99 latency exceeding 500ms.
What to measure
| Metric | Target | Alert threshold |
|---|---|---|
| Success rate (HTTP 2xx + valid JSON-RPC response) | >99.9% | <99% over 5min |
| p50 latency | <50ms reads, <200ms writes | n/a (track trend) |
| p95 latency | <200ms reads, <500ms writes | >500ms over 5min |
| p99 latency | <500ms reads, <1000ms writes | >1500ms over 5min |
| Error rate by type | n/a | Any 4xx surge above baseline |
| Failover trigger count | <1/hour | >5/hour (your primary is degrading) |
Code example: lightweight observability wrapper
type RpcMetric = {
chain: string
method: string
endpoint: string
durationMs: number
status: 'success' | 'error' | 'timeout'
errorCode?: string | number
}
async function measuredCall(
client: any,
chain: string,
method: string,
params: any[],
endpoint: string,
metricSink: (m: RpcMetric) => void,
) {
const start = performance.now()
try {
const result = await client.request({ method, params })
metricSink({
chain, method, endpoint,
durationMs: performance.now() - start,
status: 'success',
})
return result
} catch (err: any) {
metricSink({
chain, method, endpoint,
durationMs: performance.now() - start,
status: err.code === 'TIMEOUT' ? 'timeout' : 'error',
errorCode: err.code ?? err.status,
})
throw err
}
}Pipe metricSink into Datadog, Grafana, Honeycomb, or your in-house metrics stack. Tag every metric by chain and endpoint so you can compare provider performance per network.
SLI and SLO targets
Service Level Indicators (SLIs) measure something. Service Level Objectives (SLOs) commit to a target. For dApp RPC, sensible production SLOs look like this:
- 99.9% of RPC reads return a valid response within 500ms (rolling 30-day window)
- 99.5% of RPC writes (eth_sendRawTransaction or equivalent) submit within 2 seconds
- 99.9% of finalized state reads return consistent data
Track these in a dashboard. Page on-call only when the SLO is at risk of breaking, not on individual blips.
Choosing infrastructure that is actually reliable
Reliable RPC infrastructure has three properties: isolated capacity (so your traffic does not compete with other tenants), geographic distribution (so a regional outage does not take you down), predictable performance (no credit-weighted billing that makes critical methods 75x more expensive than basic ones). Shared-tier endpoints from public providers can hit two of these. Dedicated capacity or enterprise plans hit all three.
Shared vs dedicated tradeoffs
Shared RPC plans pool customers behind common infrastructure. They are cheap, fast to set up, fine for most workloads under 30 requests per second. Above that threshold, or when tail latency starts mattering, the noisy-neighbor problem becomes real. A neighbor running a 24/7 analytics job can spike your p99 latency without you knowing why.
For a full decision framework, see our guide on shared vs dedicated RPC nodes.
Compliance signals as reliability proxies
Compliance certifications are an underrated reliability signal. ISO 27001 certified infrastructure means the provider has audited processes for incident response, capacity planning, operational continuity. The certification does not magically prevent outages, but it commits the provider to recovery procedures most uncertified providers do not have to follow.
BoltRPC operates infrastructure with 2 billion daily request capacity across 24 production networks, accessible through a single API key, operated by an ISO/IEC 27001:2022 certified team. Flat-monthly pricing with published per-method RU weights (reads = 3 RU, eth_getLogs = 4 RU, writes = 5 RU, debug_* = 6 RU) keeps variance bounded. Heavy methods still cost more than cheap ones, but 33% more instead of 7500% more on opaque credit-weighted plans. Your monthly bill is forecastable before you ship.
Geographic distribution
If your dApp serves users globally, your RPC layer should be globally distributed with automatic failover. A primary endpoint in one region plus a secondary in another region covers most outage scenarios. Three regions covers anything short of an internet-wide event.
Compare your endpoint’s response time from at least three geographic vantage points. AWS regions like us-east-1, eu-west-1, ap-southeast-1 are good test sites because they map roughly to your user distribution.
Should I run my own RPC node?
Most dApp teams should not run their own RPC node. Running production-grade nodes requires expertise in chain-specific operational quirks, capacity planning for state growth, monitoring infrastructure, 24/7 on-call coverage. The total cost of ownership for a single well-operated node typically exceeds enterprise-tier RPC plans, the lead time to fix a deep operational issue is days, not minutes.
When it makes sense
- You have sustained workloads above 200 requests per second on a single chain
- You need custom RPC methods or non-standard configurations
- Sovereignty over your node operation is a regulatory or strategic requirement
- You already have an in-house infrastructure team with chain expertise
When it does not
- You operate across multiple chains (running 5 nodes is 5x the operational burden)
- Your team’s primary expertise is product or smart contracts, not infrastructure
- Your workload is bursty rather than sustained (you pay for peak capacity 24/7)
- Latency-sensitive workloads where regional distribution matters (you would need nodes in multiple regions)
The cost reality
A single well-operated Ethereum archive node runs roughly $400 to $800 per month in cloud costs alone, before salaries for the ops engineer. Multiply by the number of chains. Compare against a multi-chain enterprise-tier plan from a managed provider that covers all your networks under one contract, with ISO 27001 certified operations and a dedicated team member on call. Self-hosting becomes attractive only at very specific operational scales.
The RPC reliability checklist
Use this as the pre-production review for any dApp:
- Multi-provider failover configured with at least 2 endpoints, primary plus secondary
- Health checks running on a 10-30 second interval, marking endpoints dead after 3 consecutive failures
- Retry logic with exponential backoff and jitter, capped at 3 attempts
- Non-retryable errors (400, 401, 403) explicitly excluded from retry paths
- Request timeouts set to 2 seconds for reads, 5 seconds for writes
- Success rate monitoring with alert at 99% over 5 minutes
- p99 latency monitoring with alert at 1500ms over 5 minutes
- Per-endpoint attribution in metrics (so you know which provider failed)
- Idempotency keys on write paths where the protocol allows
- Documented runbook for “primary RPC is down” with the exact failover steps
If you cannot tick all 10, you have known reliability gaps.
FAQ
How do I monitor RPC reliability in production?
Track success rate, latency distribution (p50, p95, p99), error breakdown by type, per-endpoint attribution. Pipe metrics into Datadog, Grafana, or Honeycomb. Alert on success rate dropping below 99% over a 5-minute window or p99 latency exceeding 1500ms.
What is a good p99 latency target for blockchain RPC?
For Ethereum and EVM mainnet reads, target p99 under 200ms on your primary endpoint and under 500ms across the failover chain. Solana reads should target under 100ms p99 because the chain produces blocks every 400ms. Write operations get more relaxed targets, typically 1000ms p99 for reads and 2000ms for writes.
How many RPC providers should a production dApp use?
Two providers as a minimum, three providers as the production standard. One provider is a single point of failure. Two providers covers single-provider outages but not correlated regional incidents. Three providers spread across different infrastructures eliminates almost all common failure modes.
Do I need a dedicated RPC node for reliability?
Not for most workloads. Multi-provider failover on shared tiers handles reliability for the majority of dApps. Dedicated nodes become necessary when sustained throughput exceeds 30 requests per second, when tail latency spikes start breaking time-sensitive operations, or when compliance requires isolated infrastructure. See our shared vs dedicated RPC nodes guide for the decision framework.
How does ISO 27001 certification affect RPC reliability?
ISO 27001 certification commits the provider to audited processes for incident response, capacity planning, change management, operational continuity. It does not prevent outages but it signals operational maturity. For enterprise dApps with compliance requirements, ISO 27001 certified infrastructure is often a procurement prerequisite.
Production-ready RPC starts with the right foundation
Reliability is not a feature you bolt on. It is the result of stacking five independent layers: request optimization, multi-provider failover, retry logic, monitoring, isolated infrastructure. Skip any layer and you build the page that wakes someone up at 3am.
BoltRPC provides the infrastructure layer of this stack: ISO 27001 certified team, 24 production networks, flat-monthly pricing with published per-method RU weights (no opaque credit multipliers), single API key. Start with a free 14-day trial at trial.boltrpc.io. No credit card required.
