eth_getLogs Pipeline: Index Events at Scale

Build a production-grade eth_getLogs pipeline: chunking, deduplication, reorg handling, retry logic, and storage patterns for indexing events at scale.

eth_getLogs Pipeline: Index Events at Scale

eth_getLogs is the backbone of blockchain data infrastructure. Every indexer, analytics platform, portfolio tracker, and protocol dashboard uses it. A single call returns all events matching a filter across a block range. Powerful, but unreliable without the right patterns around it.

This guide builds a production pipeline: chunked queries, deduplication, reorg handling, retry logic, and storage.


Why eth_getLogs Needs a Pipeline

A raw eth_getLogs call works fine for small block ranges. At production scale, three problems appear:

1. Timeouts: Large block ranges (100,000+ blocks) take too long. Providers time out or cap at a maximum range.

2. Reorgs: A block you already indexed may be replaced by a competing block. Your data becomes wrong.

3. Rate limits: Continuous high-frequency queries drain rate limits and budgets on compute-unit-priced providers.

A proper pipeline handles all three automatically.


The Basic eth_getLogs Call

import { ethers } from 'ethers';

// batchMaxCount: 1 stops ethers batching requests. With batching on, an endpoint that
// refuses eth_getLogs shows up as "missing response for request" instead of its own message.
const provider = new ethers.JsonRpcProvider(
  'https://endpoints.boltrpc.io/rpc/YOUR_API_KEY/ethereum',
  undefined,
  { staticNetwork: true, batchMaxCount: 1 }
);

// Get all Transfer events for USDC in the last 1,000 blocks
const currentBlock = await provider.getBlockNumber();
const transferTopic = ethers.id('Transfer(address,address,uint256)');

const logs = await provider.getLogs({
  address: USDC_ADDRESS,
  topics: [transferTopic],
  fromBlock: currentBlock - 1000,
  toBlock: 'latest',
});

console.log(`Found ${logs.length} Transfer events`);

This works for small ranges. For historical indexing or continuous pipelines, use everything below.


Chunked Queries

There is no standard block range limit. On 24 October 2026 we measured public endpoints on 15 chains: the widest window they accepted on a filter that matches nothing ran from 10 blocks to the whole chain, and many endpoints also cap the number of logs (5,000 to 200,000) before they cap the range. Error codes and messages differ per provider, so a fixed chunk size and a fixed list of error strings will both break. The full measurements and every error string are in eth_getLogs block range limits by chain, measured.

The chunker below adapts instead. It starts at 2,000 blocks and doubles after each success. On an error it reads the message: it throws if the method is off or a key is needed, backs off on a rate limit, and otherwise shrinks the window to the hint in the message or halves it. A range limit is remembered for the rest of the run, a result-count limit is not.

// Everything a provider says about a failure ends up somewhere in the error.
// ethers v6 puts it in different fields depending on the HTTP status.
function bodyMessage(body) {
  try { return JSON.parse(body)?.error?.message; } catch { return undefined; }
}
function errorText(err) {
  return [err?.error?.message, err?.info?.error?.message, bodyMessage(err?.info?.responseBody), err?.info?.responseBody, err?.shortMessage]
    .filter(Boolean)
    .join(' | ');
}

// How big a window does the provider say would have worked?
// Returns a block count, or null when the message carries no hint.
function suggestedWindow(text) {
  // Hex pairs (Linea, MEV Blocker, blastapi):
  //   "Try with this block range [0x1EE7816, 0x1EE908A]"
  //   "this block range should work: [0x190bb90, 0x190bb99]"
  let m = text.match(/\[\s*(0x[0-9a-f]+)\s*,\s*(0x[0-9a-f]+)\s*\]/i);
  if (m) return Number(BigInt(m[2]) - BigInt(m[1])) + 1;
  // Decimal pair: "retry with the range 53027049-53027329"
  m = text.match(/range\s+(\d+)\s*-\s*(\d+)/i);
  if (m) return Number(m[2]) - Number(m[1]) + 1;
  // "narrow to <= 73541 blocks for this filter"
  m = text.match(/narrow to <=\s*(\d+)\s*blocks/i);
  if (m) return Number(m[1]);
  return null;
}

// A stated maximum, when there is one: "limited to a 10,000 range", "exceed maximum block range: 50000"
function statedMax(text) {
  const m = text.match(/(?:limited to a|exceed maximum block range:|must not exceed|up to a|exceeds the maximum of|exceeds limit of)\s*([\d,]+)/i);
  return m ? Number(m[1].replace(/,/g, '')) : null;
}

const FATAL = /method .*not supported|is disabled|archive requests require|specify an address|unauthori[sz]ed|api key/i;
const RATE = /rate.?limit|too many requests|request limit reached|usage limit|429/i;
const SIZE = /more than \d+ (?:results|bytes)|exceeds max results|too many results|logs matched by query exceeds|response (?:size )?(?:too large|exceeded)|query timeout|timed out|timeout/i;

async function getLogsAdaptive(provider, filter, fromBlock, toBlock, opts = {}) {
  const { start = 2000, max = 100_000, onStep = () => {} } = opts;
  const out = [];
  let from = fromBlock;
  let window = start;
  let rangeCeiling = max; // learned from range errors: stays
  let grow = 2;           // doubles until a result-count error, then creeps
  let rateRetries = 0;

  while (from <= toBlock) {
    const to = Math.min(from + window - 1, toBlock);
    try {
      const logs = await provider.getLogs({ ...filter, fromBlock: from, toBlock: to });
      onStep({ from, to, window, ok: true, logs: logs.length });
      out.push(...logs);
      from = to + 1;
      rateRetries = 0;
      window = Math.min(Math.ceil(window * grow), rangeCeiling); // grow after a success
    } catch (err) {
      const text = errorText(err);
      if (FATAL.test(text)) throw new Error(`not retryable: ${text.split(' | ')[0]}`);
      if (RATE.test(text) && !SIZE.test(text)) {
        if (++rateRetries > 5) throw new Error(`rate limited: ${text}`);
        const wait = 500 * 2 ** rateRetries;
        onStep({ from, to, window, ok: false, why: `rate limit, wait ${wait}ms` });
        await new Promise((r) => setTimeout(r, wait));
        continue;
      }
      if (window === 1) throw new Error(`a single block fails, so the endpoint cannot serve this filter: ${text.split(' | ')[0]}`);

      const isResultError = SIZE.test(text);
      // A stated maximum is only a block count on range errors; on result errors it is a log count ("limit of 10000").
      const hint = suggestedWindow(text) ?? (isResultError ? null : statedMax(text));
      // A hint is the biggest window that fit this time; aim 20% under it for result errors, because density moves.
      const target = hint === null ? Math.floor(window / 2) : isResultError ? Math.floor(hint * 0.8) : hint;
      const next = Math.max(1, Math.min(target, window - 1));
      if (isResultError) grow = 1.1; // dense data: stop doubling into the same wall
      if (!isResultError) rangeCeiling = Math.min(rangeCeiling, next); // a range limit does not move with traffic
      onStep({ from, to, window, ok: false, why: text.split(' | ')[0].slice(0, 90), next, hinted: hint !== null });
      window = next;
    }
  }
  return out;
}

// Keeps the call shape used by the indexer and reorg code below.
// It uses the `provider` from the first example.
const getLogsChunked = (filter, fromBlock, toBlock, chunkSize = 2000) =>
  getLogsAdaptive(provider, filter, fromBlock, toBlock, { start: chunkSize });

We ran getLogsAdaptive against Linea, MEV Blocker, publicnode on Base, dRPC, Sonic, Arbitrum One, Monad, Plasma and the BNB Chain dataseeds. It followed hints where the message had one, halved where it had none, and stopped on an endpoint that refuses eth_getLogs. The message at that point is the endpoint’s own only with batchMaxCount: 1 on the provider, as in the first example. With ethers’ default batching on, the same refusals ended as “missing response for request” or “method not available”. The rate limit retry branch was reached once, on mainnet.base.org, but ethers retries a 429 itself, so we stopped that run after about nine minutes. The post eth_getLogs block range limits by chain, measured lists the results and the two bugs the tests caught, and has a web3.py version.

Use it the way the original chunker was used:

// Index 6 months of USDC transfers
const sixMonthsAgo = currentBlock - (6 * 30 * 24 * 300); // ~6 months at 12s/block
const historicalLogs = await getLogsChunked(
  { address: USDC_ADDRESS, topics: [transferTopic] },
  sixMonthsAgo,
  currentBlock
);

Retry with Exponential Backoff

Transient failures (network blips, provider timeouts) are normal. Retry automatically:

async function getLogsWithRetry(filter, maxRetries = 3, baseDelay = 1000) {
  for (let attempt = 0; attempt <= maxRetries; attempt++) {
    try {
      return await provider.getLogs(filter);
    } catch (error) {
      if (attempt === maxRetries) throw error;

      const isRetryable =
        error.message?.includes('timeout') ||
        error.message?.includes('rate limit') ||
        error.code === 'NETWORK_ERROR' ||
        error.code === 'SERVER_ERROR';

      if (!isRetryable) throw error;

      const delay = baseDelay * Math.pow(2, attempt);
      console.warn(`getLogs failed (attempt ${attempt + 1}/${maxRetries + 1}). Retrying in ${delay}ms`);
      await new Promise(resolve => setTimeout(resolve, delay));
    }
  }
}

The Full Indexer Class

class EventIndexer {
  constructor(config) {
    this.provider = config.provider;
    this.wsProvider = config.wsProvider;
    this.filter = config.filter;
    this.chunkSize = config.chunkSize || 2000;
    this.storage = config.storage; // Database adapter
    this.lastIndexedBlock = null;
    this.confirmations = config.confirmations || 12; // Wait for 12 blocks before finalizing
  }

  async start(fromBlock) {
    // Load last indexed block from storage
    this.lastIndexedBlock = await this.storage.getLastIndexedBlock() || fromBlock;

    console.log(`Starting indexer from block ${this.lastIndexedBlock}`);

    // Catch up on historical data
    await this.catchUp();

    // Switch to live mode
    await this.subscribeLive();
  }

  async catchUp() {
    const currentBlock = await this.provider.getBlockNumber();
    const safeBlock = currentBlock - this.confirmations; // Only index confirmed blocks

    if (this.lastIndexedBlock >= safeBlock) {
      console.log('Already caught up');
      return;
    }

    console.log(`Catching up: ${this.lastIndexedBlock} → ${safeBlock}`);

    const logs = await getLogsChunked(
      this.filter,
      this.lastIndexedBlock + 1,
      safeBlock,
      this.chunkSize
    );

    await this.processLogs(logs);
    this.lastIndexedBlock = safeBlock;
    await this.storage.saveLastIndexedBlock(safeBlock);

    console.log(`Caught up to block ${safeBlock}`);
  }

  async subscribeLive() {
    // Use WebSocket for real-time new blocks
    this.wsProvider.on('block', async (blockNumber) => {
      const safeBlock = blockNumber - this.confirmations;

      if (safeBlock <= this.lastIndexedBlock) return;

      // Index the new confirmed blocks
      const logs = await getLogsWithRetry({
        ...this.filter,
        fromBlock: this.lastIndexedBlock + 1,
        toBlock: safeBlock,
      });

      if (logs.length > 0) {
        await this.processLogs(logs);
      }

      this.lastIndexedBlock = safeBlock;
      await this.storage.saveLastIndexedBlock(safeBlock);
    });

    console.log('Live indexing started');
  }

  async processLogs(logs) {
    if (logs.length === 0) return;

    // Deduplicate by transaction hash + log index
    const unique = deduplicateLogs(logs);

    // Parse logs
    const parsed = unique.map(log => this.parseLog(log)).filter(Boolean);

    // Store in batches
    await this.storage.batchInsert(parsed);
    console.log(`Processed ${parsed.length} events`);
  }

  parseLog(log) {
    try {
      const parsed = this.filter.interface?.parseLog(log);
      return {
        blockNumber: log.blockNumber,
        transactionHash: log.transactionHash,
        logIndex: log.index,
        address: log.address,
        eventName: parsed?.name,
        args: parsed?.args,
        raw: log,
      };
    } catch {
      return null;
    }
  }
}

Deduplication

Logs can appear duplicates during reorgs. Deduplicate by the combination of transactionHash + logIndex:

function deduplicateLogs(logs) {
  const seen = new Set();
  return logs.filter(log => {
    const key = `${log.transactionHash}-${log.index}`;
    if (seen.has(key)) return false;
    seen.add(key);
    return true;
  });
}

Reorg Handling

A reorg means blocks you indexed get replaced. Transactions in those blocks may be removed, reordered, or changed. To handle reorgs correctly:

async function handleReorg(provider, storage, safeDepth = 12) {
  const currentBlock = await provider.getBlockNumber();

  // Re-verify the last N blocks
  for (let i = safeDepth; i >= 0; i--) {
    const blockNumber = currentBlock - i;
    const block = await provider.getBlock(blockNumber);
    const storedHash = await storage.getBlockHash(blockNumber);

    if (storedHash && storedHash !== block.hash) {
      console.warn(`Reorg detected at block ${blockNumber}`);

      // Remove all events from the reorged block onwards
      await storage.deleteFromBlock(blockNumber);

      // Re-index from the reorg point
      const logs = await getLogsChunked(filter, blockNumber, currentBlock - safeDepth);
      await storage.batchInsert(logs.map(parseLog));
      await storage.saveBlockHash(blockNumber, block.hash);

      break;
    }
  }
}

Simpler approach: Only index blocks with 12+ confirmations. The this.confirmations = 12 setting in the indexer above does this automatically. Blocks older than 12 confirmations have a statistically negligible reorg probability.


Topic Filtering

Narrow your queries with specific topic filters to reduce result set size:

const transferTopic = ethers.id('Transfer(address,address,uint256)');

// All Transfer events
const allTransfers = { topics: [transferTopic] };

// Only transfers FROM a specific address
const fromAddress = ethers.zeroPadValue('0xYourAddress', 32);
const transfersFrom = { topics: [transferTopic, fromAddress] };

// Only transfers TO a specific address
const toAddress = ethers.zeroPadValue('0xYourAddress', 32);
const transfersTo = { topics: [transferTopic, null, toAddress] };

// Transfers between two specific addresses
const exactTransfer = { topics: [transferTopic, fromAddress, toAddress] };

// Multiple possible senders (OR logic)
const fromEither = {
  topics: [
    transferTopic,
    [
      ethers.zeroPadValue(ADDRESS_A, 32),
      ethers.zeroPadValue(ADDRESS_B, 32),
    ]
  ]
};

Storage Patterns

// Minimal storage adapter interface
class PostgresStorage {
  async getLastIndexedBlock() {
    const result = await db.query(
      'SELECT block_number FROM indexer_state WHERE id = 1'
    );
    return result.rows[0]?.block_number || null;
  }

  async saveLastIndexedBlock(blockNumber) {
    await db.query(
      'INSERT INTO indexer_state (id, block_number) VALUES (1, $1) ON CONFLICT (id) DO UPDATE SET block_number = $1',
      [blockNumber]
    );
  }

  async batchInsert(events) {
    if (events.length === 0) return;

    // Use ON CONFLICT DO NOTHING for idempotency
    const values = events.map((e, i) =>
      `($${i * 5 + 1}, $${i * 5 + 2}, $${i * 5 + 3}, $${i * 5 + 4}, $${i * 5 + 5})`
    ).join(',');

    await db.query(
      `INSERT INTO events (block_number, tx_hash, log_index, event_name, args)
       VALUES ${values}
       ON CONFLICT (tx_hash, log_index) DO NOTHING`,
      events.flatMap(e => [e.blockNumber, e.transactionHash, e.logIndex, e.eventName, JSON.stringify(e.args)])
    );
  }
}

FAQ

What is the maximum block range for eth_getLogs?

It varies by provider and by chain, and there is no standard. On 24 October 2026 we measured public endpoints from a 10-block window to no block cap at all on a filter that matches nothing, and many also cap results at 5,000 to 200,000 logs, which often bites first. See eth_getLogs block range limits by chain, measured. Some endpoints handle much larger ranges. The chunked query pattern in this guide handles any limit automatically.

How do I index from the genesis block?

Use the chunked query pattern with fromBlock: 0. For Ethereum mainnet this means indexing 20+ million blocks, which takes hours to days depending on event frequency. Start with a more recent block if you only need recent history.

Can I run multiple indexers in parallel?

Yes. Shard by block range or by contract address. One indexer handles blocks 0-5,000,000, another handles 5,000,001-10,000,000, etc. Merge results in your storage layer. Ensure your deduplication handles overlapping ranges at shard boundaries.

Is eth_getLogs affected by compute unit pricing?

Yes. On providers using compute unit pricing, eth_getLogs typically carries a significantly higher CU weight than simple reads. Alchemy, for example, lists it at 60 CU per call against 10 for eth_blockNumber (its table as read on 24 October 2026). A continuous indexer running many queries per hour can exhaust plans quickly on CU-priced providers. BoltRPC pricing is straightforward: fixed monthly plans with no surprise bills regardless of how heavily you use eth_getLogs.

How do I handle topics for anonymous events?

Anonymous events have no event signature topic. The first topic slot is empty. Filter by topics: [null, ...] and match by the log’s data field instead. Anonymous events are rare in modern contracts.


BoltRPC provides the eth_getLogs access your indexer needs. Fixed monthly plans mean predictable costs as your indexer scales.

Start your free 14-day trial: dashboard.boltrpc.io

Related: The Graph vs Direct RPC | Optimizing RPC Calls for DeFi

Questions we get about this.

Content is for informational purposes only and does not constitute financial, legal or technical advice. Code examples and configurations are provided as-is. Verify against official documentation and test in your own environment before deploying to production.

More from the blog.

All articles →
Ethereum

Glamsterdam Upgrade: What Changes for RPC Users

Glamsterdam brings ePBS and block-level access lists to Ethereum. What changes in JSON-RPC headers, logs and gas estimates, and where the fork stood 7 Nov 2026.

Nov 10, 2026 · 13 min read
RPC

Public RPC Endpoint Rate Limits, Measured

Free public RPC endpoints rarely publish limits. We measured six official ones on 7 Nov 2026: 429 shapes, batch caps, missing methods and when to use a key.

Nov 10, 2026 · 17 min read
Pricing

Alchemy Alternative: What Switching Really Costs

Switching from Alchemy pays off for standard JSON-RPC at volume. A worked 50M-call bill, what you give up and the migration steps.

Oct 24, 2026 · 18 min read