How does GitHub API rate limiting work, and what strategies do you use for efficient pagination and avoiding rate limit exhaustion in large-scale automation?

Imagine you are at an all-you-can-eat buffet with a twist: you get exactly 5,000 plate refills per hour, and the kitchen tracks your remaining servings on a display board. If you waste plates by grabbing food you already have (uncached requests) or by taking one item per plate instead of loading up (non-batched queries), you will hit your limit before dessert. Smart diners check the display board (rate limit headers), reuse plates when the food has not changed (conditional requests), and stack multiple items per trip (GraphQL batching).

GitHub's REST API returns three critical headers with every response: X-RateLimit-Limit (your ceiling, typically 5,000), X-RateLimit-Remaining (requests left in the current window), and X-RateLimit-Reset (Unix timestamp when the window resets). When remaining hits zero, the API returns 403 with a 'rate limit exceeded' message. The search API has its own stricter limit: 30 requests per minute for authenticated users. Secondary rate limits (formerly abuse limits) are less documented but enforce concurrency constraints—no more than 100 concurrent requests, and algorithmic detection flags clients making too many requests to a single endpoint in a short period.

Conditional requests are the single most impactful optimization. By sending the ETag from a previous response as an 'If-None-Match' header, or a date as 'If-Modified-Since', you get a 304 Not Modified response when data has not changed—and this response does NOT count against your rate limit. For monitoring scenarios like polling for new pull requests on payments-api, conditional requests can reduce effective API consumption by 80-90 percent. The GitHub CLI 'gh api' command supports this natively with the '--cache' flag.

Pagination strategy matters enormously for data-heavy operations. REST API responses are paginated at 30 items by default (max 100 via '?per_page=100'), with Link headers providing next, prev, first, and last URLs. Always set per_page=100 to minimize requests. For sequential page traversal, follow the 'next' Link header rather than constructing page numbers manually, as items can shift between pages during concurrent modifications. The GraphQL API uses cursor-based pagination with 'after' and 'first' parameters, which is more reliable than offset pagination. GraphQL also solves the N+1 query problem: instead of making one REST call per pull request to fetch reviews, a single GraphQL query can fetch 100 PRs with their reviews, check runs, and labels in one request consuming roughly 100-200 points from the 5,000-point hourly GraphQL budget.

In production, a large-scale automation system processing 50 repositories in the acme-corp organization should implement a rate-limit-aware HTTP client. Before each request, check X-RateLimit-Remaining; if below a threshold (say 100), sleep until X-RateLimit-Reset. Implement exponential backoff with jitter for 429 and 403 responses. Use a request queue that serializes calls to rate-limited endpoints while allowing parallelism across non-rate-limited ones. For webhook-based architectures, prefer receiving push-based events rather than polling—a single webhook subscription replaces hundreds of polling API calls.

The most dangerous gotcha is the secondary rate limit, which has no published threshold and is enforced algorithmically. Making rapid parallel requests to the same endpoint, even well within the primary rate limit, can trigger a secondary limit response with a 'Retry-After' header. The fix is to add a 1-second delay between mutating requests (POST, PATCH, DELETE) and limit concurrent requests to any single endpoint. Another subtle issue: GitHub App installation tokens and user tokens have separate rate limit pools, but requests from the same IP address can still trigger secondary limits regardless of authentication method.