How Readability Algorithms Extract Article Content

When you subscribe to a blog that only offers a snippet in its RSS feed, you’re left with a choice. Click through to the website and wade through popups, newsletter modals, and cookie banners, or use full text extraction to get clean content directly in your reader. That extraction process relies on readability algorithms, and understanding how they work helps explain why they sometimes nail the content and other times miss the mark entirely.

Mozilla’s Readability library, which powers Firefox’s Reader Mode and many RSS readers, has been around for over a decade. It processes hundreds of millions of pages, so its edge cases have been thoroughly tested by real-world usage. The algorithm takes a straightforward approach to a messy problem. Web pages are full of noise: navigation bars, sidebars, footers, ads, comment sections, and social sharing widgets. The goal is to strip all of that away and leave only the core article content.

The process starts by walking through every paragraph element in the document. Each paragraph gets scored based on several factors. The length of text matters because real article content tends to have substantial paragraphs rather than single sentences. The presence of commas is another signal, since well-written prose typically includes them. The algorithm also looks at the class and ID names of ancestor elements. Classes like “article,” “content,” or “post” boost the score, while classes like “comment,” “sidebar,” or “ad” reduce it. These class name patterns are hard-coded heuristics drawn from observing how publishers structure their HTML across thousands of sites.

Once individual paragraphs are scored, the algorithm propagates those scores up the DOM tree to parent elements. The subtree with the highest cumulative score becomes the chosen article container. Everything outside that subtree gets discarded. The algorithm then performs a cleanup pass, removing empty divs, social widgets, and links that lack surrounding text content. What remains is the extracted article.

This approach works remarkably well for traditional long-form blog posts and news articles. The structure of these pages tends to follow predictable patterns, with the main content sitting in a single dominant container. When you read a 2000-word essay on a personal blog, Readability can reliably separate the essay from the navigation menu and footer copyright notice.

The limitations become apparent when pages deviate from this assumed structure. Readability assumes you’re looking for a single subtree that contains the entire article. Comparison pages with multiple equally-weighted sections confuse the algorithm. A product comparison page might have three columns describing different options, and Readability will pick whichever column happens to score highest, silently dropping the other two. The result looks complete enough that you might not realize content went missing.

Gallery pages and listicles present another challenge. These pages consist of many short items separated by images or embedded media. The paragraph-density heuristic that Readability relies on breaks down because there isn’t a single block of dense text. Each list item might be just a few sentences, so no individual paragraph scores particularly high. The algorithm might extract a fragment or return nothing useful at all.

Forum discussions and comment threads fall outside Readability’s intended use case entirely. The algorithm was designed for articles with a clear main body, not for threaded conversations where every post has equal weight. If you try to extract a forum thread, Readability will likely grab the original post and maybe the first few replies, then abandon the rest. This makes sense for its original purpose but falls short if you’re trying to capture discussion-heavy content.

Single-page applications introduce a different kind of problem. Readability operates on the DOM as it exists at a given moment. If a page loads its content dynamically through JavaScript after the initial HTML render, Readability sees nothing to extract. It cannot trigger JavaScript execution or wait for content to appear. This means any RSS pipeline using Readability must either render the full page with a headless browser first or accept that SPA-rendered content will be incomplete.

Paywalled content creates a more subtle issue. Many news sites render the first couple paragraphs to anonymous visitors before gating the rest behind a subscription. Readability extracts those visible paragraphs faithfully and returns them without any indication that you received a preview rather than the full article. You might read 200 words thinking you have the complete piece, when the actual article runs 1500 words. Detecting this requires comparing the extracted length against metadata from structured data like JSON-LD, which adds complexity beyond what the core algorithm handles.

Infinite-scroll pages face a similar partial-content problem. If a page only renders its first chunk of content in the initial HTML and lazy-loads more as you scroll, Readability only sees what’s already in the DOM. This isn’t a bug in the extraction logic. The content genuinely isn’t there yet. A robust extraction pipeline needs to either wait for JavaScript hydration to complete or gracefully handle the reality that you’ll only get partial results.

Despite these limitations, readability algorithms remain the most practical approach for most RSS use cases. The alternative would be maintaining custom extraction rules for thousands of individual websites, which becomes unsustainable as sites change their layouts. A heuristic-based approach that works well on 80 percent of pages is more valuable than a perfect solution that only works on 20 percent.

Some extraction tools take a different approach. Trafilatura, for instance, uses more complex heuristics and can handle certain edge cases that Readability misses. Machine learning-based tools like Dragnet analyze block-level features to classify content versus chrome. These approaches can outperform Readability on specific tasks, especially separating article content from comments, but they tend to be slower and more complex to deploy.

The tradeoff between speed and accuracy matters when you’re processing feeds at scale. Readability runs in milliseconds within a browser tab. It requires no network calls or external API costs. For an RSS reader that needs to extract full text from dozens of feeds on a regular schedule, this performance characteristic makes a real difference. A slower extraction method might work fine for occasional use but becomes a bottleneck when you’re fetching hundreds of articles per day.

Understanding these strengths and weaknesses helps set realistic expectations for full text extraction. When a feed displays clean article content, it’s because the page structure aligned well with the algorithm’s assumptions. When extraction fails or returns incomplete content, you’re seeing one of the known edge cases at work. The algorithm isn’t broken. It’s operating within its designed scope and encountering a page structure it wasn’t optimized to handle.

This knowledge also informs how you curate your feeds. Sources that use clean, semantic HTML tend to extract reliably. Sites that rely heavily on JavaScript rendering or unconventional layouts may consistently frustrate full text extraction. You can adjust your workflow accordingly, keeping problematic sources in your reader for headlines but planning to visit the actual site when you want to read the full piece.