Discover Feeds

Feedscout discovers RSS, Atom, JSON Feed, and RDF feeds from webpages using four discovery methods.

Basic Usage

import { discoverFeeds } from 'feedscout'

const feeds = await discoverFeeds('https://example.com')

Each result contains feed metadata:

{
  url: 'https://example.com/feed.xml',
  isValid: true,
  format: 'rss',        // 'rss' | 'atom' | 'json' | 'rdf'
  title: 'Example Blog',
  description: 'A blog about examples',
  siteUrl: 'https://example.com/',
  method: 'html',
}

Results from the Platform method also include a hint that identifies the type of feed.

Discovery Methods

Method	Source
Platform	Popular platform URL patterns
HTML	<link> and <a> elements
Headers	HTTP Link headers
Guess	Common feed paths

How Discovery Works

1. URI Collection: Each enabled method extracts potential feed URLs.
2. Deduplication: Duplicate URLs are removed.
3. Validation: Each URL is fetched and parsed to confirm it's a valid feed.
4. Results: Valid feeds are returned with metadata (format, title, etc.).

If the input is already a valid feed, it is returned as the only result and no methods run.

Specifying Methods

By default, all discovery methods are used (platform, html, headers, guess). You can customize which methods to use with methods option.

Array Syntax

Use an array to enable methods with their default options:

const feeds = await discoverFeeds(url, {
  methods: ['platform', 'html', 'headers', 'guess'],
})

Object Syntax

Use an object to customize individual method options:

const feeds = await discoverFeeds(url, {
  methods: {
    platform: true, // Use defaults
    html: {
      anchorLabels: ['rss', 'feed'],
      anchorUris: ['/feed', '/rss'],
    },
    headers: true, // Use defaults
    guess: {
      uris: ['/feed', '/rss.xml', '/atom.xml'],
    },
  },
})

Set a method to true to use defaults, or provide an options object to customize.

Using Existing Content

If you already have the HTML content and headers, pass them directly to avoid an extra fetch:

// Response fetched someplace else
const response = await fetch('https://example.com')

const feeds = await discoverFeeds(
  {
    url: 'https://example.com',
    content: await response.text(),
    headers: response.headers,
  },
  { methods: ['html', 'headers'] },
)

Options

Stop on First Method

Stop after the first discovery method that finds a valid feed. Methods are validated in order: platform, html, headers, guess. A method whose URLs all turn out invalid does not stop discovery. Useful for platforms (like YouTube) where early methods find the correct feed and later methods (like guess) generate many unnecessary URLs:

const feeds = await discoverFeeds(url, {
  methods: ['platform', 'html', 'headers', 'guess'],
  stopOnFirstMethod: true,
})
// Methods after the first one with a valid feed are skipped

Stop on First Result

Return immediately after finding a valid result (feed, blogroll):

const feeds = await discoverFeeds(url, {
  methods: ['html', 'guess'],
  stopOnFirstResult: true,
})
// Returns array with at most 1 result

Concurrency

Control how many URLs are validated in parallel:

const feeds = await discoverFeeds(url, {
  methods: ['html', 'guess'],
  concurrency: 5, // Default is 3
})

Include Invalid

Include invalid results for debugging:

const feeds = await discoverFeeds(url, {
  methods: ['html', 'guess'],
  includeInvalid: true,
})

for (const feed of feeds) {
  if (feed.isValid) {
    console.log(`Found: ${feed.title}`)
  } else {
    console.log(`Invalid: ${feed.url}`, feed.error)
  }
}

Progress Tracking

Monitor discovery progress with a callback:

const feeds = await discoverFeeds(url, {
  methods: ['html', 'guess'],
  onProgress: ({ tested, total, found, current }) => {
    console.log(`[${tested}/${total}] Testing: ${current}`)
    console.log(`Found so far: ${found}`)
  },
})

Follow the stages of discovery with onStep, for example to show which method is running:

const feeds = await discoverFeeds(url, {
  onStep: (step) => {
    if (step.step === 'validate' && step.status === 'start') {
      console.log(`Testing ${step.total} URLs from ${step.method}`)
    }
  },
})