Feedscout

Customize URL Resolution ​

Feedscout resolves discovered URLs before validation to ensure consistent results. You can provide a custom resolution function to change this behavior.

Default Behavior ​

By default, Feedscout resolves relative URLs against the base URL:

typescript
// Base URL: https://example.com/blog/
// Discovered: /feed.xml
// Resolved: https://example.com/feed.xml

// Base URL: https://example.com/blog/
// Discovered: ../rss
// Resolved: https://example.com/rss

It also repairs URLs a page often gets wrong:

typescript
// HTML entities are decoded
// Discovered: /feed?format=rss&lang=en
// Resolved: https://example.com/feed?format=rss&lang=en

// Feed and podcast schemes become https
// Discovered: feed://example.com/rss.xml
// Resolved: https://example.com/rss.xml

// Malformed http protocols are fixed
// Discovered: htp://example.com/feed.xml
// Resolved: http://example.com/feed.xml

An entity is decoded only with its trailing semicolon, so a query such as ?id=1&copy=2 stays as it is. A URL the default cannot parse is kept as discovered and reported to onError.

Custom Resolution ​

Provide a resolveUrlFn to customize URL resolution:

typescript
import type { DiscoverResolveUrlFn } from 'feedscout'

const customResolve: DiscoverResolveUrlFn = (url, baseUrl) => {
  // Resolve relative URLs
  const resolved = new URL(url, baseUrl).href

  // Remove tracking parameters
  const parsed = new URL(resolved)
  parsed.searchParams.delete('utm_source')
  parsed.searchParams.delete('utm_medium')
  parsed.searchParams.delete('utm_campaign')

  return parsed.href
}

const feeds = await discoverFeeds(url, {
  methods: ['html', 'guess'],
  resolveUrlFn: customResolve,
})

// Also works with discoverHubs
const hubs = await discoverHubs(url, {
  resolveUrlFn: customResolve,
})

Interface ​

typescript
type DiscoverResolveUrlFn = (url: string, baseUrl: string | undefined) => string | undefined

Return undefined when the URL cannot be resolved. Feedscout then keeps the URL as discovered. The same happens when the function throws, and the error is reported to onError, so a malformed href on a page does not stop discovery.

A resolved URL with a scheme other than http or https, such as a javascript: or mailto: link, is dropped, since it cannot be fetched. The Guess method finds nothing on a page without an http origin, such as a saved file: page, since the URLs it builds start from that origin.

Use Cases ​

Removing Query Parameters ​

Strip unnecessary query parameters:

typescript
const resolveUrl: DiscoverResolveUrlFn = (url, baseUrl) => {
  const resolved = new URL(url, baseUrl)

  // Keep only essential parameters
  const essentialParams = ['format', 'type']
  const params = new URLSearchParams()

  for (const key of essentialParams) {
    const value = resolved.searchParams.get(key)
    if (value) params.set(key, value)
  }

  resolved.search = params.toString()
  return resolved.href
}

Forcing HTTPS ​

Upgrade HTTP URLs to HTTPS:

typescript
const resolveUrl: DiscoverResolveUrlFn = (url, baseUrl) => {
  const resolved = new URL(url, baseUrl)
  resolved.protocol = 'https:'
  return resolved.href
}

Removing Trailing Slashes ​

Remove trailing slashes from paths:

typescript
const resolveUrl: DiscoverResolveUrlFn = (url, baseUrl) => {
  const resolved = new URL(url, baseUrl)
  resolved.pathname = resolved.pathname.replace(/\/+$/, '')
  return resolved.href
}

Custom Domain Handling ​

Rewrite URLs for specific domains:

typescript
const resolveUrl: DiscoverResolveUrlFn = (url, baseUrl) => {
  const resolved = new URL(url, baseUrl)

  // Use feeds subdomain for specific sites
  if (resolved.hostname === 'example.com') {
    resolved.hostname = 'feeds.example.com'
  }

  return resolved.href
}

Combining with Other Options ​

URL resolution works with discoverFeeds, discoverBlogrolls, discoverFavicons, and discoverHubs:

typescript
const feeds = await discoverFeeds(url, {
  methods: ['html', 'guess'],
  resolveUrlFn: customResolve,
  extractFn: customExtractor,
  concurrency: 3,
})

const hubs = await discoverHubs(url, {
  methods: ['headers', 'html'],
  resolveUrlFn: customResolve,
})