We're back after a server migration that caused effbot.org to fall over a bit harder than expected. Expect some glitches.

Using Channel Providers Outside the effbot.exe/effnews Application

January 27, 2003 | Fredrik Lundh

This note discusses how to use effnews provider scripts in your own applications.

Loading Providers

A provider is a Python script that defines two objects, a urls sequence, and a callable simpleprovider object (usually a function).

The following function loads all provider scripts in a given directory (in a default effnews install, the directory is c:\effbot.exe\effnews):

Example: Loading available providers
import os, glob

def getproviders(directory):
    providers = {}
    for file in glob.glob(os.path.join(directory, "*.provider")):
        ns = {}
            execfile(file, ns)
            for url in ns.get("urls"):
                if isinstance(url, type("")):
                    providers[url] = ns
    return providers

The function returns a dictionary, mapping URLs to module namespaces.

Using a Provider

Once you’ve loaded the provider dictionary, you can download entire web pages, and use the appropriate provider to parse them. Here’s an example, which prints channel information and items to standard output:

Example: Loading data using a provider
import urllib

class Context:

    def push(self, tag, **args):
        print "---", tag
        for key, value in args.items():
            print key, value

providers = getproviders("c:/effbot.exe/effnews")

def loadurl(url):

    provider = providers.get(url)

    file = urllib.urlopen(url)

    if provider:
        c = Context()
        provider["simpleprovider"](c, file.read())

        ... rss_loader(url, file) ...


The rss_loader function isn’t shown in this example. You can for example use the parser from the effnews toolkit to implement the function.