(from Google Operating System Blog 6.16.09)

These instructions will configure Firefox to use Gmail or your Google Apps hosting account to handle mailto links:

1. First, type about:config in the address bar, search for gecko.handlerService.allowRegisterFromDifferentHost, and double click to set the value to true.
2. Then, log into GMail or your Google Apps mail URL, paste this line of JavaScript into your address bar, and hit enter:
   For Gmail: (compressed for formatting purposes - just copy & paste)
   javascript:window.navigator.registerProtocolHandler("mailto","https://mail.google.com/mail/?extsrc=mailto&url=%s","GMail")
3. For Google Apps:(compressed for formatting purposes - just copy & paste)
   javascript:window.navigator.registerProtocolHandler("mailto","https://mail.google.com/a/example.com/mail/?extsrc=mailto&url=%s","GApps")
   Replace “example.com” with your domain.
4. Finally, to set the default email client in Firefox without installing extensions:
   * go to Tools > Options
   * select the Applications tab
   * search for mail
   * select Use Gmail or GApps from the list of actions associated with the mailto protocol
   * click OK to save the settings.