I'm searching for a method to configure wordpress on my small RHEL4 machine to transmit out email notices using gnupg (sendmail).

Can anybody point me within the right direction?

WordPress uses it's own function wp_mail() to transmit out emails. As it is a 'pluggable' function, you are able to override it with your personal version inside a WordPress plugin

/**
 * Plugin Name: GNUPG Mail
 * Description: Send mail using GNUPG.
 */

function wp_mail($to, $subject, $message, $headers = '', $attachments = array())
{
    // use the PHP GnuPG library here to send mail.
}