Cover V11, I11

Figure 1
Figure 2
Figure 3
Figure 4
Figure 5
Listing 1
Listing 2
Listing 3


Mail Routing Using LDAP

Enrique Flores R.

In an effort to mitigate the spam and virus problems in our user community, the IT group at Cypress Semiconductor Corporation has implemented Brightmail Solution Suite, an anti-spam/anti-virus mailwall solution. Brightmail's product refers to spam and infected email as "graymail". For each user, the mailwall software creates individual graymail mailboxes where illegitimate email is sidelined. Through the use of a Web interface, users are given the option to delete or retrieve their graymail messages. Retrieved graymail, or email not filtered by the mailwall rule sets, is forwarded to our site mail hub, where it is either spooled for local users or relayed to the appropriate remote site. Figure 1 details the specifics of our mail flow. Figure 2 is a flowchart depicting the sidelining process.

In our environment, we store graymail in an NFS-mounted qtree on one of our Network Appliance filers. Because of the NFS component of our mailwall configuration, we chose maildrop as the Sendmail local mailer for graymail delivery. Maildir was the chosen format for the graymail mailboxes. Graymail is first sidelined in /usr/local/mailwall/gmspool. Next, Sendmail queues it in /var/spool/graymqueue before maildrop delivers it to the mailboxes on the NFS mount located at /var/graymail. Every ten minutes, a "harvester" program collects graymail, connects to Sendmail, and sends the messages to the NFS-mounted graymail mailboxes.

We began to experience serious performance problems with our mailwall systems soon after introducing this service. Further investigation showed that the maildrop mailer was re-queueing graymail addressed to unknown users. The logs recorded the following error:

Deferred: local mailer (/usr/local/bin/maildrop) exited with EX_TEMPFAIL
After a short time, the graymqueue directory would grow extremely large with graymail that was failing to de-queue. The re-queueing taking place was rapidly depleting system resources and halting service. The only way to temporarily recover was to create a new graymqueue directory and restart Sendmail. This procedure had to be repeated each time the system's backlog reached a dangerous level.

Using LDAP

To alleviate this condition, we chose to address the root of the problem -- processing mail to unknown users. We configured our mail systems to handle only mail directed to valid users or aliases. As a result, maildrop would only process graymail addressed to real users. To do this, we chose LDAP to handle the validation and routing of email. In our corporation, all our users have entries (accounts) in our LDAP systems. By referencing our LDAP systems, only mail verified by LDAP would be accepted.



We recently completed a redesign of our Directory Information Tree (DIT). A DIT is a namespace that provides the means by which directory data is named and referenced. As part of our redesign, we created several branch points (organizational units) and categorized data into functional units. Figure 3 is a diagram of our current directory layout.

The root of our tree is From that point down, Our DIT is divided into five units:

  • ou=People, -- Where corporate user data is stored. For this Sendmail configuration, each user entry was modified to include mail routing information.
  • ou=Groups, -- Holds group data.
  • ou=Applications, -- Where we store application-specific data. This is where we store configuration information for our LDAP-enabled applications.
  • ou=Resources, -- Stores resource information such as printers, conference rooms, phone bridge codes, etc.
  • ou=Services, -- The branch that contains data for our network services. NIS information and Sendmail configuration data would be stored here.

We have future plans to LDAP-enable our NIS domain ( For this reason, we created an organizational unit (ou) named under the ou=Services branch. We also created branch points for our current NIS services. In the near future, our NIS maps will be stored under their respective ou.

With Sendmail's extensive LDAP support (as of v8.12), we also plan to store Sendmail configuration data, such as aliases, maps, and classes in LDAP. With this in mind, an ou=Sendmail branch under ou=Services was created, as well as ou's for aliases, maps, and classes. For our configuration, we only implemented the storage of aliases in LDAP. We have left maps and classes for a future project.


For this Sendmail/LDAP project, I converted our external aliases file to LDIF (LDAP Data Interchange Format) format. See Listing 1 for a list of objectclasses and attributes used for our aliases. These are all standard objectclasses and attributes present in our directory server software; no schema modifications were necessary. We plan to store internal (NIS) aliases under ou=aliases,, ou=Services, External aliases are stored under ou=aliases, ou=Sendmail, ou=Services,

In preparation for storing NIS maps in LDAP, I designed the format for our NIS aliases map at the same time as the creation of the format for our external aliases. The LDIF formats for our internal and external aliases differ slightly. See Listing 2 for LDIF examples of internal and external mail aliases.

The distinguishing elements between internal and external LDIF entries are the objectclasses. Internal entries include the nisMap, nisObject, and mailGroup objectclasses. External entries only include the mailGroup objectclass. In our example, this is the characteristic that separates internal and external aliases in LDAP.

Because our Sendmail configuration is only concerned with incoming/external mail, I configured Sendmail to deliver exclusively to aliases with the mailGroup objectclass. When Sendmail is searching the directory for aliases, it will ignore any entries possessing the "nis" objectclasses, despite the fact that these entries also contain the mailGroup objectclass. A detailed explanation of how this is done will follow later in this article.


The second piece of our configuration deals with mail routing. All user entries under the ou=People, branch were modified to include mail routing information. Refer to Listing 1 for a list of objectclasses and attributes that have been added. As an example of a user entry with mail routing attributes, I have included a portion of my LDAP entry in Listing 2. Later in this article is an explanation of how the routing decisions are made.

There is no distinction made between internal and external users. All our users in LDAP possess an mail address and accept mail from external sources.


LDAP and Sendmail

The fundamental software used for our Sendmail/LDAP configuration is as follows:

  • Solaris 8
  • Netscape Directory Server 4.15
  • Sendmail 8.12.1

Solaris 8 was used because of its native support for LDAP. The LDAP libraries are supplied by the SUNWlldap package. These libraries are referenced during the compilation of Sendmail.

For the LDAP server, we used our existing directory server software, Netscape Directory Server (v4.15). I made no changes to the directory server's schema. I took advantage of the software's standard objectclasses and attributes relevant to email. Please note that the configuration information detailed here can also be applied to open source directory software (e.g., OpenLDAP). We chose not to use open source LDAP software because of current dependencies and standardization on Netscape Directory Server.

The LDAP features present in Sendmail 8.12.1 made this version of Sendmail appropriate for our needs. In version 8.12.1, LDAP can be used for aliases, maps, and classes by either specifying your own LDAP map specification or using the built-in default LDAP map specification. LDAP can also be used for mail routing purposes. This feature enables LDAP-based rerouting of a particular address to either a different host or a different address. As mentioned previously, we implemented LDAP for aliases and routing.

In this article, I am only detailing the installation and configuration of Sendmail. For information regarding the installation of Netscape Directory Server, visit:

Supporting Software

In addition to a directory server and Sendmail, the following software plays a supporting role in the Sendmail configuration:

  • libgcc 3.0.3 libraries
  • Berkeley DB 3.2.9

I downloaded Solaris packages for this software at:
Both of these packages were uncompressed and then installed with the following command (installs in /usr/local):

# pkgadd -d libgcc-3.0.3-sol8-sparc-local
# pkgadd -d db-3.2.9-sol8-sparc-local
The libgcc libraries are needed because of the library requirements found in Berkeley DB. For this reason, the libgcc package was installed first. Berkeley DB provides the database formats that can be used for Sendmail alias files and general maps (aliases, virtusertable, etc.).

NEWDB is the format used in our configuration. NEWDB is included automatically in the Sendmail build if the Build script can find a library named libdb.a or In order for the Build script to be cognizant of NEWDB library and header files needed for compilation, the following links were created:

# ln -s /usr/local/BerkeleyDB.3.2/lib/libdb.a /usr/lib/libdb.a
# ln -s /usr/local/BerkeleyDB.3.2/include/db.h /usr/include/db.h
# ln -s /usr/local/BerkeleyDB.3.2/include/db_cxx.h /usr/include/db_cxx.h
Once the links are made, the next step is to build Sendmail.

Compiling and Installing Sendmail 8.12.1

The current release of Sendmail can be found at I downloaded the gzipped source at:
Uncompress and untar the source with the following command:

# gunzip -c sendmail.8.12.1.tar.gz | tar xvf -
A directory named "sendmail-8.12.1" will be created. Before beginning the compiling process, it is important to read through the various README files. There are new features in this version of Sendmail, and the documentation should be studied before moving forward.


Starting with version 8.12, Sendmail is more security conscious. By default, the software will install without set-user-ID root. To do this, the software must run as a set-group-ID program. Additionally, there are two Sendmail processes that must be started. One is a daemon process, and the other a submission program. There are also two cf files that are used -- for the daemon and for the submission program.

As part of the security enhancements to the software, a user account and a group must be created specifically for the Sendmail processes. It is recommended that the user and group both be named "smmsp" with a uid and gid of 25. You must ensure that the security configuration meets the needs and policies of your own organization. I suggest the following permissions for the files and spool areas used by Sendmail:

-r-xr-sr-x    root   smmsp   /usr/lib/sendmail
drwxrwx---    smmsp  smmsp   /var/spool/clientmqueue
drwx------    root   root    /var/spool/mqueue
-r--r--r--    root   root    /etc/mail/
-r--r--r--    root   root    /etc/mail/
For more information regarding the security features of version 8.12, refer to the "SECURITY" document in the source distribution.


The instructions and syntax detailed here are for Solaris 8 only. Refer to the site configuration document in devtools/Site/README for additional information.

To build Sendmail with LDAPMAP (map used for LDAP features), the following two entries must be made in the site.config.m4 file:

APPENDDEF('confLIBS', '-lldap')  
The next step is to change to the sendmail/ directory. In this directory, compile and install Sendmail by running:

# ./Build
# ./Build install
Sendmail will install in /usr/lib. The next step is to create a file (the file is created and placed in /etc/mail during the compile). The Sendmail download comes with generic cf files that can be used. One of these can be used to test the startup of Sendmail. Once the file is created, start the Sendmail processes by entering:

# /usr/lib/sendmail -L sm-mta -bd -q15m
# /usr/lib/sendmail -L sm-msp-queue -Ac -q15m
To verify the version of Sendmail and the features it was compiled with, run the following:

# /usr/lib/sendmail -d0.1 -bt  
Version 8.12.1

============ SYSTEM IDENTITY (after readcf) ============
      (short domain name) $w = qbert
  (canonical domain name) $j =
         (subdomain name) $m =
              (node name) $k = qbert

ADDRESS TEST MODE (ruleset 3 NOT automatically invoked)
Enter <ruleset> <address>
For each of the associated Sendmail utilities (makemap, mailstats, etc.), read the README file in the utility's directory. Before installing a utility, be sure to make a backup copy of the utility being replaced. To install, run ./Build followed by ./Build install.

Sendmail Configuration

Now that the steps to building Sendmail have been listed, I can expand on our configuration details. There are two mc files that make up our mailwall system. One configuration is for our Sendmail process that accepts and filters incoming mail. The second configuration is for the process that moves graymail to the graymail mailboxes.

Configuration Details

The first mc file is configured to accept incoming mail on the standard SMTP port, port 25. This configuration interfaces with our mailwall filter through Sendmail's MILTER API. The mailwall filter program is named "bmifilter" and it has been configured to operate on port 5513 (defined by INPUT_MAIL_FILTER). Listing 3 contains the mc file specifics for our port 25 Sendmail instance.

Before the LDAP pieces were added to our system, mail not caught by the mailwall filter was relayed to our site mail hub. With LDAP-based routing configured, mail is relayed to the user's specific "mailhost" value, as indicated in their LDAP entry. In the mc file, LDAP routing is defined by the FEATURE() command in the following manner:

FEATURE('ldap_routing', <mailHost>, <mailRoutingAddress>, <bounce>, <detail>)
  • <mailHost> is a map definition describing how to look up an alternative mail host for a particular address.
  • <mailRoutingAddress> is a map definition describing how to look up an alternative address for a particular address.
  • <bounce> argument, if present and not the word "passthru", dictates that mail should be bounced if neither a mailHost nor mailRoutingAddress is found.
  • <detail> indicates which actions to take if the address contains +detail information. strip tries the lookup with the +detail and if no matches are found, strips the +detail and tries the lookup again. preserve does the same as "strip", but if a mailRoutingAddress match is found, the +detail information is copied to the new address.

In our mc file, the <bounce> and <detail> parameters are left blank. For the <mailHost> and <mailRoutingAddress> attributes, the LDAP search will look through LDAP entries containing the "mailRecipient" objectclass. At our site, the mailhost and mailroutingaddress values are set. Therefore, mail is relayed to the listed mailhost using the mailroutingaddress. A user's mailrouting address is his uid plus his mailhost (e.g., This information was gathered from our existing aliases file and imported into all LDAP user entries.

Sendmail makes routing decisions based on the following matrix:

mailHost is             mailRoutingAddress is   Results in
-----------             ---------------------   ----------
set to a "local" host   set                     mail delivered to

set to a "local" host   not set                 delivered to
                                                original address

set to a remote host    set                     mailRoutingAddress
                                                relayed to mailHost

set to a remote host    not set                 original address
                                                relayed to mailHost

not set                 set                     mail delivered to

not set                 not set                 delivered to
                                                original address *OR*
                                                bounced as unknown user

Mail not routed by the ldap_routing FEATURE is handed to the local mailer. Mail that gets this far is mail addressed to an LDAP entry that does not contain the mailRecipient objectclass (i.e., a mail alias). At this point, the ALIAS_FILE definition is consulted to determine where to deliver the message. An LDAP search query is the first parameter in the alias definition. The query reads:

ldap:-k (&(objectClass=mailGroup)(!(objectclass=nisMap))(mail=%0)) -v mgrpRFC822MailMember
%0 is the address that has been passed to the local mailer. The search filter is prepended with a &, which equates to the Boolean operator AND. The search filter is translated as follows:

Find an entry that contains:

  • A mail attribute set to the address in question (mail=%0), and
  • An objectclass of mailGroup (objectclass=mailGroup), and
  • Does NOT contain an objectclass of nisMap (!(objectclass=nisMap))
  • Once such an entry is found, deliver the message to the address(es) set in the mgrpRFC822MailMember attribute.

This search filter will ensure that only mail for external aliases is accepted. Recall that both internal and external aliases contain the mailGroup objectclass, but external aliases do not contain the nisMap objectclass. Therefore, this filter will only find external mail aliases. Anything not found in LDAP will be searched for in /etc/mail/aliases (the second parameter in ALIAS_FILE).

Other LDAP Particulars

confLDAP_DEFAULT_SPEC defines the default LDAP settings. This value contains LDAP-specific settings such as -h host and -p port. Our example is set to use the standard LDAP port (port 389) and to search starting from the root of the DIT (-b Redundancy has also been designed. By setting the host portion to -h ldap1 ldap2 ldap3, three directory servers will be searched in sequence. If ldap1 is not found, then ldap2 is contacted. Finally, queries will be sent to ldap3 if ldap2 is not available.

The domain routed via LDAP is defined by LDAPROUTE_DOMAIN(). Alternatively, LDAPROUTE_DOMAIN_FILE() can be used to list a file that will hold the domain information. In Listing 3, I have included a commented-out example of the LDAPROUTE_DOMAIN_FILE syntax.

Graymail Sendmail

The second configuration runs on port 5512 and is responsible for processing graymail. This configuration is very similar to the first, with the exception of ldap_routing. In this instance, LDAP is strictly used as an ALIAS_FILE reference. Since graymail is being delivered to both aliases (mailGroup objectclass) and regular users (mailRecipient objectclass), both types of recipients must be included in the ALIAS_FILE search query. Listing 3 also lists the mc file for our port 5512 Sendmail instance.

According to our configuration, if LDAP does not return a result, then the next source looked at is /etc/mail/aliases. If nothing is found in the local aliases file, then the local user relay (LUSER_RELAY) handles the message. LUSER_RELAY is set to local:nobody. This setting results in graymail being sent to "nobody" if the recipient is not found in LDAP or in the local aliases. Since "nobody" is set to /dev/null in the aliases file, mail not validated locally or by LDAP is deleted.

Also note that an SMTP MAILER is not included in this mc file. The only MAILER configured is "local". We only want a local mailer because the job of this process is to deliver to graymail mailboxes on a "local" NFS mount (/var/graymail). When the cf file is built, by default mail.local is set as the local mailer (Mlocal). The cf file must be edited manually in order to replace mail.local with maildrop. Listing 3 contains the local mailer definition (path, flags, etc.) from our cf file.

Testing LDAP Interaction

Once the cf file is built, Sendmail's interaction with LDAP can be tested in the following manner:

# /usr/lib/sendmail -bt
ADDRESS TEST MODE (ruleset 3 NOT automatically invoked)
Enter <ruleset> <address>
> /map ldapmh
map_lookup: ldapmh ( returns (0)
> /map ldapmra
map_lookup: ldapmra ( returns (0)
In this example, I am looking up my (ehf) mailhost and mailroutingaddress attributes through the ldapmh and ldapmra maps. These maps were made available by the LDAPMAP definition added to the site.config.m4 file during the build procedure. A successful match will return the respective value provided by LDAP. An unsuccessful match will return the statement "no match".


The positive aspect of this configuration is the ability to centrally manage mail routing and alias information along with the rest of our user profiles. LDAP provides an easy and convenient way to administer user information. Our heavy reliance on LDAP is also a negative point. In our situation, if LDAP is not available, mail cannot be routed and aliases cannot be expanded. Mail from the outside world stops completely. Our significant dependency on directory data demands that we provide a highly available LDAP service. To prevent any mail service outage, the failover configuration implemented by confLDAP_DEFAULT_SPEC() will ensure a very low probability that Sendmail will be unable to contact a directory server. Figures 4 and 5 diagram the final architecture for mail flow and graymail sidelining.

Incorporating LDAP into our Sendmail build solved our problem with re-queuing graymail. By deleting unsolicited mail with LUSER_RELAY(), we were able to greatly diminish the amount of processed graymail. Most importantly, by only routing mail addressed to users and aliases found in LDAP, the objective of validating and processing legitimate mail was met.


1. T. Bialaski, M. Haines, Solaris and LDAP Naming Services: Deploying LDAP in the Enterprise, Sun Microsystems Press, 2001.

2. Sendmail 8.12.1 documentation, "Sendmail Configuration Files".



Enrique Flores is a Sr. UNIX Systems Administrator at Cypress Semiconductor Corporation in San Jose, California. He has worked with LDAP since 1998, and began working with Sendmail in 2000. Enrique received his undergraduate degree from UCLA in 1997 and was enrolled in graduate studies at SUNY, Albany until the end of 1998. He can be contacted at: