9. Hlfsd

Hlfsd is a daemon which implements a filesystem containing a symbolic link to subdirectory within a user's home directory, depending on the user which accessed that link. It was primarily designed to redirect incoming mail to users' home directories, so that it can be read from anywhere. It was designed and implemented by Erez Zadok and Alexander Dupuy, at the Computer Science Department of Columbia University. A paper on Hlfsd was presented at the Usenix LISA VII conference in 1993.

Hlfsd operates by mounting itself as an NFS server for the directory containing linkname, which defaults to `/hlfs/home'. Lookups within that directory are handled by Hlfsd, which uses the password map to determine how to resolve the lookup. The directory will be created if it doesn't already exist. The symbolic link will be to the accessing user's home directory, with subdir appended to it. If not specified, subdir defaults to `.hlfsdir'. This directory will also be created if it does not already exist.

A `SIGTERM' sent to Hlfsd will cause it to shutdown. A `SIGHUP' will flush the internal caches, and reload the password map. It will also close and reopen the log file, to enable the original log file to be removed or rotated. A `SIGUSR1' will cause it to dump its internal table of user IDs and home directories to the file `/tmp/hlfsddump'.

9.1 Introduction to Hlfsd

Electronic mail has become one of the major applications for many computer networks, and use of this service is expected to increase over time, as networks proliferate and become faster. Providing a convenient environment for users to read, compose, and send electronic mail has become a requirement for systems administrators (SAs).

Widely used methods for handling mail usually require users to be logged into a designated "home" machine, where their mailbox files reside. Only on that one machine can they read newly arrived mail. Since users have to be logged into that system to read their mail, they often find it convenient to run all of their other processes on that system as well, including memory and CPU-intensive jobs. For example, in our department, we have allocated and configured several multi-processor servers to handle such demanding CPU/memory applications, but these were underutilized, in large part due to the inconvenience of not being able to read mail on those machines. (No home directories were located on these designated CPU-servers, since we did not want NFS service for users' home directories to have to compete with CPU-intensive jobs. At the same time, we discouraged users from running demanding applications on their home machines.)

Many different solutions have been proposed to allow users to read their mail on any host. However, all of these solutions fail in one or more of several ways:

• they introduce new single points of failure
• they require using different mail transfer agents (MTAs) or user agents (UAs)
• they do not solve the problem for all cases, i.e. the solution is only partially successful for a particular environment.

We have designed a simple filesystem, called the Home-Link File System, to provide the ability to deliver mail to users' home directories, without modification to mail-related applications. We have endeavored to make it as stable as possible. Of great importance to us was to make sure the HLFS daemon, `hlfsd' , would not hang under any circumstances, and would take the next-best action when faced with problems. Compared to alternative methods, Hlfsd is a stable, more general solution, and easier to install/use. In fact, in some ways, we have even managed to improve the reliability and security of mail service.

Our server implements a small filesystem containing a symbolic link to a subdirectory of the invoking user's home directory, and named symbolic links to users' mailbox files.

The Hlfsd server finds out the uid of the process that is accessing its mount point, and resolves the pathname component `home' as a symbolic link to a subdirectory within the home directory given by the uid's entry in the password file. If the gid of the process that attempts to access a mailbox file is a special one (called HLFS_GID), then the server maps the name of the next pathname component directly to the user's mailbox. This is necessary so that access to a mailbox file by users other than the owner can succeed. The server has safety features in case of failures such as hung filesystems or home directory filesystems that are inaccessible or full.

On most of our machines, mail gets delivered to the directory `/var/spool/mail'. Many programs, including UAs, depend on that path. Hlfsd creates a directory `/mail', and mounts itself on top of that directory. Hlfsd implements the path name component called `home', pointing to a subdirectory of the user's home directory. We have made `/var/spool/mail' a symbolic link to `/mail/home', so that accessing `/var/spool/mail' actually causes access to a subdirectory within a user's home directory.

The following table shows an example of how resolving the pathname `/var/mail/NAME' to `/users/ezk/.mailspool/NAME' proceeds.

9.2 Background to Mail Delivery

This section provides an in-depth discussion of why available methods for delivering mail to home directories are not as good as the one used by Hlfsd.

9.2.1 Single-Host Mail Spool Directory

The most common method for mail delivery is for mail to be appended to a mailbox file in a standard spool directory on the designated "mail home" machine of the user. The greatest advantage of this method is that it is the default method most vendors provide with their systems, thus very little (if any) configuration is required on the SA's part. All they need to set up are mail aliases directing mail to the host on which the user's mailbox file is assigned. (Otherwise, mail is delivered locally, and users find mailboxes on many machines.)

As users become more sophisticated, and aided by windowing systems, they find themselves logging in on multiple hosts at once, performing several tasks concurrently. They ask to be able to read their mail on any host on the network, not just the one designated as their "mail home".

9.2.2 Centralized Mail Spool Directory

A popular method for providing mail readability from any host is to have all mail delivered to a mail spool directory on a designated "mail-server" which is exported via NFS to all of the hosts on the network. Configuring such a system is relatively easy. On most systems, the bulk of the work is a one-time addition to one or two configuration files in `/etc'. The file-server's spool directory is then hard-mounted across every machine on the local network. In small environments with only a handful of hosts this can be an acceptable solution. In our department, with a couple of hundred active hosts and thousands of mail messages processed daily, this was deemed completely unacceptable, as it introduced several types of problems:

Scalability and Performance

As more and more machines get added to the network, more mail traffic has to go over NFS to and from the mail-server. Users like to run mail-watchers, and read their mail often. The stress on the shared infrastructure increases with every user and host added; loads on the mail server would most certainly be high since all mail delivery goes through that one machine.(1) This leads to lower reliability and performance. To reduce the number of concurrent connections between clients and the server host, some SAs have resorted to automounting the mail-spool directory. But this solution only makes things worse: since users often run mail watchers, and many popular applications such as `trn', `emacs', `csh' or `ksh' check periodically for new mail, the automounted directory would be effectively permanently mounted. If it gets unmounted automatically by the automounter program, it is most likely to get mounted shortly afterwards, consuming more I/O resources by the constant cycle of mount and umount calls.

Reliability

The mail-server host and its network connectivity must be very reliable. Worse, since the spool directory has to be hard-mounted,(2) many processes which access the spool directory (various shells, `login', `emacs', etc.) would be hung as long as connectivity to the mail-server is severed. To improve reliability, SAs may choose to backup the mail-server's spool partition several times a day. This may make things worse since reading or delivering mail while backups are in progress may cause backups to be inconsistent; more backups consume more backup-media resources, and increase the load on the mail-server host.

9.2.3 Distributed Mail Spool Service

Despite the existence of a few systems that support delivery to users' home directories, mail delivery to home directories hasn't caught on. We believe the main reason is that there are too many programs that "know" where mailbox files reside. Besides the obvious (the delivery program `/bin/mail' and mail readers like `/usr/ucb/Mail', `mush', `mm', etc.), other programs that know mailbox location are login, from, almost every shell, `xbiff', `xmailbox', and even some programs not directly related to mail, such as `emacs' and `trn'. Although some of these programs can be configured to look in different directories with the use of environment variables and other resources, many of them cannot. The overall porting work is significant.

Other methods that have yet to catch on require the use of a special mail-reading server, such as IMAP or POP. The main disadvantage of these systems is that UAs need to be modified to use these services -- a long and involved task. That is why they are not popular at this time.

Several other ideas have been proposed and even used in various environments. None of them is robust. They are mostly very specialized, inflexible, and do not extend to the general case. Some of the ideas are plain bad, potentially leading to lost or corrupt mail:

automounters

Using an automounter such as Amd to provide a set of symbolic links from the normal spool directory to user home directories is not sufficient. UAs rename, unlink, and recreate the mailbox as a regular file, therefore it must be a real file, not a symbolic link. Furthermore, it must reside in a real directory which is writable by the UAs and MTAs. This method may also require populating `/var/spool/mail' with symbolic links and making sure they are updated. Making Amd manage that directory directly fails, since many various lock files need to be managed as well. Also, Amd does not provide all of the NFS operations which are required to write mail such as write, create, remove, and unlink.

$MAIL

Setting this variable to an automounted directory pointing to the user's mail spool host only solves the problem for those programs which know and use $MAIL. Many programs don't, therefore this solution is partial and of limited flexibility. Also, it requires the SAs or the users to set it themselves -- an added level of inconvenience and possible failures.

/bin/mail

Using a different mail delivery agent could be the solution. One such example is `hdmail'. However, `hdmail' still requires modifying all UAs, the MTA's configuration, installing new daemons, and changing login scripts. This makes the system less upgradable or compatible with others, and adds one more complicated system for SAs to deal with. It is not a complete solution because it still requires each user have their $MAIL variable setup correctly, and that every program use this variable.

9.2.4 Why Deliver Into the Home Directory?

There are several major reasons why SAs might want to deliver mail directly into the users' home directories:

Location

Many mail readers need to move mail from the spool directory to the user's home directory. It speeds up this operation if the two are on the same filesystem. If for some reason the user's home directory is inaccessible, it isn't that useful to be able to read mail, since there is no place to move it to. In some cases, trying to move mail to a non-existent or hung filesystem may result in mail loss.

Distribution

Having all mail spool directories spread among the many more filesystems minimizes the chances that complete environments will grind to a halt when a single server is down. It does increase the chance that there will be someone who is not able to read their mail when a machine is down, but that is usually preferred to having no one be able to read their mail because a centralized mail server is down. The problem of losing some mail due to the (presumably) higher chances that a user's machine is down is minimized in HLFS.

Security

Delivering mail to users' home directories has another advantage -- enhanced security and privacy. Since a shared system mail spool directory has to be world-readable and searchable, any user can see whether other users have mail, when they last received new mail, or when they last read their mail. Programs such as `finger' display this information, which some consider an infringement of privacy. While it is possible to disable this feature of `finger' so that remote users cannot see a mailbox file's status, this doesn't prevent local users from getting the information. Furthermore, there are more programs which make use of this information. In shared environments, disabling such programs has to be done on a system-wide basis, but with mail delivered to users' home directories, users less concerned with privacy who do want to let others know when they last received or read mail can easily do so using file protection bits.

In summary, delivering mail to home directories provides users the functionality sought, and also avoids most of the problems just discussed.

9.3 Using Hlfsd

9.3.1 Controlling Hlfsd

Much the same way Amd is controlled by `ctl-amd', so does Hlfsd get controlled by the `ctl-hlfsd' script:

ctl-hlfsd start

Start a new Hlfsd.

ctl-hlfsd stop

Stop a running Hlfsd.

ctl-hlfsd restart

Stop a running Hlfsd, wait for 10 seconds, and then start a new one. It is hoped that within 10 seconds, the previously running Hlfsd terminate properly; otherwise, starting a second one could cause system lockup.

For example, on our systems, we start Hlfsd within `ctl-hlfsd' as follows on Solaris 2 systems:

hlfsd -a /var/alt_mail -x all -l /var/log/hlfsd /mail/home .mailspool

The directory `/var/alt_mail' is a directory in the root partition where alternate mail will be delivered into, when it cannot be delivered into the user's home directory.

Normal mail gets delivered into `/var/mail', but on our systems, that is a symbolic link to `/mail/home'. `/mail' is managed by Hlfsd, which creates a dynamic symlink named `home', pointing to the subdirectory `.mailspool' within the accessing user's home directory. This results in mail which normally should go to `/var/mail/$USER', to go to `$HOME/.mailspool/$USER'.

Hlfsd does not create the `/var/mail' symlink. This needs to be created (manually) once on each host, by the system administrators, as follows:

mv /var/mail /var/alt_mail
ln -s /mail/home /var/mail

Hlfsd also responds to the following signals:

A `SIGHUP' signal sent to Hlfsd will force it to reload the password map immediately.

A `SIGUSR1' signal sent to Hlfsd will cause it to dump its internal password map to the file `/usr/tmp/hlfsd.dump.XXXXXX', where `XXXXXX' will be replaced by a random string generated by mktemp(3) or (the more secure) mkstemp(3).

9.3.2 Hlfsd Options

-a alt_dir

Alternate directory. The name of the directory to which the symbolic link returned by Hlfsd will point, if it cannot access the home directory of the user. This defaults to `/var/hlfs'. This directory will be created if it doesn't exist. It is expected that either users will read these files, or the system administrators will run a script to resend this "lost mail" to its owner.

-c cache-interval

Caching interval. Hlfsd will cache the validity of home directories for this interval, in seconds. Entries which have been verified within the last cache-interval seconds will not be verified again, since the operation could be expensive, and the entries are most likely still valid. After the interval has expired, Hlfsd will re-verify the validity of the user's home directory, and reset the cache time-counter. The default value for cache-interval is 300 seconds (5 minutes).

-f

Force fast startup. This option tells Hlfsd to skip startup-time consistency checks such as existence of mount directory, alternate spool directory, symlink to be hidden under the mount directory, their permissions and validity.

-g group

Set the special group HLFS_GID to group. Programs such as `/usr/ucb/from' or `/usr/sbin/in.comsat', which access the mailboxes of other users, must be setgid `HLFS_GID' to work properly. The default group is `hlfs'. If no group is provided, and there is no group `hlfs', this feature is disabled.

-h

Help. Print a brief help message, and exit.

-i reload-interval

Map-reloading interval. Each reload-interval seconds, Hlfsd will reload the password map. Hlfsd needs the password map for the UIDs and home directory pathnames. Hlfsd schedules a `SIGALRM' to reload the password maps. A `SIGHUP' sent to Hlfsd will force it to reload the maps immediately. The default value for reload-interval is 900 seconds (15 minutes.)

-l logfile

Specify a log file to which Hlfsd will record events. If logfile is the string `syslog' then the log messages will be sent to the system log daemon by syslog(3), using the `LOG_DAEMON' facility. This is also the default.

-n

No verify. Hlfsd will not verify the validity of the symbolic link it will be returning, or that the user's home directory contains sufficient disk-space for spooling. This can speed up Hlfsd at the cost of possibly returning symbolic links to home directories which are not currently accessible or are full. By default, Hlfsd validates the symbolic-link in the background. The -n option overrides the meaning of the -c option, since no caching is necessary.

-o mount-options

Mount options which Hlfsd will use to mount itself on top of dirname. By default, mount-options is set to `ro'. If the system supports symbolic-link caching, default options are set to `ro,nocache'.

-p

Print PID. Outputs the process-id of Hlfsd to standard output where it can be saved into a file.

-v

Version. Displays version information to standard error.

-x log-options

Specify run-time logging options. The options are a comma separated list chosen from: `fatal', `error', `user', `warn', `info', `map', `stats', `all'.

-C

Force Hlfsd to run on systems that cannot turn off the NFS attribute-cache. Use of this option on those systems is discouraged, as it may result in loss or misdelivery of mail. The option is ignored on systems that can turn off the attribute-cache.

-D log-options

Select from a variety of debugging options. Prefixing an option with the string `no' reverses the effect of that option. Options are cumulative. The most useful option is `all'. Since this option is only used for debugging other options are not documented here. A fuller description is available in the program source.

-P password-file

Read the user-name, user-id, and home directory information from the file password-file. Normally, Hlfsd will use getpwent(3) to read the password database. This option allows you to override the default database, and is useful if you want to map users' mail files to a directory other than their home directory. Only the username, uid, and home-directory fields of the file password-file are read and checked. All other fields are ignored. The file password-file must otherwise be compliant with Unix Version 7 colon-delimited format passwd(4).

9.3.3 Hlfsd Files

The following files are used by Hlfsd:

`/hlfs'

directory under which Hlfsd mounts itself and manages the symbolic link `home'.

`.hlfsdir'

default sub-directory in the user's home directory, to which the `home' symbolic link returned by Hlfsd points.

`/var/hlfs'

directory to which `home' symbolic link returned by Hlfsd points if it is unable to verify the that user's home directory is accessible.

`/usr/tmp/hlfsd.dump.XXXXXX'

file to which Hlfsd will dump its internal password map when it receives the `SIGUSR1' signal. `XXXXXX' will be replaced by a random string generated by mktemp(3) or (the more secure) mkstemp(3).

For discussion on other files used by Hlfsd, see See section lostaltmail, and lostaltmail.conf-sample.

This document was generated by Erez Zadok on November, 27 2006 using texi2html 1.76.