[Unionfs] Strange behavior when adding a branch

[Erez Zadok]

In message <Pine.LNX.4.64.0707081951380.30636 at zebday>, Yoav Weiss writes:
> Hi,
> 
> I tested unionfs's ability to add branches to a live system, and 
> encountered behavior that seems wrong.
> 
> When a process has an open file descriptor and a branch is added while the 
> file is being read, data from the new branch is returned.  This leads to 
> unexpected results in some situations, and I don't see where such behavior 
> is desirable.
> 
> I'd expect the old descriptor to remain linked to the file in the old 
> branch, and the overwritten file getting a new inode so that new processes 
> opening the file will see the new copy.  This is similar to the 
> non-unionfs situation where a file is unlinked and replaced while being 
> used by some process.  The old process keeps a link to the deleted file, 
> and a new file with a different inode will be available for new processes. 
> Any reason unionfs should behave differently rather than preserve these 
> semantics ?
[...]

BTW, from your script, I see you're using unionctl and probably the older
unionfs 1.x; branch management has changed significantly in 2.0 (for the
better).  What I describe below is for unionfs 2.x, which may behave
differently for you.  (Can you try unionfs2 please ?)

> As the script shows, the process gets data from the new branch without 
> reopening the file.  Is this the expected behavior ?  Is there a known way 
> to avoid it and use branches safely ?

Yoav/Shaya, it depends what users really want.

First, understand that there's little precedent to what Unionfs does.  In
traditional posix OSs, a file or directory has a clear, unmistakable,
single, unique location -- a one-to-one mapping of files to their locations.
With Unionfs, however, you've got yourself a fan-out situation -- a
one-to-MANY mapping of files to their locations.  Any time you have "MANY"
to choose from, you have to answer the question "which one do I pick"?

Worse, branch management is a nasty action to take, no matter what.  Again,
there's no precedent to having a file system who's content changes mid way.
Inserting/removing branches is like replacing one drive of a RAID5 array and
expecting the array to consistently reconfigure itself and present a
coherent new view of files and directories.

If you remember throughout the history of unionfs 1.x and now 2.x, we've had
various modes of operations (e.g., "delete all" and "delete first"), but we
deprecated modes of operation which didn't appear to be used by users too
much.  We wanted to keep the code as simple and functional as possible.
Simple, clear, well understood semantics are very important for code
maintainability (and user sanity :-)

The situation you mention above, I believe, is one in which you insert a new
top-level branch, which contains files with the same name, but different
content.  And you have files with the same name open on the older branches.

Right now the semantics are simple: a file's content is going to point to
the left-most version of that file (once you go through the file/dentry
revalidation upon entry into any Unionfs method).  So for files, we re-open
the file on the leftmost lower branch that it can be found.  For
directories, we open all of the lower directories with the same name, and
merge them left-to-right.

Now, if I recall, we've had cases where users inserted a new leftmost branch
and they *wanted* to immediately see the new content: that was the main
reason they inserted the new leftmost branch (e.g., for software package
versioning).

We also have cases where many users modify files on the lower branches
directly, and expect those modifications to be immediately visible through
the union, even to processes with open files.  In the last few months we've
made a lot of progress on supporting these kinds of features, and ensuring a
uniform semantics when it comes to branch management and modifying lower
branches (all falling under the general umbrella of "cache coherency").

There are also more complicated cases involving copyup.  Suppose you have
this situation:

1. you have a union with branch /a which has a file /a/f.  branch /a is
   readonly.

2. you open the file 'f' for writing through the union

3. now you insert a new leftmost branch /b which also has a file /b/f, and
   at the same time, you mark branch /a as readonly.

4. now you try to read file 'f' through the union: according to your
   suggestion, unionfs should return the older content from /a/f.  Currently
   unionfs will return the newer content from /b/f.

5. now suppose you try to write to file 'f' through the union.  Since branch
   /a was marked as readonly, unionfs shouldn't write to that file.  So that
   should result in a copyup of /a/f onto /b/f, right?  But even that's not
   so clear: you have a 3-way merge problem now.  You have the original file
   content /a/f, the newer content /b/f, and the also new content of /a/f
   which you're trying to write into.  So which of the two "newer" ones
   should we use and when?  It's confusing for sure, and it may depend on
   the specific needs of users at that time.  Should we have allowed the
   read of 'f' to go to the old content when a new one was inserted?  Should
   we have copied-up the old content of 'f' from /a/f to /b/f on the first
   read?  On the first write?

Folks, I don't mind changing Unionfs's semantics if that's what most user
want.  But I'd like to hear that this is indeed what users want.  I don't
want to make a change that will break the desired behavior for existing
users who may depend on that behavior.  If the issue is confined to having
two different ways in which unionfs might handle *open* files, then which
one of them should we pick: stick with the old content or go for the new?
Or maybe it should be a user option?  Ideally, I'd like to avoid having many
different modes of operations to choose from at mount time.

Cheers,
Erez.