[Unionfs] unionfs preventing paging

[Christian Hitz]

Hi folks,

we were searching for explanations for unexpected out-of-memory 
situations and have isolated unionfs as a cause.

Background:
embedded ARM-based system, Linux 2.6.24.7 kernel, 32MB memory, no swap
RootFS: jffs2-ro partition overlayed with a second jffs2-rw partition 
using unionfs 2.3.3 (same effect with 2.4).

Observed effect:
Starting an application (binary size: 11MB) [1] from the unionfs 
partition leads to significant less pageable memory than starting the 
same application from a non-unionfs partition.

Test case:
The available free and pageable memory was determined by gradually 
filling a tmpfs partition and so forcing the kernel to free up memory by 
paging out application code.

Starting the application from unionfs we could fill 10MB of data into 
the tmpfs [2] before the oom_killer started to kill processes. The 'RES' 
value of the application (as observed with top) remained constant. 
Running the same application from a non-unionfs partition the tmpfs 
could be filled to 19MB before the oom_killer became active. The 'RES' 
of the application was reduced to <1MB during the procedure.

This difference of ~9MB available memory corresponds to the pageable 
parts of the application.

Would you have an explanation as to why Linux does not page out the 
application when started from unionfs?

Kind regards,
Christian Hitz

[1] The test application was generated with the attached script 
(generate.sh). The script generates a big application of which most of 
the code is only executed once. So most of this code should be pageable. 
About 200000 iterations are needed for an ~10MB application (on ARM).
[2] done with repeated invocations of:
          dd if=/def/zero bs=1024 count=1024 of=fill1

-- 
Christian Hitz
Engineer

BridgeCo AG              Direct   +41 44 802 33 22
Ringstrasse 14           Fax      +41 44 802 33 39
CH-8600 Duebendorf       christian.hitz at bridgeco.net
Switzerland              http://www.bridgeco.net
-------------- next part --------------
#!/bin/sh
if [ $# -lt 2 ]; then
echo "usage: $0 num_iterations out_file"
exit
fi

functioncnt=0
echo "#include <stdio.h>" > $2
echo "" >> $2
echo "int global_counter = 0;" >> $2
echo "" >> $2
while [ $functioncnt -lt $1 ]
    do
    echo "void function_$functioncnt() {" >> $2
    echo "    int i;" >> $2
    echo "    for ( i = 0 ; i < 16 ; i++ ) {" >> $2
    echo "        global_counter++;" >> $2
    echo "    }" >> $2
    echo "" >> $2
    echo "    for ( ; i >= 0 ; i-- ) {" >> $2
    echo "        global_counter--;" >> $2
    echo "    }" >> $2
    echo "" >> $2
    echo "    global_counter += 2;" >> $2
    echo "    global_counter += 2;" >> $2
    echo "    global_counter += 2;" >> $2
    echo "    global_counter += 2;" >> $2
    echo "    global_counter += 2;" >> $2
    echo "    global_counter += 2;" >> $2
    echo "    global_counter += 2;" >> $2
    echo "    global_counter += 2;" >> $2
    echo "    global_counter += 2;" >> $2
    echo "    global_counter += 2;" >> $2
    echo "}" >> $2
    echo "" >> $2
    functioncnt=`expr $functioncnt + 1`
    done

echo "int main(int argc, char *argv[]) {" >> $2
echo "    int l_cnt = 0;" >> $2
echo "" >> $2
functioncnt=0
while [ $functioncnt -lt $1 ]
    do
    echo "    function_$functioncnt();" >> $2
    functioncnt=`expr $functioncnt + 1`
    done

echo "" >> $2
echo "    for ( ;; ) {" >> $2
echo "        if ( !((l_cnt++)%800000) ) usleep(1);" >> $2
echo "    }" >> $2
echo "" >> $2
echo "    return 0;" >> $2
echo "}" >> $2