CRUX PPC

author: Acrux

---

Hugetlbfs How-TO

This How-To describes how to get hugetlbfs file system working on CRUX PPC .

Power Architecture based systems normally support 4KB pages as the operating system base page size for memory pages. With newer 32/64bit systems, an operating system can change this to be a 64KB page size. In addition to the memory base page size, each systems allow for the additional use of 16MB pages. POWER5+ or later can also use 16GB page sizes. In Linux, these pages are referred to as HugePages.
Libhugetlbfs provides for the ability to transparently leverage these much larger pages whatever base page size (4KB or 64KB) is being used. Applications can use 16MB huge pages to fulfill malloc() requests without being recompiled by using LD_PRELOAD.

## POWER4+ 2/way 1.45GHz 6GB ram
root:~# gcc -O3 -mcpu=power4  a.c
root:~# ./a.out
12.506792
root:~# echo 348 > /proc/sys/vm/nr_hugepages
root:~# LD_PRELOAD=/usr/lib64/libhugetlbfs.so HUGETLB_MORECORE=yes ./a.out
5.665106

## dual PPC970MP 2.5GHz 14GB ram
root:~# gcc -O3 -mcpu=970  a.c
root:~# ./a.out
8.730244
root:~# echo 768 > /proc/sys/vm/nr_hugepages
root:~# LD_PRELOAD=/usr/lib64/libhugetlbfs.so HUGETLB_MORECORE=yes ./a.out
2.231789

1. You must configure kernel to use hugetlbfs file system. Download a 2.6.26 or newer kernel

2. When configuring the kernel, enable:

File Systems --->

    Pseudo filesystems --->

      [*] HugeTLB file system support

3. Build the ppc64 kernel as usual

# make all
# make modules_install
# cp vmlinux /boot/vmlinux

4. Create and mount the libhugetlbfs file system

# mkdir /mnt/libhugetlbfs
# chmod 777 /mnt/libhugetlbfs

5. Edit the /etc/fstab file to include the following line:

hugetlbfs     /mnt/libhugetlbfs   hugetlbfs  defaults       0     0

and reboot.

6. Install libhugetlbfs with your port manager (as ilenia)

# ilenia -U libhugetlbfs

7. Reserve huge pages for compute jobs to use, by providing the number of desired pages in the nr_hugepages file. Clear any existing huge pages before you reserve a new number of huge pages, as shown in the following example:

# echo 0 > /proc/sys/vm/nr_hugepages
# echo 100 > /proc/sys/vm/nr_hugepages

8. Optional. Edit the /etc/sysctl.conf file to include the following parameters:

vm.nr_hugepages=0 
vm.nr_hugepages=100 

Run the sysctl -p command to reload settings from the default sysctl configuration file.

Further reading and information: kernel/Documentation/vm/hugetlbpage.txt