Tutoriale Online

Tutoriale Online Invata Online cum se face si cum este corect. Learn Online How Must To Do !

Archive for the ‘Large Memory Optimization (Big Pages’ Category

Online Tutoriale Large Memory Optimization (Big Pages, Huge Pages)

Posted by ascultradio on September 3, 2009

Large Memory Optimization (Big Pages, Huge Pages) :

Big Pages in RHEL2.1 and Huge Pages in RHEL 3/4 are very useful for large Oracle SGA sizes and in general for systems with large amount of physical memory. It optimizes the use of Translation Lookaside Buffers (TLB), locks these larger pages in RAM, and the system has less bookkeeping work to do for that part of virtual memory due to larger page sizes. This is a useful feature that should be used on x86 and x86-64 platforms. The default page size in Linux for x86 is 4KB.

Physical memory is partitioned into pages which are the basic unit of memory management. When a Linux process accesses a virtual address, the CPU must translate it into a physical address. Therefore, for each Linux process the kernel maintains a page table which is used by the CPU to translate virtual addresses into physical addresses. But before the CPU can do the translation it has to perform several physical memory reads to retrieve page table information. To speed up this translation process for future references to the same virtual address, the CPU saves information for recently accessed virtual addresses in its Translation Lookaside Buffers (TLB) which is a small but very fast cache in the CPU. The use of this cache makes virtual memory access very fast. Since TLB misses are expensive, TLB hits can be improved by mapping large contiguous physical memory regions by a small number of pages. So fewer TLB entries are required to cover larger virtual address ranges. A reduced page table size also means a reduction in memory management overhead. To use larger page sizes for shared memory, Big Pages (RHEL 2.1) or Huge Pages (RHEL 3/4) must be enabled which also locks these pages in physical memory.

Big Pages in RHEL 2.1 and Huge Pages in RHEL 3/4

In RHEL 2.1 large memory pages can be configured using the Big Pages (bigpages) feature. In RHEL 3/4 Red Hat replaced Big Pages with a feature called Huge Pages (hugetlb) which behaves a little bit different. The Huge Pages feature in RHEL 3/4 allows you to dynamically allocate large memory pages without a reboot. Allocating and changing Big Pages in RHEL 2.1 always required a reboot. However, if memory gets too fragmented in RHEL 3/4 allocation of physically contiguous memory pages can fail and a reboot may become necessary.

The advantages of Big Pages and Huge Pages are:

  • Increased performance by through increased TLB hits
  • Pages are locked in memory and are never swapped out which guarantees that shared memory like SGA remains in RAM
  • Contiguous pages are preallocated and cannot be used for anything else but for System V shared memory (e.g. SGA)
  • Less bookkeeping work for the kernel for that part of virtual memory due to larger page sizes

Usage of Big Pages and Huge Pages in Oracle 9i and 10g

Big pages are supported implicitly in RHEL 2.1. But Huge Pages in RHEL 3/4 need to be requested explicitly by the application by using the SHM_HUGETLB flag when invoking the shmget() system call. This ensures that shared memory segments are allocated out of the Huge Pages pool. This is done automatically in Oracle 10g and 9i R2 (9.2.0.6) but earlier Oracle 9i R2 versions require a patch, see Metalink Note:262004.1.

Sizing Big Pages and Huge Pages

With the Big Pages and Huge Pages feature you specify how many physically contiguous large memory pages should be allocated and pinned in RAM for shared memory like Oracle SGA. For example, if you have three Oracle instances running on a single system with 2 GB SGA each, then at least 6 GB of large pages should be allocated. This will ensure that all three SGAs use large pages and remain in main physical memory. Furthermore, if you use ASM on the same system, then I recommend to add an additional 200MB. I’ve seen ASM instances creating between 70 MB and 150 MB shared memory segments. And there might be other non-Oracle processes that allocate shared memory segments as well.

It is, however, not recommended to allocate too many Big or Huge Pages. These preallocated pages can only be used for shared memory. This means that unused Big or Huge Pages won’t be available for other use than for shared memory allocations even if the system runs out of memory and starts swapping. Also take note that Huge Pages are not used for the ramfs shared memory filesystem, see Huge Pages and Shared Memory Filesystem in RHEL 3/4, but Big Pages can be used for the shm filesystem in RHEL 2.1.

Checking Shared Memory Before Starting Oracle Databases

It is very important to always check the shared memory segments before starting an instance. If an abandoned shared memory segment from e.g. an instance crash is not removed, it will remain allocated in the Big Pages or Huge Pages pool. This could mean that new allocated shared memory segments for the new instance SGA won’t fit into the Big Pages or Huge Pages pool. For more information on removing shared memory, see Removing Shared Memory.

Configuring Big Pages in RHEL 2.1

Before configuring Big Pages, ensure to have read Sizing Big Pages and Huge Pages.

Note that Big Pages in x86 RHEL 2.1 can only be allocated and pinned above (approx) 860MB of physical RAM which is known as Highmem or high memory region in x86. Thus, Big Pages cannot be larger than Highmem. The total amount of memory in the high region can be obtained by reading the memory statistic HighTotal from the /proc/meminfo

file:

$ grep "HighTotal" /proc/meminfo
HighTotal:     9043840 kB
$

The Big Pages feature can be enabled with the following command:

# echo "1" > /proc/sys/kernel/shm-use-bigpages

Alternatively, you can use sysctl(8) to change it:

# sysctl -w kernel.shm-use-bigpages=1

To make the change permanent, add the following line to the file /etc/sysctl.conf. This file is used during the boot process.

echo "kernel.shm-use-bigpages=1" >> /etc/sysctl.conf

Setting kernel.shm-use-bigpages to 2 enables the Big Pages feature for the shmfs shared memory filesystem. Setting kernel.shm-use-bigpages to 0 disables the Big Pages feature.

In RHEL 2.1 the size of the Big Pages pool is configured by adding a parameter to the kernel boot command. For example, if you use GRUB and you want to set the Big Pages pool to 1000 MB, edit the /etc/grub.conf file and add the “bigpages” parameter as follows:

default=0
timeout=10
title Red Hat Linux Advanced Server (2.4.9-e.40enterprise)
        root (hd0,0)
        kernel /vmlinuz-2.4.9-e.40enterprise ro root=/dev/sda2 bigpages=1000MB
        initrd /initrd-2.4.9-e.40enterprise.img
title Red Hat Linux Advanced Server (2.4.9-e.40smp)
        root (hd0,0)
        kernel /vmlinuz-2.4.9-e.40smp ro root=/dev/sda2
        initrd /initrd-2.4.9-e.40smp.img

After this change the system must be rebooted:

# shutdown -r now

After a system reboot the 1000 MB Big Pages pool should show up under BigPagesFree in /proc/meminfo

.

grep BigPagesFree /proc/meminfo

Note that if HighTotal in /proc/meminfo is 0 KB, then BigPagesFree will always be 0 KB as well since Big Pages can only be allocated and pinned above (approx) 860MB of physical RAM.

Configuring Huge Pages in RHEL 3

Before configuring Huge Pages, ensure to have read Sizing Big Pages and Huge Pages.

In RHEL 3 the desired size of the Huge Pages pool is specified in megabytes. The size of the pool should be configured by the incremental size of the Huge Page size. To obtain the size of Huge Pages, execute the following command:

$ grep Hugepagesize /proc/meminfo
Hugepagesize:     2048 kB
$

The number of Huge Pages can be configured and activated by setting hugetlb_pool in the proc filesystem. For example, to allocate a 1GB Huge Page pool, execute:

# echo 1024 > /proc/sys/vm/hugetlb_pool

Alternatively, you can use sysctl(8)

to change it:

# sysctl -w vm.hugetlb_pool=1024

To make the change permanent, add the following line to the file /etc/sysctl.conf. This file is used during the boot process. The Huge Pages pool is usually guaranteed if requested at boot time:

# echo "vm.hugetlb_pool=1024" >> /etc/sysctl.conf

If you allocate a large number of Huge Pages, the execution of the above commands can take a while. To verify whether the kernel was able to allocate the requested number of Huge Pages, execute:

$ grep HugePages_Total /proc/meminfo
HugePages_Total:   512
$

The output shows that 512 Huge Pages have been allocated. Since the size of Huge Pages on my system is 2048 KB, a Huge Page pool of 1GB has been allocated and pinned in physical memory.

If HugePages_Total is lower than what was requested with hugetlb_pool, then the system does either not have enough memory or there are not enough physically contiguous free pages. In the latter case the system needs to be rebooted which should give you a better chance of getting the memory.

To get the number of free Huge Pages on the system, execute:

$ grep HugePages_Free /proc/meminfo

Free system memory will automatically be decreased by the size of the Huge Pages pool allocation regardless whether the pool is being used by an application like Oracle DB or not:

$ grep MemFree /proc/meminfo

After an Oracle DB startup you can verify the usage of Huge Pages by checking whether the number of free Huge Pages has decreased:

$ grep HugePages_Free /proc/meminfo

To free the Huge Pages pool, you can execute:

# echo 0 > /proc/sys/vm/hugetlb_pool

This command usually takes a while to finish.

Configuring Huge Pages in RHEL 4

Before configuring Huge Pages, ensure to have read Sizing Big Pages and Huge Pages.

In RHEL 4 the size of the Huge Pages pool is specified by the desired number of Huge Pages. To calculate the number of Huge Pages you first need to know the Huge Page size. To obtain the size of Huge Pages, execute the following command:

$ grep Hugepagesize /proc/meminfo
Hugepagesize:     2048 kB
$

The output shows that the size of a Huge Page on my system is 2MB. This means if I want to allocate a 1GB Huge Pages pool, then I have to allocate 512 Huge Pages. The number of Huge Pages can be configured and activated by setting nr_hugepages in the proc filesystem. For example, to allocate 512 Huge Pages, execute:

# echo 512 > /proc/sys/vm/nr_hugepages

Alternatively, you can use sysctl(8) to change it:

# sysctl -w vm.nr_hugepages=512

To make the change permanent, add the following line to the file /etc/sysctl.conf. This file is used during the boot process. The Huge Pages pool is usually guaranteed if requested at boot time:

# echo "vm.nr_hugepages=512" >> /etc/sysctl.conf

If you allocate a large number of Huge Pages, the execution of the above commands can take a while. To verify whether the kernel was able to allocate the requested number of Huge Pages, run:

$ grep HugePages_Total /proc/meminfo
HugePages_Total:   512
$

The output shows that 512 Huge Pages have been allocated. Since the size of Huge Pages is 2048 KB, a Huge Page pool of 1GB has been allocated and pinned in physical memory.

If HugePages_Total is lower than what was requested with nr_hugepages, then the system does either not have enough memory or there are not enough physically contiguous free pages. In the latter case the system needs to be rebooted which should give you a better chance of getting the memory.

To get the number of free Huge Pages on the system, execute:

$ grep HugePages_Free /proc/meminfo

Free system memory will automatically be decreased by the size of the Huge Pages pool allocation regardless whether the pool is being used by an application like Oracle DB or not:

$ grep MemFree /proc/meminfo

NOTE: In order that an Oracle database can use Huge Pages in RHEL 4, you also need to increase the ulimit parameter “memlock” for the oracle user in /etc/security/limits.conf if “max locked memory” is not unlimited or too small, see ulimit -a or ulimit -l. For example:

oracle           soft    memlock         1048576
oracle           hard    memlock         1048576

The memlock parameter specifies how much memory the oracle user can lock into its address space. Note that Huge Pages are locked in physical memory. The memlock setting is specified in KB and must match the memory size of the number of Huge Pages that Oracle should be able to allocate. So if the Oracle database should be able to use 512 Huge Pages, then memlock must be set to at least 512 * Hugepagesize, which is on my system 1048576 KB (512*1024*2). If memlock is too small, then no single Huge Page will be allocated when the Oracle database starts. For more information on setting shell limits, see Setting Shell Limits for the Oracle User.

Now login as the oracle user again and verify the new memlock setting by executing ulimit -l before starting the database.

After an Oracle DB startup you can verify the usage of Huge Pages by checking whether the number of free Huge Pages has decreased:

$ grep HugePages_Free /proc/meminfo

To free the Huge Pages pool, you can execute:

# echo 0 > /proc/sys/vm/nr_hugepages

This command usually takes a while to finish.

Huge Pages and Shared Memory Filesystem in RHEL 3/4

In the following example I will show that the Huge Pages pool is not being used by the ramfs shared memory filesystems. The ramfs shared memory filesystems can be used for Configuring Very Large Memory (VLM).

The ipcs command shows only System V shared memory segments. It does not display shared memory of a shared memory filesystems. The following command shows System V shared memory segments on a node running a database with an SGA of 2.6 GB:

# ipcs -m

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status
0x98ab8248 1081344    oracle    600        77594624   0
0xe2e331e4 1245185    oracle    600        2736783360 0

The first shared memory segment of 74 MB was created by the ASM instance. The second shared memory segment of 2.6 GB was created by the database instance.

On this database system the size of the database buffer cache is 2 GB:

  db_block_buffers = 262144
  db_block_size    = 8192

The following command shows that Oracle allocated a shared memory file of 2GB (262144*8192=2147483648) for the buffer cache on the ramfs shared memory filesystem:

# mount | grep ramfs
ramfs on /dev/shm type ramfs (rw)
# ls -al /dev/shm
total 204
drwxr-xr-x    1 oracle   dba             0 Oct 30 16:00 .
drwxr-xr-x   22 root     root       204800 Oct 30 16:00 ..
-rw-r-----    1 oracle   dba      2147483648 Nov  1 16:46 ora_orcl1_1277954

The next command shows how many Huge Pages are currently being used on this system:

$ grep Huge /proc/meminfo
        HugePages_Total:  1536
        HugePages_Free:    194
        Hugepagesize:     2048 kB
$

The output shows that 1342 (1536-194) Huge Pages are being used. This translates into 2814377984 (1342*2048*1024) bytes being allocated in the Huge Pages pool. This number matches the size of both shared memory segments (2736783360+77594624=2814377984) displayed by the ipcs command above.

This shows that the Huge Pages pool is not being used for the ramfs shared memory filesystem. Hence, you do not need to increase the Huge Pages pool if you use the ramfs shared memory filesystem.

Advertisements

Posted in Large Memory Optimization (Big Pages | Tagged: , | Leave a Comment »