Last time installing an Oracle DB on Solaris 11.4 SPARC I realized missing extra large memory page sizes. I want to see 16GB huge pages on SPARC but only got 2GB pages as largest allocated memory by Oracle. Well, still bigger than on x86, where you will get 4k/2M/1G but dynamic large pages chosen by the database like needed is a very nice feature on SPARC:
Multiple Page Size Support
MPSS feature in Oracle Solaris allows an application to use different page sizes for different regions of virtual memory. Larger page sizes let the Translation Lookaside Buffer (TLB) map more physical memory using the fixed number of TLB entries. Larger pages may therefore reduce the cost of virtual-to-physical memory mapping and increase overall system performance.
First of all we need a domain which provides 16GB pages which is controlled by the SPARC hypervisor (aka LDOM). This is documented by the LDOM parameter “effective-max-pagesize” and “hardware-max-pagesize”.
To get a given effective-max-pagesize of 16GB the LDOM must be assigned in a layout that includes at least one MBLOCK that has 4 aligned physical and contiguous ranges of 16GB. That means at least one MBLOCK with 4x16GB (64GB) and this MBLOCK *must* be aligned to a 16GB hardware address.
Alignment can be a tricky part since “LDOMs” reserve a small amout of memory for internal use which means the first available block might not be aligned to our needed 16GB hardware address. For example you could use hardware addresses for that like:
root@t7primary01:~# ldm set-mem mblock=0x400000000:64g my64Gdomain
root@t7primary01:~# ldm list-constraints my64domain | grep page
effective-max-pagesize=16GB
hardware-max-pagesize=16GB
root@t7primary01:~#
Pfua… ok… now it should work but still… no 16GB pages…
After struggling around with support I got an answer from a kernel developer… extra large pages are disabled on “small” systems with less than 512GB of memory (in pages). There were some issues (internal Bug) but with latest and greatest versions systems actually run satisfactorily.
Anyhow it is still disabled by this threshold (might chance again), but yes, it also might not really be necessary on smaller systems. If you still want to use extra large pages, you can adjust the threshold:
root@t7primary01:~# ldm list-constraints primary | grep -i 0x2 0x2000000000 256G root@t7primary01:~# root@t7primary01:~# grep xlarge /etc/system set xlarge_mem_threshold = 0x1900000 root@t7primary01:~# pagesize -a 8192 65536 4194304 268435456 2147483648 17179869184 root@t7primary01:~# pmap -sx $(ps -ef -o pid,comm | awk '/smon/ {print $1}') | grep osm 0000000380000000 8192 8192 - 8192 4M rwxsR-- [ osm shmid=0x0 ] 0000000380800000 4096 4096 - 4096 - rwxsR-- [ osm shmid=0x0 ] 00000003C0000000 262144 262144 - 262144 256M rwxsRi- [ osm shmid=0x4 ] 0000000400000000 117440512 117440512 - 117440512 16G rwxsRi- [ osm shmid=0x1 ] 0000002000000000 6291456 6291456 - 6291456 2G rwxsRi- [ osm shmid=0x2 ] 0000002180000000 1835008 1835008 - 1835008 256M rwxsRi- [ osm shmid=0x3 ] root@t7primary01:~#
Have fun showing your DBAs a possible platform performance feature…