We enabled the Oracle Database InMemory option on an Oracle SuperCluster M7 and wanted to see if our SPARC DAX engines are used. With newer Solaris releases you could use daxstat to get some more info next to busstat or dtrace but issuing daxstat brought an error:
root@OSC:~# daxstat Traceback (most recent call last): File "/usr/bin/daxstat", line 969, in <module> sys.exit(main()) File "/usr/bin/daxstat", line 962, in main return process_opts() File "/usr/bin/daxstat", line 905, in process_opts dax_ids, dax_queue_ids = derive_dax_opts(args, parser) File "/usr/bin/daxstat", line 844, in derive_dax_opts dax_ids = find_ids(query, parser, None) File "/usr/bin/daxstat", line 683, in find_ids all_dax_kstats = RCU.list_objects(kbind.Kstat(), query) File "/usr/lib/python2.7/vendor-packages/rad/connect.py", line 391, in list_objects a RADInterface object File "/usr/lib/python2.7/vendor-packages/rad/client.py", line 213, in _raise_error packer.pack_int((timestamp % 1000000) * 1000) rad.client.NotFoundError: Error listing com.oracle.solaris.rad.kstat:type=Kstat: not found (3)
In my installation the following was not installed:
root@OSC:~# pkg list -a | grep kstat library/python-2/python-kstat 5.11-0.175.2.0.0.27.0 --o system/management/rad/module/rad-kstat 0.5.11-0.175.3.17.0.1.0 --- root@OSC:~# pkg install system/management/rad/module/rad-kstat [...] root@OSC:~# pkg list | grep rad system/management/rad 0.5.11-0.175.3.21.0.4.0 i-- system/management/rad/client/rad-c 0.5.11-0.175.3.21.0.3.0 i-- system/management/rad/client/rad-python 0.5.11-0.175.3.17.0.1.0 i-- system/management/rad/module/rad-kstat 0.5.11-0.175.3.17.0.1.0 i-- system/management/rad/module/rad-smf 0.5.11-0.175.3.17.0.1.0 i-- root@OSC:~# root@OSC:~# svcadm disable svc:/system/rad:local svc:/system/rad:local-http root@OSC:~# svcadm enable svc:/system/rad:local svc:/system/rad:local-http
And yes, that was it 😉
root@OSC:~# daxstat -ad 60 DAX commands fallbacks input output %busy ALL 32541246 15222 1.0G 78.0M 0 ALL 5760 0 6.0M 0.0M 0 ALL 2240 0 6.0M 0.0M 0 root@OSC:~# daxstat 1 1 DAX commands fallbacks input output %busy 0 7062 0 0.0M 0.0M 0 1 7071 0 0.0M 0.0M 0 2 7071 0 0.0M 0.0M 0 3 7067 0 0.0M 0.0M 0 4 7073 0 0.0M 0.0M 0 5 7071 0 0.0M 0.0M 0 6 7066 0 0.0M 0.0M 0 7 7067 0 0.0M 0.0M 0 8 4078650 1878 0.0M 0.0M 0 9 4078651 1941 0.0M 0.0M 0 10 4078699 1870 0.0M 0.0M 0 11 4078674 1914 0.0M 0.0M 0 12 4078720 1923 0.0M 0.0M 0 13 4078723 1929 0.0M 0.0M 0 14 4078706 1897 0.0M 0.0M 0 15 4078721 1871 0.0M 0.0M 0 16 5696 0 0.0M 0.0M 0 17 5705 0 0.0M 0.0M 0 18 5704 0 0.0M 0.0M 0 19 5704 0 0.0M 0.0M 0 20 5702 0 0.0M 0.0M 0 21 5703 0 0.0M 0.0M 0 22 5700 0 0.0M 0.0M 0 23 5702 0 0.0M 0.0M 0
As you can see, the dax engines are working…
In this example I rearranged some cores to the IM-zone to get more DAX pipelines and could use DAX engines from more than one chip, because we are also using more memory than one socket owns (around 1.2 TB). I read that “DAX units and pipelines are not hardwired to certains cores and you can submit work to any DAX unit on a CPU” in this article which explains DAX very good. So I thought it makes sense to spread cores for the zone around sockets. After changing the core pinning from one socket to three sockest I saw three times 8 units in use. That could be the reason why only the middle is more busy than the rest… might be a NUMA effect, I will test to repopulate the IM store to see if it spreads the load soon…