Author: pressy

Starting with Solaris 11.4 you got a new interface for device latency without using dtrace. The information was always there but now you can use a “human readable” command. That might bring easier understanding and analyzing the disk subsystem.

I ran an IO calibrate in a 18c database which resides on NVMe flash drives:

root@t7primary01:~# iostat -x -L c5t1d0 c1t1d0 1
                     extended device statistics
device     r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b
blkdev2   29.9   13.1  185.7   81.4  0.0  0.0    0.0    0.1   0   0
latency          range         count      density distribution
                  <4us             0        0.00%        0.00%
                 4-8us          3527        0.02%        0.02%
                8-16us       1224166        5.74%        5.76%
               16-32us       4020858       18.85%       24.61%
               32-64us       1225365        5.75%       30.35%
              64-128us      13243355       62.10%       92.45%
             128-256us       1561222        7.32%       99.77%
             256-512us         34758        0.16%       99.93%
            512-1024us          4642        0.02%       99.96%
                 1-2ms          3624        0.02%       99.97%
                 2-4ms          5758        0.03%      100.00%
                 4-8ms            97        0.00%      100.00%
                8-16ms            33        0.00%      100.00%
                 >16ms             0        0.00%      100.00%
                 total      21327405
blkdev3   32.0   14.0  198.5   86.8  0.0  0.0    0.0    0.1   0   0
latency          range         count      density distribution
                  <4us             0        0.00%        0.00%
                 4-8us          2848        0.01%        0.01%
                8-16us       1280834        5.62%        5.64%
               16-32us       4197957       18.44%       24.07%
               32-64us       1355014        5.95%       30.02%
              64-128us      14167408       62.22%       92.24%
             128-256us       1734203        7.62%       99.86%
             256-512us         29629        0.13%       99.99%
            512-1024us           895        0.00%       99.99%
                 1-2ms           614        0.00%      100.00%
                 2-4ms           939        0.00%      100.00%
                 4-8ms           104        0.00%      100.00%
                8-16ms            27        0.00%      100.00%
                 >16ms             0        0.00%      100.00%
                     extended device statistics
device     r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b
blkdev2 49234.1    1.0 307781.4   20.0  0.0  5.4    0.0    0.1   0  20
latency          range         count      density distribution
                 <16us             0        0.00%        0.00%
               16-32us             1        0.00%        0.00%
               32-64us            21        0.04%        0.05%
              64-128us         42324       88.06%       88.10%
             128-256us          5647       11.75%       99.85%
             256-512us            67        0.14%       99.99%
            512-1024us             1        0.00%       99.99%
                 1-2ms             1        0.00%      100.00%
                 2-4ms             2        0.00%      100.00%
                  >4ms             0        0.00%      100.00%
                 total         48064
blkdev3 52145.2    0.0 325741.9    0.0  0.0  5.7    0.0    0.1   0  18
latency          range         count      density distribution
                 <32us             0        0.00%        0.00%
               32-64us            28        0.05%        0.05%
              64-128us         44430       87.26%       87.31%
             128-256us          6374       12.52%       99.83%
             256-512us            84        0.16%       99.99%
            512-1024us             3        0.01%      100.00%
               >1024us             0        0.00%      100.00%
                 total         50919

That’s a nice overview…

Just read a nice blog about ZFS performance I would like to recommend:
Six Metrics for Measuring ZFS Pool Performance

Good explanations and maybe great input on your next layout… It’s from a BSD engineer, but the basics are true for all ZFS implementations…

Check it out! 🙂

Last time I had big fun upgrading a bunch of Solaris clusters from solaris 11.3 and cluster 4.3 to solaris 11.4 and cluster 4.4 on some T5 servers. Having Solaris Cluster installed gave me an error on pkg update… some days later I had to install some other Solaris Clusters and got a similar message regarding some other dependencies… in both situations, solaris cluster update to 4.4 fails, the solution was including the ha-cluster repository also as a, I would call it, search first or search order…

Some errors I saw:

root@sol11.3server:~# pkg update -nvv
 Creating Plan (Running solver): -
 pkg update: No solution was found to satisfy constraints
 No solution found to update to latest available versions.
 This may indicate an overly constrained set of packages are installed.

 latest incorporations:

 pkg://solaris/consolidation/SunVTS/SunVTS-incorporation@8.2.2,5.11-11.4.0.0.1.3.8:20180328T214527Z
 pkg://solaris/consolidation/X/X-incorporation@11.4,5.11-11.4.3.0.1.5.0:20181107T002739Z
 pkg://solaris/consolidation/cde/cde-incorporation@11.4,5.11-11.4.0.0.1.14.0:20180814T153157Z
 pkg://solaris/consolidation/cns/cns-incorporation@11.4,5.11-11.4.0.0.0.11.0:20171211T161950Z
 pkg://solaris/consolidation/dbtg/dbtg-incorporation@0.5.12,5.12-5.12.0.0.0.87.0:20151116T001936Z
 pkg://solaris/consolidation/desktop/desktop-incorporation@11.4,5.11-11.4.0.0.0.12.0:20180103T012026Z
 pkg://solaris/consolidation/desktop/gnome-incorporation@11.4,5.11-11.4.1.0.1.1.0:20180829T212505Z
 pkg://solaris/consolidation/ips/ips-incorporation@11.4,5.11-11.4.0.0.1.14.0:20180814T145051Z
 pkg://solaris/consolidation/l10n/l10n-incorporation@11.4,5.11-11.4.0.0.1.14.1:20180814T145027Z
 pkg://solaris/consolidation/ldoms/ldoms-incorporation@11.4,5.11-11.4.0.0.1.14.0:20180814T173637Z
 pkg://solaris/consolidation/man/man-incorporation@5.12,5.12-5.12.0.0.0.101.0:20160612T200100Z
 pkg://solaris/consolidation/osnet/osnet-incorporation@11.4,5.11-11.4.3.0.1.5.0:20181107T002954Z
 pkg://solaris/consolidation/sfw/sfw-incorporation@11.4,5.11-11.4.0.0.1.14.0:20180814T153159Z
 pkg://solaris/consolidation/sic_team/sic_team-incorporation@0.11.5,5.11-11.4.0.0.1.7.0:20180521T161050Z
 pkg://solaris/consolidation/smcc/smcc-incorporation@11.4,5.11-11.4.0.0.0.1.0:20170724T145344Z
 pkg://solaris/consolidation/solaris_re/solaris_re-incorporation@11.4,5.11-11.4.3.0.1.5.0:20181107T004038Z
 pkg://solaris/consolidation/sunpro/sunpro-incorporation@11.4,5.11-11.4.0.0.1.4.0:20180409T193641Z
 pkg://solaris/consolidation/userland/userland-incorporation@11.4,5.11-11.4.3.0.1.4.0:20181024T214602Z
 pkg://solaris/entire@11.4,5.11-11.4.3.0.1.5.0:20181107T004627Z

 Dependency analysis is unable to determine the cause.
 Try specifying expected versions to obtain more detailed error messages.

What I learned from Oracle Support is to set a new option called “-p” issuing set-publisher, where the man page says “If no publisher is specified, all publishers in repo_uri are added or updated as appropriate.” My support engineer explained that there are dependencies which are not in the right repo for any reason, so it seems to be fixed in a future release easily, but right now it is like it is… solaris-dev and cluster-dev team, it’s your game 😉

This magic “-p” makes some strange publisher settings but after using that the upgrade and the installation worked like it should without errors. Downloaded the cluster release files and shared them per NFS:

sol11.3server:~# pkg publisher
PUBLISHER                   TYPE     STATUS P LOCATION
solaris                     origin   online F http://reposerver/solaris-support
ha-cluster                  origin   online F http://reposerver/ha-cluster-support/
solarisstudio               origin   online F http://reposerver/solarisstudio-support/
sol11.3server:~#
sol11.3server:~# pkg unset-publisher solaris
sol11.3server:~# pkg unset-publisher ha-cluster
sol11.3server:~# pkg unset-publisher solarisstudio
sol11.3server:~# pkg set-publisher -g http://reposerver/solaris-support
sol11.3server:~# pkg set-publisher -p file:////net/reposerver/myCluster_release_repo
sol11.3server:~# pkg set-publisher -g http://reposerver/ha-cluster-support/ ha-cluster
sol11.3server:~# pkg set-publisher -g http://reposerver/solarisstudio-support/ solarisstudio
sol11.3server:~#
sol11.3server:~# pkg publisher
PUBLISHER                   TYPE     STATUS P LOCATION
solaris                     origin   online F file:///net/reposerver/myCluster_release_repo/
solaris                     origin   online F http://reposerver/solaris-support/
ha-cluster                  origin   online F file:///net/reposerver/myCluster_release_repo/
ha-cluster                  origin   online F http://reposerver/ha-cluster-support/
solarisstudio               origin   online F http://reposerver/solarisstudio-support/
sol11.3server:~#

This also works for Oracle’s “public” repositories but you will have to accept ALL necessary publishers for your SSL/HTTPS key and certificate on http://pkg-register.oracle.com/ and download these files again to set your publisher over ssl to pkg.oracle.com, also the important release publisher in that step!

root@sol11.4server:~# cat /etc/release
                            Oracle Solaris 11.4 SPARC
  Copyright (c) 1983, 2018, Oracle and/or its affiliates.  All rights reserved.
                           Assembled 06 November 2018
root@sol11.4server:~# cat /etc/cluster/release
             Oracle Solaris Cluster 4.4.1.2.0 for Solaris 11 sparc
  Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.
root@sol11.4server:~# uname -a
SunOS sol11.4server 5.11 11.4.3.5.0 sun4v sparc sun4v
root@sol11.4server:~#

Happy upgrading and installing 😉