zfsize – small script to find the biggest files on ZFS

I found some time to script and was looking into zdb and what could be done with it. I would say that it is a nice feature to ask “what is your biggest file in that filesystem” (rather a mega find command). You could also find files which where deleted but are still in the ZFS because a process is using it. I also wrote a script for that, you will find it in my next post.

root@server:~# ./zfsize.sh -h
small script to find the biggest files on ZFS
Usage:
                ./zfsize.sh -z <ZFS> [-o tempdir] [-c count]
root@server:~# ./zfsize.sh -z rpool/downloads -c 2
ZFS = rpool/downloads
Mountpoint = /downloads
TempDir = /tmp
This may take a while ...
/downloads/sol-10-u11-ga-sparc-dvd.iso  2207.50 MB
/downloads/sol-11_1-repo-full.iso       2896.00 MB
root@server:~#

#!/usr/bin/bash
#set -x
###################################################
#
# zfsize v0.1
#
# ZFS file sizes
#
# small script to find the biggest files on ZFS
#
# 16.09.2016, written by Martin Presslaber
#
###################################################
help ()
{
		print "small script to find the biggest files on ZFS"
                print "Usage:"
                print "\t\t$0 -z <ZFS> [-o tempdir] [-c count]"
}
########## preTESTS #############
OS=`uname -s`
RELEASE=`uname -r`
VERS=`uname -v`
ZONE=`zonename`
if [[ $OS != SunOS ]]
then
        print "This script will only work on Solaris"
        exit 1
fi
[[ $ZONE == global ]] || print "This script will only work in the global zone"
[[ $VERS == 1[1-9].[1-9] ]] && SOLARIS=new
if [ ${RELEASE#*.} -gt 10 ] ;
then
        ID=$(/usr/bin/whoami)
else
        ID=$(/usr/ucb/whoami)
fi
if [ $ID != "root" ]; then
        echo "$ID, you must be root to run this program."
        exit 1
fi
if [ $# -lt 1 ]
        then
                help && exit 1
        fi
#[[ $1 != "-[az]" ]] && help && exit 1
########## Options ###########
TEMPDIR="/tmp"
while getopts "z:o:c:h" args
do
        case $args in
        z)
                ZFS=$OPTARG
                ZFSlist=`zfs list $ZFS 2>/dev/null | nawk -v ZFS=$ZFS '$1~ZFS {print $0}'`
                [[ $ZFSlist == "" ]] && print "$ZFS does not seem to be a ZFS" && exit 1
                ZFSmountpoint=`zfs list $ZFS 2>/dev/null | nawk -v ZFS=$ZFS '$1~ZFS {print $NF}'`
        ;;

        o)
        TEMPDIR=$OPTARG
        [[ -d $TEMPDIR ]] || print "$TEMPDIR does not exist!" && exit 1
        ;;

	c)
	COUNT="-$OPTARG"
	;;

        h|*)
                help && exit 1
        ;;
        esac
done
shift $(($OPTIND -1))

######### Let's go #########
print "ZFS = $ZFS"
print "Mountpoint = $ZFSmountpoint"
print "TempDir = $TEMPDIR"
print "This may take a while ... "

zdb -dddd $ZFS |\
nawk -v MP=$ZFSmountpoint 'BEGIN { printf("FILE\tSIZE\n"); }
$0 ~/ZFS plain file$/ { interested = 1; }
interested && $1 == "path" { printf(MP"%s", $2); }
interested && $1 == "size" { printf("\t%.2f MB\n", $2/1024/1024); }
interested && $1 == "Object" { interested = 0; }'  > $TEMPDIR/zfsize.tmp
sort -nk 2,2 $TEMPDIR/zfsize.tmp > $TEMPDIR/zfsize-sorted.tmp
tail $COUNT $TEMPDIR/zfsize-sorted.tmp
# clean up
rm $TEMPDIR/zfsize.tmp
rm $TEMPDIR/zfsize-sorted.tmp
##################### EOF #####################

Using different HW Features in a Box

I wrote a small article for my company how you could use Oracle’s new SPARC hardware for different layers in your datacentre… original in German, could be found on SPARC T7-1 testing In-Memory, DAX and Crypto Engines
Some findings and interesting points translated for my blog:

So what I thought about are classic tasks normally found on several servers, build in one box. All of them could benefit from different features which come with M7 or S7 chips.
The database in the backend will profit from the big memory bandwidth and the SQL Offload Engines called DAX, data analytics accelerators. In the combination Oracle says in their PowerPoints the database could scan up to 170 billion rows per second with those streaming engines with a measured bandwidth from 160GB/sec per socket. Wow… and that’s measurement; the M7 processor hardware facts are talking about 4 memory controller units per socket which could handle 333 GB/sec raw memory bandwidth per processor. (It seems that DDR4 is the “bottleneck” not the CPU…) compared to the latest Xeon E7 88xx v4 (Q2/16) with 102GB/sec mentioned on Intel’s ARK technical details pages.

The next layer could be the application itself. With 8 threads per core a perfect fit for a high user load and with critical threads the process has more exclusive access to the hardware. Perfect for running a wide mix of workloads, some will be designed for throughput, others for low latency.

The third level could be something like a reverse proxy with a SSO backend or something. The proxy could take the application sessions if not already encrypted and use the build in cryptographic accelerators on the processor to encrypt. Solaris itself and some standard applications using these engines already, like Apache, IPsec, Java, KSSL, OpenSSL, ZFS Crypto. But not only Oracle software like the database and WebLogic are supporting Solaris’ Crypto Framework, also IBM with DB2, Informix, IBM HTTP Server or WebSphere are certified with IBM Global Security Kit to use SPARC’s hardware encryption (IBM GSKit v8).

Oracle SPARC processors can handle 15 industry standard algorithms and a bunch of random number generators (AES, Camellia, CRC32c, DES, 3DES, DH, DSA, ECC, MD5, RSA, SHA-1, SHA-224, SHA-256, SHA-384, SHA-512). (BTW; Xeons would have 7 crypto instructions and 5 on-chip accelerated algorithms; IBM Power8 = 6 instructions and 8 accelerated.)

The last level could be the way to the internet, separated to the other domains. Solaris offers a build-in firewall, load-balancer or other web utilities to handle the connections. Having Solaris on SPARC in the front helps you easily to prevent so called script-kiddies using their found hacks and attacks because on one side SPARC is big endian based, so standard attacks will run into the “wrong direction” compared to little endian on x86. On the other side the new SPARC processors are protected by “silicon-secure-memory”. When an application requests some new memory to use via malloc(), the operating system tags the block of memory with a version number, and gives the app a pointer to that memory. Whenever a pointer is used to access a block of memory, the pointer’s version number must match the memory block’s version number, or an exception will be triggered. The version numbers are checked in real-time by the processor with a tiny overhead – an extra one percent of execution time, according to Oracle’s benchmarks. (more infos at theregister )
So imaging using all of these features a whole datacentre could be hosted on a single server or if it comes down to availability you could build a cluster with failover or live migration between the servers.

t7datacenter

solaris repo

Local Repository

Hopefully you have a lot of Solaris systems; so it might make sense to create your own mirrored local repository for Solaris and other packages.

Let's start creating the repo with the GA build avaiable from the download sites for everyone:

root@psvsparc1:/downloads/11.3repo# ls
install-repo.ksh           sol-11_3-repo_2of5.zip     sol-11_3-repo_4of5.zip     sol-11_3-repo_md5sums.txt
sol-11_3-repo_1of5.zip     sol-11_3-repo_3of5.zip     sol-11_3-repo_5of5.zip
root@psvsparc1:/downloads/11.3repo# install-^C
root@psvsparc1:/downloads/11.3repo# chmod +x install-repo.ksh
root@psvsparc1:/downloads/11.3repo# ./install-repo.ksh -d /ai/repo/
Using sol-11_3-repo download.
Uncompressing sol-11_3-repo_1of5.zip...done.
Uncompressing sol-11_3-repo_2of5.zip...done.
Uncompressing sol-11_3-repo_3of5.zip...done.
Uncompressing sol-11_3-repo_4of5.zip...done.
Uncompressing sol-11_3-repo_5of5.zip...done.
Repository can be found in /ai/repo/.
root@psvsparc1:/downloads/11.3repo# pkgrepo rebuild -s /ai/repo/
Initiating repository rebuild.

OK, now we could use this local repo but we also want other servers to use it. We need a new service for that:

root@psvsparc1:~# svccfg -s application/pkg/server setprop pkg/inst_root=/ai/repo
root@psvsparc1:~# svccfg -s application/pkg/server setprop pkg/readonly=true
root@psvsparc1:~# svccfg -s application/pkg/server setprop pkg/port=8080
root@psvsparc1:~# svcadm refresh application/pkg/server
root@psvsparc1:~# svcadm enable application/pkg/server

That's it... now let's connect the client:

root@client:~# pkg unset-publisher solaris
Updating package cache                           1/1
root@client:~# pkg set-publisher -O http://psvsparc1:8080 solaris
root@client:~# pkg publisher
PUBLISHER                   TYPE     STATUS P LOCATION
solaris                     origin   online F http://psvsparc1:8080/

OK... at the end of the day we want to get updates... you will need an active support contract and then you can connect to the support repository... this example updates the local solaris repository to the latest or newest patchset / SRU avaiable at oracle. SSL files can be obtained from http://pkg-register.oracle.com/

root@psvsparc1:~# pkgrecv -s https://pkg.oracle.com/solaris/support -d /ai/repo \
> --key /var/pkg/ssl/pkg.oracle.com.key.pem \
> --cert /var/pkg/ssl/pkg.oracle.com.certificate.pem -m latest '*'
Processing packages for publisher solaris ...
Retrieving and evaluating 6949 package(s)...
PROCESS                                         ITEMS    GET (MB)   SEND (MB)
Completed                                   1191/1191   2836/2836   4383/4383

root@psvsparc1:~#
root@psvsparc1:~# pkgrepo -s /ai/repo refresh
Initiating repository refresh.
root@psvsparc1:~#

The client sees the GA version, his own installation and the newest one:

root@client:~# pkg list -af entire
NAME (PUBLISHER)                                  VERSION                    IFO
entire (solaris)                                  0.5.11-0.175.3.11.0.6.0    ---
entire (solaris)                                  0.5.11-0.175.3.10.0.7.0    i--
entire (solaris)                                  0.5.11-0.175.3.1.0.5.0     ---
root@client:~# 
other repositories

You might want to use other Oracle software repositories... no problem:

root@psvsparc1:~#  zfs create -o atime=off rpool/ai/repo-ss
root@psvsparc1:~#  pkgrepo create /ai/repo-ss
root@psvsparc1:~#  pkgrecv --key /var/pkg/ssl/pkg.oracle.com.key.pem --cert /var/pkg/ssl/pkg.oracle.com.certificate.pem -s https://pkg.oracle.com/solarisstudio/support/ -d  /ai/repo-ss '*'
Retrieving and evaluating 347 package(s)...
PROCESS                                         ITEMS    GET (MB)   SEND (MB)
Completed                                     130/130   4667/4667 14730/14730

root@psvsparc1:~# pkgrepo -s /ai/repo-ss/ refresh
root@psvsparc1:~# pkgrepo get -s /ai/repo-ss
SECTION    PROPERTY                     VALUE
publisher  prefix                       ""
repository check-certificate-revocation False
repository signature-required-names     ()
repository trust-anchor-directory       /etc/certs/CA/
repository version                      4
root@psvsparc1:~# svccfg -s pkg/server add solarisstudio
root@psvsparc1:~# svccfg -s pkg/server:solarisstudio addpg pkg application
root@psvsparc1:~# svccfg -s pkg/server:solarisstudio setprop pkg/port=8082
root@psvsparc1:~# svccfg -s pkg/server:solarisstudio setprop pkg/inst_root=/ai/repo-ss
root@psvsparc1:~# svccfg -s pkg/server:solarisstudio addpg general framework
root@psvsparc1:~# svccfg -s pkg/server:solarisstudio addpropvalue general/enabled boolean: true
root@psvsparc1:~# svccfg -s pkg/server list
:properties
default
solarisstudio
root@psvsparc1:~# svcadm enable application/pkg/server:solarisstudio
root@psvsparc1:~#
root@psvsparc1:~#
root@psvsparc1:~#

And another one:

root@psvsparc1:~# svccfg -s pkg/server add ha-cluster
root@psvsparc1:~# svccfg -s pkg/server:ha-cluster addpg pkg application
root@psvsparc1:~# svccfg -s pkg/server:ha-cluster setprop pkg/port=8081
root@psvsparc1:~# svccfg -s pkg/server:ha-cluster setprop pkg/inst_root=/ai/repo-sc
root@psvsparc1:~# svccfg -s pkg/server:ha-cluster addpg general framework
root@psvsparc1:~# svccfg -s pkg/server:ha-cluster addpropvalue general/enabled boolean: true
root@psvsparc1:~# svccfg -s pkg/server list
:properties
default
solarisstudio
ha-cluster
root@psvsparc1:~#
root@psvsparc1:~# svcadm enable application/pkg/server:ha-cluster
root@psvsparc1:~# svcs -a | grep ha-cluster
online         14:34:23 svc:/application/pkg/server:ha-cluster
root@psvsparc1:~#
root@psvsparc1:~# netstat -aun | grep 808
      *.8082               *.*            root       4414 pkg.depotd          0      0  128000      0 LISTEN
      *.8081               *.*            root       3940 pkg.depotd          0      0  128000      0 LISTEN
      *.8080               *.*            root       2081 pkg.depotd          0      0  128000      0 LISTEN
root@psvsparc1:~#

Client:

root@client:~# pkg set-publisher -O http://psvsparc1:8081 ha-cluster
root@client:~# pkg publisher
PUBLISHER                   TYPE     STATUS P LOCATION
solaris                     origin   online F http://psvsparc1:8080/
ha-cluster                  origin   online F http://psvsparc1:8081/
root@client:~# pkg list -af ha-cluster/group-package/ha-cluster-framework-full
NAME (PUBLISHER)                                  VERSION                    IFO
ha-cluster/group-package/ha-cluster-framework-full (ha-cluster) 4.3-0.24.0                 ---
ha-cluster/group-package/ha-cluster-framework-full (ha-cluster) 4.2-0.30.0                 ---
ha-cluster/group-package/ha-cluster-framework-full (ha-cluster) 4.1-0.18.2                 ---
ha-cluster/group-package/ha-cluster-framework-full (ha-cluster) 4.0.0-0.22.1               ---
root@client:~# pkg set-publisher -O http://psvsparc1:8082 solarisstudio
root@client:~# pkg publisher
PUBLISHER                   TYPE     STATUS P LOCATION
solaris                     origin   online F http://psvsparc1:8080/
ha-cluster                  origin   online F http://psvsparc1:8081/
solarisstudio               origin   online F http://psvsparc1:8082/
root@client:~#

Performance

Tried once to update around 50 Solaris boxes and ran into a time-out on some... found that on docs.oracle.com

# svccfg -s application/pkg/server:default setprop pkg/threads=200
# svcadm refresh application/pkg/server:default
# svcadm restart application/pkg/server:default