Using on-the-fly decompression for UFS filesystems

by Bernd Schemmer, December 2008

Homepage: http://www.bnsmb.de/

Table of contents

    

Introduction

Purpose

This article describes the usage of fiocompress and dcfs to create compressed files on UFS filesystems.
fiocompress and dcfs are new  in Solaris 10 Update 6 for SPARC.
They also exist in OpenSolaris for SPARC since Newboot for SPARC was integrated.

Scope

Due to the fact there is only a little information about fiocompress and dcfs, this article may contain wrong or incomplete information and should be read as draft documentation by a user of the tool.

Prerequisites

Solaris 10 Update 6 for SPARC or a current release of OpenSolaris for SPARC

Please be aware that these tools are marked with
 

Note that this utility is not a Committed interface. See attributes(5).

in the man page.

References

man pages

fiocompress(1M) man page

dcfs (7FS) man page

Source code

fiocompress source code

dcfs source code

Blog entries

Porting OpenSolaris Project Indiana to SPARC - Part 2

Compression in Ramdisk - Dcfs (Moinak's blog)

Installation

dcfs is part of the Solaris package  SUNWckr - Core Solaris Kernel (Root); fiocompress is part of the Solaris package SUNWcsr Core Solaris, (Root). These packages should be available on every Solaris 10 Update 6 (SPARC) or OpenSolaris installation.

Usage

The program fiocompress(1M) can be used to compress or decompress files on UFS formatted filesystems. fiocompress uses the dcfs(7FS) filesystem which uses gzip to compress or decompress the files.

Note: dcfs is a "layered filesystem" - you can not use it as primary filesystem.

The man page for fiocompress says

The primary use of fiocompress is to compress files in the boot archive.

But that's surely not the only useful usage of this tool.

The usage syntax for fiocompress is

/sbin/fiocompress  -c [-m] [-b block_size] | -d {file}

The parameter -c is used to compress a file and the parameter -d is used to decompress a file (see fiocompress(1M) for a more detailed documentation for fiocompress).

Example usage:

[Fri Dec 05 22:15:07 root@sol9 /var/develop/fiocompress]
# fiocompress -c SunTools.pkg SunTools.pkg.compressed

[Fri Dec 05 22:15:25 root@sol9 /var/develop/fiocompress]
# ls -l SunTools.pkg SunTools.pkg.compressed
-rw-r--r--   1 root     root     7895552 Dec  5 22:14 SunTools.pkg
-rw-r--r--   1 root     root     2030863 Dec  5 22:15 SunTools.pkg.compressed

[Fri Dec 05 22:16:30 root@sol9 /var/develop/fiocompress]
# file SunTools.pkg.compressed
SunTools.pkg.compressed:        data

To decompress the file again use

[Fri Dec 05 22:17:19 root@sol9 /var/develop/fiocompress]
# fiocompress -d SunTools.pkg.compressed SunTools.pkg.decompressed

[Fri Dec 05 22:17:26 root@sol9 /var/develop/fiocompress]
# ls -l SunTools.pkg.compressed SunTools.pkg.decompressed  SunTools.pkg
-rw-r--r--   1 root     root     7895552 Dec  5 22:14 SunTools.pkg
-rw-r--r--   1 root     root     2030863 Dec  5 22:15 SunTools.pkg.compressed
-rw-r--r--   1 root     root     7895552 Dec  5 22:17 SunTools.pkg.decompressed

[Fri Dec 05 22:17:44 root@sol9 /var/develop/fiocompress]
# diff SunTools.pkg SunTools.pkg.decompressed

So, that's not different to the usage of gzip or any other program to compress  files - so why should you use fiocompress?

Advanced Usage

The real advantage of fiocompress is hidden in the man page for fiocompress; there it says:

The following options are supported:

...

-m

    Mark the compressed file for automatic decompression on read. Can be used only in conjunction with -c.
...

So, let's test this option:

[Fri Dec 05 22:21:09 root@sol9 /var/develop/fiocompress]
# fiocompress -m -c  SunTools.pkg SunTools.pkg.compressed_with_m

[Fri Dec 05 22:21:37 root@sol9 /var/develop/fiocompress]
# ls -l SunTools.pkg SunTools.pkg.compressed_with_m
-rw-r--r--   1 root     root     7895552 Dec  5 22:14 SunTools.pkg
-rw-r--r--   1 root     root     7895552 Dec  5 22:21 SunTools.pkg.compressed_with_m

[Fri Dec 05 22:21:47 root@sol9 /var/develop/fiocompress]
# du SunTools.pkg SunTools.pkg.compressed_with_m
15440   SunTools.pkg
3984    SunTools.pkg.compressed_with_m

[Fri Dec 05 22:21:53 root@sol9 /var/develop/fiocompress]
# file SunTools.pkg.compressed_with_m
SunTools.pkg.compressed_with_m: package datastream

[Fri Dec 05 22:22:00 root@sol9 /var/develop/fiocompress]
# diff SunTools.pkg.compressed_with_m SunTools.pkg

Hopefully you see the difference:  The dcfs filesystem adds a decompression on the fly for UFS formatted filesystems!

That's a really cool feature.

Using this feature you can compress your files and still use them without thinking about the compression also on UFS formatted filesystems.

One disadvantage of fiocompress is that it only supports read-only operations, e.g. copying a file compressed with the -m option decompresses the file, e.g.

[Fri Dec 05 22:22:09 root@sol9 /var/develop/fiocompress]
# cp SunTools.pkg.compressed_with_m SunTools.pkg.compressed_with_m.copy

[Fri Dec 05 22:24:25 root@sol9 /var/develop/fiocompress]
# du -ks SunTools.pkg.compressed_with_m SunTools.pkg.compressed_with_m.copy
1992    SunTools.pkg.compressed_with_m
7720    SunTools.pkg.compressed_with_m.copy

From another point of view this an advantage because you do not have to think about how to decompress a file ...

And you can not edit or change a compressed file without first decompressing it. But for files that are only used in read-only mode fiocompress is really useful.

Examples

Create a directory with compressed man pages

# Create the directory structure
#

[Fri Dec 05 22:49:32 root@sol9 /var/develop/fiocompress]
# mkdir man

[Fri Dec 05 22:49:35 root@sol9 /var/develop/fiocompress]
# cd /usr/share/man ; find . -type d  | cpio -pdum /var/develop/fiocompress/man/
0 blocks

# copy and compress the files
#

[Fri Dec 05 22:50:51 root@sol9 /usr/share/man]
# time  for i in $( find . -type f ) ; do echo fiocompress -m -c $i /var/develop/fiocompress/man/$i ; done

[Fri Dec 05 22:51:34 root@sol9 /usr/share/man]
# time  for i in $( find . -type f ) ; do  fiocompress -m -c $i /var/develop/fiocompress/man/$i ; done

real    3m16.741s
user    0m37.202s
sys     1m39.979s

# check the results
#

[Fri Dec 05 23:03:49 root@sol9 /usr/share/man]
# du -ks /usr/share/man /var/develop/fiocompress/man
58941   /usr/share/man
28890   /var/develop/fiocompress/man

# and test the new man directory tree
#

[Fri Dec 05 23:04:40 root@sol9 /usr/share/man]
# man -M /var/develop/fiocompress/man fiocompress
Reformatting page.  Please Wait... done

System Administration Commands                    fiocompress(1M)

NAME
     fiocompress - file compression utility

SYNOPSIS
     /sbin/fiocompress  -c [-m] [-b block_size] | -d {file}

DESCRIPTION
     The fiocompress utility is  a  file  compression  tool  that
     works  together  with  the  dcfs(7FS) file system to perform
     per-file compression. You can use fiocompress to  decompress
     a  compressed  file or mark a compressed file as compressed,
     causing automatic decompression on read. The primary use  of
     fiocompress is to compress files in the boot archive.

     Note that this utility is not  a  Committed  interface.  See
     attributes(5).

OPTIONS
     The following options are supported:

     -b block_size

         Specify a block size for compression. The default  block
         size is 8192.i

     -c

         Compress the specified file.

     -d

         Decompress the specified file.

     -m

         Mark the compressed file for automatic decompression  on
         read. Can be used only in conjunction with -c.

EXIT STATUS
     0

Compressing ISO images

# Compress the ISO image
#

[Sat Dec 06 09:44:58 root@sol9 /]
# time fiocompress -m -c  /export/iso/Solaris10/sol-10-u5-doc-ga-dvd.iso /var/daten/sol-10-u5-doc-ga-dvd.iso.compressed

real    1m23.849s
user    1m15.220s
sys     0m4.894s

# check the result
#
[Sat Dec 06 09:47:05 root@sol9 /]
# ls -l

[Sat Dec 06 09:55:16 root@sol9 /]
# ls -l  /export/iso/Solaris10/sol-10-u5-doc-ga-dvd.iso /var/daten/sol-10-u5-doc-ga-dvd.iso.compressed
-rw-rw-r--   1 root     root     426311680 Apr  5  2008 /export/iso/Solaris10/sol-10-u5-doc-ga-dvd.iso
-rw-r--r--   1 root     root     426311680 Dec  6 09:47 /var/daten/sol-10-u5-doc-ga-dvd.iso.compressed

[Sat Dec 06 09:55:23 root@sol9 /]
# du -ks  /export/iso/Solaris10/sol-10-u5-doc-ga-dvd.iso /var/daten/sol-10-u5-doc-ga-dvd.iso.compressed
416536  /export/iso/Solaris10/sol-10-u5-doc-ga-dvd.iso
305624  /var/daten/sol-10-u5-doc-ga-dvd.iso.compressed

# Use the compressed ISO image
#
[Sat Dec 06 09:55:28 root@sol9 /]
# lofiadm -a /var/daten/sol-10-u5-doc-ga-dvd.iso.compressed
/dev/lofi/24

[Sat Dec 06 09:56:01 root@sol9 /]
# mkdir /tmp/sol10u5

[Sat Dec 06 09:56:07 root@sol9 /]
# mount -o ro -F hsfs /dev/lofi/24 /tmp/sol10u5

[Sat Dec 06 09:56:24 root@sol9 /]
# ls /tmp/sol10u5
Copyright   css         graphics    index.html  pdf

Troubleshooting

fiocompress does not preserve the attributes of the compressed files

This is a bug in the current version of fiocompress; see the blog entry Porting OpenSolaris Project Indiana to SPARC - Part 2 (Update 17.04.2012: The blog entry is not there anymore ...) for a workaround for this bug.

The bugid for this bug is  6724205.

Size restriction

There seems to be a size restriction for fiocompress or dcfs, example:

[Sat Dec 06 10:01:51 root@sol9 /]
# ls -l /export/iso/Solaris10/sol-10-u6-ga1-sparc-dvd.iso
-rw-r--r--   1 root     root     2661416960 Oct 28 20:12 /export/iso/Solaris10/sol-10-u6-ga1-sparc-dvd.iso

[Sat Dec 06 10:01:52 root@sol9 /]
# du -h /export/iso/Solaris10/sol-10-u6-ga1-sparc-dvd.iso
 2.5G   /export/iso/Solaris10/sol-10-u6-ga1-sparc-dvd.iso

[Sat Dec 06 10:02:23 root@sol9 /]
# fiocompress -m -c /export/iso/Solaris10/sol-10-u6-ga1-sparc-dvd.iso /var/daten/sol-10-u6-ga1-sparc-dvd.iso
fiocompress: cannot open /export/iso/Solaris10/sol-10-u6-ga1-sparc-dvd.iso - Value too large for defined data type