BitWackr Blog

Deduplication beyond backup

One Gigabyte per Second In-Line Disk Data Compression Using an “Off-the-Shelf” Server

leave a comment »

Exar’s BitWackr C Series adds disk data compression to “off-the-shelf” Windows and Linux servers. BitWackr C Series is a combination of software and specialized hardware (a data reduction ASIC installed on a PCIe card) that simply, effectively and affordably extends hardware-accelerated data compression to disk storage volumes. Suitable for a broad range of Windows and Linux applications including biomedical research, scientific data analysis, HPC visualization, HPC simulation, geo-seismic investigation, data warehousing, data analytics, backup to disk, NAS, email, archive and other storage-capacity-intensive applications, BitWackr C Series typically cuts both storage requirements and the energy required to power and cool disk storage by up to 80% for very little money.

Unlike data compression performed in software using general-purpose processors, BitWackr C Series reduces data and storage requirements without adding to CPU workload or system overhead. This frees CPU cycles to perform higher value computational tasks and does not elongate processing times. Offloading in-line data compression to a PCIe card translates into an imperceptible change in application performance or responsiveness.

A prospective customer expressed interest in equipping off-the-shelf servers with disk data compression that was capable of delivering 880 Megabytes per second of in-line data compression to a disk storage subsystem. We accepted this challenge, but we also thought it might be interesting to demonstrate in-line data compression using commodity server hardware at an even higher rate, so we set a target of writing data at one Gigabyte per second (1 GB/second or 1024 MB/second) and set out to demonstrate that performance level.

The test system was configured as follows:

 

At the start of the test, we installed a single Exar Express DR1625 data reduction card in the Dell server as the compression hardware for the BitWackr C Series software. We next configured SATA SSDs as the “back end” data store and used LVM to create volumes to which the BitWackr C Series would write compressed data. The Linux “spew” command sent random data to each BitWackr-compressed volume.

The first set of tests resulted in aggregate throughput in the range of 400 – 500 MB/second. BitWackr was performing as expected, but we observed that the performance constraint was the bandwidth of the storage sub-system.

Identifying a back-end data store for our tests that was capable of ingesting data at Gigabyte per second rates was problematic. In the past, Gigabyte per second rates required multiple Fibre Channel interface connections for data transfer to the storage sub-system. SATA SSDs weren’t giving us the hoped for performance, so we decided to overcome this obstacle by trying PCIe Solid State storage that claims to be capable of sustaining Gigabyte per second data rates.  

In the next test, we replaced the SATA SSDs with a PCIe solid state device as the storage sub-system. We saw write throughput improve to between 500 and 550 MB/second. Thinking that we would probably not achieve our objective using this product, we tried something new.

The next test replaced the first PCIe solid-state device with a single LSI Solid State Storage Card (LSI SSS6200) http://www.lsi.com/storage_home/products_home/solid_state_storage/sss6200/index.html  to act as the storage sub-system. We immediately saw write throughput improve to about 800 MB/second. This was close to our customers’ requirement of 880 MB/second, but since our objective was to achieve an aggregate data rate of one Gigabyte per second, we pressed on.

We next installed a second LSI SSS6200 in the Dell server and used LVM to create volumes across both LSI cards. We saw a further performance increase to 1008 MB/second. But with our goal of one Gigabyte (1024 MB) per second within reach, we decided to run one more set of tests.

For the final tests, we configured two different volume groups on each LSI SSS6200. One volume group was configured for BitWackr metadata I/O and the other as the “back end” data store.

Using this configuration, we were able to measure an aggregate performance bandwidth of 1028 MB/second, thus exceeding our objective of one Gigabyte per second. The Linux “top” command confirmed that the Dell R710’s CPUs were coasting along because the “heavy lifting” of Gigabyte per second data compression was being effectively offloaded to the Exar DR1625 card. 

While demonstrating Gigabyte per second in-line disk data compression using commodity server hardware may sound a bit like a science fair project, there was sound business and technical justification for performing this test. We first wanted to prove to prospective server and appliance builders who are solving data-intensive challenges that in-line disk data compression at Gigabyte per second rates was achievable without the need for specialized server hardware.

Finally, we wanted to make sure that Exar customers understand that highly effective, high performance disk data compression for biomedical research, scientific data analysis, HPC visualization, HPC simulation, geo-seismic investigation, data warehousing, data analytics, backup to disk, NAS, email, archive and other storage-capacity-intensive applications can now be brought to market quickly, cost-effectively and with minimal development time or effort.

I believe we accomplished these objectives.  

Although our objective was to test and demonstrate BitWackr C Series performance, we also learned a great deal about the I/O characteristics of PCIe storage from two leading vendors. We needed a high performance storage back-end to support BitWackr’s operational capabilities and this class of storage device delivered the performance we needed.

Exar’s BitWackr C Series data compression is available to OEMs, SIs and VARs for immediate evaluation. The product has a list price of well under $1,000 for both the hardware and software in single unit quantities.

Advertisements

Written by BitWackr

August 13, 2010 at 1:06 pm

Posted in Uncategorized

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: