Compress a String for writing into a Database

Jun 15, 2009 at 12:39 PM
Edited Jun 15, 2009 at 1:23 PM

Hi Guys,

I have this code that was working with the inbuilt gzip compression but as the ratio was appalling I decided to give DotNetZip a try instead. I am compressing HTML into a SQLite DB so I would expect the compression ratio to be very good.

I have compiled the following code by following various examples but to be honest I don't really understand it so I can't tell what the issue is.

I am able to deflate the string "hello world" and then inflate it back to "hello world" so functionally it seems fine, but with inbuilt compression turned on my db goes from 7.25mb to 7.23mb. And with DotNetZip compression my db went to 7.89mb, so I am now really confused.

 

Imports System.IO.Compression
Imports System.Text
Imports System.IO
Imports Ionic

Public Class ZipString
Public Shared Function Deflate(ByVal text As String) As String
Dim buffer As Byte() = Encoding.UTF8.GetBytes(text)
Dim ms As New MemoryStream()
'Using zip As New GZipStream(ms, CompressionMode.Compress, True)
Using Zip As New Ionic.Zlib.ZlibStream(ms, _
Zlib.CompressionMode.Compress, Zlib.CompressionLevel.BEST_SPEED, True)
Zip.Write(buffer, 0, buffer.Length)
End Using

ms.Position = 0
Dim outStream As New MemoryStream()

Dim compressed As Byte() = New Byte(ms.Length - 1) {}
ms.Read(compressed, 0, compressed.Length)

Dim gzBuffer As Byte() = New Byte(compressed.Length + 3) {}
System.Buffer.BlockCopy(compressed, 0, gzBuffer, 4, compressed.Length)
System.Buffer.BlockCopy(BitConverter.GetBytes(buffer.Length), 0, gzBuffer, 0, 4)
Return Convert.ToBase64String(gzBuffer)
End Function

Public Shared Function Inflate(ByVal compressedText As String) As String
Dim gzBuffer As Byte() = Convert.FromBase64String(compressedText)
Using ms As New MemoryStream()
Dim msgLength As Integer = BitConverter.ToInt32(gzBuffer, 0)
ms.Write(gzBuffer, 4, gzBuffer.Length - 4)

Dim buffer As Byte() = New Byte(msgLength - 1) {}

ms.Position = 0
'Using zip As New GZipStream(ms, CompressionMode.Decompress)
Using Zip As New Ionic.Zlib.ZlibStream(ms, _
Zlib.CompressionMode.Decompress, Zlib.CompressionLevel.BEST_SPEED, True)
Zip.Read(buffer, 0, buffer.Length)
End Using

Return Encoding.UTF8.GetString(buffer)
End Using
End Function
End Class



Any help is greatly appreciated.

Edit: Warning people do not copy and paste from word into this forum, it goes mental!

 

 

Jun 15, 2009 at 3:13 PM

Turns out I was being dumb and looking in the wrong place I was passing it into my DB all wrong.

 

That code actually works fine, see it as my contribution to the community :)

 

Thanks for a great component.

Coordinator
Jun 15, 2009 at 6:55 PM

great, Glad you figured it out.

 

Coordinator
Jun 16, 2009 at 11:32 AM

Hey, L33t, couple comments:

The compression level is ignored when CompressionMode is Decompress.  Therefore...this code:

Using Zip As New Ionic.Zlib.ZlibStream(ms, _
                                       Zlib.CompressionMode.Decompress,_
                                       Zlib.CompressionLevel.BEST_SPEED, _
                                       True)
    Zip.Read(buffer, 0, buffer.Length)
End Using

...might be better written as:

Using Zip As New Ionic.Zlib.ZlibStream(ms, _
                                       Zlib.CompressionMode.Decompress,_
                                       True)
    Zip.Read(buffer, 0, buffer.Length)
End Using

Also, you know that by using BEST_SPEED, you are getting the least effective compression possible. You mentioned that the built-in DeflateStream wasn't effective for you. Using BEST_SPEED you may be missing out on 20% compression. Maybe worth testing to find out the difference between BEST_SPEED and BEST_COMPRESSION in your case.

By the way, the enums BEST_SPEED and BEST_COMPRESSION change to BestSpeed and BestCompression in the current version of v1.8.