{"id":945,"date":"2011-08-13T12:11:36","date_gmt":"2011-08-13T16:11:36","guid":{"rendered":"http:\/\/yuguangzhang.com\/blog\/?p=945"},"modified":"2015-11-19T13:53:59","modified_gmt":"2015-11-19T18:53:59","slug":"javascript-lzma-decompression","status":"publish","type":"post","link":"http:\/\/yuguangzhang.com\/blog\/javascript-lzma-decompression\/","title":{"rendered":"Javascript LZMA Decompression"},"content":{"rendered":"<p>In modern browsers, g-zip compression is a standard feature. The typical compression ratio for a plain text file is 30%, reducing the download time of web content by 70% and making it load 2-3 times faster. In spite of the speed up, g-zip is an old algorithm based on <a title=\"LZ77\" href=\"http:\/\/en.wikipedia.org\/wiki\/LZ77_and_LZ78\">LZ77<\/a>. Since then, newer algorithms have been invented, with LZMA being the standard. <a title=\"LZMA typically produces files half the size compared to g-zip\" href=\"http:\/\/tukaani.org\/lzma\/benchmarks.html\">On Linux, LZMA typically produces files half the size compared to g-zip<\/a>. This tutorial will show you how to use an LZMA compressed file produced by the standard lzma command on Unix machines directly in a client side web application. The rest of the post assumes you have the JavaScript libraries for <a href=\"https:\/\/github.com\/nmrugg\/LZMA-JS\">LZMA<\/a> and <a href=\"https:\/\/github.com\/vjeux\/jsDataView\">binary AJAX<\/a> set up.<\/p>\n<p><strong>First, Make a Compressed File<\/strong><\/p>\n<p>[cc lang=&#8221;bash&#8221;]<\/p>\n<p>echo &#8220;Hello, world.&#8221; | lzma -3 > hello.lzma<\/p>\n<p>[\/cc]<\/p>\n<p><strong>Next, \u00a0Read Binary Data<\/strong><\/p>\n<p>[cc lang=&#8221;html&#8221;]<\/p>\n<p><script src=\"..\/src\/jquery-1.4.4-binary-ajax.js\"><\/script><br \/>\n<script src=\"..\/src\/jdataview.js\"><\/script><br \/>\n<script>\nfunction unzip(data) {\n    \/\/ Make a view on the data\n    var view = new jDataView(data);<\/p>\n<p>    var int_arr = new Array;<\/p>\n<p>    while (view.tell() < view.length) {\n\n        int_arr.push(view.getUint8(view.tell()));\n\n    }\n    console.log(int_arr.length);\n    console.log(int_arr);\n\n}\n\n\/\/ Download the file\n$.get('hello.lzma', unzip, 'binary');\n<\/script><br \/>\n[\/cc]<br \/>\nThis is a pretty simple step, except the while loop counter may be unintuitive. getUint8 increments the file pointer, though it wasn't documented in <a href=\"http:\/\/www.khronos.org\/registry\/webgl\/doc\/spec\/TypedArray-spec.html\">the API specification<\/a>. I spent an hour or so comparing the output in hex. One of the problem was that<br \/>\n[cc]<br \/>\n5d 00 00 08 00 0d 00 00 00 00 00 00 00 00<br \/>\n[\/cc]<br \/>\nis the same as<br \/>\n[cc]<br \/>\n5d 00 00 08 00 0d ff ff ff ff ff ff ff ff<br \/>\n[\/cc]<br \/>\nin little Endian. You can <a href=\"http:\/\/nmrugg.github.com\/LZMA-JS\/demos\/advanced_demo.html\">try it in the decompressor<\/a>, just replace the bytes in the hello world lzma on compression level 3. However, I figured out the problem as soon as I compared view.length and int_arr.length. They were multiples of 2! That always has significance in computing, in this case it meant I was reading every other byte. After correcting the while loop, I moved onto decoding the binary.<\/p>\n<p><strong>Third, Enjoy the Decoding<\/strong><\/p>\n<p>Yes, this is a rather boring thing to do while waiting, but do enjoy it.<br \/>\n[cc lang=\"javascript\"]<br \/>\n    lzma.decompress(int_arr, function(result) {<br \/>\n        $('body').append($('<textarea><\/textarea>').val(result));<br \/>\n    })<br \/>\n[\/cc]<\/p>\n<p><strong>Benefits<\/strong><\/p>\n<p>Using LZMA compression rather than g-zip, I was able to reduce a g-zipped file to 2\/3 of its size, reducing the download time by 33%. The LZMA decompression algorithm could be improved to use an array to store results, joining them at the end, rather than appending to a string. It is not recommended to use this method unless you have large files. The libraries themselves take up about 50kb with g-zip. Furthermore, it is unsuitable for downloads where files are sent directly to the user, without being used by the application, since the user would have the decompression utilities.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>In modern browsers, g-zip compression is a standard feature. The typical compression ratio for a plain text file is 30%, reducing the download time of web content by 70% and making it load 2-3 times faster. In spite of the speed up, g-zip is an old algorithm based on LZ77. Since then, newer algorithms have [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_import_markdown_pro_load_document_selector":0,"_import_markdown_pro_submit_text_textarea":"","footnotes":""},"categories":[5,23],"tags":[144,143,133,145],"class_list":["post-945","post","type-post","status-publish","format-standard","hentry","category-linux","category-programming","tag-binary-ajax","tag-compression","tag-javascript","tag-performance"],"aioseo_notices":[],"_links":{"self":[{"href":"http:\/\/yuguangzhang.com\/blog\/wp-json\/wp\/v2\/posts\/945","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/yuguangzhang.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/yuguangzhang.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/yuguangzhang.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/yuguangzhang.com\/blog\/wp-json\/wp\/v2\/comments?post=945"}],"version-history":[{"count":1,"href":"http:\/\/yuguangzhang.com\/blog\/wp-json\/wp\/v2\/posts\/945\/revisions"}],"predecessor-version":[{"id":1450,"href":"http:\/\/yuguangzhang.com\/blog\/wp-json\/wp\/v2\/posts\/945\/revisions\/1450"}],"wp:attachment":[{"href":"http:\/\/yuguangzhang.com\/blog\/wp-json\/wp\/v2\/media?parent=945"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/yuguangzhang.com\/blog\/wp-json\/wp\/v2\/categories?post=945"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/yuguangzhang.com\/blog\/wp-json\/wp\/v2\/tags?post=945"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}