GZIP/BZIP2 progress indicator

Have you ever needed to figure out how long gzip/bzip compression would take for the file and found it difficult to present it to the user ? Especially for scripts dealing with large files, this would be one of the major usability issues. Well there's a solution for the problem with the use of tool called pipe viewer. You can find more details on how to use it on it's home page, but I'll show you how I've used it to show progress for bzip2 compression of large files.

filename="file_to_bzip2"
size=`stat -L "$filename" -c "%s"`
pv -s $size "$filename" | bzip2 -c > "$filename.bz2"

Generates following output:

# output
2.75GB 0:32:47 [ 1.5MB/s] [===================>     ] 61% ETA 0:20:16

GoF design patterns reference card

Gang Of Four design patterns is probably most known developer book ever. As it is little too verbose on single patterns, people created reference cards for it. Most popular would probably be one at DZone. Recently I've tried to find one, that I can put on my Kindle and eventually found it on Google docs, so I've thought maybe I should share it.

YouTube video offline mode

Recently I enrolled to AI class online course. Each week they submit dozen of explanatory and Q&A videos on youtube and reference them from AI-class site. As I mostly have time to study those videos when I'm offline I've decided to explore possibilities of getting videos to local computer for offline viewing. Purpose of this post is just to summarize steps which I did in order to achieve this goal on linux.


Get tools

First thing first, I had to equip myself with tools.

youtube-dl
After unsuccessful attempt to use clive/cclive, I decided to use youtube-dl which you can download at github project page. Although it is single purpose youtube only downloader it works and does exactly what I need. You need python interpreter to run it.
grake
Grake is utility to parse youtube links out of HTML and it's hosted at code.google.com. Grake is Perl module, but installation is straightforward, as one only has to follow instructions in INSTALL file.


Get list of videos
First step is using youtube.com to find playlist of videos. For example I've just typed 'AI class unit9 videos' to google/youtube to get this link: 'http://www.youtube.com/playlist?list=PL9163DC3C43AF7612'.


Next step is to download list using Grake and edit it

$ cd ~/Videos/Unit9
$ grake http://www.youtube.com/playlist?list=PL9163DC3C43AF7612 < Unit9.lst

Downloaded file looks like:

http://youtube.com/watch?v=videoseries
http://youtube.com/watch?v=DgH6NaJHfVQ
http://youtube.com/watch?v=9D35JSWSJAg
http://youtube.com/watch?v=9QMZQkKuYjo
http://youtube.com/watch?v=YfSBYf9h7qk
...

I've just deleted first line using vim.


Get videos
Last step is to get the videos by using youtube-dl. I've used this:

$ cat Unit9.lst | xargs youtube-dl -tA

With -tA options I told downloader to use video name as filename instead of funky hash and autoprefix it with number as it's downloaded.

How to convert multi-page HTML e-book for Kindle

Recently I've got link to the interesting e-book Architecture of Open Source applications. Because I prefer to read books on my Kindle and there was no MOBI version I've decided to prepare it myself. Here is step-by-step guide on how to convert multi-page HTML to format suitable for Kindle on your Ubuntu/Debian/Other linux.

Getting data

First of all we need to get HTML/CSS and image files to local machine. On my machine it's as simple as:

$mkdir ~/aosabook; cd ~/aosabook
$wget -I en,images -nd -r -k -l 3 http://www.aosabook.org/en

We want to download all documents recursively but only from en and images directories, don't create directory structure to local copy and replace paths in html documents so they're locally referenced. For more details check man wget.

Convert to single HTML

Now we have all data downloaded in ~/aosabook and to check whether book is readable we just have to open file:///home/aosabook/ in browser.
Because the structure of the web is multi-page, we have to do additional step. Convert the multi page document into single page. I've used htmldoc utility for this. Run htmldoc and do following:

• input tab
  • choose Document Type: Book
  • Use add files button to add all html files from ~/aosabook/
  • Select cover image
• output tab
  • set output to file
  • set output path to ~/aosabook/aosabook.html
  • set output format to html

Play with some other options namely width (set it to 600px for kindle 3) and hit generate.

Convert to Mobi

In order to convert to MOBI format suitable for Kindle, I've just added generated aosabook.html as book to Calibre (you use calibre for your kindle management right?) and clicked on book to convert to Mobi and upload to device.