QNAP SSH/RSYNC traffic shaped on uplink from Orange fiber due to CoS/QoS/DSCP

I've spent some time on this and there were some things I learned in the process that are not very straightforward. 

It all started with me finding out the sync process for data from my NAS got super slow (128KB/s -> 1 MBit/s). It uses ssh with rsync and I was able to replicate this by checking scp transfer out being capped as well.

After digging around I've isolated it to the provider (or their router). There is nothing on the router that would indicate throttling of ssh. When I tried SCP towards a different port I was still capped. WTH? When I quickly spun-up openvpn instance the throttling went away even with the extra overhead and using rsync/ssh.

There must be something that detects SSH with deep packet inspection, I've concluded and called my provider (Orange SK). They said all should be okay and told me to call them when I'm on site. Since that is going to be in two weeks I kept digging. Reaching out to my friends on my facebook, one of the guys (Juraj Lutter) mentioned that I should check DSCP.

And indeed setting -o IPQoS throughput with my scp command made my transfer fast again. The simple explanation is that default is lowlatency(for interactive) and throughput for non-interactive, but that gets mixed up with rsync. 

Now on to setting it up in QNAP. Oh the horrors - patching config generation by using sed over /etc/init.d/login.sh to get proper /etc/config/ssh/sshd_config was the easy part. Figuring out that even if your HOME points to /root the /share/homes/admin/.ssh/config gets checked during ssh/scp init was the harder part. Anyways after I've set it everywhere the backup now works fine even for rsync transfers. 

Remember: If your ssh/scp is rate-limited with some operators, make sure to set "IPQoS threshold" in your ssh/config.

DZone shortest code to print first 30 fibonacci numbers without using numbers

There's interesting challenge on Dzone website. The task is to print Fibonacci sequence for first 30 numbers in the format "order: fib number". Once constraint is that you can't use any numbers in the code. Shortest answer wins.
My take on that problem in Ruby is here (85characters):

Code snippet - 30 fibonacci number without using numbers in ruby on Snipplr

[edit] During morning run I've realised that using array is just PITA and by getting rid of indexes for last and previous elements I can save some space (64 characters):

Code snippet - 30 fibonacci numbers without using numbers in Ruby on Snipplr

[edit] Since there was groovy solution with 64 characters I've shaved off another two getting down to 62.

Code snippet - 30 Fibonacci Numbers Without Using Numbers In Ruby(62) on Snipplr

Later when I read discussion they have dropped requirement to use : in the output so I decided to update solution and found another nifty trick to get rid of another two characters ending up at 58(60) chars which can be easily reduced to 57 by removing whitespace after p.

Code snippet - 30 Fibonacci Numbers Without Using Numbers In Ruby(58) on Snipplr

One-liners: Deformalize

Recently I needed quick hack for following situation. Given an input like this:

A 1
A 2
B 3
C 4
C 5
C 6
C 7
D 8

I wanted to generate following:

text A -p 1 -p 2
text B -p 3
text C -p 4 -p 5 -p 6 -p 7
text D -p 8

After some thought I have generated it with this awk one-liner.

cat file | awk '{ if (X==$1) { printf " -p "$2 ; } else { printf "\ntext " $1 " -p "$2 ; X=$1 } }'

What would you do? Is there an easier way ?

ENV is not a Hash but Object in Ruby

Recently I hit this funny problem in ruby, when trying to merge options to ENV. Although ENV supports most of Hash functions, it's not really Hash instance:

irb(main):001:0> ENV.class
=> Object
irb(main):002:0> env = { 'X'=>1 }
=> {"X"=>1}

irb(main):003:0> env.class
=> Hash

There are some useful methods missing from ENV like merge!

irb(main):004:0> ENV.merge(env)
NoMethodError: undefined method `merge' for #<object:0xb7f54814>
       from (irb):4

Workaround is simple enough, merge! hash manualy:

irb(main):012:0> env.each { |k,v| ENV[k.to_s]=v.to_s }
=> {"X"=>1}
irb(main):013:0> ENV["X"]
=> "1"

Work around core library class mocks/stubs in ruby

Just a short post about mocking/stubbing your Ruby functions. Let's have a class like this:

class ConfigurableClass
  attr_reader :option
  def configure_from_file(file)
    json = JSON(IO.read(file))
    @option = json['my']['option']
  end
end

One way to test this would be to mock IO.read with Mocha

describe ConfigurableClass do
 it "can configure from json" do
   data =  '{ "my": { "option" : "hello" } }'
     IO.any_instance.expect(:read).with("x.json").returns(data)
     cc = ConfigurableClass.new
     cc.configure_from_file("x.json")
     cc.option.should eq "hello"
   end
end

This is tricky if your code does IO.read anywhere else which you obviously don't want to mock. However with some refactoring code is much cleaner.

class ConfigurableClass
  attr_reader :option
  def read_config_file(file)
    IO.read(file)
  end

  def configure_from_file(file)
    json = JSON(read_config_file)
    @option = json['my']['option']
  end
end

Then in the rspec it's easy to mock only function specific to ConfigurableClass.

describe ConfigurableClass do
  it "can configure from json" do
    data =  '{ "my": { "option" : "hello" } }'
    ConfigurableClass.any_instance.expect(read_config_file).with("x.json").returns(data)
    cc = ConfigurableClass.new
    cc.configure_from_file("x.json")
    cc.option.should eq "hello"
end

GZIP/BZIP2 progress indicator

Have you ever needed to figure out how long gzip/bzip compression would take for the file and found it difficult to present it to the user ? Especially for scripts dealing with large files, this would be one of the major usability issues. Well there's a solution for the problem with the use of tool called pipe viewer. You can find more details on how to use it on it's home page, but I'll show you how I've used it to show progress for bzip2 compression of large files.

filename="file_to_bzip2"
size=`stat -L "$filename" -c "%s"`
pv -s $size "$filename" | bzip2 -c > "$filename.bz2"

Generates following output:

# output
2.75GB 0:32:47 [ 1.5MB/s] [===================>     ] 61% ETA 0:20:16

GoF design patterns reference card

Gang Of Four design patterns is probably most known developer book ever. As it is little too verbose on single patterns, people created reference cards for it. Most popular would probably be one at DZone. Recently I've tried to find one, that I can put on my Kindle and eventually found it on Google docs, so I've thought maybe I should share it.

Design of t-shirt for cryptoclass 2012

YouTube video offline mode

Recently I enrolled to AI class online course. Each week they submit dozen of explanatory and Q&A videos on youtube and reference them from AI-class site. As I mostly have time to study those videos when I'm offline I've decided to explore possibilities of getting videos to local computer for offline viewing. Purpose of this post is just to summarize steps which I did in order to achieve this goal on linux.


Get tools

First thing first, I had to equip myself with tools.

youtube-dl
After unsuccessful attempt to use clive/cclive, I decided to use youtube-dl which you can download at github project page. Although it is single purpose youtube only downloader it works and does exactly what I need. You need python interpreter to run it.
grake
Grake is utility to parse youtube links out of HTML and it's hosted at code.google.com. Grake is Perl module, but installation is straightforward, as one only has to follow instructions in INSTALL file.


Get list of videos
First step is using youtube.com to find playlist of videos. For example I've just typed 'AI class unit9 videos' to google/youtube to get this link: 'http://www.youtube.com/playlist?list=PL9163DC3C43AF7612'.


Next step is to download list using Grake and edit it

$ cd ~/Videos/Unit9
$ grake http://www.youtube.com/playlist?list=PL9163DC3C43AF7612 < Unit9.lst

Downloaded file looks like:

http://youtube.com/watch?v=videoseries
http://youtube.com/watch?v=DgH6NaJHfVQ
http://youtube.com/watch?v=9D35JSWSJAg
http://youtube.com/watch?v=9QMZQkKuYjo
http://youtube.com/watch?v=YfSBYf9h7qk
...

I've just deleted first line using vim.


Get videos
Last step is to get the videos by using youtube-dl. I've used this:

$ cat Unit9.lst | xargs youtube-dl -tA

With -tA options I told downloader to use video name as filename instead of funky hash and autoprefix it with number as it's downloaded.

Groovy Junit tests in Eclipse Indigo

I'm playing with Groovy and Grails a little bit and recently I had a problem running unit-test from recent Eclipse Indigo (3.7) with groovy-eclipse plugin installed.

Problem manifests itself with following error, when I try to righ-click and "run as" junit test-case.

junit.framework.AssertionFailedError: No tests found in path.to.your.class

I've spent some time trying to find solution and it was obvious. Think it might be helpful to keep solution in some form. I'll probably post more Groovy/Grails related stuff in the future, so why not start with this.

Let's have simple class MyClass

package sk.adino.poc.groovy.unittests

class MyClass {
  x() { return 1;}
}

and test class MyClassTest in same package

package sk.adino.poc.groovy.unittests

class MyClassTest extends GroovyTestCase {
  testX { assert x==1 }
}

Right-click in eclipse and run as JUnit test should work, but you have to:

1. have groovy-eclipse plugin installed
2. add external JAR pointing to junit-x.y.z.jar from your groovy installation lib directory - otherwise it won't even compile
3. add void as a return type of your testX function so it should look like:

package sk.adino.poc.groovy.unittests

class MyClassTest extends GroovyTestCase {
  void testX { assert x==1 }
}

Java puzzler - Letter went missing

Recently I've implemented function that takes String and populates long so that it contains character values from the back of the string. However my unit test is failing because one Letter is missing. Are you able to put it to the right place so that unit tests will not fail without debugging the code?

import static org.junit.Assert.*;
import org.junit.Test;

public class StringToolsTest {

	static final char [] data = {0xca, 0xfe, 0xeb, 0xab, 0xee};
	static final String string5 = String.valueOf(data);
	static final String string0 = "";
	static final String string1 = "a";
	static final String string9 = "987654321";
	
	@Test
	public void testFillBackwards() {
		long l = 0;
		assertEquals(0xcafeebabeeL, StringTools.fillBackwards(string5, l));
		assertEquals(0L, StringTools.fillBackwards(string0, l));
		assertEquals(0x61L, StringTools.fillBackwards(string1, l));
		assertEquals(0x3837363534333231L, StringTools.fillBackwards(string9, l));
	}
}

public class StringTools {
	public static long fillBackwards(String s, long d)
	{
		long result = 0L;
		byte[] bytes = s.getBytes();
		for (int i=0; i<Math.min(8,bytes.length); i++){
			result |= ((bytes[bytes.length-i-1] & 0xff) << (8*i));
		}
		return result;
	}
}

Solution bellow (highlight to show):

The problem is on the line 29 in "& 0xFF". I should be doing &0xFFL instead. Why ? bytes[x]&0xff is by default int (both operands are widened to int), but then << is applied, result of which is type of left operand (java spec) and this is |= to long result. Solution is to make & operator to work on long values by adding L to 0xFF.

Puzzler on snipplr.

Question Interview

One of my friends got this question on his interview: "Given is an arbitrary long string, revert order of words in the string". If the input is "Hello World!", function should modify string to: "World! Hello". Function must have this interface: "void revert(char * string)". It must work in linear time O(n) and have no special memory requirements M(1).

My solution (spoiler alert)

Java puzzler - How many integers are in the box ?

This is most basis Java Puzzler. Question is, how will the output look like ?

• Will it print out powers of 10 starting from 1 ending at 10000000.
• Or will it print just 1.
• Or will it print any other number.
• Or will it loop forever.
• Or anything else.

package sk.adino.puzzlers;

import static org.junit.Assert.*;
import org.junit.Test;

public class AutoboxingTest {

	@Test
	public void test() {
		int i=1;
		while(i<10000000) {
			System.out.println("value:" + i);
			Integer j=i;
			Integer k=i;
			if (j==k){
				i=i*10;
			}else break;
		}
		assertEquals(i,1000);
	}
}

Puzzler @ Snipplr

Puzzler for C developer role

Today, I've created puzzler for C developer at company I work for. This is one of rare cases where we can publish the code, and it's few lines anyhow, so here I give it a shot. You can try to solve the puzzler, as described in comments. It's very easy and you should have no problem solving it without Solaris. All you need is a C compiler.

Bonus question: why did I choose (R) and (C) instead of R and C ?

Bellow is the code which you can find also on snipplr:

1. /** This program should be compiled using following command:
3.  $ cc -g -o test_hr_test test_hr_test.c
5.  Original developer Malvin left company to pursue his dream and went
6.  to live with wild ligers. No-one ever heard any word about him since
7.  he left. We've managed to recover source code from his source repo.,
8.  but program seems to be broken. Strangely Malvin used iso-8859-1 as
9.  his console encoding.
11.  Hints:
12.  - program does not compile
13.  - we were not able to recover whole function permutateBlockRev()
14.  - program crashes on our X86 solaris 10 matchine (not on linux)
16.  Are you the one which can help us recover the password ?
17.  Send it to: rpc-hr@ri-rpc.sk
18.  */
20. #include <stdio.h>
21. #include <string.h>
23. /** bit manipulation macros */
24. #define CLRBIT( STR, IDX ) ( (STR)[(IDX)/8] &= ~(0x01 << (7 - ((IDX)%8))) )
25. #define SETBIT( STR, IDX ) ( (STR)[(IDX)/8] |= (0x01 << (7 - ((IDX)%8))) )
26. #define GETBIT( STR, IDX ) (( ((STR)[(IDX)/8]) >> (7 - ((IDX)%8)) ) & 0x01)
28. /** permutation works on unsigned char blocks of size 8 */
29. #define BLOCK_SIZE 8
30. typedef unsigned char[BLOCK_SIZE] block;
32. /** Function does reverse permutation to DES initial block permutation as described at
33.  http://en.wikipedia.org/wiki/DES_supplementary_material
34.  @param b - block which will be permutated in-place
35.  @note indexes start from 0 contrary to wikipedia initialization vector - C like index
36.  */
37. void permutateBlockRev(block b)
38. {
39. int i;
40. static const int initialPermuteMap[64] =
41. {
42. 57, 49, 41, 33, 25, 17, 9, 1,
43. 59, 51, 43, 35, 27, 19, 11, 3,
44. 61, 53, 45, 37, 29, 21, 13, 5,
45. 63, 55, 47, 39, 31, 23, 15, 7,
46. 56, 48, 40, 32, 24, 16, 8, 0,
47. 58, 50, 42, 34, 26, 18, 10, 2,
48. 60, 52, 44, 36, 28, 20, 12, 4,
49. 62, 54, 46, 38, 30, 22, 14, 6
50. };
52. /* There are few lines of code missing */
53. }
55. int main (void)
56. {
57. unsigned char str[BLOCK_SIZE + 1] = "DummyStr";
58. block permutated = { 0x7a, 0xd2, 0x21, 0x3c, 0x05, 0x85, 0x8d, 0x71 };
60. permutateBlockRev(permutated);
61. strcpy(str, permutated);
62. printf("Password is: %s\n", str);
63. }

How to convert multi-page HTML e-book for Kindle

Recently I've got link to the interesting e-book Architecture of Open Source applications. Because I prefer to read books on my Kindle and there was no MOBI version I've decided to prepare it myself. Here is step-by-step guide on how to convert multi-page HTML to format suitable for Kindle on your Ubuntu/Debian/Other linux.

Getting data

First of all we need to get HTML/CSS and image files to local machine. On my machine it's as simple as:

$mkdir ~/aosabook; cd ~/aosabook
$wget -I en,images -nd -r -k -l 3 http://www.aosabook.org/en

We want to download all documents recursively but only from en and images directories, don't create directory structure to local copy and replace paths in html documents so they're locally referenced. For more details check man wget.

Convert to single HTML

Now we have all data downloaded in ~/aosabook and to check whether book is readable we just have to open file:///home/aosabook/ in browser.
Because the structure of the web is multi-page, we have to do additional step. Convert the multi page document into single page. I've used htmldoc utility for this. Run htmldoc and do following:

• input tab
  • choose Document Type: Book
  • Use add files button to add all html files from ~/aosabook/
  • Select cover image
• output tab
  • set output to file
  • set output path to ~/aosabook/aosabook.html
  • set output format to html

Play with some other options namely width (set it to 600px for kindle 3) and hit generate.

Convert to Mobi

In order to convert to MOBI format suitable for Kindle, I've just added generated aosabook.html as book to Calibre (you use calibre for your kindle management right?) and clicked on book to convert to Mobi and upload to device.

WTF #2

This must be work of some dark evil mastermind.

void  DECIMAL_TO_HEXA( int num, char *hex )
{
     int aux_hex[32],i=0;
     char hexa[32], resu[32];
 
     while(num>0) {
         aux_hex[i]=num%16;
         num/=16;
         i++;
     }
 
     resu[0]='\0';
     i--;
     for(i;i>=0;i--) {
         if (aux_hex[i]<10) { 
                 sprintf(hexa, "%d", aux_hex[i]); 
                 strcat(resu, hexa); 
         }
         else if (aux_hex[i]==10) strcat(resu, "A"); 
         else if (aux_hex[i]==11) strcat(resu, "B"); 
         else if (aux_hex[i]==12) strcat(resu, "C"); 
         else if (aux_hex[i]==13) strcat(resu, "D"); 
         else if (aux_hex[i]==14) strcat(resu, "E"); 
         else if (aux_hex[i]==15) strcat(resu, "F"); 
     } 
     strcpy(hex, resu);
}

C/C++ puzzler #2 - Is it typo ?

Ok, here is next C/C++ specific puzzler. Puzzler here is to guess, what will be result of program ?

1. It won't compile


2. It will compile, but it will segfault when we try to run it


3. It will run and print 4 times 30


4. Anything else you can think of

Highlight text bellow code to find out correct answer.

/* compile with cc -g -o test_typedef_sizeof test_typedef_sizeof.c */
#include <stdio.h>
#include <string.h>

typedef char name_t[30];

void fn(name_t value)
{
    printf("%d\n", sizeof(name_t));
    printf("%d\n", sizeof(value));
}

int main (void)
{
    name_t aa;

    printf("%d\n", sizeof(name_t));
    printf("%d\n", sizeof(aa));

    fn(aa);
    return 0;
}

I'll just show you program output:

30
30
30
4

Is it not, what you have expected ? Fine, let's walk the program together. On entry to main() program prints sizeof(user defined type name_t), which has length of 30 chars. Next it'll print sizeof(instance of name_t) which again is 30 chars long. Then it calls function fn(instance of name_t), In fn() it prints sizeof(user defined type name_t) which is again 30 bytes. But when it tries to print sizeof(value) it prints 4, because arrays are passed as reference to first element in C/C++. And the sizeof(pointer) is 4 respectively 8 for 64 bit system.

Java EE factsheets

Stefan Jäger recently posted bunch of Java EE fact-sheets on his blog. Stefan created fact-sheets for: SF and SL session beans, MDB's and JMS, Transactions and security, and finally JPA. Very nice, and available as PDF downloads.

C/C++ puzzler #1 - Where do you want to go today ?

First puzzler is in C/C++ and it is based on puzzler found in book Java Puzzlers by Neal Gafter and Joshua Bloch.
Problem:
We have following C program:

/* compile with: cc -g -o test_xxx test_xxx.c */
#include <stdio.h>
#include <stdlib.h>

int main (void)
{
printf("firefox.exe");
http://www.google.com;
printf(":new\n");
exit(0);
}

Options:
What will the program do:
a) print firefox.exe
b) run firefox.exe open http://www.google.com in new tab
c) fail to compile
d) other


Solution: (highlight to see)

First look at source, and you probably think that this couldn't compile. In fact it does. Real puzzler is, why does it compile. Problem seems to be in line http://www.google.com; But let's take a closer look. Is it problem ? It compile because of long forgotten feature of C, which is labels. (Except for kernel source, you'll find plenty of labels there ;) What compiler see is label http: followed by comment //www.google.com;

Puzzlers series

I guess everyone have been in situation when he/she has written code, which didn't behave as it should. Most of these times, problem was of PEBKAC (problem exists between keyboard and chair) type. Computers (nearly) always do, what they're told to do. Under influence of Java Puzzlers book by Neal Gafter and Joshua Bloch, I've decided to start series of posts to this blog, which would cover situations when program doesn't do what we think it should. At first posts will be mostly in C/C++, but i think more languages will be covered in the future. Posts will be formated, so that you'll see only problem with possible answers. Solution will have font color set to background color, or some close color, so you'd have to highlight block in order to see it. You're more than welcome to add your own puzzlers. Just contact me.