QNAP SSH/RSYNC traffic shaped on uplink from Orange fiber due to CoS/QoS/DSCP

I've spent some time on this and there were some things I learned in the process that are not very straightforward. 

It all started with me finding out the sync process for data from my NAS got super slow (128KB/s -> 1 MBit/s). It uses ssh with rsync and I was able to replicate this by checking scp transfer out being capped as well.

After digging around I've isolated it to the provider (or their router). There is nothing on the router that would indicate throttling of ssh. When I tried SCP towards a different port I was still capped. WTH? When I quickly spun-up openvpn instance the throttling went away even with the extra overhead and using rsync/ssh.

There must be something that detects SSH with deep packet inspection, I've concluded and called my provider (Orange SK). They said all should be okay and told me to call them when I'm on site. Since that is going to be in two weeks I kept digging. Reaching out to my friends on my facebook, one of the guys (Juraj Lutter) mentioned that I should check DSCP.

And indeed setting -o IPQoS throughput with my scp command made my transfer fast again. The simple explanation is that default is lowlatency(for interactive) and throughput for non-interactive, but that gets mixed up with rsync. 

Now on to setting it up in QNAP. Oh the horrors - patching config generation by using sed over /etc/init.d/login.sh to get proper /etc/config/ssh/sshd_config was the easy part. Figuring out that even if your HOME points to /root the /share/homes/admin/.ssh/config gets checked during ssh/scp init was the harder part. Anyways after I've set it everywhere the backup now works fine even for rsync transfers. 

Remember: If your ssh/scp is rate-limited with some operators, make sure to set "IPQoS threshold" in your ssh/config.

DZone shortest code to print first 30 fibonacci numbers without using numbers

There's interesting challenge on Dzone website. The task is to print Fibonacci sequence for first 30 numbers in the format "order: fib number". Once constraint is that you can't use any numbers in the code. Shortest answer wins.
My take on that problem in Ruby is here (85characters):

Code snippet - 30 fibonacci number without using numbers in ruby on Snipplr

[edit] During morning run I've realised that using array is just PITA and by getting rid of indexes for last and previous elements I can save some space (64 characters):

Code snippet - 30 fibonacci numbers without using numbers in Ruby on Snipplr

[edit] Since there was groovy solution with 64 characters I've shaved off another two getting down to 62.

Code snippet - 30 Fibonacci Numbers Without Using Numbers In Ruby(62) on Snipplr

Later when I read discussion they have dropped requirement to use : in the output so I decided to update solution and found another nifty trick to get rid of another two characters ending up at 58(60) chars which can be easily reduced to 57 by removing whitespace after p.

Code snippet - 30 Fibonacci Numbers Without Using Numbers In Ruby(58) on Snipplr

One-liners: Deformalize

Recently I needed quick hack for following situation. Given an input like this:

A 1
A 2
B 3
C 4
C 5
C 6
C 7
D 8

I wanted to generate following:

text A -p 1 -p 2
text B -p 3
text C -p 4 -p 5 -p 6 -p 7
text D -p 8

After some thought I have generated it with this awk one-liner.

cat file | awk '{ if (X==$1) { printf " -p "$2 ; } else { printf "\ntext " $1 " -p "$2 ; X=$1 } }'

What would you do? Is there an easier way ?

ENV is not a Hash but Object in Ruby

Recently I hit this funny problem in ruby, when trying to merge options to ENV. Although ENV supports most of Hash functions, it's not really Hash instance:

irb(main):001:0> ENV.class
=> Object
irb(main):002:0> env = { 'X'=>1 }
=> {"X"=>1}

irb(main):003:0> env.class
=> Hash

There are some useful methods missing from ENV like merge!

irb(main):004:0> ENV.merge(env)
NoMethodError: undefined method `merge' for #<object:0xb7f54814>
       from (irb):4

Workaround is simple enough, merge! hash manualy:

irb(main):012:0> env.each { |k,v| ENV[k.to_s]=v.to_s }
=> {"X"=>1}
irb(main):013:0> ENV["X"]
=> "1"

Work around core library class mocks/stubs in ruby

Just a short post about mocking/stubbing your Ruby functions. Let's have a class like this:

class ConfigurableClass
  attr_reader :option
  def configure_from_file(file)
    json = JSON(IO.read(file))
    @option = json['my']['option']
  end
end

One way to test this would be to mock IO.read with Mocha

describe ConfigurableClass do
 it "can configure from json" do
   data =  '{ "my": { "option" : "hello" } }'
     IO.any_instance.expect(:read).with("x.json").returns(data)
     cc = ConfigurableClass.new
     cc.configure_from_file("x.json")
     cc.option.should eq "hello"
   end
end

This is tricky if your code does IO.read anywhere else which you obviously don't want to mock. However with some refactoring code is much cleaner.

class ConfigurableClass
  attr_reader :option
  def read_config_file(file)
    IO.read(file)
  end

  def configure_from_file(file)
    json = JSON(read_config_file)
    @option = json['my']['option']
  end
end

Then in the rspec it's easy to mock only function specific to ConfigurableClass.

describe ConfigurableClass do
  it "can configure from json" do
    data =  '{ "my": { "option" : "hello" } }'
    ConfigurableClass.any_instance.expect(read_config_file).with("x.json").returns(data)
    cc = ConfigurableClass.new
    cc.configure_from_file("x.json")
    cc.option.should eq "hello"
end

GZIP/BZIP2 progress indicator

Have you ever needed to figure out how long gzip/bzip compression would take for the file and found it difficult to present it to the user ? Especially for scripts dealing with large files, this would be one of the major usability issues. Well there's a solution for the problem with the use of tool called pipe viewer. You can find more details on how to use it on it's home page, but I'll show you how I've used it to show progress for bzip2 compression of large files.

filename="file_to_bzip2"
size=`stat -L "$filename" -c "%s"`
pv -s $size "$filename" | bzip2 -c > "$filename.bz2"

Generates following output:

# output
2.75GB 0:32:47 [ 1.5MB/s] [===================>     ] 61% ETA 0:20:16

GoF design patterns reference card

Gang Of Four design patterns is probably most known developer book ever. As it is little too verbose on single patterns, people created reference cards for it. Most popular would probably be one at DZone. Recently I've tried to find one, that I can put on my Kindle and eventually found it on Google docs, so I've thought maybe I should share it.