QNAP SSH/RSYNC traffic shaped on uplink from Orange fiber due to CoS/QoS/DSCP

I've spent some time on this and there were some things I learned in the process that are not very straightforward. 

It all started with me finding out the sync process for data from my NAS got super slow (128KB/s -> 1 MBit/s). It uses ssh with rsync and I was able to replicate this by checking scp transfer out being capped as well.

After digging around I've isolated it to the provider (or their router). There is nothing on the router that would indicate throttling of ssh. When I tried SCP towards a different port I was still capped. WTH? When I quickly spun-up openvpn instance the throttling went away even with the extra overhead and using rsync/ssh.

There must be something that detects SSH with deep packet inspection, I've concluded and called my provider (Orange SK). They said all should be okay and told me to call them when I'm on site. Since that is going to be in two weeks I kept digging. Reaching out to my friends on my facebook, one of the guys (Juraj Lutter) mentioned that I should check DSCP.

And indeed setting -o IPQoS throughput with my scp command made my transfer fast again. The simple explanation is that default is lowlatency(for interactive) and throughput for non-interactive, but that gets mixed up with rsync. 

Now on to setting it up in QNAP. Oh the horrors - patching config generation by using sed over /etc/init.d/login.sh to get proper /etc/config/ssh/sshd_config was the easy part. Figuring out that even if your HOME points to /root the /share/homes/admin/.ssh/config gets checked during ssh/scp init was the harder part. Anyways after I've set it everywhere the backup now works fine even for rsync transfers. 

Remember: If your ssh/scp is rate-limited with some operators, make sure to set "IPQoS threshold" in your ssh/config.

One-liners: Deformalize

Recently I needed quick hack for following situation. Given an input like this:

A 1
A 2
B 3
C 4
C 5
C 6
C 7
D 8

I wanted to generate following:

text A -p 1 -p 2
text B -p 3
text C -p 4 -p 5 -p 6 -p 7
text D -p 8

After some thought I have generated it with this awk one-liner.

cat file | awk '{ if (X==$1) { printf " -p "$2 ; } else { printf "\ntext " $1 " -p "$2 ; X=$1 } }'

What would you do? Is there an easier way ?

GZIP/BZIP2 progress indicator

Have you ever needed to figure out how long gzip/bzip compression would take for the file and found it difficult to present it to the user ? Especially for scripts dealing with large files, this would be one of the major usability issues. Well there's a solution for the problem with the use of tool called pipe viewer. You can find more details on how to use it on it's home page, but I'll show you how I've used it to show progress for bzip2 compression of large files.

filename="file_to_bzip2"
size=`stat -L "$filename" -c "%s"`
pv -s $size "$filename" | bzip2 -c > "$filename.bz2"

Generates following output:

# output
2.75GB 0:32:47 [ 1.5MB/s] [===================>     ] 61% ETA 0:20:16

YouTube video offline mode

Recently I enrolled to AI class online course. Each week they submit dozen of explanatory and Q&A videos on youtube and reference them from AI-class site. As I mostly have time to study those videos when I'm offline I've decided to explore possibilities of getting videos to local computer for offline viewing. Purpose of this post is just to summarize steps which I did in order to achieve this goal on linux.


Get tools

First thing first, I had to equip myself with tools.

youtube-dl
After unsuccessful attempt to use clive/cclive, I decided to use youtube-dl which you can download at github project page. Although it is single purpose youtube only downloader it works and does exactly what I need. You need python interpreter to run it.
grake
Grake is utility to parse youtube links out of HTML and it's hosted at code.google.com. Grake is Perl module, but installation is straightforward, as one only has to follow instructions in INSTALL file.


Get list of videos
First step is using youtube.com to find playlist of videos. For example I've just typed 'AI class unit9 videos' to google/youtube to get this link: 'http://www.youtube.com/playlist?list=PL9163DC3C43AF7612'.


Next step is to download list using Grake and edit it

$ cd ~/Videos/Unit9
$ grake http://www.youtube.com/playlist?list=PL9163DC3C43AF7612 < Unit9.lst

Downloaded file looks like:

http://youtube.com/watch?v=videoseries
http://youtube.com/watch?v=DgH6NaJHfVQ
http://youtube.com/watch?v=9D35JSWSJAg
http://youtube.com/watch?v=9QMZQkKuYjo
http://youtube.com/watch?v=YfSBYf9h7qk
...

I've just deleted first line using vim.


Get videos
Last step is to get the videos by using youtube-dl. I've used this:

$ cat Unit9.lst | xargs youtube-dl -tA

With -tA options I told downloader to use video name as filename instead of funky hash and autoprefix it with number as it's downloaded.