Google Online Services via Command Line Interface CLI
This exploration trigger was a quasi-failed YouTube upload. In fact, I am not even sure that the upload failed. All I know is that I started an upload of a long video which quickly saturated my meagre (0.5 Mbps) uplink. My Internet became unusable while I had to wait an estimated 600 minutes. Needless to say, I let it go overnight, only to come back to this in the morning:
YouTube isn’t telling me much, except that it’s “processing” it (though the 0:00 suggests failure):
I’ve decided that enough is enough and looked at a few alternatives (rather than simply using a browser other than Chromium).
0 Solution
After all, since YouTube has removed any time limitation to the videos I can upload I might as well start producing more videos. In High Definition, file sizes can easily reach several gigabytes even with the time limit (15 minutes). For instance, here is a browser upload of a 720p video of 11 or so minutes after about 8 hours, overnight:
Obviously, it’s inhuman to wait that long and have your Internet “incapacitated” throughout this ordeal. If you use VoIP, anybody calling you in this interval might not be able to hear you. Since the upload cannot be sped up due to physical limitations, it has to be managed. There are 3 ways to manage it that I can think of:
- Use a separate application for such uploads that allows specifying bandwidth limits. If such an application does not exist, use some kind of proxy server on the same computer that allows traffic shaping / limiting on a per-application basis, and upload to YouTube only from a specific browser thus controlled.
- Identify a persistent property (server IP if fixed or TCP / UDP ports) of this upload process that allows it to be managed / limited on the router via QoS rules.
- Perform such uploads only from specific computers and manage their connection on the router via QoS rules.
The second idea has low chances of success, while the third is particularly enticing since if we discover some linux tools we might be able to run such a process right from the NAS where the files to be uploaded usually reside.
Let us now look at the applications we can identify.
1 VLMC
VideoLAN Movie Creator is a free, open source video editor from the same guys who brought us the amazing VLC. It borrows from that project the wide OS and codecs support. Though currently in a very early release (“pre-alpha”), it shows promise and it already features YouTube integration. If I’d be using it to upload videos to YouTube I could also edit them before. On the downside, this program is quite buggy still so I risk having the “Aw, Snap!” moment.
2 WMM
Windows Movie Maker (or Live! Movie Maker) is a video editor provided freely by Microsoft as part of its “Live” suite. It is a mature product and it even has video acceleration if you have a newer video card. However, they’ve been moving away from YouTube integration into a partnership with Vimeo, which makes sense when you’re competing with Google. Even if YouTube upload is kept in the feature set, I’d prefer a quicker, faster option for upload as I don’t always need to edit videos before upload.
3 DVDVideoSoft
This company makes a suite of applications that include downloading YouTube videos, converting them to MP3, editing, burning and uploading to YouTube. Everything is modularized and can be downloaded separately (some work even as Chrome extensions). This is a good option if memory or CPU load are problems you are confronted with often (I know I am). They have a nice video explaining how to upload. The programs are not open source, but the developer claims there are no spyware or trojans included.
4 Expression
Microsoft Expression is, in its free incarnation, a screencast application. Currently at version 4 SP2, it can use a plugin to publish directly to YouTube.
5 Gdata
Now we’re getting close!
Python-gdata is a set of Python scripts that allow using Google Services via CLI. There are two applications of interest based on it:
- Youtube-upload is a command-line Python script that easily uploads videos to – you guessed it! – Youtube. The goal is to install it on my NAS, but first I installed it on my Cygwin clone, MobaXterm. The command to upload a video would be:
$ youtube-upload --email=myemail@gmail.com --password=mypassword
--title="A.S. Mutter" --description="A.S. Mutter plays Beethoven"
--category=Music --keywords="mutter, beethoven" anne_sophie_mutter.flv - GoogleCL allows full access to Google services via the command line, as shown in example scripts. It’s been covered in a manual, on lh (1, 2), by publicint (.msi Python install), muo, ln, mte. An upload example is:
$ google youtube post --category People killer_robots.avi
Youtube-upload works quite well, but one has to specify even the title and keywords, otherwise it complains. It also complains that pycurl is missing, meaning it cannot use HTTP and must use the “basic gdata API.”
On Cygwin / MobaXterm, GoogleCL launches Lynx to request access. Google online doesn’t like it and then it bans it. However, after quitting Lynx an interesting message was left on the screen:
Please specify user: …
6 [main] python 5040 fhandler_dev_zero::fixup_mmap_after_fork: requested 0x7F540000 != 0x0 mem alloc base 0x7F450000, state 0x2000, size 65536, Win32 error 487
3927 [main] python 5040 C:\Users\__\AppData\Local\Temp\ib_MobaXterm\bin\python.exe: *** fatal error - recreate_mmaps_after_fork_failed
1492 [main] python 4208 fhandler_dev_zero::fixup_mmap_after_fork: requested 0x7F390000 != 0x0 mem alloc base 0x0, state 0x10000, size 393216, Win32 error 487
1918 [main] python 4208 C:\Users\__\AppData\Local\Temp\ib_MobaXterm\bin\python.exe: *** fatal error - recreate_mmaps_after_fork_failed
Please log in and/or grant access via your browser at https://www.google.com/accounts/OAuthAuthorizeToken?oauth_token=___&hd=default then hit enter.
After opening that URL in my browser and clicking “Grant access” I finally got what I wanted:
You have successfully granted GoogleCL myuser@mymachine access to your Google Account. You can revoke access at any time under 'My Account'. Learn more
I now have to add the “—category” option in the .config file in my /home directory and maybe set an alias to make uploads easier.
The GoogleCL advantage is that once you set your username authentication you don’t have to set it again. Likewise, it is more forgiving in terms of the metadata one has to specify for the upload, not to mention all the other Google services it can access. Youtube-upload may be good when uploading videos to a different, seldom-used account. I will be installing both on my NAS.
local traffic control
I was able to find three application that might be able to limit traffic on the local Windows computer.
- NetLimiter Lite seems to have good reviews and I intend to try it first.
- NetBalancer has generated more buzz, but its free version is limited to 5 processes and rules at a time and some of the buzz is negative. Martin Brinkmann explains how to use it but in comments there are some negative reports: What awful software! It ‘temporarily’ disconnected my internet connection (usb tether) and has not restored it. Now my computer won’t even recognise the modem (no problem on other computers). I’ve tried system restores, uninstalling/reinstalling hardware, half a day on and no luck. Netbalancer has changed some setting that has completely destroyed my internet connection. Thanks guys!
Vikitech has its own writeup. - Traffic Shaper XP is quite advanced and works on Windows 2000, XP or 2003 Server. It has a Deployment Overview and its fans.
Linux has trickle, wondershaper and iproute / netem.
router QoS
Uploads from my NAS and QoS on the router is the solution I lean toward. I have already setup QoS on my DD-WRT router but I haven’t done much testing. The problem is that a lot of the reports on what works and what doesn’t are contradictory, making testing a necessity, much like what Tony Fortunato of techfirm (see also his summary of features) did.
Some of the aforementioned contradictory information:
- dd-wrt QoS. This is the sine qua non starting point.
- DefunctPrecept has his own thoughts on what each QoS level means.
- Mark Maunder of Feedjit fame recommends shaping the internal interface via SSH console (rather than the GUI) because “I’ve configured the rules on the internal interface even though most QoS rules are generally configured on an external interface because it’s the only thing that really really seems to work. The Cisco engineers among you may disagree, but go try it yourself before you comment.”
Enter the following command which clears all traffic control settings on interface br0:
tc qdisc del dev br0 root
Then enter the following:
tc qdisc add dev br0 root handle 1: cbq \
avpkt 1000 bandwidth 2mbittc class add dev br0 parent 1: classid 1:1 cbq \
rate 700kbit allot 1500 prio 5 bounded isolatedtc filter add dev br0 parent 1: protocol ip \
prio 16 u32 match ip dst 192.168.1.133 flowid 1:1tc filter add dev br0 parent 1: protocol ip \
prio 16 u32 match ip src 192.168.1.133 flowid 1:1These commands will rate limit the IP address 192.168.1.133 to 700 kilobits per second.
- The Mythbusters guys host on their site a walkthrough by Matt Braga. So does Wi-Fi Planet. They are almost direct applications of the FAQ above.
- Tomato distro takes a somewhat different (and some say better) approach.
- Gargoyle router has quota management but supports only a few routers.
The dd-wrt forum is another excellent source of information. We learn that there is a WRT54 Script Generator by Robert “Robson” Mytkowski which is a GUI program that will generate IP Tables script which you can “cut and paste” and worked at least with the SP1 version, with these caveats:
There are following problems in generated script:
a) it is using command "modprobe" which is not present on DD-WRT standard. Therefore some kernel modules are not loaded and most of the next commands fail. Need to be replaced to "insmod".
b) DD-WRT iptables kernel support have "iprange" feature disabled (any iptables command trying to use "-m iprange" fails). Therefore rules which use ip ranges are not working without modifications. This can be workarounded by replacing ip ranges to subnet masks, i.e.:
-m iprange --dst-range 192.168.1.1-192.168.1.254
should be replaced with:
-d 192.168.1.0/24
("-m iprange --src-range" should be replaced with subnet "-s" option).
WRTbwmon will run on any linux powered router (OpenWRT, DD-WRT, Tomato) and provides per user bandwidth monitoring capabilities and generates usage reports – integrated in tomatousb router.
On to installing the scripts on my NAS now..
Sources / More info: aw,snap, man-tc, sgen
Long time ago, others swore by a shaper.sh script.
#!/bin/sh
TC=/usr/sbin/tc
IPTABLES=/usr/sbin/iptables
# PPPoE, DSL16000, 18140/925 sync at my modem, NEVER expect to have for example 6000 when you ordered DSL6000
# RATE should be somewhat about 85% of you upstream, this works well for me, do not complain about "losing" 15%
DEV=ppp0
DOWNLINK=17800
RATE=760
# we want just the status when we call ./shaper.sh status
if [ "$1" = "status" ]
then
$TC -s qdisc ls dev $DEV
$TC -s class ls dev $DEV
exit
fi
# clean existing down- and uplink qdiscs, hide errors
$TC qdisc del dev $DEV root 2> /dev/null > /dev/null
$TC qdisc del dev $DEV ingress 2> /dev/null > /dev/null
# kill the qdiscs to disable QoS (well iptables will still mark packets but there are no handles for them)
if [ "$1" = "stop" ]
then
exit
fi
###### marking ######
## flush the handler
$IPTABLES -t mangle --flush
# rtp/voice get mark 3
$IPTABLES -t mangle -A POSTROUTING -p udp -m length --length 150:250 -j MARK --set-mark 3
$IPTABLES -t mangle -A POSTROUTING -p udp --source-port 11000:11009 -j MARK --set-mark 3
$IPTABLES -t mangle -A POSTROUTING -m layer7 --l7proto sip -j MARK --set-mark 3
$IPTABLES -t mangle -A POSTROUTING -m layer7 --l7proto teamspeak --set-mark 3
# gaming gets mark 3 well
$IPTABLES -t mangle -A POSTROUTING -m layer7 --l7proto quake-halflife -j MARK --set-mark 3
$IPTABLES -t mangle -A POSTROUTING -m layer7 --l7proto counterstrike-source -j MARK --set-mark 3
# p2p + ftp gets mark 6
$IPTABLES -t mangle -A POSTROUTING -m layer7 --l7proto edonkey -j MARK --set-mark 6
$IPTABLES -t mangle -A POSTROUTING -m layer7 --l7proto gnutella -j MARK --set-mark 6
$IPTABLES -t mangle -A POSTROUTING -m layer7 --l7proto bittorrent -j MARK --set-mark 6
$IPTABLES -t mangle -A POSTROUTING -m layer7 --l7proto ftp -j MARK --set-mark 6
# for shaping (modem queue never gets full)
# this is VERY important, would ruin all our work
$TC qdisc add dev ${DEV} root handle 1: tbf rate ${RATE}kbit burst 4k latency 30ms
# priorization 1: voip/gaming | 2: default | 3: p2p
$TC qdisc add dev ${DEV} parent 1: handle 10: prio bands 3 priomap 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
$TC qdisc add dev ${DEV} parent 10:1 handle 100: pfifo
$TC qdisc add dev ${DEV} parent 10:2 handle 200: pfifo limit 200
$TC qdisc add dev ${DEV} parent 10:3 handle 300: pfifo limit 700
#-------------------------------------
# filtering
#-------------------------------------
# 10:1 voip
$TC filter add dev ${DEV} parent 10:0 protocol ip prio 10 \
handle 0x3 fw flowid 10:1
# 10:3 p2p
$TC filter add dev ${DEV} parent 10:0 protocol ip prio 20 \
handle 0x6 fw flowid 10:3
# default: All non VoIP traffic on band 2
$TC filter add dev ${DEV} parent 10:0 prio 30 protocol ip u32 \
match ip src 0.0.0.0/0 \
flowid 10:2
########## downlink #############
# slow downloads down to somewhat less than the real speed to prevent
# queuing at our ISP. Tune to see how high you can set it.
# ISPs tend to have *huge* queues to make sure big downloads are fast
#
# attach ingress policer:
$TC qdisc add dev $DEV handle ffff: ingress
# filter *everything* to it (0.0.0.0/0), drop everything that's
# coming in too fast:
$TC filter add dev $DEV parent ffff: protocol ip prio 50 u32 match ip src \
0.0.0.0/0 police rate ${DOWNLINK}kbit burst 10k drop flowid :1
which has finally evolved into:
#!/bin/sh
#######################################################
# npshaper v0.1
#######################################################
# Wan link download speed in Kbits (set to 80%-90% of link capacity; 6000 -> 4800)
DOWNLOAD=4800
# Wan link upload speed in Kbits (set to 80%-90% of link capacity; 600 -> 480)
UPLOAD=480
# Download burst size in Kbytes
D_BURST=50
# Upload burst size in Kbytes
U_BURST=5
# 'Home server' (always on machine used for serving webpages / FTP / P2P / ...)
HOMESERVER_IP=192.168.0.9
#
# Ports used by the 'home server' services
#
HOMESERVER_HTTP_PORT=80
HOMESERVER_HTTPS_PORT=443
HOMESERVER_FTP_PORT=4521
HOMESERVER_HFS_PORT=4580
HOMESERVER_EMULE_TCP_PORT=4662
HOMESERVER_EMULE_UDP_PORT=4672
HOMESERVER_BITTORRENT_PORT=6881
#######################################################
WAN=$(nvram get wan_ifname)
LAN=$(nvram get lan_ifname)
DEBUG=0
if [ "$1" = "start" ]
then
echo "Starting..."
[ $DEBUG -eq 1 ] && insmod ipt_LOG >&- 2>&-
insmod cls_fw >&- 2>&-
#insmod sch_hfsc >&- 2>&-
insmod sch_htb >&- 2>&-
insmod ipt_CONNMARK >&- 2>&-
insmod ipt_length >&- 2>&-
insmod ipt_limit >&- 2>&-
insmod ipt_tos >&- 2>&-
#insmod sch_ingress >&- 2>&-
insmod ipt_layer7 >&- 2>&-
#insmod ipt_ipp2p >&- 2>&-
#insmod ipt_multiport >&- 2>&-
#insmod cls_u32 >&- 2>&-
# Remove previous settings
tc qdisc del dev $WAN root >&- 2>&-
tc qdisc del dev $LAN root >&- 2>&-
##### WAN #####
echo "Setting up Wan interface traffic classes..."
tc qdisc add dev $WAN root handle 1: htb
tc class add dev $WAN parent 1: classid 1:1 htb rate ${UPLOAD}kbit ceil ${UPLOAD}kbit burst ${U_BURST}k cburst ${U_BURST}k
tc class add dev $WAN parent 1:1 classid 1:10 htb rate $(($UPLOAD*5/10))kbit ceil ${UPLOAD}kbit burst ${U_BURST}k cburst ${U_BURST}k prio 0
tc class add dev $WAN parent 1:1 classid 1:20 htb rate $(($UPLOAD*3/10))kbit ceil ${UPLOAD}kbit burst ${U_BURST}k cburst ${U_BURST}k prio 1
tc class add dev $WAN parent 1:1 classid 1:30 htb rate $(($UPLOAD*2/10))kbit ceil ${UPLOAD}kbit burst ${U_BURST}k cburst ${U_BURST}k prio 2
tc filter add dev $WAN parent 1: prio 1 protocol ip handle 1 fw flowid 1:10
tc filter add dev $WAN parent 1: prio 2 protocol ip handle 2 fw flowid 1:20
tc filter add dev $WAN parent 1: prio 3 protocol ip handle 3 fw flowid 1:30
##### LAN #####
echo "Setting up Lan interface traffic classes..."
tc qdisc add dev $LAN root handle 1: htb
tc class add dev $LAN parent 1: classid 1:1 htb rate ${DOWNLOAD}kbit ceil ${DOWNLOAD}kbit burst ${D_BURST}k cburst ${D_BURST}k
tc class add dev $LAN parent 1:1 classid 1:10 htb rate $(($DOWNLOAD*5/10))kbit ceil ${DOWNLOAD}kbit burst ${D_BURST}k cburst ${D_BURST}k prio 0
tc class add dev $LAN parent 1:1 classid 1:20 htb rate $(($DOWNLOAD*3/10))kbit ceil ${DOWNLOAD}kbit burst ${D_BURST}k cburst ${D_BURST}k prio 1
tc class add dev $LAN parent 1:1 classid 1:30 htb rate $(($DOWNLOAD*2/10))kbit ceil ${DOWNLOAD}kbit burst ${D_BURST}k cburst ${D_BURST}k prio 2
tc filter add dev $LAN parent 1: prio 1 protocol ip handle 1 fw flowid 1:10
tc filter add dev $LAN parent 1: prio 2 protocol ip handle 2 fw flowid 1:20
tc filter add dev $LAN parent 1: prio 3 protocol ip handle 3 fw flowid 1:30
######################################## MARK CHAIN ##################################################
echo "Setting up classification chains..."
# Remove previous settings
iptables -t mangle -F
iptables -t mangle -X
# Wan ('upload' traffic) classification chain
iptables -t mangle -N wan_mark_chain
iptables -t mangle -A POSTROUTING -o $WAN -j wan_mark_chain
# Lan ('download' traffic) classification chain
iptables -t mangle -N lan_mark_chain
iptables -t mangle -A POSTROUTING -o $LAN -j lan_mark_chain
# Restore any saved connection mark (connection already marked and tracked)
iptables -t mangle -A wan_mark_chain -j CONNMARK --restore-mark
iptables -t mangle -A lan_mark_chain -j CONNMARK --restore-mark
### RULES BEGIN #####################################
# DNS (outgoing) queries - Express
iptables -t mangle -A wan_mark_chain -m mark --mark 0 -p udp --dport 53 -j MARK --set-mark 1
# HTTP on home server - Bulk
iptables -t mangle -A wan_mark_chain -m mark --mark 0 -s $HOMESERVER_IP -p tcp --sport $HOMESERVER_HTTP_PORT -j MARK --set-mark 3
iptables -t mangle -A lan_mark_chain -m mark --mark 0 -d $HOMESERVER_IP -p tcp --dport $HOMESERVER_HTTP_PORT -j MARK --set-mark 3
# HTTPS on home server - Bulk
iptables -t mangle -A wan_mark_chain -m mark --mark 0 -s $HOMESERVER_IP -p tcp --sport $HOMESERVER_HTTPS_PORT -j MARK --set-mark 3
iptables -t mangle -A lan_mark_chain -m mark --mark 0 -d $HOMESERVER_IP -p tcp --dport $HOMESERVER_HTTPS_PORT -j MARK --set-mark 3
# FTP on home server - Bulk
iptables -t mangle -A wan_mark_chain -m mark --mark 0 -s $HOMESERVER_IP -p tcp --sport $HOMESERVER_FTP_PORT -j MARK --set-mark 3
iptables -t mangle -A lan_mark_chain -m mark --mark 0 -d $HOMESERVER_IP -p udp --dport $HOMESERVER_FTP_PORT -j MARK --set-mark 3
# HFS on home server - Bulk
iptables -t mangle -A wan_mark_chain -m mark --mark 0 -s $HOMESERVER_IP -p tcp --sport $HOMESERVER_HFS_PORT -j MARK --set-mark 3
iptables -t mangle -A lan_mark_chain -m mark --mark 0 -d $HOMESERVER_IP -p tcp --dport $HOMESERVER_HFS_PORT -j MARK --set-mark 3
# Edonkey on home server - Bulk
iptables -t mangle -A wan_mark_chain -m mark --mark 0 -s $HOMESERVER_IP -p tcp --sport $HOMESERVER_EMULE_TCP_PORT -j MARK --set-mark 3
iptables -t mangle -A lan_mark_chain -m mark --mark 0 -d $HOMESERVER_IP -p tcp --dport $HOMESERVER_EMULE_TCP_PORT -j MARK --set-mark 3
iptables -t mangle -A wan_mark_chain -m mark --mark 0 -s $HOMESERVER_IP -p udp --sport $HOMESERVER_EMULE_UDP_PORT -j MARK --set-mark 3
iptables -t mangle -A lan_mark_chain -m mark --mark 0 -d $HOMESERVER_IP -p udp --dport $HOMESERVER_EMULE_UDP_PORT -j MARK --set-mark 3
# Bittorrent on home server - Bulk
iptables -t mangle -A wan_mark_chain -m mark --mark 0 -s $HOMESERVER_IP -p tcp --sport $HOMESERVER_BITTORRENT_PORT -j MARK --set-mark 3
iptables -t mangle -A lan_mark_chain -m mark --mark 0 -d $HOMESERVER_IP -p tcp --dport $HOMESERVER_BITTORRENT_PORT -j MARK --set-mark 3
iptables -t mangle -A wan_mark_chain -m mark --mark 0 -s $HOMESERVER_IP -p udp --sport $HOMESERVER_BITTORRENT_PORT -j MARK --set-mark 3
iptables -t mangle -A lan_mark_chain -m mark --mark 0 -d $HOMESERVER_IP -p udp --dport $HOMESERVER_BITTORRENT_PORT -j MARK --set-mark 3
# Edonkey 'catch-all' - Bulk
iptables -t mangle -A wan_mark_chain -m mark --mark 0 -m layer7 --l7proto edonkey -j MARK --set-mark 3
iptables -t mangle -A lan_mark_chain -m mark --mark 0 -m layer7 --l7proto edonkey -j MARK --set-mark 3
# Bittorrent 'catch-all' - Bulk
iptables -t mangle -A wan_mark_chain -m mark --mark 0 -m layer7 --l7proto bittorrent -j MARK --set-mark 3
iptables -t mangle -A lan_mark_chain -m mark --mark 0 -m layer7 --l7proto bittorrent -j MARK --set-mark 3
### RULES END #####################################
# Save mark so we track the full connection
iptables -t mangle -A wan_mark_chain -j CONNMARK --save-mark
iptables -t mangle -A lan_mark_chain -j CONNMARK --save-mark
# ACK packets and suck (connection control) - Express
iptables -t mangle -A wan_mark_chain -p tcp -m length --length :128 --tcp-flags SYN,RST,ACK ACK -j MARK --set-mark 1
iptables -t mangle -A lan_mark_chain -p tcp -m length --length :128 --tcp-flags SYN,RST,ACK ACK -j MARK --set-mark 1
# ICMP (ping and such) - Express
iptables -t mangle -A wan_mark_chain -p icmp -j MARK --set-mark 1
iptables -t mangle -A lan_mark_chain -p icmp -j MARK --set-mark 1
# TOS Minimize-Delay - Express
iptables -t mangle -A wan_mark_chain -m tos --tos Minimize-Delay -j MARK --set-mark 1
iptables -t mangle -A lan_mark_chain -m tos --tos Minimize-Delay -j MARK --set-mark 1
# Default (anything else) - Normal
iptables -t mangle -A wan_mark_chain -m mark --mark 0 -j MARK --set-mark 2
iptables -t mangle -A lan_mark_chain -m mark --mark 0 -j MARK --set-mark 2
######################################################################################################
echo "Setting up debugging..."
[ $DEBUG -eq 1 ] && iptables -t mangle -A wan_mark_chain -m mark --mark 1 -j LOG --log-prefix wan_qos_express::
[ $DEBUG -eq 1 ] && iptables -t mangle -A wan_mark_chain -m mark --mark 2 -j LOG --log-prefix wan_qos_normal::
[ $DEBUG -eq 1 ] && iptables -t mangle -A wan_mark_chain -m mark --mark 3 -j LOG --log-prefix wan_qos_bulk::
[ $DEBUG -eq 1 ] && iptables -t mangle -A lan_mark_chain -m mark --mark 1 -j LOG --log-prefix lan_qos_express::
[ $DEBUG -eq 1 ] && iptables -t mangle -A lan_mark_chain -m mark --mark 2 -j LOG --log-prefix lan_qos_normal::
[ $DEBUG -eq 1 ] && iptables -t mangle -A lan_mark_chain -m mark --mark 3 -j LOG --log-prefix lan_qos_bulk::
echo "Setting up accounting..."
iptables -t mangle -A wan_mark_chain -m mark --mark 1 -j RETURN
iptables -t mangle -A wan_mark_chain -m mark --mark 2 -j RETURN
iptables -t mangle -A wan_mark_chain -m mark --mark 3 -j RETURN
iptables -t mangle -A lan_mark_chain -m mark --mark 1 -j RETURN
iptables -t mangle -A lan_mark_chain -m mark --mark 2 -j RETURN
iptables -t mangle -A lan_mark_chain -m mark --mark 3 -j RETURN
echo "...OK, all done."
fi
########################################
if [ "$1" = "status" ]
then
echo "--- Current status ---"
echo "--- WAN (Upload) ---"
tc -s qdisc ls dev $WAN
tc -s class ls dev $WAN
echo ""
echo "--- LAN (Download) ---"
tc -s qdisc ls dev $LAN
tc -s class ls dev $LAN
echo ""
echo "--- Classification chains ---"
iptables -L -v -t mangle
echo ""
fi
if [ "$1" = "stats" ]
then
LAN_EXPRESS_PACKETS=`iptables -L -v -n -t mangle | grep "RETURN" | grep "match 0x1" | head -n 1 | awk '{print $1}'`
LAN_NORMAL_PACKETS=`iptables -L -v -n -t mangle | grep "RETURN" | grep "match 0x2" | head -n 1 | awk '{print $1}'`
LAN_BULK_PACKETS=`iptables -L -v -n -t mangle | grep "RETURN" | grep "match 0x3" | head -n 1 | awk '{print $1}'`
LAN_EXPRESS_BYTES=`iptables -L -v -n -t mangle | grep "RETURN" | grep "match 0x1" | head -n 1 | awk '{print $2}'`
LAN_NORMAL_BYTES=`iptables -L -v -n -t mangle | grep "RETURN" | grep "match 0x2" | head -n 1 | awk '{print $2}'`
LAN_BULK_BYTES=`iptables -L -v -n -t mangle | grep "RETURN" | grep "match 0x3" | head -n 1 | awk '{print $2}'`
WAN_EXPRESS_PACKETS=`iptables -L -v -n -t mangle | grep "RETURN" | grep "match 0x1" | tail -n 1 | awk '{print $1}'`
WAN_NORMAL_PACKETS=`iptables -L -v -n -t mangle | grep "RETURN" | grep "match 0x2" | tail -n 1 | awk '{print $1}'`
WAN_BULK_PACKETS=`iptables -L -v -n -t mangle | grep "RETURN" | grep "match 0x3" | tail -n 1 | awk '{print $1}'`
WAN_EXPRESS_BYTES=`iptables -L -v -n -t mangle | grep "RETURN" | grep "match 0x1" | tail -n 1 | awk '{print $2}'`
WAN_NORMAL_BYTES=`iptables -L -v -n -t mangle | grep "RETURN" | grep "match 0x2" | tail -n 1 | awk '{print $2}'`
WAN_BULK_BYTES=`iptables -L -v -n -t mangle | grep "RETURN" | grep "match 0x3" | tail -n 1 | awk '{print $2}'`
echo "Traffic stats:"
echo "D/U Class Packets Bytes"
echo "D Express $LAN_EXPRESS_PACKETS $LAN_EXPRESS_BYTES"
echo "D Normal $LAN_NORMAL_PACKETS $LAN_NORMAL_BYTES"
echo "D Bulk $LAN_BULK_PACKETS $LAN_BULK_BYTES"
echo "U Express $WAN_EXPRESS_PACKETS $WAN_EXPRESS_BYTES"
echo "U Normal $WAN_NORMAL_PACKETS $WAN_NORMAL_BYTES"
echo "U Bulk $WAN_BULK_PACKETS $WAN_BULK_BYTES"
fiOk, so what does this script do?
Well, it creates the next traffic classes:Code:
UPLOAD (WAN OUTPUT)
1 ----+ 1:1
|
|----> 1:10 Express
|----> 1:20 Normal
|----> 1:30 Bulk
DOWNLOAD (LAN OUTPUT)
1 ----+ 1:1
|
|----> 1:10 Express
|----> 1:20 Normal
|----> 1:30 BulkUsage:
npshaper.sh start - Start the shapper (set the QOS rules)
npshaper.sh status - Print the QOS rules and stats
npshaper.sh stats - Print the basic express/normal/bulk classes stats
The script will set rules for shapping:
- Downloading traffic on the LAN output side (traffic that comes from the internet, gets queued on the router or droped if needed, before going into the lan)
- Uploading traffic on the WAN side (traffic going from the lan, gets queued on the router or droped if needed, before going to internet).
Traffic is asigned to one of the three classes by packet marking them with marks 1 (traffic control, pings, and such goes into express 1:10), 2 (default, goes into 1:20) or 3 (bulk, like p2p, goes into 1:30)
The "### RULES BEGIN ### ... ### RULES END ###" section is meant to be edited so you can add custom rules to set what is 'bulk', 'normal' or 'express' traffic (via the marks).
By default it classifies P2P, web serving or FTP data comming from, or going to, the 'home server' (I have an always on computer that I use as a web server and P2P client) as 'bulk'.
This way, the rest of the computers on the network ( doing web surfing, gaming, voice...) won't even notice (no high latency) when eMule is running, or when somebody is using the 'home server' FTP.
Though the script is far from perfect, I think it has one main advantage: it classifies both the 'download' and 'upload' connections!
CaScAdE script, on the downloading side, just shapes the wan ingress (download), so it does "drop anything that is coming in too fast"; but does not classify it so some traffic classes get priority and guaranted bandwidth.
As an example of the 'results', this is my current 'status' (tc qdiscs, tc classes and iptables mangle chain)Code:
# /jffs/etc/bbshaper.sh status
--- Current status ---
--- WAN (Upload) ---
qdisc htb 1: r2q 10 default 0 direct_packets_stat 23
Sent 108459650 bytes 214524 pkts (dropped 973, overlimits 3658)
class htb 1:1 root rate 480000bit ceil 480000bit burst 5Kb cburst 5Kb
Sent 108446797 bytes 214511 pkts (dropped 0, overlimits 0)
rate 186152bit 82pps
lended: 12754 borrowed: 0 giants: 0
tokens: 130762 ctokens: 130762
class htb 1:10 parent 1:1 prio 0 rate 240000bit ceil 480000bit burst 5Kb cburst 5Kb
Sent 4078000 bytes 60349 pkts (dropped 0, overlimits 0)
rate 28344bit 61pps
lended: 60349 borrowed: 0 giants: 0
tokens: 261523 ctokens: 130762
class htb 1:20 parent 1:1 prio 1 rate 144000bit ceil 480000bit burst 5Kb cburst 5Kb
Sent 54175080 bytes 87561 pkts (dropped 715, overlimits 0)
rate 111776bit 12pps
lended: 84236 borrowed: 3325 giants: 0
tokens: 208386 ctokens: 63049
class htb 1:30 parent 1:1 prio 2 rate 96000bit ceil 480000bit burst 5Kb cburst 5Kb
Sent 50193717 bytes 66601 pkts (dropped 258, overlimits 0)
rate 49816bit 9pps
lended: 57172 borrowed: 9429 giants: 0
tokens: 75518 ctokens: 123956
--- LAN (Download) ---
qdisc htb 1: r2q 10 default 0 direct_packets_stat 273
Sent 46355930 bytes 173969 pkts (dropped 65, overlimits 107)
class htb 1:1 root rate 4800Kbit ceil 4800Kbit burst 50Kb cburst 50Kb
Sent 46347701 bytes 173699 pkts (dropped 0, overlimits 0)
rate 1250Kbit 124pps
lended: 9061 borrowed: 0 giants: 0
tokens: 126438 ctokens: 126438
class htb 1:10 parent 1:1 prio 0 rate 2400Kbit ceil 4800Kbit burst 50Kb cburst 50Kb
Sent 5037602 bytes 77708 pkts (dropped 0, overlimits 0)
rate 6944bit 14pps
lended: 77708 borrowed: 0 giants: 0
tokens: 263751 ctokens: 131876
class htb 1:20 parent 1:1 prio 1 rate 1440Kbit ceil 4800Kbit burst 50Kb cburst 50Kb
Sent 36566602 bytes 59775 pkts (dropped 65, overlimits 0)
rate 1245Kbit 104pps
lended: 50714 borrowed: 9061 giants: 0
tokens: -7882 ctokens: 126438
class htb 1:30 parent 1:1 prio 2 rate 960000bit ceil 4800Kbit burst 50Kb cburst 50Kb
Sent 4743497 bytes 36216 pkts (dropped 0, overlimits 0)
rate 5904bit 6pps
lended: 36216 borrowed: 0 giants: 0
tokens: 658142 ctokens: 131628
--- Classification chains ---
Chain PREROUTING (policy ACCEPT 890K packets, 290M bytes)
pkts bytes target prot opt in out source destination
Chain INPUT (policy ACCEPT 373K packets, 77M bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 1584K packets, 691M bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 339K packets, 76M bytes)
pkts bytes target prot opt in out source destination
Chain POSTROUTING (policy ACCEPT 1950K packets, 777M bytes)
pkts bytes target prot opt in out source destination
215K 107M wan_mark_chain all -- any vlan1 anywhere anywhere
175K 44M lan_mark_chain all -- any br0 anywhere anywhere
Chain lan_mark_chain (1 references)
pkts bytes target prot opt in out source destination
175K 44M CONNMARK all -- any any anywhere anywhere CONNMARK restore
11 540 MARK tcp -- any any anywhere 192.168.0.9 MARK match 0x0 tcp dpt:www MARK set 0x3
0 0 MARK tcp -- any any anywhere 192.168.0.9 MARK match 0x0 tcp dpt:https MARK set 0x3
0 0 MARK udp -- any any anywhere 192.168.0.9 MARK match 0x0 udp dpt:4521 MARK set 0x3
9 448 MARK tcp -- any any anywhere 192.168.0.9 MARK match 0x0 tcp dpt:4580 MARK set 0x3
2444 122K MARK tcp -- any any anywhere 192.168.0.9 MARK match 0x0 tcp dpt:4662 MARK set 0x3
7374 536K MARK udp -- any any anywhere 192.168.0.9 MARK match 0x0 udp dpt:4672 MARK set 0x3
337 16667 MARK tcp -- any any anywhere 192.168.0.9 MARK match 0x0 tcp dpt:6881 MARK set 0x3
8259 1065K MARK udp -- any any anywhere 192.168.0.9 MARK match 0x0 udp dpt:6881 MARK set 0x3
0 0 MARK all -- any any anywhere anywhere MARK match 0x0 LAYER7 l7proto edonkey MARK set 0x3
0 0 MARK all -- any any anywhere anywhere MARK match 0x0 LAYER7 l7proto bittorrent MARK set 0x3
175K 44M CONNMARK all -- any any anywhere anywhere CONNMARK save
76563 3862K MARK tcp -- any any anywhere anywhere length 0:128 tcp flags:SYN,RST,ACK/ACK MARK set 0x1
1130 84914 MARK icmp -- any any anywhere anywhere MARK set 0x1
0 0 MARK all -- any any anywhere anywhere TOS match Minimize-Delay MARK set 0x1
60697 36M MARK all -- any any anywhere anywhere MARK match 0x0 MARK set 0x2
77695 3949K RETURN all -- any any anywhere anywhere MARK match 0x1
60697 36M RETURN all -- any any anywhere anywhere MARK match 0x2
36198 4235K RETURN all -- any any anywhere anywhere MARK match 0x3
Chain wan_mark_chain (1 references)
pkts bytes target prot opt in out source destination
215K 107M CONNMARK all -- any any anywhere anywhere CONNMARK restore
29 1914 MARK udp -- any any anywhere anywhere MARK match 0x0 udp dpt:domain MARK set 0x1
0 0 MARK tcp -- any any 192.168.0.9 anywhere MARK match 0x0 tcp spt:www MARK set 0x3
0 0 MARK tcp -- any any 192.168.0.9 anywhere MARK match 0x0 tcp spt:https MARK set 0x3
0 0 MARK tcp -- any any 192.168.0.9 anywhere MARK match 0x0 tcp spt:4521 MARK set 0x3
7 287 MARK tcp -- any any 192.168.0.9 anywhere MARK match 0x0 tcp spt:4580 MARK set 0x3
49 4630 MARK tcp -- any any 192.168.0.9 anywhere MARK match 0x0 tcp spt:4662 MARK set 0x3
5089 325K MARK udp -- any any 192.168.0.9 anywhere MARK match 0x0 udp spt:4672 MARK set 0x3
88 11038 MARK tcp -- any any 192.168.0.9 anywhere MARK match 0x0 tcp spt:6881 MARK set 0x3
2767 363K MARK udp -- any any 192.168.0.9 anywhere MARK match 0x0 udp spt:6881 MARK set 0x3
21 2350 MARK all -- any any anywhere anywhere MARK match 0x0 LAYER7 l7proto edonkey MARK set 0x3
42 4536 MARK all -- any any anywhere anywhere MARK match 0x0 LAYER7 l7proto bittorrent MARK set 0x3
215K 107M CONNMARK all -- any any anywhere anywhere CONNMARK save
59261 3160K MARK tcp -- any any anywhere anywhere length 0:128 tcp flags:SYN,RST,ACK/ACK MARK set 0x1
679 40746 MARK icmp -- any any anywhere anywhere MARK set 0x1
0 0 MARK all -- any any anywhere anywhere TOS match Minimize-Delay MARK set 0x1
88275 54M MARK all -- any any anywhere anywhere MARK match 0x0 MARK set 0x2
60356 3233K RETURN all -- any any anywhere anywhere MARK match 0x1
88275 54M RETURN all -- any any anywhere anywhere MARK match 0x2
66819 49M RETURN all -- any any anywhere anywhere MARK match 0x3
Traffic stats:
D/U Class Packets Bytes
D Express 77714 3950K
D Normal 60956 37M
D Bulk 36213 4237K
U Express 60701 3251K
U Normal 88301 54M
U Bulk 66849 49M
Comments