After wasting time and intense frustration, I have decided that waiting for videos of photos to upload to online services is not something I want to do from my laptop. I shall thus explore alternatives..
This exploration trigger was a quasi-failed YouTube upload. In fact, I am not even sure that the upload failed. All I know is that I started an upload of a long video which quickly saturated my meagre (0.5 Mbps) uplink. My Internet became unusable while I had to wait an estimated 600 minutes. Needless to say, I let it go overnight, only to come back to this in the morning:
YouTube isn’t telling me much, except that it’s “processing” it (though the 0:00 suggests failure):
I’ve decided that enough is enough and looked at a few alternatives (rather than simply using a browser other than Chromium).
After all, since YouTube has removed any time limitation to the videos I can upload I might as well start producing more videos. In High Definition, file sizes can easily reach several gigabytes even with the time limit (15 minutes). For instance, here is a browser upload of a 720p video of 11 or so minutes after about 8 hours, overnight:
Obviously, it’s inhuman to wait that long and have your Internet “incapacitated” throughout this ordeal. If you use VoIP, anybody calling you in this interval might not be able to hear you. Since the upload cannot be sped up due to physical limitations, it has to be managed. There are 3 ways to manage it that I can think of:
- Use a separate application for such uploads that allows specifying bandwidth limits. If such an application does not exist, use some kind of proxy server on the same computer that allows traffic shaping / limiting on a per-application basis, and upload to YouTube only from a specific browser thus controlled.
- Identify a persistent property (server IP if fixed or TCP / UDP ports) of this upload process that allows it to be managed / limited on the router via QoS rules.
- Perform such uploads only from specific computers and manage their connection on the router via QoS rules.
The second idea has low chances of success, while the third is particularly enticing since if we discover some linux tools we might be able to run such a process right from the NAS where the files to be uploaded usually reside.
Let us now look at the applications we can identify.
VideoLAN Movie Creator is a free, open source video editor from the same guys who brought us the amazing VLC. It borrows from that project the wide OS and codecs support. Though currently in a very early release (“pre-alpha”), it shows promise and it already features YouTube integration. If I’d be using it to upload videos to YouTube I could also edit them before. On the downside, this program is quite buggy still so I risk having the “Aw, Snap!” moment.
Windows Movie Maker (or Live! Movie Maker) is a video editor provided freely by Microsoft as part of its “Live” suite. It is a mature product and it even has video acceleration if you have a newer video card. However, they’ve been moving away from YouTube integration into a partnership with Vimeo, which makes sense when you’re competing with Google. Even if YouTube upload is kept in the feature set, I’d prefer a quicker, faster option for upload as I don’t always need to edit videos before upload.
This company makes a suite of applications that include downloading YouTube videos, converting them to MP3, editing, burning and uploading to YouTube. Everything is modularized and can be downloaded separately (some work even as Chrome extensions). This is a good option if memory or CPU load are problems you are confronted with often (I know I am). They have a nice video explaining how to upload. The programs are not open source, but the developer claims there are no spyware or trojans included.
Microsoft Expression is, in its free incarnation, a screencast application. Currently at version 4 SP2, it can use a plugin to publish directly to YouTube.
Now we’re getting close!
Python-gdata is a set of Python scripts that allow using Google Services via CLI. There are two applications of interest based on it:
- Youtube-upload is a command-line Python script that easily uploads videos to – you guessed it! – Youtube. The goal is to install it on my NAS, but first I installed it on my Cygwin clone, MobaXterm. The command to upload a video would be:
$ youtube-upload --firstname.lastname@example.org --password=mypassword
--title="A.S. Mutter" --description="A.S. Mutter plays Beethoven"
--category=Music --keywords="mutter, beethoven" anne_sophie_mutter.flv
- GoogleCL allows full access to Google services via the command line, as shown in example scripts. It’s been covered in a manual, on lh (1, 2), by publicint (.msi Python install), muo, ln, mte. An upload example is:
$ google youtube post --category People killer_robots.avi
Youtube-upload works quite well, but one has to specify even the title and keywords, otherwise it complains. It also complains that pycurl is missing, meaning it cannot use HTTP and must use the “basic gdata API.”
On Cygwin / MobaXterm, GoogleCL launches Lynx to request access. Google online doesn’t like it and then it bans it. However, after quitting Lynx an interesting message was left on the screen:
Please specify user: …
6 [main] python 5040 fhandler_dev_zero::fixup_mmap_after_fork: requested 0x7F540000 != 0x0 mem alloc base 0x7F450000, state 0x2000, size 65536, Win32 error 487
3927 [main] python 5040 C:\Users\__\AppData\Local\Temp\ib_MobaXterm\bin\python.exe: *** fatal error - recreate_mmaps_after_fork_failed
1492 [main] python 4208 fhandler_dev_zero::fixup_mmap_after_fork: requested 0x7F390000 != 0x0 mem alloc base 0x0, state 0x10000, size 393216, Win32 error 487
1918 [main] python 4208 C:\Users\__\AppData\Local\Temp\ib_MobaXterm\bin\python.exe: *** fatal error - recreate_mmaps_after_fork_failed
Please log in and/or grant access via your browser at https://www.google.com/accounts/OAuthAuthorizeToken?oauth_token=___&hd=default then hit enter.
You have successfully granted GoogleCL myuser@mymachine access to your Google Account. You can revoke access at any time under 'My Account'. Learn more
I now have to add the “—category” option in the .config file in my /home directory and maybe set an alias to make uploads easier.
The GoogleCL advantage is that once you set your username authentication you don’t have to set it again. Likewise, it is more forgiving in terms of the metadata one has to specify for the upload, not to mention all the other Google services it can access. Youtube-upload may be good when uploading videos to a different, seldom-used account. I will be installing both on my NAS.
I was able to find three application that might be able to limit traffic on the local Windows computer.
- NetLimiter Lite seems to have good reviews and I intend to try it first.
- NetBalancer has generated more buzz, but its free version is limited to 5 processes and rules at a time and some of the buzz is negative. Martin Brinkmann explains how to use it but in comments there are some negative reports: What awful software! It ‘temporarily’ disconnected my internet connection (usb tether) and has not restored it. Now my computer won’t even recognise the modem (no problem on other computers). I’ve tried system restores, uninstalling/reinstalling hardware, half a day on and no luck. Netbalancer has changed some setting that has completely destroyed my internet connection. Thanks guys!
Vikitech has its own writeup.
- Traffic Shaper XP is quite advanced and works on Windows 2000, XP or 2003 Server. It has a Deployment Overview and its fans.
Uploads from my NAS and QoS on the router is the solution I lean toward. I have already setup QoS on my DD-WRT router but I haven’t done much testing. The problem is that a lot of the reports on what works and what doesn’t are contradictory, making testing a necessity, much like what Tony Fortunato of techfirm (see also his summary of features) did.
Some of the aforementioned contradictory information:
- dd-wrt QoS. This is the sine qua non starting point.
- DefunctPrecept has his own thoughts on what each QoS level means.
- Mark Maunder of Feedjit fame recommends shaping the internal interface via SSH console (rather than the GUI) because “I’ve configured the rules on the internal interface even though most QoS rules are generally configured on an external interface because it’s the only thing that really really seems to work. The Cisco engineers among you may disagree, but go try it yourself before you comment.”
Enter the following command which clears all traffic control settings on interface br0:
tc qdisc del dev br0 root
Then enter the following:
tc qdisc add dev br0 root handle 1: cbq \
avpkt 1000 bandwidth 2mbit
tc class add dev br0 parent 1: classid 1:1 cbq \
rate 700kbit allot 1500 prio 5 bounded isolated
tc filter add dev br0 parent 1: protocol ip \
prio 16 u32 match ip dst 192.168.1.133 flowid 1:1
tc filter add dev br0 parent 1: protocol ip \
prio 16 u32 match ip src 192.168.1.133 flowid 1:1
These commands will rate limit the IP address 192.168.1.133 to 700 kilobits per second.
- The Mythbusters guys host on their site a walkthrough by Matt Braga. So does Wi-Fi Planet. They are almost direct applications of the FAQ above.
- Tomato distro takes a somewhat different (and some say better) approach.
- Gargoyle router has quota management but supports only a few routers.
The dd-wrt forum is another excellent source of information. We learn that there is a WRT54 Script Generator by Robert “Robson” Mytkowski which is a GUI program that will generate IP Tables script which you can “cut and paste” and worked at least with the SP1 version, with these caveats:
There are following problems in generated script:
a) it is using command "modprobe" which is not present on DD-WRT standard. Therefore some kernel modules are not loaded and most of the next commands fail. Need to be replaced to "insmod".
b) DD-WRT iptables kernel support have "iprange" feature disabled (any iptables command trying to use "-m iprange" fails). Therefore rules which use ip ranges are not working without modifications. This can be workarounded by replacing ip ranges to subnet masks, i.e.:
-m iprange --dst-range 192.168.1.1-192.168.1.254
should be replaced with:
("-m iprange --src-range" should be replaced with subnet "-s" option).
On to installing the scripts on my NAS now..