This post is about my open source remote bandwidth usage stats logger project.

I’ve been wanting to log the internet bandwidth usage at my home for quite some time since I don’t really find the stats put out by my ISP to be completely accurate. But I have multiple devices at home that access the internet so it is not feasible to install a bandwidth monitor individually on all these devices as I’d still need to add up all logs and many of such devices don’t even have a way to install a monitoring software installed (e.g. my PS3, my TV, my media players, etc). So, I thought of monitoring the usage at my router. It is a smart one (Asus wl-500w) and I can install various linux software on it but then it requires a hard disk to run most of them which I don’t turn on all the time. Moreover, even if I could run the software directly from router flash, it doesn’t have enough space to store the logs (neither is it a good idea to keep writing to the flash often).

Hence I came up with the idea of a light weight remote bandwidth usage logger where in the router has to do the bare minimum possible (here, it just has to make a GET call to a webpage with current status of received and transmitted bytes, accomplished by a single line of wget command). The system is completely open source and is intelligent enough to keep track of things even if router reboots or the bytes count wraps around the 4 GB limit (because of 32 bit data structures used). The logging and parsing is all done on server side and provides various useful things.

Few of the features of the system:

• Very lightweight in CPU/RAM resource usage on host as well as server
• Outputs easy to browse html and graphs
• User defined directory structure to decide the granularity that is wanted
• yearly, monthly, daily stats and graphs
• Provides quick access to extra stats like last time the host connected, last known host IP etc
• Provides total download/upload numbers as well as detailed
• Estimates uptime of remote host
• Can use a DNS name or IP to avoid (or call out) spurious amounts of usage logged by a spurious IP

Remote Bandwidth Stats Yearly Stats

Remote Bandwidth Stats Monthly Stats

Remote Bandwidth Stats Daily Stats

© Shantanu Goel | This article, titled Remote Bandwidth Stats, was originally published on Shantanu's Technophilic Musings

I’ve been playing around with linux uinput infrastructure for one of my ideas (Details coming soon ). “uinput” is basically a mechanism in linux that allows user space applications to inject input events (mouse/keyboard or anything else) into the system. But the problem I faced was that my programs were compiling and running without any errors but I just couldn’t see the effect happening, i.e., the events weren’t getting injected into the system. I am using Ubuntu Maverick Meerkat (10.10) and when I searched on the net, I saw various other folks are facing the same issue. I finally got it working and here is how.

First problem I came across was that I was using “/dev/input/uinput” path in my programs but it seems it is changed to “/dev/uinput” in Ubuntu. Changing this got me past some runtime errors I was getting.

Then I thought that the issue was because uinput module wasn’t getting loaded because even after specifically doing a modprobe, I still couldn’t see it loaded when I did lsmod. This was strange because the /dev/uinput node was present. On looking under /lib, I couldn’t even find the uinput.ko file. But then I found that the reason for this is that in Ubuntu Maverick, uinput is not a module anymore. It is built into the kernel now. (See this launchpad bug.)

So, having figured that the above was a false alarm, I was at a loss how to explain this. But then I tried just changing the permissions of the uinput node to 0666 (i.e. writable by all users) and ran my program without sudo and it worked like a charm. Searching for sudo along with uinput lead me to these bugs which confirms that the problem is because of this only. So, if you are facing this issue, you can either change the uinput node permissions or install the sudo patch described in the bug discussion page.

Let me know if this worked for you as well

© Shantanu Goel | This article, titled Ubuntu Maverick uinput Problem Solved, was originally published on Shantanu's Technophilic Musings

If you have come across a tutorial or just someone on a forum who tells you to install something in Debian/Ubuntu that involves using apt-get, it is ok for you but when they tell you that you need to use a caret symbol (^) at the end, that’s where you become curious. What is even more weird is that when you search for the name of the package that the given command seems to install cannot be found using apt-cache search. e.g. You will see this used most often when someone tells you how to install LAMP server setup (Linux-Apache-MySQL-PHP) by using the command “sudo apt-get install lamp-server^”. If you miss the caret at the end or try to search for lamp-server, it just doesn’t work.

Well, the answer is that the caret symbol is a short form for performing a task that otherwise the program “tasksel” would have done with the given package name. tasksel is a program to ease the installation of commonly used things that go together for a particular use. e.g. In the above instance of LAMP, the four packages and their dependencies are always used together, so tasksel provides a sort of a meta-package or meta-task that can be run by the user with a single command and then tasksel will take it upon itself to install all of them and set them up correctly for your use. Now, apt-get provides a way to perform that same task by itself without you having to install tasksel first and all you have to do is to give that same package name to apt-get but just append a caret at the end to tell apt-get that it is a tasksel package/task identifier and not a regular package name in debian/ubuntu repositories.

Hope that clears things up

© Shantanu Goel | This article, titled What does caret (^) in an apt-get command mean?, was originally published on Shantanu's Technophilic Musings

An application review process was announced today for getting your apps into Ubuntu “extras” repository. But like most other announcements of new things, this one was met as well with as much criticism as the praise it got. A lot of the criticism is FUD though, although mostly unintentional and caused by ignorance of some facts behind it. I’ll try to address some of these concerns here to the best of my knowledge from what I’ve been reading over the past some time about this development.

Before we begin, here are the terms listed out for an app to be eligible for the new repository:

only new applications that are not present in an existing official Ubuntu repository (such as main/universe) are eligible in this process (e.g a new version of an application in an existing official repository is not eligible). Also no other software can depend on the application being submitted (e.g. development libraries are not eligible), only executable applications (and content that is part of them) are eligible, and not stand-alone content, documentation or media, and applications must be Open Source and available under an OSI approved license.

Now, the concerns that are floating around are many. Some are genuine queries and many are straight attacks dishing out evil conspiracy theories . The concerns can be summarised as:

• Closed source and paid apps are being given the boot by Ubuntu. They should have allowed them.
• Why can’t we have updates for existing apps in this repository?
• Why only standalone apps are allowed. Libraries are equally important.
• Why should the software be packaged for /opt. This involves more headache for the developers.
• Standalone media and content should have been allowed.

Well, to answer these, all you need to understand is the “premise” and concept behind the repository where this software will go and the reasoning behind the review process will unfold itself. The repository governed by this review process is meant to be for “maintainable” pieces of software that would generally go to other universe/main etc repos but couldn’t make it into this release of Ubuntu because the developers were a bit late. This process and feature now gives a chance to the developers to still get their app to public instead of waiting for next release cycle and also allows end-users to use new apps without having to upgrade their version of Ubuntu . There is a separate “partners” repository where closed source and paid apps can go so there is no conflict there.

Only allowing standalone apps and not updates and dependencies is so obvious. It would soon go out of hand exponentially if you allowed dependencies since one thing needs another library updated, then that would need another and yet another and so on and before you know it, you would be back at square one, that is upgrading to the next version of Ubuntu itself.

The only thing that I found out of the place was not allowing standalone content but probably they didn’t allow it because such content can be had by end users without a repo easily and would ease the burden of repo maintainers and reviewers by lessening the entries.

I’d also like to point out that this is just a beginning and the process (and the people behind it) would definitely be maturing from all the feedback they get from the community. Think of it as an alpha release . Jono Bacon summed it up very nicely in this comment of his that he made elsewhere:

this is an *experimental* process; it is a work in progress, it will evolve, and the kinks will be ironed out. The process was approved by the Ubuntu Technical Board who are all seasoned Ubuntu developers (Matt Zimmerman, Colin Watson, Martin Pitt, Mark Shuttleworth, and Kees Cook), and much of what you refer to as "restrictions" were requested by them for the purpose of delivering a nimble process that is stable and secure.

To be clear: you can think of this process as a "light-weight" equivalent to our traditional MOTU development process; it is intended to provide a method in which app devs can get content in Ubuntu, whereas our traditional development processes have be orientated around the needs of Operating System integrators.

As such, this is a first step; sure it is not perfect, but I would like to invite you to make it better; I think this would be a more valuable contribution than just ranting about it on here. Our goal is to make Ubuntu better and to make it more attractive to app developers, and this process will evolve and improve (as was made clear throughout the discussion of this).

 

So, what do you think about it? I, for one, am pretty excited..

© Shantanu Goel | This article, titled Clearing the FUD around Ubuntu Application Review Process, was originally published on Shantanu's Technophilic Musings

Skype on linux works great but the problem that I faced was that it does not handle the buttons on the bluetooth headsets. My wife uses skype to call into her work related conference calls and was pretty frustrated that she had to keep sitting in front of my laptop (her laptop didn’t allow to install skype) just to switch mute on and off. So, I whipped up this little trick to do so. Here is how:

1. First I looked at the skype API (pretty simple, based on dbus). Also found a sample script somewhere that allowed to answer and hang up calls through skype API.

2. Modified the script to switch the mute on and off instead of answering/hanging up. Script can be downloaded at the end of this article. You can also modify it to take some other action that you want.

3. Install “blueman”. If you are using Ubuntu, just type “sudo apt-get install blueman” in shell.

4. Once blueman is installed and you have done your headset pairing etc, right click on the blueman icon in the system tray and click on “Local Services”.

5. In local services, click on headset tab and you can see a input box where you can give the path of a program/script which will be run whenever the “answer button” on the headset is pressed. This is important to note that blueman only supports the answer button and nothing else and I didn’t have the inclination to develop a full-fledged AVRCP compliant program to capture other buttons as all I was interested in mute functionality. You can see the screenshot of this below:

6. Make sure that you give the path where you saved the script that you downloaded in step 4 and you are all set now. Make a call and click on the answer button of your headset to see the magic happen

Let me know if you have your own hacks like this or know of a way to have a full fledged capturing of all headset buttons.

DOWNLOAD: Skype Bluetooth Mute Button Script

© Shantanu Goel | This article, titled Mute (or Answer) Skype Calls with BT Headset Button in Linux, was originally published on Shantanu's Technophilic Musings

This article will tell you a simple way to login automatically to beam cable (Beam-Telecom) internet connection instead of using the web login manually. Not only that, I’ll also tell you how to do this auto-login through your router if you have one with a custom linux firmware. I took a beam cable connection a few days ago. The price is excellent and speeds are good but the only thing that bothered me was the web based login. You need to open up a browser and login through their online portal before you can access the internet. Now, if you don’t log off, then the login may persist even across PC/router reboots but many times it doesn’t. But in these times, it becomes a pain in the wrong place when I am just looking to play online on my PS3 or one of my various net-capable devices, scattered around the house, are trying to pull data and this forces me to open up my laptop just to login. Even for people who use the internet only on their PC, they might not be using the browser all the time and find it a hassle, minor one but still a hassle, to open a browser and login. Hence, I set out to find a way to do this automatically. (BTW, I’ll also be listing out how I came onto the solution in case you are in interested. If you are not, feel free to skip some paragraphs below to move to the solution)

First off, I came to know that Beam cable supports PPPoE logins, but for the life of me, I (and any of my friends) couldn’t get it working. So, I thought of an alternative approach. Since, this was just a login through a web page, there was a good chance that I could automate it through perl/wget/curl etc. I immediately opened up their portal and took a look at the source code in the hopes of finding out the login form, which I could use to find the POST data that they want and submit it through wget. But what I found was that they are using javascript to do the login process, so no direct way of finding this out. And on top of that, the functions are not embedded in the page but loaded through an external script.

Hmm, a setback but a minor one. Javascript is still a client-side executed environment so all hope wasn’t lost. I could have waded through the browser cache to find the js files loaded by the portal and went through the functions to reconstruct the data that I need but I chose an easier way. There is this excellent addon for firefox called “Live Headers”. It shows you all the transactions happening between your browser and the server. So, I just fired it up, filled in my login details in the portal and hit “Login”. On seeing the live headers log, I found out the exact url as well as the POST data that they were sending to do the actual login and from there it was a piece of cake. With “wget”, it is as simple as executing

wget <url> –post-data=<data>

So, by substituting the information that I got, this is the command that you need to execute on your PC (For linux/mac it is already installed generally and for windows, you will need to download from here)

wget http://123.176.37.2/newportal/Ajax.php –post-data="function=ExecuteLogin&user=YOURBEAMUSERNAME&pwd=YOURBEAMPASSWORD&remember=false&timestamp=1271186686298"

Please remember to replace YOURBEAMUSERNAME and YOURBEAMPASSWORD with your actual username and password respectively. Also, please note that the above command is all one single line (it might appear in multiple lines in your browser).

So, everytime you need to login just run this command. Now, how to do this automatically? For linux, just put it inside a shell script for linux and set it to run automatically on bootup. For windows, paste the command in a text file, and save it as beam_login.bat and put it in your start up folder so that it runs everytime on boot. (You might want to put some delays in there as it might take some time to get connected to the network or you can query the network interface to run it when the network is up. Let me know if you want to know how to do this for your platform and I can help you).

So, this is for the PC, now how about if I need to use non-browser capable devices. This is easy if you have a router with a custom linux firmware (because they generally have wget installed or allow you to install it). I have an Asus wl-500w with oleg’s custom firmware installed and it was a piece of cake with it. I just created the following shell script

#!/bin/sh

if [ -e Ajax.php ]
then
  rm Ajax.php
fi

#Limit the number of retries to prevent the router from going into continuous loop
num_retries=10

while [ $num_retries -gt 0 ]
do
wget http://123.176.37.2/newportal/Ajax.php –post-data="function=ExecuteLogin&user=YOURBEAMUSERNAME&pwd=YOURBEAMPASSWORD&remember=false&timestamp=1271186686298"
if [ -e Ajax.php ]
then
  rm Ajax.php
  break
else
  sleep 2s
  num_retries=`expr $num_retries – 1`
fi
done

Then I set it to run on boot and voila!! A tip for other users of any of the asus routers with oleg’s firmware (like wl-500g, wl-520gu, wl-500gp etc). The firmware has some scripts in the /local/sbin folder that are run at various times. I suggest you put the above code inside the post-firewall script. (Do not forget to run “flashfs save && flasfs commit && flashfs enable” to save your changes). Now, every time my router reboots, it automatically logs in to the connection

Hope I have been clear in the isntructions. If you are unclear about anything or need any help in implementing it for your specific platform, do let me know.

© Shantanu Goel | This article, titled Automatic Login For Beam Cable, was originally published on Shantanu's Technophilic Musings

If you can’t get tail command to continuously monitor a file, then read on. I was working on a script yesterday, a part of which depended on continuous monitoring of a text file. I had used our trusty old “tail” command for this but while testing by manually putting in some data into the file, it was failing but curiously it was working fine when used in actual scenario. Befuddled, I did a simple test. I created a simple text file “a.txt” with a few lines of data and then ran the following command.

tail -f a.txt

It showed the last few lines of the file and kept waiting. So far so good. Then I opened the file in vim editor, wrote a few more lines, saved the file and then waited but nothing in the window that was running the tail command. Thinking that the data might be buffered and not flushed to the disc yet, I ran the sync command but still nothing.

Then I got a hint that when I used the “-F” or “–follow=name” option instead of “-f”, the tail command was able to detect the change just fine, the only problem being that in this mode, it prints the last few lines again, not just the newly added line. The main difference in these new options is that tail command tracks the file for changes by its name and not by the file descriptor, and then it dawned on me. The problem is not in the tail command but my testing method itself. When I save the file opened in vim, it creates a new file with a new inode while the one opened by tail is still the old one (which is now a temporary file which has actually been deleted). When I quit tail, then the kernel deletes the file automatically. This is also confirmed by running “lsof | grep a.txt” (lsof lists the open files and then we find the ones related to a.txt). The output shown is;

tail 11966 shantanu 3r REG 8,6 8 224954 /home/shantanu/dev/perl/plot/a.txt~ (deleted)

vim 12576 shantanu 9u REG 8,6 12288 210918 /home/shantanu/dev/perl/plot/.a.txt.swp

which shows what we had discussed above. This gets worked around when I use the -F option because then tail periodically reopens the file by name and reads it again, thus bypassing the above issue. Then I simply tried running tail again on the same file and doing something like “echo abc >> a.txt” and I could see the behaviour as expected with tail immediately detecting the change and displaying it in its window. Hope this helps if you have been pulling out your hair thinking you have gone crazy as your favourite little tool that you have been using for so many years has suddenly stopped working and no one else apart from you is even complaining

© Shantanu Goel | This article, titled Continuous Monitoring With Tail Fails, was originally published on Shantanu's Technophilic Musings

I generally move from LTS to LTS releases of Ubuntu but reluctantly had to update my system to Karmic when my old HDD gave in. I also thought of trying out Kubuntu (KDE based) this time as I had heard that its doing a lot of interesting things (Akonadi/Nepomuk etc) and that with 4.3.1 it is stable as well. The installation went fine, all over within less than half an hour and then the problems started. Here are the 2 main issues that I faced along with their solutions, in the hope that if someone else runs into same issues does not have to waste time (and hair) on it

1. Bootloader Issue: I dual boot my system along with Win XP (Needed for some office work) and generally install the bootloader onto a separate boot partition instead of overwriting the MBR. But this time when I did the same, I couldn’t boot (not even the grub menu). When I did this, the very helpful message that I got on restarting after installing kubuntu was “Error loading operating system”. I could still go back to my good windows install by setting the boot flag onto its partition. I mucked around with it a lot, trying to install grub2 again and found that actually it fails when I try to do the above (install on a partition). I also tried copying the boot sector from my /boot partition to C: and use ntldr to boot into linux but that also didn’t work. This lead me to believe that the same thing happened during my main installation and the installer failed to tell me anything about it. I tried then to install it on MBR but that also mucked up things and I couldn’t even get the error message, a cursor just kept blinking. Finally, the solution. I had to install kubuntu again and this time selected to overwrite MBR (this is default, BTW).

2. Wi-Fi Connection Issue: Second immediate issue was with knetworkmanager. It failed to start my wi-fi module (even after installing restricted drivers). I disabled and enabled wi-fi, so that brought some life to it (wi-fi led started glowing) but it still couldn’t scan for any access points. I could do the scan through command line and see my AP but my AP is WPA2 secured and I didn’t want to mess around with wpa-supplicant stuff. Solution: Installed Wicd. And it worked automagically.

Hope this helps someone else who faces the same issue. Looks like my idea of moving to kubuntu wasn’t a good one (especially because I had 2 lockups during the 1 hour I used it for in the wee hours of this morning). I’ll probably give it another week or so before taking the decision whether to move back to ubuntu or not.

One good thing to take away from this though: For the first time I could really appreciate the live CD installer that these linux distros give. As I could search about the various issues I had without having a second computer. But then again, it’s not that good as well because it means that so far I never had any installation issues at all…

© Shantanu Goel | This article, titled grub and knetworkmanager problems in Kubuntu Karmic, was originally published on Shantanu's Technophilic Musings

I recently got hold of a Guitar Hero controller for my PS3 but the game still had some time to come so I tried it out on my laptop with Frets on Fire (rather FoFix). But when I started it, it was crashing with some weird error about not being able to open mixer devices. Searching on the google, I came to know this is happening because of Pulse Audio as this application needs direct access to the sound devices and finally I came upon pasuspender. This little tool allows you to do that with ease. Just run it in the following way:

pasuspender — ./FretsOnFire

And it will start working just fine. You can use the same trick for any other application that needs direct access to the sound devices. What pasuspender does is that it will suspend pulse audio and allow the application to access them directly and when the application is terminated, normal functioning will resume.

© Shantanu Goel | This article, titled Making Frets On Fire Work With Pulse Aludio – pasuspender, was originally published on Shantanu's Technophilic Musings

Update: It has come to my notice that many people do not know about this (and other problems listed in comments of this post). Please digg and reddit this post so that more people can be made aware of it before they upgrade unknowingly.

You should not upgrade to Ubuntu 9.04 (aka Jaunty Jackalope), released a few weeks ago, if:

1. You have an older ATI graphics card (prior to R500, .e.g xpress 200m)
2. You like to have 3D support (read compiz/games, etc)

This is because of the following reasons:

• Ubuntu Jaunty Jackalope has the new version of X.org server (1.6)
• The open source graphics drivers for ATI do not have 3D support
• The only version of the ATI properietary drivers (fglrx\Catalyst) compatible with X.org server 1.6 is 9.4
• Support for older cards (mentioned above) was dropped from fglrx 9.4

So, basically right now you are in a soup and it seems to me that this is going to remain this way unless someone figures out how to backport x.org 1.5 onto Jauntu, which is going to be a hell of a task, IMHO

But, if you are a simple person, who looks beyond the gloss of the 3D and marvels at the beauty that the rest of the system holds, then go right ahead and dive in

© Shantanu Goel | This article, titled Why You Should Not Upgrade to Ubuntu Jaunty Jackalope (9.04), was originally published on Shantanu's Technophilic Musings