A simple Ruby rapidshare downloader

This is a very simple ruby program that i wrote to automate the process of downloading files from rapidshare, without having a premium account.
The program file is available for download here, to use it you have to install ruby and rubygems (under Linux look them up in your distro package manager, under Windows go here to download ruby and here to download rubygems) if you don't have them installed yet, and once you have rubygems, install a couple of libraries the program needs this way:

#>gem install mechanize --include-dependencies

I didn't tried it under windows but it shouldn't have any problem, maybe you have to escape the directory path where you want to store the files, so 'c:\myfolder\somewhere' should be 'c:\\myfolder\\somewhere'.

Execute the program without any arguments and the help is displayed.

usage:
$> rapidshare_downloader RAPIDSHARE_URLS...(separated by spaces) DOWNLOAD_DIRECTORY

Example:
$> rapidshare_downloader http://rapidshare.com/something1 http://rapidshare.com/something2 /home/myname/somedirectorypath

Or you can also call it this way(which is more practical):

$> rapidshare_downloader DOWNLOAD_FILE
where DOWNLOADFILE is a text file containing on each line a rapidshare address, and in the last line an existing directory in the filesystem

Example of a DOWNLOADFILE:
http://rapidshare.com/something1
http://rapidshare.com/something2
...
/home/myname/somedirectorypath

Empty lines and lines starting with '#' are not read.


Some comments if you are interested in how it works.

I used a very good http client library called Mechanize, because the one that comes with ruby (Net::HTTP) is a little bit hard to manage, for example cookies are not handled automatically, and to submit a form you need to pass in the parameters one by one, you cannot post the html.
The use of Mechanize is really simple and intuitive, it handles cookies and redirection alone, it's possible to simulate button clicks, submit forms, etc.
The only drawback is that the documentation is not complete, so sometimes you have to guess.
It is available as a gem, so installing it is as simple as doing:

$>sudo gem install mechanize --include-dependencies

it will also download hpricot'which is a library that mechanize uses. Hpricot is used to parse html and sometimes you have to create some hpricot objects to interact with mechanize.

The code is easy to understand, what I do was a little hack to the html. First I make a get request with the url address that rapidshare gives, then I submit the first form in the page which is the one that submits if you click on the "Free user" button, in the response there is a form in a javascript string which will be place into the DOM once a "setTimeout" javascript function executes (this time is also validated in the server, so it's impossible to submit the form before time), so I extract this form and submit it after the time passes. I do this for each file.

Another important thing to notice, is that the file is saved in memory first and after all the file is downloaded, then it get saved in the hard disk.
That is why I release the reference to the file and call the garbage collector after the file is saved, I take a look and it seems to not be working, but I still had plenty of memory when i tried it, maybe thats why the garbage collector didn't collect it, or the top command was not showing the right things...
So to run the program you will need memory depending on the size of the files, or at least each file (if the gc works).

Any questions or suggestions, please write to me.

PD. the file is now posted at github and this will be the latest version, this is the clone url: git://github.com/bsampietro/rapidshare_downloader.git

Connect to a VPN using the PPTP (Microsoft propietary) protocol under openSuse 11.0 Linux

I use openSuse but it surely work under other distros too.
I did some research and I didn't found much information about this on the web, that is why I am writing this little article.
Install the “pptp” package, it comes with two binaries:
“pptp” and “pptp-command”, we will be using “pptp-command” for everything.
As root run:
# pptp-command

it will present four options:
1.)start
2.) stop
3.) setup
4.) quit

We need first to setup the connection so we select: 3
it will present 8 options

1.) Manage CHAP secrets
2.) Manage PAP secrets
3.) List PPTP Tunnels
4.) Add a NEW PPTP Tunnel
5.) Delete a PPTP Tunnel
6.) Configure resolv.conf
7.) Select a default tunnel
8.) Quit

First we need to authenticate, so we have to select 1 or 2 depending on the kind of authentication the server use.
When I configure it, the server I was trying to connect use CHAP so lets select 1
It will present:

1.) List CHAP secrets
2.) Add a New CHAP secret
3.) Delete a CHAP secret
4.) Quit

We need to add a new CHAP secret, so we select: 2
first the program ask for the Local Name,

Local Name: myusername

there put the username of your vpn account, then it asks for the Remote Name [PPTP]

Remote Name [PPTP]:

there in most cases we can leave the default (see the explanation), so we left it blank... then it asks for the password of your vpn account

Password: mypassword

there we put the password.
Once this is done the CHAP secret is added, then we go back(or we can do any other task that the program offers), we press 4.

The main menu appears again and we need to add a new PPTP tunnel so we select option 4

Add a NEW PPTP Tunnel.

1.) Other
Which configuration would you like to use?: 1
Tunnel Name: thenameofmytunnel
Server IP: serverdir
What route(s) would you like to add when the tunnel comes up?
This is usually a route to your internal network behind the PPTP server.
You can use TUNNEL_DEV and DEF_GW as in /etc/pptp.d/ config file
TUNNEL_DEV is replaced by the device of the tunnel interface.
DEF_GW is replaced by the existing default gateway.
The syntax to use is the same as the route(8) command.
Enter a blank line to stop.
route: add-net 192.168.10.0 netmask 255.255.255.0 dev TUNNEL_DEV
Local Name and Remote Name should match a configured CHAP or PAP secret.
Local Name is probably your NT domain\username.
NOTE: Any backslashes (\) must be doubled (\\).

Local Name: myusername
Remote Name [PPTP]:
Adding thenameofmytunnel - serverdir - myusername
Added tunnel thenameofmytunnel

It asks for the configuration to use, we select Other (there Is no any other choice), then it asks for the “Tunnel Name” we put there the name which you want to identify your tunnel, put it the way you want.
Then it asks for the Server IP, put there the ip or name of the server which you want to connect to.
Then it asks for the route, using the same syntax of the linux route command, you can put in there(for example): add-net 192.168.10.0 netmask 255.255.255.0 dev TUNNEL_DEV, so that the operating system knows that every packet send to the network 192.168.10.0 with the netmask 255.255.255.0 should use the device of the tunnel interface, here the program replaces TUNNEL_DEV with the name of the device of your vpn connection (generally ppp0). For this you should know the internal of the network behind your vpn connection.
You could also left this blank and add the route from the operating system like this:
#route add -net 192.168.10.0 netmask 255.255.255.0 dev ppp0
but in this case it will be lost when you reboot the machine.

At last it ask for the “Local Name” and the “Remote Name”, this have to match the ones you put in your CHAP secrets, remember that the local name is the username of your vpn account and the Remote Name we leave it blank.

Once this is done we just quit and then we type again the pptp-command as root but here we select 1 (start), you could have also typed:
#pptp-command start

It asks for the connection you want to use, and thats it!, you should be connecting to your vpn.

If you need to connect to a machine in your vpn that uses the RDP protocol which come installed with windows, you can download the package “rdesktop” and type:
>rdesktop -zfk keymap machineip , you can look at the installed keymaps doing a
>locate keymap and see the ones that are installed inside ..rdesktop/keymaps this is important because by default it uses the english keymap, so if your keyboard is not english then you might have the keys changed.

You can take a look at http://pptpclient.sourceforge.net for more details

Some tweaks on linux.

I have recently install openSuSE 11.0 on my new Compaq Presario F756 laptop.

I am going to tell you about a few little tweaks you can do to prepare your system, I use opensuse, but you can apply them to other linux distros as well.

Partitioning:

Partitioning a harddisk is always complicated because you never know the amount of space you will use on each partition, and if you need to resize a partition you have to backup and all stuff...

I have a 120 GB disc and what I did was to reserve 12 GB for the root partition, 7 GB for /home, 1 GB for /swap(i have 1 GB of RAM and I know I am not going to need any more swap space, but this totally depends on how you use your system, anyway if you need more space you can create a swap file on any partition and use it) and the rest I assign it as a big partition to store music, movies and large files like iso images for example.

When you format a partition with the ext3 filesystem, by default it allocates a 5% of the space to the superuser, but if it is not a system partition for example the partition i keep for big files, it does not make a lot of sense, and if the partition is big this will take you out a lot of space. So what we can do, is use the tune2fs command like this:

# tune2fs -m 0 /dev/xxx

this is the explanation for the -m option:

(-m reserved-blocks-percentage
Specify the percentage of the filesystem blocks reserved for the super-user. This avoids fragmentation,
and allows root-owned daemons, such as syslogd(to continue to function correctly after non-privileged
processes are prevented from writing to the filesystem. The default percentage is 5%.)

I dont know if you can do this, once you have data on the partition, do it at your own risk.

TMPFS (Temporary file system)

This is a very useful stuff and I have not heard very much about it around there. I find it when I was looking how to setup a ramdisk for linux, and I find this that is much better.

A tmpfs is a filesystem that is stored in RAM, and it also can dynamically grow as you put data on it, if your linux kernel is 2.4 or above you can use this feature.

It is very useful to store temporary files (like recently decompress files, small downloads from the internet or browsers cache) to avoid fragmentation and innecesary writes to the disk, it also release you from the need of manually deleting those temporary files, because once you power off or reboot the system they disappear... you have to be careful also.

You can mount a filesystem of this type any time you want this way:

# mount -t tmpfs tmpfs /tmp

(It is possible to add another parameters but this is the simplest way)

or if you want it to be mounted at system startup, add this line to your /etc/fstab:

tmpfs /somewhere tmpfs auto,user,sync,rw 0 0

Once you do this you can assign the firefox cache to this file system (browsers cache completely full your harddisk with a lot of little temporary files) this way:

In the location bar, type about:config and hit enter, then right-click in the white-space, and choose New->String,
add the name “browser.cache.disk.parent_directory” (without quotes) and click ok, In the next box, enter the full patch to the directory you want to store your cache in. Something like “/somewhere/firefoxcache”

Then restart firefox and done.

Useful things for firefox:

Install the google toolbar, it is very useful to store bookmarks (on the server) and avoids going to the google page everytime you need to search for something, it has several other useful functions.

Another very useful tool if you are a web developer is Firebug, excelent tool for debugging javascript, and a lot of other stuff.

I know there are plenty of other very useful add-ons, I have to spend some time digging.

Wireless

Another thing I had to do and is known for not being very linux friendly is configure a wireless card.

If you have an atheros card you can choose between mad-wifi which is an open source projects that provides linux native drivers for atheros cards, or ndiswrapper which is a program that installs windows XP drivers in linux. If your card is not atheros, I think the only choice is ndiswrapper, or maybe your card manufacturer made some linux native drivers.

I have an atheros wi-fi card but I decided to go with ndiswrapper instead of native mad-wifi because I did some research and the common mad-wifi download will not work with my wireless card.

So I install ndiswrapper, follow the instructions here.

it is just:

# ndiswrapper -i /path/to/your/inffile/xxx.inf

you need to have both windows drivers files, the inf file and the sys file in the same directory.

Then if all goes well you can verify it with this:

# ndiswrapper – l

If you did it right it will display something like this:

Installed ndis drivers:  driver present, hardware present

Where shows the name of the windows driver installed with ndiswrapper.

If it says "invalid driver" then you need to uninstall that driver and try another one.

To uninstall a driver, you need to type:

ndiswrapper -e 

Then install the module with: 

modprobe ndiswrapper and restart your network manager with: 

# /etc/init.d/network restart

To load the driver at boot, add the line: “modprobe ndiswrapper” at the beginning to your /etc/init.d/network startup script (I have read that if after “modprobing” ndiswrapper you configure everything with yast, you don't need to edit the /etc/init.d/network script, but for me it didn't work. This is the only thing that is specific for opensuse), maybe there is a neater way of doing this, but this just works.

So after doing all this, I use yast to configure the wireless, just create a new connection with Device Type → Wireless and in Module Name type “ndiswrapper”. Then hit next and put there the name (ESSID) of your wireless LAN and the type of authentication you use in your router, (I try to use KnetworkManager to manage the authentication and leave empty the settings in yast but with wap-psk which is the one that I use it was impossible) and that's it, reboot and if you put the authentication type, the code, and the ESSID right, you will have wireless.

Some useful programs:

First add the Packman repositories which are available in the “software repositories” in yast along with the standard SuSE ones.

openSuSE does not come with the useful command “locate”, so install it via yast, the name of the package is findutils-locate.

Another useful tool is the command line tool “rar” to compress and decompress rar archives, which is very popular in windows world.

Talking about windows, there are certain windows programs that maybe useful and don't have a linux version, for this you can install “wine” and you can run most windows little programs, and the heavy ones also with a little tweaking.

For playing audio, I use to love winamp in windows. In linux there is audacious which is a winamp clone and works better than the old xmms, in SuSE repositories(standard ones plus packman) there are input plugins for every possible sound file format including wma (which sometimes we can not avoid)

For playing video there is mplayer, which I think is a very good player, although I have never tried VLC which I heard is good also.

Well, this were some basic things i did to put linux to work the way i like, other recomendations are welcome!!