You are here

Command-line downloading files behind an authentication wall

Submitted by Druss on Sat, 2012-06-30 00:25

I often find myself in situations where I want to download a file from the command-line using tools such as wget. While there are many reasons to doing so, today I needed to do it due to the size of the file and the fact that I needed it on a webserver of mine. So instead of downloading it first to my desktop and then uploading it to my server, I wanted to simply download the file directly from my webserver via the command-line.

While this would normally be a cinch using wget, said file was behind a login wall. I'm not talking about HTTP authentication (which are reasonably well supported by wget). Instead, I was dealing with site session logins. I suppose that these can also be handled by wget with some skulduggery, I wanted to try another tack, namely Lynx or its more modern (or so I've been told) equivalent, Links.

So I SSHd into my server and started (already installed) Links. I was presented with a completely blank screen on start. Having used Lynx a couple of times back in my sadomasochist years, I remembered to press the g key to goto a web page. In the dialog box that popped up, I entered the URL of my target site. Presented with a login page, I entered my details (navigating through the form with arrow keys), tabbed a few times and "clicked" OK with my Enter key. I was informed that my login form had been submitted successfully. That was it. I was not taken to the next page as presumably that was being handled by JavaScript or something similarly weird from a text browser's POV.

I found out that now that I was logged into the site, I could access previously inaccessible locations within it by simply navigating to them directly. So, all I had to do was copy the URL of the download link to the file I was targeting and paste it in Links. IOW, pressing the g key and entering the URL was all it took for me to start downloading the file. I was also presented with options to background this task in case I wanted to do other things.

The download is proceeding smoothly. Only 2 more GB to go.

Hope this helps someone out there :)