MicroMacro 2 - Download a file over HTTP

For only tutorials. If you would like to post a tutorial here, first post it elsewhere (scripts, general, etc.) and private message me (Administrator) a link to the thread. I'll move it for you.
Post Reply
Message
Author
User avatar
Administrator
Site Admin
Posts: 5306
Joined: Sat Jan 05, 2008 4:21 pm

MicroMacro 2 - Download a file over HTTP

#1 Post by Administrator » Thu Mar 19, 2015 11:23 am

After some much needed reworking of the socket class, it is now possible to connect to a webserver and download a file from it. In this example, we'll download a copy of the RoM bot scripts from: https://solarstrike.net/project/download/14

To do this, we open a socket and send the request to the webserver: GET /project/download/14 HTTP/1.1\r\nHost: solarstrike.net\r\n\r\n
Once the request is received by the server, it will send us a reply telling us some information about the file, and then the contents of it. The reply will be formatted like this:
HTTP/1.1 200 OK
Date: Thu, 19 Mar 2015 16:12:01 GMT
Server: Apache
X-Powered-By: PHP/5.6.3
Cache-Control: public
Content-Disposition: attachment; filename="rombot.3.29.r788.zip"
Accept-Ranges: bytes
Content-Transfer-Encoding: binary
Set-Cookie: laravel_session=<snipped>; expires=Thu, 19-Mar-2015 18:12:01 GMT; Max-Age=7200; path=/; httponly
Last-Modified: Fri, 26 Dec 2014 17:54:37 GMT
Content-Length: 261060
Content-Type: application/zip

< actual binary file content begins here >
Note: Each file line ending has "\r\n". When we see "\r\n\r\n" is when we know where the file's content begins.

While we could parse and make use of anything from that header (the true filename, content-type, etc.), we are only going to bother with the content-length in this example. We will check the received content and compare it against the expected file size to see when we have completed the transfer and then terminate the connection.


In macro.init(), we will open our socket and send the request to the server:

Code: Select all

	local host = "solarstrike.net";
	local connected,err = socket:connect(host, 80);
	print("Connected:", connected, err);

	local proto = "http";
	local file = "/project/download/14";
	local get = sprintf("GET %s HTTP/1.1\r\nHost: %s\r\n\r\n", file, host);

	print("Send:", socket:send(get));
	outfile = io.open("download.zip", "wb");
	assert(outfile);

Our macro.main() simply continues execution until signaled to stop with the variable 'done':

Code: Select all

function macro.main()
	if( done ) then
		printf("Total received bytes: %d\n\n", totalReceived);
		outfile:close();
		return false;
	else
		return true; -- Keep on waiting until we're done.
	end
end
macro.event() is where the bulk of the work is done. Particularly, on our 'socketreceieved' event. It might look complicated, but it isn't. We simple read the packets and check for usable data, split it up by how we want to use it, and save the actual file content into download.zip. Once we reach the full filesize, we close the connection with socket:close() and set our 'done' variable to 'true' to signal script termination.

Just for good measure, we'll even display a fancy progress bar that shows the user how much of the file has been downloaded.

Code: Select all

function macro.event(e, ...)
	if( e == "socketreceived" ) then -- For every packet we receive...
		local sockId,data = ...;

		if( not beginSaving  ) then
			-- Parse needed header content and make use of it.
			contentLength = data:match("Content%-Length:%s+(%d+)");
			if( contentLength ) then
				contentLength = tonumber(contentLength);
				printf("Expected file size: %d\n", contentLength);
			end

			-- A \r\n\r\n signals the start of the content, so we'll save anything past it.
			local foundPos = data:find("\r\n\r\n");
			if( foundPos ) then
				print("Data:", string.sub(data, 0, foundPos));
				print("Begin saving\n\n");
				data = string.sub(data, foundPos + string.len("\r\n\r\n"));
				beginSaving = true;

				printf(progressFmt, 0, ''); -- Start displaying our progress bar (blank right now)
			end
		end

		if( beginSaving ) then
			-- We've received another piece of the file, so save it and update progress.
			outfile:write(data);
			totalReceived = totalReceived + string.len(data);

			local progressRatio = totalReceived / contentLength;
			local goodCount = 0;
			if( progressRatio > 0 ) then
				goodCount = math.floor(progressRatio * progressBarWidth);
			else
				goodCount = 0;
			end
			printf(progressFmt, math.floor(progressRatio * 100), string.rep('=', goodCount));

			if( totalReceived == contentLength ) then
				print("\n\nSuccessfully completed file transfer.");
				socket:close();
				done = true;
			end
		end
	end
Attachments
http.download.lua
(2.57 KiB) Downloaded 315 times

Post Reply

Who is online

Users browsing this forum: No registered users and 5 guests