Friday, December 27, 2013

How to Build a Robot in Browser

Robot, bot, browserbot, botser, botowser? I actually used a ReBot -- REcommendation Bot before. Many possible names, the same meaning -- you need a software robot running in your browser.

Why browser? Some of the reasons are:

REASON 1. The stuff you will feed to your robot is in the web -- something you view in your tabs or possibly something that your robot will open and read on its own (possible!, done that already!),
REASON 2. Final results of your robot's digestion tract (I avoid saying poop, obviously) are also web-based -- the closest example is storing your stuff in cloud drive, where I personally use the Dropbox JS API which I wrote myself (comes with the code below).
REASON 3. You need your robot to possess maximum achievable compatibility, where web technologies is obviously the way to go. What did the Firefox guy recently say about Firefox OS release? -- "All other platforms are beautiful rose gardens surrounded by unreasonably high fences".
REASON 4. ... fill in with your personal reasons ... of which I have a couple but will not write them here to stay focused on the main topic.


So, what do you need to write a robot? Not much, it comes out. See the points below.

POINT 1. Use Chrome. See illustration below about how chrome designs its extensions. Firefox and other browsers have their own designs, but I find Chrome the easiest to use. In, fact I failed to get an example Firefox extension to work in the first place. Does not indicate my stupidity... it indicates how clumsy the design is -- take my word for it. Besides, Chrome makes it easy to debug extensions. You can open consoles for each of the three below components -- float, inpage and background.





POINT 2.
About extension components. Use them wisely, meaning "in accordance to their purpose". *.bg.js (background) will start running immediately when the extension is loaded. *.inpage.js will run on each page which matches your prefix. *.float.js will show the pretty (hopefully) GUI when user clicks on the icon that shows up in the browser. Yes, you can create a pretty icon for your extension.

POINT 3. If you work with one URL prefix, it is easy -- just write the matching rule in your manifest.json. However, if your robot wants to digest many different pages (like mine does), then you need to write the matching rule as "matches":["http://*/*"] (exact line in manifest.json) and fork into individual processors from inside your *.inpage.js. Increases complexity, but totally worthwhile given the increased scope/coverage -- ultimately you are building a Swiss knife of web page parsing.

POINT3. Do not be afraid to throw all the web technology you have at your problem. Specifically, note the following unique aspects of Chrome extensions. (1) you have DOM in all three components, ... it would seem that there should be no DOM in *.bg.js but there is (!) one .. DOM is important when you need to set timers to ... hm ... time/pace things such as Dropbox accesses, Google Map requests, etc.... I use jQuery's Timer extension which needs a DOM to work. (2) you are free of Same Origin policy inside your extensions so contact whatever service you want... I use Dropbox for cloud-side storage, for example. (3) you can work with all the advanced features such as jQuery (load its JS from manifest.json), CANVAS/SVG, local storage, etc.



That's pretty much it. You can see my working testcases at this GitHub repository. It is a ready-to-use Chrome extension which you can load in its current form -- just point Chrome to this folder and tell it to load the extension. The icon should pop up in the bar immediately. Obviously I have no background or inpage scripts, but the testcases will show you where I am shooting at.

I will not go into details about the serverless.js file which contains all my custom components including those of GUI nature. The extension has several working testcases. Some of them are not finished yet ... like Stringex and possibly CloudStorage, but the rest will work. I am particularly proud of NICECOVERring and SidePaneStack components which I will use as low-level QUI components. Those definitely work and you are free to play around with them.







No comments:

Post a Comment