A Firefox cluster driven by JavaScript, Perl, and PL/PgSQL
----

A Firefox X<cluster> driven by
I<JavaScript>, I<Perl>, & I<PL/PgSQL>

☺{{#author|agentzh@yahoo.cn}}☺
{{#author|章亦春 (agentzh)}}

{{#date|2009.2}}
----

 "How about using X<Firefox> in a crawler CI<cluster>?"
 "Man, you're C<crazy>!"

----

   CM<✓> We're running X<24> headless firefox processes
        on X<8> production machines (Linux) and their
        load is around X<3.0>.
   CM<✓> We get CI<100,000> web pages crawled and analyzed
        by my our Firefox cluster X<every hour>.

----

{{img src="images/cluster-arch.png"}}
----

{{img src="images/firefox-guts.png"}}
----

TAG<☆>  We use Firefox X<extensions> to I<control> Firefox's Gecko
CI<from inside> rather than talk to it from outside.

----

    CM</* crawler.js */>
    KW<var> V<browser> = V<document>.getElementById(C<'my-browser'>);
    KW<var> V<browserListener> = KW<new> BrowserListener(browser);
    V<browserListener>.register();
    KW<var> V<openresty> = KW<new> OpenResty.Client(
        {
            server: C<'http://api.openresty.org'>,
            user: C<'listhunter.Firefox'>
        }
    );
    V<openresty>.callback = doTasks;
    V<openresty>.get(C<'/=/view/FirefoxGetTasks/count/200'>);

----

    KW<function> doTasks(V<tasks>, V<ind>) {
        KW<if> (V<ind> == KW<null>) V<ind> = 0;
        KW<var> V<task> = V<tasks>[V<ind>];
        KW<if> (V<task> == KW<null>) KW<return>;
        V<browserListener>.loadPage(
            KW<function> (V<url>, V<done>) {
                KW<if> (V<done>) {
                    analyze(V<browser>.contentDocument);
                }
                doTasks(V<tasks>, V<ind> + 1);
            },
            X<3> CM</* timeout in sec */>
        );
    }

----

  CM<☺> We did CI<NOT> patch Firefox
  with only two small X<exceptions>:

  ➥ Redirect X<Error Console> outputs to CI<stderr>
  ➥ Ignore X<CSS MIME> type CI<mismatch>

----

TAG<☆>  The X<prefetchers> CI<prefetch> the web page content
via the X<HTTP proxy> with cache so that X<Firefox> can
load stuffs from the CI<cache> directly.
----

{{img src="images/prefetcher-guts.png"}}
----

{{img src="images/proxy-guts.png"}}{{img src="images/proxy-guts2.png"}}


----

CM<☺> I added an CI<OverrideExpire> config directive to X<mod_cache>
so that it CI<forgets> overything about RFC.

{{img src="images/feather.gif"}}

----

CM<☺> I implemented a X<mod_libmemcached_cache> module
so that we can have CI<distributive> cache storage for X<mod_cache>

{{img src="images/feather.gif"}}

----

V<Sample benchmark with 59 URLs, 200 currency>

    mod_disk_cache + SATA disk    200 ~ 300 QPS
    mod_disk_cache + tmpfs        400 ~ 500 QPS
    C<mod_libmemcached_cache>       X<2200+>     QPS

----

{{img src="images/resty-guts.png"}}
----

CM<☺> OpenResty is a X<REST> wrapper for PostgreSQL.
It is trivial to CI<expose> PL/PgSQL functions/stored procedures
to the outside world via I<web services> without loosing security.

{{img src="images/pg.png"}}

----

{{img src="images/resty-cluster.png"}}
----

  V<List Hunter>
  ➥  Is the web page a X<list page> or a X<content page>?
  ➥  Extract links in the "CI<main> CI<list>" in list pages.

----

{{img src="images/listhunter-main.png"}}
----

{{img src="images/listhunter-main-list.png"}}
----

  V<Comment Hunter>

  ➥  Extract user X<comments> from
     C<arbitrary> web pages

----

{{img src="images/commenthunter-main.png"}}
----

CI<Test results> from our surfer girls
(with 100 random Chinese commercial sites):

+ Precision ratio: X<97.6%>
+ Recall ratio: X<91.2.%>

----

V<☺> X<Vision>-based filters to rule out
CI<non-comment> lists

{{img src="images/vision.png"}}
----

    V<element>.offsetWidth X<*> V<element>.offsetHeight   CM<// node area>
    V<element>.offsetWidth X</> V<element>.offsetHeight   CM<// node shape>

    CM<// x coordinate of element's left-upper corner>
    V<element>.offsetLeft X<+> absolute x coordiate of V<element>.offsetParent

    CM<// y coordinate of element's left-upper corner>
    V<element>.offsetTop X<+> absolute y coordiate of V<element>.offsetParent

----

{{img src="images/commenthunter-vert.png"}}
----

{{img src="images/commenthunter-hlist.png"}}
----

{{img src="images/commenthunter-linkarea.png"}}
----

V<☺>  I<Ranking testing> is CI<expensive>
but CI<necessary> for the last filter

----

{{img src="images/commenthunter-rank.png"}}
----

V<♡> X<Perl>'s Test::Simple CI<love> for
I<extension JavaScript>

{{img src="images/love-letter.jpg"}}
----

{{img src="images/listhunter-tests.png"}}
----

{{img src="images/listhunter-tests2.png"}}
----

{{img src="images/listhunter-tests-text.png"}}
----

    Test.GuiMode = false;
    Test.X<plan>(2 * V<list>.length);
    KW<for> (KW<var> V<i> = 0; V<i> < V<list>.length; V<i>++) {
        Test.X<ok>(V<i> >= 0, C<'i is always non-negative'>);
        Test.X<is>(V<i> * 2, V<i> + V<i>, C<'i x 2 = i + i'>);
    }
    Test.X<summary>();

----

V<Comment Hunter>: X<JavaScript> & X<Perl> code CI<only>

{{img src="images/js-icon.jpg"}}   {{img src="images/perl_camel.jpg"}}
----

      V<$ find js -name '*.js' | xargs wc -l>
         27 js/cli-prefs.js
        332 js/main.js
          3 js/test-data.js
        374 js/haiway-miner.js
         26 js/box.js
         32 js/util.js
          7 js/env.js
         62 js/benchmark-timer.js
         18 js/samples.js
        160 js/test.js
        329 js/filters.js
        151 js/browser-listener.js
        137 js/test-more.js
       X<1658 total>


----

      V<$ find lib -name '*.pm' | xargs wc -l>
         39 lib/CommentHunter/View/Test.pm
        106 lib/CommentHunter/View/Main.pm
         34 lib/CommentHunter/View/Overlay.pm
         52 lib/CommentHunter/App.pm
        X<231 total>

----

CI<Powered> by my X<XUL::App> framework

{{img src="images/moosecamel.png"}}
----

A CI<Hello World> extension in X<XUL::App>

{{img src="images/helloworld.jpg"}}
----

    CM<# File lib/HelloWorld/App.pm>
    KW<package> HelloWorld::App;
    KW<our> V<$$VERSION>; BEGIN { V<$$VERSION> = '0.01' }
    KW<use> XUL::App::Schema;
    KW<use> XUL::App schema {
        X<<xulfile 'hellowin.xul' => >>
            X<generated from 'HelloWorld::View::HelloWin',>
            X<includes qw( jquery.js hellowin.js );>

        xpifile 'helloworld.xpi' =>
            name is 'HelloWorld',
            id is 'helloworld@agentz.agentz-office', CM<# FIXME>
            version is V<$$VERSION>,
            targets {
                Firefox => ['2.0' => '3.0a5'], CM<# FIXME>
            },
            creator is 'The HelloWorld development team',
            V<...>

----

  V<Ruby>: "We have this CI<gorgeous> syntax!"
  V<Perl>: "Hey, we do X<as well> ;)"

----

   CM<# File lib/HelloWorld/View/HelloWin.pm>
   KW<package> HelloWorld::View::HelloWin;
   KW<use base> 'XUL::App::View::Base';
   KW<use> Template::Declare::Tags 'XUL';
    template main => KW<sub> {
        show 'header';  CM<# from XUL::App::View::Base>
        window {
            attr {
                id => "helloworld-hellowin",
                xmlns => V<$::XUL_NAME_SPACE>,
                title => _('Hello World ') .
                    V<$$HelloWorld::App::VERSION>,
                V<...>
            }
            X<label { _("Hello, world!") } >
        }
        V<...>

----

   V<$> CI<xulapp> X<bundle> .
    Writing file hellowin.xul
    Writing bundle file X<./helloworld.xpi>
   V<$> 

----

 Our KW<helloworld.xpi> bundle TAG<➥>

   CM<✓> contains CI<0> Perl
   CM<✓> has CI<0> dependencies
      (except Firefox itself)
   CM<✓> runs happily CI<everywhere>
      (Win32, Linux, Mac, and etc.)

----

 The CI<future>

   CM<✓> CI<Opensource> everything we have :)
   CM<✓> More hunters, more fun: I<Table Hunter>, I<Title Hunter>,
         I<Ranking Hunter>, I<Ads Hunter>, I<Summary Hunter>, ...
   CM<✓> Automatic C/C++ X<XPCOM> wrapper generator for XUL::App.
   CM<✓> Bring Firefox extension love to Apple's CI<WebKit>
         (A WebKit crawler cluster?)

----

☺ CI<Any questions>? ☺