pdf
|<<
<
>
>>|
/
Applications of ngx_openresty and perl at lz.taobao.com ---- Applications of {{#x|ngx_openresty}} and {{#x|perl}} at lz.taobao.com ☺{{#author|agentzh@gmail.com}}☺ {{#author|章亦春 (agentzh)}} {{#date|2011.07}} ---- {{#x|♡}} We are {{#x|lz.taobao.com}} {{http://lz.taobao.com}} {{img src="images/lz-logo.gif" width="247" height="50"}} 我们是淘宝量子统计 ---- ➥ we offer web/sale/ads {{#ci|analitcal}} services for our {{#i|Taobao sellers}}. {{img src="images/taobao-logo.png" width="140" height="35"}} ➥ 我们为我们的淘宝卖家提供 网络/销售/广告的{{#ci|统计分析}}服务 ---- {{#x|♡}} We're analyzing {{#ci|tera bytes}} of raw data {{#x|every day}} 我们每天分析 {{#ci|TB}} 级别的数据 ---- {{#x|♡}} We do have {{#ci|1,800,000+}} of Taobao seller users {{#x|registered}} for our web app. 我们的 web 应用有一百八十多万的淘宝卖家注册用户 ---- {{img src="images/lz-login.jpg" width="982" height="582"}} ---- {{img src="images/lz-shopsummary.png" width="1059" height="723"}} ---- {{img src="images/lz-srccomp.png" width="1059" height="723"}} ---- {{img src="images/cluster-arch.png" width="1124" height="428"}} ---- {{#x|♡}} Our frontend web app is running directly in your {{#ci|web browser}}! 我们的前端 web 应用是直接运行在 你的{{#x|网络浏览器}}里的！ ---- {{img src="images/browser-app.png" width="730" height="416"}} ---- {{#x|♡}} TT == Perl {{#ci|Template Toolkit}} {{http://search.cpan.org/perldoc?Template::Manual}} ---- {{#x|♡}} We only use the TT {{#ci|templating language}}, {{#x|not}} the Perl TT engine itself 我们只是使用了 TT 这门模版语言, 而不是 Perl TT 引擎本身 ---- {{#x|♡}} We use the {{#x|Jemplate}} compiler to {{#ci|compile}} our TT templates to standalone {{#x|JavaScript}} code {{http://search.cpan.org/perldoc?Jemplate}} 我们使用 Jemplate 编译器把我们的 TT 模版编译为独立的 JavaScript 代码 ---- {{img src="images/jemplate.png" width="400" height="291"}} ---- {{#x|♡}} {{#ci|Client-side}} templating is a lot of {{#x|fun}}! 客户端模版编程非常有趣！ ---- {{#x|♡}} ngx_openresty is a massively {{#ci|enhanced}} and {{#ci|extended}} version of {{#x|nginx}} {{http://openresty.org}} ngx_openresty 是 nginx 的一个 进行进行过大量增强和扩展后的版本 ---- {{img src="images/openresty-org.jpg" width="908" height="616"}} ---- {{#x|♡}} The {{#x|original idea}} of ngx_openresty comes from the man simply known as {{#ci|chaoslawful}} {{https://github.com/chaoslawful/}} ngx_openresty 的最初的想法来自一个 只知道叫做 chaoslawful 的男人 ---- {{img src="images/chaoscat.png" width="800" height="600"}} ---- {{#x|♡}} Our server-side code just provides {{#x|web services}} and {{#ci|secrity protection}}. 我们的服务器端代码只是提供 支持性的数据服务和安全保护。 ---- {{img src="images/lightface.png" width="1147" height="642"}} ---- {{#x|♡}} There's {{#ci|0}} lines of {{#x|Perl}} in the online ngx_openresty server 在线的 ngx_openresty 服务器中 有 0 行 Perl 代码 ---- {{#x|♡}} Synchronous {{#ci|non-blocking}} Lua code 同步的非阻塞的 Lua 代码 ---- {{#kw|upstream}} main_db { {{#kw|drizzle_server}} 127.0.0.1:3306 {{#kw|user}}=monty {{#kw|password}}=some_pass {{#kw|dbname}}=test {{#kw|protocol}}=mysql; {{#kw|drizzle_keepalive}} {{#kw|max}}=10 {{#kw|overflow}}=reject {{#kw|mode}}=single; } ---- {{#kw|location}} /mysql { {{#kw|set_unescape_uri}} {{#v|$sql $arg_sql}}; {{#kw|set_unescape_uri}} {{#v|$backend $arg_backend}}; {{#kw|drizzle_query}} {{#v|$sql}}; {{#kw|drizzle_pass}} {{#v|$backend}}; } ---- {{#kw|location}} /main { {{#kw|content_by_lua_file}} {{#x|conf/my-app.lua}}; } ---- {{#cm|-- my-app.lua}} {{#kw|local}} {{#v|res}} = ngx.location.capture({{#x|"/mysql"}}, { args = { sql = {{#x|"select count(*) from cats"}}, backend = {{#x|"main_db"}} } }) ---- {{#x|♡}} But almost {{#ci|all}} of our Lua code for the business logic is {{#ci|generated}} and optimized by {{#x|Perl}} 但我们几乎所有用于业务逻辑的 Lua 代码 都是由 Perl 生成和优化的 ---- {{#x|♡}} We {{#ci|invented}} the {{#x|LZSQL}} little language for our business. {{http://agentzh.org/misc/nginx/lzsql-manual.html}} 我们为我们的业务发明了 LZSQL 这种小语言 ---- {{#cm|-- /=/view/itemdailyflow/type/trend}} {{#kw|int}} {{#v|$uid}}; {{#kw|text}} {{#v|$begin}}, {{#v|$end}}, {{#v|$today}}, {{#v|$url_index}}; {{#kw|symbol}} {{#v|$db}}; {{#kw|location}} {{#v|$lz_report}}; {{#v|@hist}} := {{#kw|select}} ... {{#kw|from}} {{#kw|LZDB}}.dpunit_purl_result({{#v|$db}}, {{#v|$begin}}, {{#v|$end}}, {{#v|$uid}}) {{#kw|as}} a ... {{#kw|at}} {{#v|$lz_report}}; {{#v|@rt}} := {{#kw|select}} name, count(name) {{#kw|from}} {{#kw|LZRTI}}.getPurl({{#v|$end}} {{#kw|as}} day, {{#v|$uid}}) {{#kw|group by}} name ... {{#kw|return}} {{#kw|select}} ... {{#kw|from}} {{#v|@hist}} {{#kw|union all}} {{#v|@rt}} ... ---- {{img src="images/lzsql.png" width="795" height="448"}} ---- {{#v|$}} {{#x|lzsql-compile}} -c -O2 -n src/*.lzsql {{#v|$}} {{#x|lzsql-link}} -m lightface.core \ -o lightface/core.lua src/*.oul ---- {{#x|♡}} Our LZSQL {{#ci|compiler}} is written in {{#x|Perl}}! 我们的 LZSQL 编译器是用 Perl 编写的！ ---- {{#x|☺}} Parse::RecDescent -> LZSQL {{#ci|Parser}} {{#x|☺}} Moose -> LZSQL {{#ci|AST}} {{#x|☺}} Moose -> LZSQL {{#ci|Optimizer}} {{#x|☺}} Moose -> LZSQL {{#ci|Code Emitter}} ---- {{#x|♡}} We {{#ci|can}} generate {{#x|Lua}} code, so why not {{#ci|C}}? 我们既然可以生成 Lua 代码， 那为什么不能生成 C 呢？ ---- {{#x|♡}} Our ngx_lz_st module is generated {{#ci|automatically}} from our real-time stats engine's TCP protocol {{#x|documentation}}! 我们的 ngx_lz_st 模块是从我们的 实时统计引擎的文档自动生成的！ ---- {{#x|♡}} The documentation is actually written in a declarative little {{#ci|language}} named {{#x|Ticpy}} designed by myself! 这个文档其实是用我自己设计的声明性的 名叫 Ticpy 的小语言来表达的! ---- {{#cm|// File lzrti.tcp}} ... {{#cm|// fid=7}} {{#cm|// @desc: 获取指定店铺的，指定访客的信息}} {{#cm|// @param uid: unit_id}} {{#cm|// @param uv: 访客cookie}} {{#cm|// @param all_sessions: 是否只返回p4p点击：1 是 0 否}} getUv(uid:{{#kw|4}}, uv:{{#kw|s}}, all_sessions:{{#kw|4}}): ( login_time:{{#kw|8}}, {{#cm|// session开始时间}} login_url:{{#kw|s}}, {{#cm|// session开始页面}} login_title:{{#kw|s}}, {{#cm|// session开始页面标题}} stay_time:{{#kw|4}}, {{#cm|// 当前session的时长}} page_deep:{{#kw|4}}, {{#cm|// 当前session的访问深度}} im:{{#kw|s}}, {{#cm|// 用户的旺旺}} ip:{{#kw|s}}, {{#cm|// ip地址}} location_id:{{#kw|4}}, {{#cm|// 地址id}} ... ) ---- {{img src="images/ticpy.png" width="264" height="448"}} ---- {{#v|$}} wc -l lzrti.tcp protocol.c.tt ngx_http_lz_st.c {{#x|345}} lzrti.tcp {{#x|464}} protocol.c.tt {{#x|12478}} ngx_http_lz_st.c 13287 total ---- {{#x|♡}} So...I'd rather write {{#ci|programs to write programs}} to write {{#x|programs}}... 所以。。。我宁可写 写程序的程序， 也不写程序。。。 ---- {{#x|♡}} {{#ci|Test}} our nginx {{#x|C modules}} by our opensource {{#i|Perl}} module {{#ci|Test::Nginx}}! {{http://search.cpan.org/perldoc?Test::Nginx}} 利用我们开源的 Perl 模块 Test::Nginx 来测试我们的 Nginx C 模块！ ---- {{img src="images/test-nginx.jpg" width="982" height="702"}} ---- {{#v|use Test::Nginx::Socket;}} {{#v|plan tests => 2 * blocks();}} {{#v|run_tests();}} {{#x|__DATA__}} {{#kw|=== TEST 1: sanity}} {{#kw|--- config}} location /main { echo_subrequest GET /sub; } location /sub { echo hello; } {{#kw|--- request}} GET /main {{#kw|--- response_body}} hello ---- {{#x|♡}} {{#ci|Test}} our {{#x|web serives}} by {{#i|Perl}}! 用 Perl 来测试我们的 web 服务！ ---- {{#v|use t::LZ;}} {{#v|plan tests => 1 * blocks();}} {{#v|run_tests();}} {{#x|__DATA__}} {{#kw|=== TEST 1: lzers}} {{#kw|--- url}} /=/view/lzers/~/~ {{#kw|--- res}} [ { \"cnt\" : 911619 } ] ---- {{#kw|=== TEST 5: sellerbasic}} {{#kw|--- uid: 1509}} {{#kw|--- url}} /=/view/sellerbasic/~/~ {{#kw|--- res}} [ { \"uv_times\" : 257 }, { \"login_title\" : \"店内搜索页\", \"login_url\" : \"http://xiaowangpu...\", ... ---- {{#x|♡}} Generate {{#ci|test databases}} from rules by our opensource {{#x|Cheater}} utility written by {{#i|Perl}}! {{http://search.cpan.org/perldoc?Cheater}} 利用我们开源的 Cheater 实用程序，从规则 自动生成测试数据库！ ---- {{img src="images/cheater.jpg" width="982" height="686"}} ---- {{#cm|# company.cht}} {{#cm|# Empolyee table}} {{#kw|table}} employees ( id {{#kw|serial}}; name {{#kw|text}} {{#x|/[A-Z]a-z{2,5} [A-Z]a-z{2,7}/}} {{#kw|not null unique}}; age {{#kw|integer}} {{#x|18..60}} {{#kw|not null}}; birthday {{#kw|date}}; height {{#kw|real}} {{#x|1.50 .. 1.90}} {{#kw|not null}}; grades {{#kw|text}} {{#x|{'A','B','C','D'}}} {{#kw|not null}}; department {{#kw|references}} departments.id; ) {{#cm|# Department table}} {{#kw|table}} departments ( id {{#kw|serial}}; name {{#kw|text}} {{#x|/\\w{2,10}/}} {{#kw|not null}}; ) ---- {{img src="images/cheater-mk.png" width="208" height="448"}} ---- {{#x|♡}} I've been hacking on {{#ci|GitHub}}! {{http://github.com/agentzh}} 我在 GitHub 上玩开源！ ---- {{#x|♡}} {{#ci|Follow}} me on {{#x|Sina Weibo}}! {{http://weibo.com/agentzh/}} 在新浪微博上关注我！ ---- ☺ {{#ci|Any questions}}? ☺ ----