pdf
|<<
<
>
>>|
/
Scripting libdrizzle with Lua inside Nginx ---- Scripting {{#x|libdrizzle}} with {{#x|Lua}} inside Nginx {{#author|章亦春 (agentzh)}} ☺{{#author|agentzh@gmail.com}}☺ {{#date|2012.4}} ---- \"{{#x|MySQL}} is always the {{#ci|bottleneck}}!\" \"Really?!\" ---- {{img src="images/lz-cluster.png" width="995" height="183"}} ---- {{img src="images/mysql-cache.png" width="459" height="536"}} ---- {{#x|☺}} Some {{#ci|benchmarks}} on Amazon EC2 {{#x|Small}} instances ---- {{img src="images/php-ec2-cluster.png" width="728" height="384"}} ---- {{#x|☺}} A {{#ci|Slow}} MySQL Query {{#kw|select}} {{#kw|sleep}}(1) ---- {{#x|♡}} Amazon Linux AMI {{#ci|2011.09}} {{#x|♡}} nginx {{#ci|1.0.14}} {{#x|♡}} php-fpm {{#ci|5.3.10}} ---- {{img src="images/php-arch.png" width="739" height="397"}} ---- {{img src="images/slow-cs1.png" width="640" height="500"}} ---- {{img src="images/slow-idle1.png" width="640" height="500"}} ---- {{img src="images/slow-free1.png" width="640" height="500"}} ---- {{#x|☺}} A {{#ci|Fast}} MySQL Query with a {{#x|Small}} Resultset {{#kw|select}} * {{#kw|from}} world.City {{#kw|order by}} ID {{#kw|limit}} 1 ---- {{img src="images/small-rps1.png" width="640" height="500"}} ---- {{img src="images/small-cs1.png" width="640" height="500"}} ---- {{#x|☺}} A {{#ci|Fast}} MySQL Query with a {{#x|Big}} Resultset (100 KBytes) {{#kw|select}} * {{#kw|from}} world.City {{#kw|order by}} ID {{#kw|limit}} 1000 ---- {{img src="images/big-rps1.png" width="640" height="500"}} ---- {{img src="images/big-cs1.png" width="640" height="500"}} ---- {{img src="images/serviceization.png" width="664" height="563"}} ---- {{#x|☺}} We integrated {{#ci|libdrizzle}} directly into {{#x|Nginx}}! {{http://wiki.nginx.org/HttpDrizzleModule}} ---- {{img src="images/libdrizzle.png" width="736" height="463"}} ---- {{img src="images/set-libdrizzle-events.png" width="992" height="261"}} ---- {{img src="images/upstream.png" width="901" height="297"}} ---- {{img src="images/rds.png" width="629" height="401"}} ---- {{img src="images/pool.png" width="669" height="285"}} ---- {{#x|☺}} Let's just mud with {{#ci|nginx.conf}}, the Nginx {{#x|configuration file}} ---- {{#kw|upstream}} {{#x|my_mysql_backend}} { {{#kw|drizzle_server}} {{#x|127.0.0.1:3306}} dbname={{#x|test}} password={{#x|some_pass}} user={{#x|monty}} protocol={{#x|mysql}}; {{#cm|# a connection pool that can cache up to}} {{#cm|# 200 mysql TCP connections}} {{#kw|drizzle_keepalive}} max={{#x|200}} overflow={{#x|reject}}; } ---- {{#kw|location}} ~ {{#x|'^/cat/(.*)'}} { {{#kw|set}} {{#v|$name}} {{#v|$1}}; {{#kw|set_quote_sql_str}} {{#v|$quoted_name}} {{#v|$name}}; {{#kw|drizzle_query}} {{#x|"select *}} {{#x|from cats}} {{#x|where name=}}{{#v|$quoted_name}}{{#x|"}}; {{#kw|drizzle_pass}} my_mysql_backend; {{#kw|rds_json}} on; } ---- {{#v|$}} curl 'http://localhost/cat/Jerry' {{#x|[{"name":"Jerry","age":1}]}} ---- {{#x|☺}} The {{#ci|dynamic}} {{#x|SQL}} Query for This Request {{#kw|select}} * {{#kw|from}} cats {{#kw|where}} name={{#c|'Jerry'}} ---- {{img src="images/drizzle-ec2-cluster.png" width="589" height="384"}} ---- {{#x|☺}} The {{#ci|Slow}} MySQL Query again! {{#kw|select}} {{#kw|sleep}}(1) ---- {{img src="images/slow-cs2.png" width="640" height="500"}} ---- {{img src="images/slow-idle2.png" width="640" height="500"}} ---- {{img src="images/slow-free2.png" width="640" height="500"}} ---- {{#x|☺}} The {{#ci|Fast}} MySQL Query with a {{#x|Small}} Resultset Again! {{#kw|select}} * {{#kw|from}} world.City {{#kw|order by}} ID {{#kw|limit}} 1 ---- {{img src="images/small-rps2.png" width="640" height="500"}} ---- {{img src="images/small-cs2.png" width="640" height="500"}} ---- {{#x|☺}} The {{#ci|Fast}} MySQL Query with a {{#x|Big}} Resultset (100 KBytes) Again! {{#kw|select}} * {{#kw|from}} world.City {{#kw|order by}} ID {{#kw|limit}} 1000 ---- {{img src="images/big-rps2.png" width="640" height="500"}} ---- {{img src="images/big-cs2.png" width="640" height="500"}} ---- {{#x|☺}} We also embedded {{#ci|Lua}} and {{#ci|LuaJIT}} directly into {{#x|Nginx}}! {{img src="images/lua.jpg" width="118" height="118"}} {{http://wiki.nginx.org/HttpLuaModule}} ---- {{img src="images/luavm.png" width="675" height="285"}} ---- {{#x|☺}} Use the {{#ci|Lua}} language to access the {{#x|ngx_drizzle}} module! ---- {{img src="images/subrequest.png" width="1045" height="432"}} ---- {{#kw|location}} = /api { {{#kw|content_by_lua}} ' local rds_parser = require \"rds.parser\" local cjson = require \"cjson\" local resp = {{#x|ngx.location.capture}}(\"/cat/Jerry\") local data, err = rds_parser.parse(res.body) ngx.print(cjson.encode(data.resultset)) '; } ---- {{#v|$}} curl 'http://localhost/api' {{#x|[{"name":"Jerry","age":1}]}} ---- {{#x|☺}} The {{#ci|Fast}} MySQL Query with a {{#x|Small}} Resultset Revisited! {{#kw|select}} * {{#kw|from}} world.City {{#kw|order by}} ID {{#kw|limit}} 1 ---- {{img src="images/small-rps3.png" width="640" height="500"}} ---- {{img src="images/small-cs3.png" width="640" height="500"}} ---- {{#x|☺}} The {{#ci|Fast}} MySQL Query with a {{#x|Big}} Resultset (100 KBytes) Again! {{#kw|select}} * {{#kw|from}} world.City {{#kw|order by}} ID {{#kw|limit}} 1000 ---- {{img src="images/big-rps3.png" width="640" height="500"}} ---- {{img src="images/big-cs3.png" width="640" height="500"}} ---- {{#x|☺}} I just implemented the Lua {{#ci|cosocket API}}! {{http://wiki.nginx.org/HttpLuaModule#ngx.socket.tcp}} ---- {{#cm|✓}} a socket API based on Lua {{#ci|coroutines}} {{#cm|✓}} a socket API that is {{#ci|synchronous}} {{#cm|✓}} a socket API that is {{#ci|nonblocking}} ---- {{img src="images/cosocket.png" width="845" height="297"}} ---- {{#x|☺}} I wrote the {{#x|lua-resty-mysql}} library based on the {{#ci|cosocket}} API. {{http://github.com/agentzh/lua-resty-mysql}} ---- {{#x|☺}} It is a {{#ci|pure Lua}} MySQL driver written {{#x|from scratch}}! ---- {{#kw|local}} resty_mysql = {{#kw|require}} {{#x|"resty.mysql"}} {{#kw|local}} mysql = resty_mysql:new() {{#kw|local}} ok, err = mysql:connect{ host = {{#x|"127.0.0.1"}}, port = {{#x|3306}}, database = {{#x|"world"}}, user = {{#x|"monty"}}, password = {{#x|"some_pass"}} } ---- {{#kw|local}} query = {{#x|"select * from cats"}} {{#kw|local}} rows, err, errno, sqlstate = mysql:query(query) {{#kw|for}} i, row {{#kw|in}} ipairs(rows) {{#kw|do}} {{#cm|-- process the row table}} {{#kw|end}} ---- {{#cm|-- add the current MySQL connection}} {{#cm|-- into the per-worker connection pool,}} {{#cm|-- with total capacity of 1024 connections and}} {{#cm|-- 60 seconds maximal connection idle time}} {{#kw|local}} ok, err = mysql:set_keepalive({{#x|60000}}, {{#x|1024}}) ---- {{#x|☺}} The {{#ci|Slow}} MySQL Query Revisited! {{#kw|select}} {{#kw|sleep}}(1) ---- {{img src="images/slow-cs3.png" width="640" height="500"}} ---- {{img src="images/slow-idle3.png" width="640" height="500"}} ---- {{img src="images/slow-free3.png" width="640" height="500"}} ---- {{#x|☺}} The {{#ci|Fast}} MySQL Query with a {{#x|Small}} Resultset Revisited! {{#kw|select}} * {{#kw|from}} world.City {{#kw|order by}} ID {{#kw|limit}} 1 ---- {{img src="images/small-rps4.png" width="640" height="500"}} ---- {{img src="images/small-cs4.png" width="640" height="500"}} ---- {{#x|☺}} The {{#ci|Fast}} MySQL Query with a {{#x|Big}} Resultset (100 KBytes) Revisited! {{#kw|select}} * {{#kw|from}} world.City {{#kw|order by}} ID {{#kw|limit}} 1000 ---- {{img src="images/big-rps4.png" width="640" height="500"}} ---- {{img src="images/big-cs4.png" width="640" height="500"}} ---- {{#x|☺}} How about {{#ci|comparing}} with the {{#x|NodeJS}} world? ---- {{#x|♡}} node {{#ci|0.6.14}} {{#x|♡}} node mysql {{#ci|0.9.5}} {{#x|♡}} node generic pool {{#ci|1.0.9}} ---- {{img src="images/small-rps5.png" width="640" height="500"}} ---- {{img src="images/small-cs5.png" width="640" height="500"}} ---- {{img src="images/big-rps5.png" width="640" height="500"}} ---- {{img src="images/big-cs5.png" width="640" height="500"}} ---- {{#x|☺}} {{#ci|Caching}} responses with {{#i|ngx_srcache}} + {{#i|ngx_memc}} {{http://wiki.nginx.org/HttpSRCacheModule}} {{http://wiki.nginx.org/HttpMemcModule}} ---- {{img src="images/srcache.png" width="559" height="597"}} ---- {{#cm|# configure the cache storage location}} {{#kw|location}} /memc { {{#kw|internal}}; {{#kw|set}} {{#v|$memc_key}} {{#v|$query_string}}; {{#kw|set}} {{#v|$memc_exptime}} 300; {{#kw|memc_pass}} 127.0.0.1:11211; } ---- {{#kw|location}} = /api { {{#kw|set}} {{#v|$key}} {{#x|"$uri?$args"}}; {{#kw|srcache_fetch}} {{#x|GET /memc}} {{#v|$key}}; {{#kw|srcache_store}} {{#x|PUT /memc}} {{#v|$key}}; {{#cm|# drizzle_pass/fastcgi_pass/content_by_lua/...}} } ---- {{img src="images/srcache-ec2-cluster.png" width="1001" height="384"}} ---- {{img src="images/small-rps6.png" width="640" height="500"}} ---- {{img src="images/big-rps6.png" width="640" height="500"}} ---- {{#x|☺}} Find the {{#ci|source}} for all the {{#x|benchmarks}} given here: {{http://github.com/agentzh/mysql-driver-benchmark}} ---- ☺ {{#ci|Any questions}}? ☺ {{http://openresty.org}} {{https://groups.google.com/group/openresty}}