A Tour of Perl Testing ---- A Tour of {{#x|Perl Testing}} Perl 测试之旅 ☺{{#author|agentzh@yahoo.cn}}☺ {{#author|章亦春 (agentzh)}} {{#date|2009.9}} ---- {{#ci|Why}} testing? ---- {{#x|☺}} When things have gone {{#ci|wrong}} in collaboration... {{img src="#" width="0" height="0"}} {{img src="images/driving-gone-wrong.jpg" width="500" height="333"}} ---- {{#v|agentzh}}: With {{#ci|my patch}} in r2545, Jifty's I18N should work for (at least) standalone servers. {{#v|obra}}: I'm seeing a whole bunch of {{#x|new test failures}} in the REST tests in the jifty core after your latest tests (Also, a {{#x|15% performance penalty}}) Are you seeing the same? {{#v|agentzh}}: No, I'm not. Because I'm using Windows where all the Jifty live tests are {{#ci|skipped}} by default. :( ---- {{#x|☺}} When my user file a ticket to {{#ci|report bugs}}... {{img src="#" width="0" height="0"}} {{img src="images/Report.Bug.jpg" width="250" height="203"}} ---- As seen in my UML::Class::Simple module's {{#ci|RT ticket queue}}: ---- {{#v|David Favor}}: UML::Class::Simple-0.10 {{#ci|fails}} on {{#x|latest}} perl-5.10.0-34065 stable patch: ---- PERL_DL_NONLAZY=1 /pkgs/perl-5.10.0-34065/bin/perl \"-MExtUtils::Command::MM\" \"-e\" \"test_harness(0, 'inc', 'blib/lib', 'blib/arch')\" t/basic.t t/classes-from-runtime.t t/pod-coverage.t t/pod.t t/umlclass.t t/basic.t...................invalid width and height at t/basic.t line 68 Renderer type: \"gif\" not recognized. Use one of: canon cmap cmapx cmapx_np dia dot fig gtk hpgl imap imap_np ismap mif mp pcl pdf pic plain plain-ext png ps ps2 svg svgz vml vmlz at t/basic.t line 137 # Failed test 'binary GIF data returned' # at t/basic.t line 138. # Looks like you failed 1 test of 36. Dubious, test returned 1 (wstat 256, 0x100) Failed 1/36 subtests ---- {{#v|agentzh}}: Hmm, the test failure was due to the {{#ci|lack}} of gif handler support in your graphviz installation. UML::Class::Simple {{#x|should really skip}} this GIF test in basic.t if no gif handler is found :) ---- {{#x|☺}} Test suite failures on {{#ci|various}} platforms could also be reported automatically by {{#x|bots}}! (CPAN Testers)++ ---- {{img src="#" width="0" height="0"}} {{img src="images/cpan-tester-page1.png" width="935" height="728"}} ---- {{img src="#" width="0" height="0"}} {{img src="images/cpan-tester-page2.png" width="935" height="728"}} ---- {{#x|☺}} How many {{#ci|code branches}} and {{#ci|conditionals}} we have never run yet? {{img src="#" width="0" height="0"}} {{img src="images/branches.jpg" width="700" height="525"}} ---- ➥ Run your {{#ci|test suite}} with Devel::Cover {{img src="#" width="0" height="0"}} {{img src="images/cover-summary.png" width="1037" height="444"}} ---- {{#x|☺}} What are they? ---- {{img src="#" width="0" height="0"}} {{img src="images/cover-detail-1.png" width="1037" height="676"}} ---- {{img src="#" width="0" height="0"}} {{img src="images/cover-detail-2.png" width="1037" height="676"}} ---- {{img src="#" width="0" height="0"}} {{img src="images/cover-branch.png" width="1037" height="676"}} ---- {{#x|☺}} {{#ci|Poor}}s' automated testing {{image src="images/poor.gif" width="500" height="400"}} ---- {{#cm|# preparing expected outputs}} {{#v|$}} ./my-script.pl < input.txt > output.txt {{#v|$}} less output.txt {{#cm|# verity the outputs}} {{#v|$}} mv output.txt {{#ci|output.txt.expected}} {{#cm|# hacking on my-script.pl some...}} {{#v|$}} ./my-script input.txt > output.txt {{#v|$}} {{#x|diff}} -udT {{#ci|output.txt.expected}} output.txt ---- {{#x|☺}} Testing is usually about {{#ci|comparing}} outputs with a set of common inputs. {{img src="#" width="0" height="0"}} {{img src="images/Moon_Earth_Comparison.png" width="650" height="460"}} ---- {{#cm|# test.t}} {{#kw|use}} Test::More tests => 3; is 1 + 1, 2, '1 + 1 == 2'; is 1 - 1, 0, '1 - 1 == 0'; ok defined 1, '1 is defined'; ---- {{#v|$}} {{#ci|perl}} test.t 1..3 ok 1 - 1 + 1 == 2 ok 2 - 1 - 1 == 0 ok 3 - 1 is defined ---- {{#x|☺}} Standard {{#ci|TAP}} output could be summarized by {{#x|prove}}. ---- {{#v|$}} {{#ci|prove}} test.t test....ok All tests successful. Files=1, Tests=3, 0 wallclock secs ( 0.04 cusr + 0.00 csys = 0.04 CPU) ---- {{#x|☺}} Let's {{#ci|plant}} a \"bug\" intentionally into our tests. {{image src="images/plant-bug.jpg" width="400" height="500"}} ---- {{#cm|# test.t}} {{#kw|use}} Test::More tests => 3; is 1 + 1, 2, '1 + 1 == 2'; is 1 - 1, {{#ci|1}}, '1 - 1 == 0'; ok defined 1, '1 is defined'; ---- {{#v|$}} {{#ci|perl}} test.t 1..3 ok 1 - 1 + 1 == 2 not ok 2 - 1 - 1 == 0 # Failed test {{#x|'1 - 1 == 0'}} # at test.pl line 5. # {{#x|got: '0'}} # {{#x|expected: '1'}} ok 3 - 1 is defined # Looks like you failed 1 test of 3. ---- {{#v|$}} {{#ci|prove}} test.t test.... # Failed test {{#x|'1 - 1 == 0'}} # at test.pl line 5. # {{#x|got: '0'}} # {{#x|expected: '1'}} # Looks like you failed 1 test of 3. test....dubious Test returned status 1 (wstat 256, 0x100) DIED. FAILED test 2 Failed 1/3 tests, 66.67% okay Failed Test Stat Wstat Total Fail List of Failed {{#i|-----------------------------------------------------------------------------------}} test.pl 1 256 3 1 2 Failed 1/1 test scripts. 1/3 subtests failed. Files=1, Tests=3, 0 wallclock secs ( 0.01 cusr + 0.00 csys = 0.01 CPU) Failed 1/1 test programs. 1/3 subtests failed. ---- {{#x|☺}} A few tests from the Perl 5 test suite ➥ Test {{#x|Perl 5}} with {{#x|Perl 5}} {{img src="#" width="0" height="0"}} {{img src="images/recursion-snake.jpeg" width="420" height="340"}} ---- {{#cm|# perl-current/t/op/arith.t}} print {{#x|"1..145\\n"}}; ... {{#kw|sub}} tryeq ($$$) { {{#kw|if}} ({{#v|$$_[1]}} == {{#v|$$_[2]}}) { print {{#x|"ok $$_[0]\\n"}}; } {{#kw|else}} { print {{#x|"not ok $$_[0] # $$_[1] != $$_[2]\\n"}}; } } ... {{#kw|my}} {{#v|$$T}} = 1; tryeq {{#v|$$T}}++, 13 % 4, 1; tryeq {{#v|$$T}}++, -13 % 4, 3; ... ---- {{#x|☺}} The same is also true for the {{#x|Perl 6}} test suite ➥ Test {{#x|Perl 6}} with {{#x|Perl 6}} {{img src="#" width="0" height="0"}} {{img src="images/recursion-dragon.png" width="180" height="191"}} ---- {{#cm|# rakudo/t/01-sanity/07-for.t}} {{#kw|use}} {{#kw|v6}}; {{#kw|say}} \"1..9\"; {{#kw|my}} {{#v|@array}} = <a b c>; {{#kw|my}} {{#v|$i}} = 0; {{#kw|for}} {{#v|@array}} -> {{#v|$item}} { {{#kw|if}} {{#v|@array[$i]}} {{#kw|eq}} {{#v|$item}} { {{#kw|say}} \"ok \", {{#v|$i}} + 1 } {{#kw|else}} { {{#kw|say}} \"not ok \", {{#v|$i}} + 1 } {{#v|$i}}++; } ... ---- {{#cm|# rakudo/t/spec/S32-str/ucfirst.t}} {{#kw|use v6}}; {{#kw|use}} Test; plan 5; {{#cm|# L<S32::Str/Str/ucfirst> }} is ucfirst({{#x|"hello world"}}), {{#x|"Hello world"}}, \"simple\"; ... ---- {{#x|☺}} Test::Base ➥ {{#ci|Data-driven}} test framework in Perl ---- use Test::Base; plan tests => blocks() * 1; run { my $block = shift; my $name = $block->name; my $val = eval($block->expr); if (defined $block->test_eq) { is $val, $block->test_eq, \"$name - test_eq\"; } elsif (defined $block->test_defined) { ok defined $val, \"$name - test_defined\"; } }; __DATA__ {{#x|=== TEST 1: 1 + 1 == 2}} {{#x|--- expr}}: 1 + 1 {{#x|--- test_eq}}: 2 ---- {{#x|=== TEST 2: 1 - 1 == 0}} {{#x|--- expr}}: 1 - 1 {{#x|--- test_eq}}: 0 {{#x|=== TEST 3: 1 is defined}} {{#x|--- expr}}: 1 {{#x|--- test_defined}} ---- {{#v|$}} {{#ci|prove}} test-base.t test-base....ok All tests successful. Files=1, Tests=3, 0 wallclock secs ( 0.03 cusr + 0.00 csys = 0.03 CPU) ---- {{#x|☺}} Let's see some test cases from the {{#ci|GNU make}} official test suite (using PPI to generate tests based on Test::Base from the original Perl 4 code) ---- use t::Gmake; plan tests => 3 * blocks; run_tests; __DATA__ {{#x|=== TEST 1: rules with no command}} {{#x|--- source}} a: b b: c c:; echo 'hello!' {{#x|--- stdout}} echo 'hello!' hello! {{#x|--- stderr}} {{#x|--- error_code}} 0 ---- {{#x|=== TEST 2: command at beginning}} {{#x|--- source}} a: b b: c c:; echo 'hello!' {{#x|--- stdout}} {{#x|--- stderr_like}} .*?commands commence before first target.*? {{#x|--- error_code}}: 2 ---- {{#x|☺}} Generate a {{#ci|random}} test suite from {{#x|state machines}} {{img src="#" width="0" height="0"}} {{img src="images/add.png" width="429" height="264"}} ---- {{#x|☺}} State machines {{#ci|driving}} the tests of an {{#ci|IA-32}} dissambler in my Salent project. ---- {{img src="#" width="0" height="0"}} {{img src="images/mini_state_mac.png" width="322" height="361"}} ---- {{img src="#" width="0" height="0"}} {{img src="images/state_mac.png" width="865" height="828"}} ---- {{#x|☺}} The state machine was constructed {{#ci|automatically}} from Intel's instruction table in its {{#x|manual}}. {{http://svn.berlios.de/svnroot/repos/salent/idu/encoding.txt}} ---- {{#x|☺}} Generate pretty {{#ci|HTML reports}} from TAP ouptuts {{img src="#" width="0" height="0"}} {{img src="images/01smoke.png" width="620" height="382"}} ---- {{img src="#" width="0" height="0"}} {{img src="images/02smoke.png" width="722" height="245"}} ---- {{#x|☺}} Test {{#ci|web services}} using Test::Base ---- use t::OpenResty; plan tests => 3 * blocks() - 3 * 2; run_tests; {{#x|__DATA__}} {{#x|=== TEST 1: Login w/o password}} {{#x|--- request}} GET /=/login/$TestAccount.Admin {{#x|--- response}} {\"success\":0,\"error\":\"$TestAccount.Admin is not anonymous.\"} {{#x|=== TEST 2: Delete existing models (w/o login)}} {{#x|--- request}} DELETE /=/model.js {{#x|--- response}} {\"success\":0,\"error\":\"Login required.\"} ---- {{#x|☺}} Utilize {{#ci|multiple}} cores while running {{#ci|big}} test suites. {{#v|$}} prove {{#x|-j4}} -r t ---- ☺ {{#ci|Any questions}}? ☺ ----