authorgravatar for mrpaul@aestheticwisdom.comMr. Paul <mrpaul@aestheticwisdom.com> 2021-09-29 11:31:41+07:00
committergravatar for andrew@ziglang.orgAndrew Kelley <andrew@ziglang.org> 2021-10-04 15:35:01-04:00
log65e4926c5b6ca936b460afd1d1e43474b2bc320d
tree176e9cb783ca0ffa5343419a8cc3087deae07d49
parentd4ebfa87633aa632a290dfffa38712468e5512db

langref: Explain Zig Test

Updates the Language Reference sections: Comments, Values, and Zig Test. Zig Test section moved down with the goal "make sure it can be read top to bottom sensibly" in mind (issue #1524). Comments and Values section examples changed test declarations to a main function and expect statement to print statements. A print statement was added to the "String Literals and Unicode Code Point" section's example to demonstrate the "u" format specifier. Zig Test Section: * Addresses the question: "How does the syntax work?". * Partially answers the question: "What can I do with the zig test tool?" but should be sufficient to understand the examples in all of this document. * Addresses the question: "How does a top-level test block differ from a function definition?" * Provides a example to run multiple test. Lacks clear definitions of containers, top-level, order independence, lazy analysis, resolve, reference. GitHub Issues: #8221, #8234

1 files changed, 320 insertions(+), 110 deletions(-)

doc/langref.html.in+320-110
...@@ -456,93 +456,17 @@ pub fn main() void {...@@ -456,93 +456,17 @@ pub fn main() void {
456 </p>456 </p>
457 {#see_also|Values|@import|Errors|Root Source File|Source Encoding#}457 {#see_also|Values|@import|Errors|Root Source File|Source Encoding#}
458 {#header_close#}458 {#header_close#}
459 {#header_open|Zig Test#}
460 <p>
461 <kbd>zig test</kbd> is a tool that can be used to quickly build and run Zig code
462 to make sure behavior meets expectations. {#syntax#}@import("builtin").is_test{#endsyntax#}
463 is available for code to detect whether the current build is a test build.
464 </p>
465 {#code_begin|test|detect_test#}
466const std = @import("std");
467const builtin = @import("builtin");
468const expect = std.testing.expect;
469
470test "builtin.is_test" {
471 try expect(builtin.is_test);
472}
473 {#code_end#}
474 <p>
475 Zig has lazy top level declaration analysis, which means that if a function is not called,
476 or otherwise used, it is not analyzed. This means that there may be an undiscovered
477 compile error in a function because it is never called.
478 </p>
479 {#code_begin|test|unused_fn#}
480fn unused() i32 {
481 return "wrong return type";
482}
483test "unused function" { }
484 {#code_end#}
485 <p>
486 Note that, while in {#link|Debug#} and {#link|ReleaseSafe#} modes, {#link|unreachable#} emits a
487 call to {#link|@panic#}, in {#link|ReleaseFast#} and {#link|ReleaseSmall#} modes, it is really
488 undefined behavior. The implementation of {#syntax#}std.debug.assert{#endsyntax#} is as
489 simple as:
490 </p>
491 {#code_begin|syntax|assert#}
492pub fn assert(ok: bool) void {
493 if (!ok) unreachable;
494}
495 {#code_end#}
496 <p>
497 This means that when testing in ReleaseFast or ReleaseSmall mode, {#syntax#}assert{#endsyntax#}
498 is not sufficient to check the result of a computation:
499 </p>
500 {#code_begin|syntax|assert_release_fast_mode#}
501const std = @import("std");
502const assert = std.debug.assert;
503
504test "assert in release fast mode" {
505 assert(false);
506}
507 {#code_end#}
508 <p>
509 When compiling this test in {#link|ReleaseFast#} mode, it invokes unchecked
510 {#link|Undefined Behavior#}. Since that could do anything, this documentation
511 cannot show you the output.
512 </p>
513 <p>
514 Better practice for checking the output when testing is to use {#syntax#}std.testing.expect{#endsyntax#}:
515 </p>
516 {#code_begin|test_err|test "expect in release fast mode"... FAIL (TestUnexpectedResult)#}
517 {#code_release_fast#}
518const std = @import("std");
519const expect = std.testing.expect;
520
521test "expect in release fast mode" {
522 try expect(false);
523}
524 {#code_end#}
525 <p>See the rest of the {#syntax#}std.testing{#endsyntax#} namespace for more available functions.</p>
526 <p>
527 <kbd>zig test</kbd> has a few command line parameters which affect the compilation. See
528 <kbd>zig --help</kbd> for a full list. The most interesting one is <kbd>--test-filter [text]</kbd>.
529 This makes the test build only include tests whose name contains the supplied filter text.
530 Again, thanks to lazy analysis, this can allow you to narrow a build to only a few functions in
531 isolation.
532 </p>
533 {#header_close#}
534 {#header_open|Comments#}459 {#header_open|Comments#}
535 {#code_begin|test|comments#}460 {#code_begin|exe|comments#}
536const expect = @import("std").testing.expect;461const print = @import("std").debug.print;
537462
538test "comments" {463pub fn main() void {
539 // Comments in Zig start with "//" and end at the next LF byte (end of line).464 // Comments in Zig start with "//" and end at the next LF byte (end of line).
540 // The below line is a comment, and won't be executed.465 // The line below is a comment and won't be executed.
541466
542 //expect(false);467 //print("Hello?", .{});
543468
544 const x = true; // another comment469 print("Hello, world!\n", .{}); // another comment
545 try expect(x);
546}470}
547 {#code_end#}471 {#code_end#}
548 <p>472 <p>
...@@ -896,24 +820,25 @@ pub fn main() void {...@@ -896,24 +820,25 @@ pub fn main() void {
896 in recent versions of the Unicode specification (as of Unicode 13.0).820 in recent versions of the Unicode specification (as of Unicode 13.0).
897 In Zig, a Unicode code point literal corresponds to the Unicode definition of a code point.821 In Zig, a Unicode code point literal corresponds to the Unicode definition of a code point.
898 </p>822 </p>
899 {#code_begin|test|string_literals_test#}823 {#code_begin|exe|string_literals#}
900const expect = @import("std").testing.expect;824const print = @import("std").debug.print;
901const mem = @import("std").mem;825const mem = @import("std").mem; // will be used to compare bytes
902826
903test "string literals" {827pub fn main() void {
904 const bytes = "hello";828 const bytes = "hello";
905 try expect(@TypeOf(bytes) == *const [5:0]u8);829 print("{s}\n", .{@typeName(@TypeOf(bytes))}); // *const [5:0]u8
906 try expect(bytes.len == 5);830 print("{d}\n", .{bytes.len}); // 5
907 try expect(bytes[1] == 'e');831 print("{c}\n", .{bytes[1]}); // 'e'
908 try expect(bytes[5] == 0);832 print("{d}\n", .{bytes[5]}); // 0
909 try expect('e' == '\x65');833 print("{}\n", .{'e' == '\x65'}); // true
910 try expect('\u{1f4a9}' == 128169);834 print("{d}\n", .{'\u{1f4a9}'}); // 128169
911 try expect('💯' == 128175);835 print("{d}\n", .{'💯'}); // 128175
912 try expect(mem.eql(u8, "hello", "h\x65llo"));836 print("{}\n", .{mem.eql(u8, "hello", "h\x65llo")}); // true
913 try expect("\xff"[0] == 0xff); // non-UTF-8 strings are possible with \xNN notation.837 print("0x{x}\n", .{"\xff"[0]}); // non-UTF-8 strings are possible with \xNN notation.
914}838 print("{u}\n", .{'âš¡'});
915 {#code_end#}839}
916 {#see_also|Arrays|Zig Test|Source Encoding#}840 {#code_end#}
841 {#see_also|Arrays|Source Encoding#}
917 {#header_open|Escape Sequences#}842 {#header_open|Escape Sequences#}
918 <div class="table-wrapper">843 <div class="table-wrapper">
919 <table>844 <table>
...@@ -986,7 +911,7 @@ const hello_world_in_c =...@@ -986,7 +911,7 @@ const hello_world_in_c =
986 {#header_close#}911 {#header_close#}
987 {#header_open|Assignment#}912 {#header_open|Assignment#}
988 <p>Use the {#syntax#}const{#endsyntax#} keyword to assign a value to an identifier:</p>913 <p>Use the {#syntax#}const{#endsyntax#} keyword to assign a value to an identifier:</p>
989 {#code_begin|test_err|cannot assign to constant#}914 {#code_begin|exe_build_err|constant_identifier_cannot_change#}
990const x = 1234;915const x = 1234;
991916
992fn foo() void {917fn foo() void {
...@@ -997,26 +922,26 @@ fn foo() void {...@@ -997,26 +922,26 @@ fn foo() void {
997 y += 1;922 y += 1;
998}923}
999924
1000test "assignment" {925pub fn main() void {
1001 foo();926 foo();
1002}927}
1003 {#code_end#}928 {#code_end#}
1004 <p>{#syntax#}const{#endsyntax#} applies to all of the bytes that the identifier immediately addresses. {#link|Pointers#} have their own const-ness.</p>929 <p>{#syntax#}const{#endsyntax#} applies to all of the bytes that the identifier immediately addresses. {#link|Pointers#} have their own const-ness.</p>
1005 <p>If you need a variable that you can modify, use the {#syntax#}var{#endsyntax#} keyword:</p>930 <p>If you need a variable that you can modify, use the {#syntax#}var{#endsyntax#} keyword:</p>
1006 {#code_begin|test|var_test#}931 {#code_begin|exe|mutable_var#}
1007const expect = @import("std").testing.expect;932const print = @import("std").debug.print;
1008933
1009test "var" {934pub fn main() void {
1010 var y: i32 = 5678;935 var y: i32 = 5678;
1011936
1012 y += 1;937 y += 1;
1013938
1014 try expect(y == 5679);939 print("{d}", .{y});
1015}940}
1016 {#code_end#}941 {#code_end#}
1017 <p>Variables must be initialized:</p>942 <p>Variables must be initialized:</p>
1018 {#code_begin|test_err#}943 {#code_begin|exe_build_err|var_must_be_initialized#}
1019test "initialization" {944pub fn main() void {
1020 var x: i32;945 var x: i32;
1021946
1022 x = 1;947 x = 1;
...@@ -1024,13 +949,13 @@ test "initialization" {...@@ -1024,13 +949,13 @@ test "initialization" {
1024 {#code_end#}949 {#code_end#}
1025 {#header_open|undefined#}950 {#header_open|undefined#}
1026 <p>Use {#syntax#}undefined{#endsyntax#} to leave variables uninitialized:</p>951 <p>Use {#syntax#}undefined{#endsyntax#} to leave variables uninitialized:</p>
1027 {#code_begin|test|undefined_test#}952 {#code_begin|exe|assign_undefined#}
1028const expect = @import("std").testing.expect;953const print = @import("std").debug.print;
1029954
1030test "init with undefined" {955pub fn main() void {
1031 var x: i32 = undefined;956 var x: i32 = undefined;
1032 x = 1;957 x = 1;
1033 try expect(x == 1);958 print("{d}", .{x});
1034}959}
1035 {#code_end#}960 {#code_end#}
1036 <p>961 <p>
...@@ -1047,6 +972,291 @@ test "init with undefined" {...@@ -1047,6 +972,291 @@ test "init with undefined" {
1047 {#header_close#}972 {#header_close#}
1048 {#header_close#}973 {#header_close#}
1049 {#header_close#}974 {#header_close#}
975 {#header_open|Zig Test#}
976 <p>
977 Code written within one or more {#syntax#}test{#endsyntax#} declarations can be used to ensure behavior meets expectations:
978 </p>
979 {#code_begin|test|introducing_zig_test#}
980const std = @import("std");
981
982test "expect addOne adds one to 41" {
983
984 // The Standard Library contains useful functions to help create tests.
985 // `expect` is a function that verifies its argument is true.
986 // It will return an error if its argument is false to indicate a failure.
987 // `try` is used to return an error to the test runner to notify it that the test failed.
988 try std.testing.expect(addOne(41) == 42);
989}
990
991/// The function `addOne` adds one to the number given as its argument.
992fn addOne(number: i32) i32 {
993 return number + 1;
994}
995 {#code_end#}
996 <p>
997 The <code class="file">introducing_zig_test.zig</code> code sample tests the {#link|function|Functions#}
998 {#syntax#}addOne{#endsyntax#} to ensure that it returns {#syntax#}42{#endsyntax#} given the input
999 {#syntax#}41{#endsyntax#}. From this test's perspective, the {#syntax#}addOne{#endsyntax#} function is
1000 said to be <em>code under test</em>.
1001 </p>
1002 <p>
1003 <kbd>zig test</kbd> is a tool that creates and runs a test build. By default, it builds and runs an
1004 executable program using the <em>default test runner</em> provided by the {#link|Zig Standard Library#}
1005 as its main entry point. During the build, {#syntax#}test{#endsyntax#} declarations found while
1006 {#link|resolving|Root Source File#} the given Zig source file are included for the default test runner
1007 to run and report on.
1008 </p>
1009 <aside>
1010 This documentation discusses the features of the default test runner as provided by the Zig Standard Library.
1011 Its source code is located in <code class="file">lib/std/special/test_runner.zig</code>.
1012 </aside>
1013 <p>
1014 The shell output shown above displays two lines after the <kbd>zig test</kbd> command. These lines are
1015 printed to standard error by the default test runner:
1016 </p>
1017 <dl>
1018 <dt><samp>Test [1/1] test "expect addOne adds one to 41"...</samp></dt>
1019 <dd>Lines like this indicate which test, out of the total number of tests, is being run.
1020 In this case, <samp>[1/1]</samp> indicates that the first test, out of a total of
1021 one test, is being run. Note that, when the test runner program's standard error is output
1022 to the terminal, these lines are cleared when a test succeeds.
1023 </dd>
1024 <dt><samp>All 1 tests passed.</samp></dt>
1025 <dd>This line indicates the total number of tests that have passed.</dd>
1026 </dl>
1027 {#header_open|Test Declarations#}
1028 <p>
1029 Test declarations contain the {#link|keyword|Keyword Reference#} {#syntax#}test{#endsyntax#}, followed by an
1030 optional name written as a {#link|string literal|String Literals and Unicode Code Point Literals#}, followed
1031 by a {#link|block|blocks#} containing any valid Zig code that is allowed in a {#link|function|Functions#}.
1032 </p>
1033 <aside>
1034 By convention, non-named tests should only be used to {#link|make other tests run|Nested Container Tests#}.
1035 Non-named tests cannot be {#link|filtered|Skip Tests#}.
1036 </aside>
1037 <p>
1038 Test declarations are similar to {#link|Functions#}: they have a return type and a block of code. The implicit
1039 return type of {#syntax#}test{#endsyntax#} is the {#link|Error Union Type#} {#syntax#}anyerror!void{#endsyntax#},
1040 and it cannot be changed. When a Zig source file is not built using the <kbd>zig test</kbd> tool, the test
1041 declarations are omitted from the build.
1042 </p>
1043 <p>
1044 Test declarations can be written in the same file, where code under test is written, or in a separate Zig source file.
1045 Since test declarations are top-level declarations, they are order-independent and can
1046 be written before or after the code under test.
1047 </p>
1048 {#see_also|The Global Error Set|Grammar#}
1049 {#header_close#}
1050 {#header_open|Nested Container Tests#}
1051 <p>
1052 When the <kbd>zig test</kbd> tool is building a test runner, only resolved {#syntax#}test{#endsyntax#}
1053 declarations are included in the build. Initially, only the given Zig source file's top-level
1054 declarations are resolved. Unless nested containers are referenced from a top-level test declaration,
1055 nested container tests will not be resolved.
1056 </p>
1057 <p>
1058 The code sample below uses the {#syntax#}std.testing.refAllDecls(@This()){#endsyntax#} function call to
1059 reference all of the containers that are in the file including the imported Zig source file. The code
1060 sample also shows an alternative way to reference containers using the {#syntax#}_ = C;{#endsyntax#}
1061 syntax. This syntax tells the compiler to ignore the result of the expression on the right side of the
1062 assignment operator.
1063 </p>
1064 {#code_begin|test|testdecl_container_top_level#}
1065const std = @import("std");
1066const expect = std.testing.expect;
1067
1068// Imported source file tests will run when referenced from a top-level test declaration.
1069// The next line alone does not cause "introducing_zig_test.zig" tests to run.
1070const imported_file = @import("introducing_zig_test.zig");
1071
1072test {
1073 // To run nested container tests, either, call `refAllDecls` which will
1074 // reference all declarations located in the given argument.
1075 // `@This()` is a builtin function that returns the innermost container it is called from.
1076 // In this example, the innermost container is this file (implicitly a struct).
1077 std.testing.refAllDecls(@This());
1078
1079 // or, reference each container individually from a top-level test declaration.
1080 // The `_ = C;` syntax is a no-op reference to the identifier `C`.
1081 _ = S;
1082 _ = U;
1083 _ = @import("introducing_zig_test.zig");
1084}
1085
1086const S = struct {
1087 test "S demo test" {
1088 try expect(true);
1089 }
1090
1091 const SE = enum {
1092 V,
1093
1094 // This test won't run because its container (SE) is not referenced.
1095 test "This Test Won't Run" {
1096 try expect(false);
1097 }
1098 };
1099};
1100
1101const U = union { // U is referenced by the file's top-level test declaration
1102 s: US, // and US is referenced here; therefore, "U.Us demo test" will run
1103
1104 const US = struct {
1105 test "U.US demo test" {
1106 // This test is a top-level test declaration for the struct.
1107 // The struct is nested (declared) inside of a union.
1108 try expect(true);
1109 }
1110 };
1111
1112 test "U demo test" {
1113 try expect(true);
1114 }
1115};
1116 {#code_end#}
1117 {#header_close#}
1118 {#header_open|Test Failure#}
1119 <p>
1120 The default test runner checks for an {#link|error|Errors#} returned from a test.
1121 When a test returns an error, the test is considered a failure and its {#link|error return trace|Error Return Traces#}
1122 is output to standard error. The total number of failures will be reported after all tests have run.
1123 </p>
1124 {#code_begin|test_err#}
1125const std = @import("std");
1126
1127test "expect this to fail" {
1128 try std.testing.expect(false);
1129}
1130
1131test "expect this to succeed" {
1132 try std.testing.expect(true);
1133}
1134 {#code_end#}
1135 {#header_close#}
1136 {#header_open|Skip Tests#}
1137 <p>
1138 One way to skip tests is to filter them out by using the <kbd>zig test</kbd> command line parameter
1139 <kbd>--test-filter [text]</kbd>. This makes the test build only include tests whose name contains the
1140 supplied filter text. Note that non-named tests are run even when using the <kbd>--test-filter [text]</kbd>
1141 command line parameter.
1142 </p>
1143 <p>
1144 To programmatically skip a test, make a {#syntax#}test{#endsyntax#} return the error
1145 {#syntax#}error.SkipZigTest{#endsyntax#} and the default test runner will consider the test as being skipped.
1146 The total number of skipped tests will be reported after all tests have run.
1147 </p>
1148 {#code_begin|test#}
1149test "this will be skipped" {
1150 return error.SkipZigTest;
1151}
1152 {#code_end#}
1153 <p>
1154 The default test runner skips tests containing a {#link|suspend point|Async Functions#} while the
1155 test is running using the default, blocking IO mode.
1156 (The evented IO mode is enabled using the <kbd>--test-evented-io</kbd> command line parameter.)
1157 </p>
1158 {#code_begin|test|async_skip#}
1159const std = @import("std");
1160
1161test "async skip test" {
1162 var frame = async func();
1163 const result = await frame;
1164 try std.testing.expect(result == 1);
1165}
1166
1167fn func() i32 {
1168 suspend {
1169 resume @frame();
1170 }
1171 return 1;
1172}
1173 {#code_end#}
1174 <p>
1175 In the code sample above, the test would not be skipped in blocking IO mode if the {#syntax#}nosuspend{#endsyntax#}
1176 keyword was used (see {#link|Async and Await#}).
1177 </p>
1178 {#header_close#}
1179 {#header_open|Report Memory Leaks#}
1180 <p>
1181 When code allocates {#link|Memory#} using the {#link|Zig Standard Library#}'s testing allocator,
1182 {#syntax#}std.testing.allocator{#endsyntax#}, the default test runner will report any leaks that are
1183 found from using the testing allocator:
1184 </p>
1185 {#code_begin|test_err|1 tests leaked memory#}
1186const std = @import("std");
1187
1188test "detect leak" {
1189 var list = std.ArrayList(u21).init(std.testing.allocator);
1190 // missing `defer list.deinit();`
1191 try list.append('☔');
1192
1193 try std.testing.expect(list.items.len == 1);
1194}
1195 {#code_end#}
1196 {#see_also|defer|Memory#}
1197 {#header_close#}
1198 {#header_open|Detecting Test Build#}
1199 <p>
1200 Use the {#link|compile variable|Compile Variables#} {#syntax#}@import("builtin").is_test{#endsyntax#}
1201 to detect a test build:
1202 </p>
1203 {#code_begin|test|detect_test#}
1204const std = @import("std");
1205const builtin = @import("builtin");
1206const expect = std.testing.expect;
1207
1208test "builtin.is_test" {
1209 try expect(isATest());
1210}
1211
1212fn isATest() bool {
1213 return builtin.is_test;
1214}
1215 {#code_end#}
1216 {#header_close#}
1217 {#header_open|Test Output and Logging#}
1218 <p>
1219 The default test runner and the Zig Standard Library's testing namespace output messages to standard error.
1220 </p>
1221 {#header_close#}
1222 {#header_open|The Testing Namespace#}
1223 <p>
1224 The Zig Standard Library's <code>testing</code> namespace contains useful functions to help
1225 you create tests. In addition to the <code>expect</code> function, this document uses a couple of more functions
1226 as exemplified here:
1227 </p>
1228 {#code_begin|test|testing_functions#}
1229const std = @import("std");
1230
1231test "expectEqual demo" {
1232 const expected: i32 = 42;
1233 const actual = 42;
1234
1235 // The first argument to `expectEqual` is the known, expected, result.
1236 // The second argument is the result of some expression.
1237 // The actual's type is casted to the type of expected.
1238 try std.testing.expectEqual(expected, actual);
1239}
1240
1241test "expectError demo" {
1242 const expected_error = error.DemoError;
1243 const actual_error_union: anyerror!void = error.DemoError;
1244
1245 // `expectError` will fail when the actual error is different than
1246 // the expected error.
1247 try std.testing.expectError(expected_error, actual_error_union);
1248}
1249 {#code_end#}
1250 <p>The Zig Standard Library also contains functions to compare {#link|Slices#}, strings, and more. See the rest of the
1251 {#syntax#}std.testing{#endsyntax#} namespace in the {#link|Zig Standard Library#} for more available functions.</p>
1252 {#header_close#}
1253 {#header_open|Test Tool Documentation#}
1254 <p>
1255 <kbd>zig test</kbd> has a few command line parameters which affect the compilation.
1256 See <kbd>zig test --help</kbd> for a full list.
1257 </p>
1258 {#header_close#}
1259 {#header_close#}
10501260
1051 {#header_open|Variables#}1261 {#header_open|Variables#}
1052 <p>1262 <p>