authorgravatar for andrew@ziglang.orgAndrew Kelley <andrew@ziglang.org> 2026-08-20 17:33:16-07:00
committergravatar for andrew@ziglang.orgAndrew Kelley <andrew@ziglang.org> 2026-08-20 17:35:12-07:00
log9296ec14e9efe901b397cf8560f75cacf867f170
tree8c34fd2046c87da2592b39454e833e0160900af0
parent6ddb6f5759456c319e4305e00bbb5a5a881001ae

langref: rewrite Source Encoding section

1 files changed, 17 insertions(+), 36 deletions(-)

doc/langref.html.in+17-36
...@@ -7274,46 +7274,27 @@ fn readU32Be() u32 {}...@@ -7274,46 +7274,27 @@ fn readU32Be() u32 {}
7274 </ul>7274 </ul>
7275 {#header_close#}7275 {#header_close#}
7276 {#header_close#}7276 {#header_close#}
7277
7277 {#header_open|Source Encoding#}7278 {#header_open|Source Encoding#}
7278 <p>Zig source code is encoded in UTF-8. An invalid UTF-8 byte sequence results in a compile error.</p>7279 <p>Zig source code is UTF-8 encoded. Invalid UTF-8 byte sequences are not allowed anywhere.</p>
7279 <p>Throughout all zig source code (including in comments), some code points are never allowed:</p>7280 <p>Some code points are never allowed, even in {#link|Comments#}:</p>
7280 <ul>7281 <ul>
7281 <li>Ascii control characters, except for U+000a (LF), U+000d (CR), and U+0009 (HT): U+0000 - U+0008, U+000b - U+000c, U+000e - U+0001f, U+007f.</li>7282 <li>ASCII control characters, except for U+000a (LF): U+0000...U+0009, U+000b...U+0001f, U+007f.</li>
7282 <li>Non-Ascii Unicode line endings: U+0085 (NEL), U+2028 (LS), U+2029 (PS).</li>7283 <li>Non-ASCII Unicode line endings: U+0085 (NEL), U+2028 (LS), U+2029 (PS).</li>
7284 <li>Byte order marks: U+FEFF (BOM).</li>
7283 </ul>7285 </ul>
7284 <p>7286 <p>
7285 LF (byte value 0x0a, code point U+000a, {#syntax#}'\n'{#endsyntax#}) is the line terminator in Zig source code.7287 LF (byte value 0x0a, code point U+000a, {#syntax#}'\n'{#endsyntax#}) is
7286 This byte value terminates every line of zig source code except the last line of the file.7288 the line terminator in Zig source code. This byte value terminates every
7287 It is recommended that non-empty source files end with an empty line, which means the last byte would be 0x0a (LF).7289 line of Zig source code, including last line of the file.
7288 </p>7290 </p>
7289 <p>7291 <p>These conservative rules mean that third party tools reading
7290 Each LF may be immediately preceded by a single CR (byte value 0x0d, code point U+000d, {#syntax#}'\r'{#endsyntax#})7292 already-validated Zig source code may make simplifying assumptions, such
7291 to form a Windows style line ending, but this is discouraged. Note that in multiline strings, CRLF sequences will7293 as naively separating lines based on {#syntax#}'\n'{#endsyntax#}.
7292 be encoded as LF when compiled into a zig program.7294 However, tooling such as <kbd>zig fmt</kbd> provides convenience
7293 A CR in any other context is not allowed.7295 functionality to convert invalid source encodings to valid source
7294 </p>7296 encodings, for instance by stripping byte order marks and carriage
7295 <p>7297 returns.</p>
7296 HT hard tabs (byte value 0x09, code point U+0009, {#syntax#}'\t'{#endsyntax#}) are interchangeable with
7297 SP spaces (byte value 0x20, code point U+0020, {#syntax#}' '{#endsyntax#}) as a token separator,
7298 but use of hard tabs is discouraged. See {#link|Grammar#}.
7299 </p>
7300 <p>
7301 For compatibility with other tools, the compiler ignores a UTF-8-encoded byte order mark (U+FEFF)
7302 if it is the first Unicode code point in the source text. A byte order mark is not allowed anywhere else in the source.
7303 </p>
7304 <p>
7305 Note that running <kbd>zig fmt</kbd> on a source file will implement all recommendations mentioned here.
7306 </p>
7307 <p>
7308 Note that a tool reading Zig source code can make assumptions if the source code is assumed to be correct Zig code.
7309 For example, when identifying the ends of lines, a tool can use a naive search such as <code>/\n/</code>,
7310 or an <a href="https://msdn.microsoft.com/en-us/library/dd409797.aspx">advanced</a>
7311 search such as <code>/\r\n?|[\n\u0085\u2028\u2029]/</code>, and in either case line endings will be correctly identified.
7312 For another example, when identifying the whitespace before the first token on a line,
7313 a tool can either use a naive search such as <code>/[ \t]/</code>,
7314 or an <a href="https://tc39.es/ecma262/#sec-characterclassescape">advanced</a> search such as <code>/\s/</code>,
7315 and in either case whitespace will be correctly identified.
7316 </p>
7317 {#header_close#}7298 {#header_close#}
73187299
7319 {#header_open|Keyword Reference#}7300 {#header_open|Keyword Reference#}