Commit c5f1b3224fbde8e561e240f85860dc15c88f890c | zig

author	Andrew Kelley <andrew@ziglang.org> 2022-08-31 13:15:42-07:00
committer	Andrew Kelley <andrew@ziglang.org> 2022-08-31 13:15:42-07:00
log	c5f1b3224fbde8e561e240f85860dc15c88f890c
tree	f528db7782aa89970d504f357752f7d24ade7cff
parent	9a0cb34c734d83d42c70e03e4cc292860af0a17b

CI: update_download_page: change docs deployment

The original impetus for making a change here was a typo in --add-header
causing the script to fail. However, upon inspection, I was alarmed that
we were making a --recursive upload to the *root directory* of
ziglang.org. This could result in garbage files being uploaded to the
website, or important files being overwritten. As I addressed this concern,
I decided to take on file compression as well.

Removed compression prior to sending to S3. I am vetoing pre-compressing
objects for the following reasons:
 * It prevents clients from working which do not support gzip encoding.
 * It breaks a premise that objects on S3 are stored 1-to-1 with what is
   on disk.
 * It prevents Cloudflare from using a more efficient encoding, such as
   brotli, which they have started doing recently.

These systems such as Cloudflare or Fastly already do compression on
the fly, and we should interop with these systems instead of fighting them.

Cloudfront has an arbitrary limit of 9.5 MiB for auto-compression. I looked
and did not see a way to increase this limit. The data.js file is currently
16 MiB. In order to fix this problem, we need to do one of the following things:

 * Reduce the size of data.js to less than 9.5 MiB.
 * Figure out how to adjust the Cloudfront settings to increase the max size
   for auto-compressed objects.
 * Migrate to Fastly. Fastly appears to not have this limitation. Note
   that we already plan to migrate to Fastly for the website.

1 files changed, 32 insertions(+), 21 deletions(-)

ci
- srht
  - update_download_page

ci/srht/update_download_page+32-21

...	...	@@ -101,32 +101,43 @@ CIDIR="$(pwd)"
101	101
102	102	cd "$HOME"
103	103
104		# Upload new stdlib autodocs
105		mkdir -p docs_to_upload/documentation/master/std/
106		gzip -c -9 "$ZIGDIR/docs/std/index.html" > docs_to_upload/documentation/master/std/index.html
107		gzip -c -9 "$ZIGDIR/docs/std/data.js" > docs_to_upload/documentation/master/std/data.js
108		gzip -c -9 "$ZIGDIR/docs/std/main.js" > docs_to_upload/documentation/master/std/main.js
109		gzip -c -9 "$LANGREF" > docs_to_upload/documentation/master/index.html
110		$S3CMD put -P --no-mime-magic --recursive --add-header="Content-Encoding:gzip" --add-header="Cache-Control: max-age=0, must-revalidate" "docs_to_upload/" s3://ziglang.org/
111
112		mkdir -p docs_src_to_upload/documentation/master/std/
113		cp -r "$ZIGDIR/docs/std/src" docs_src_to_upload/documentation/master/std/
114		$S3CMD put -P --no-mime-magic --recursive --add-header:"Content-Type:text/html" --add-header="Cache-Control: max-age=0, must-revalidate" "docs_src_to_upload/" s3://ziglang.org/
115
116		## Copy without compression:
117		# mkdir -p docs_to_upload/documentation/master/std/
118		# cp "$ZIGDIR/docs/std/index.html" docs_to_upload/documentation/master/std/index.html
119		# cp "$ZIGDIR/docs/std/data.js" docs_to_upload/documentation/master/std/data.js
120		# cp "$ZIGDIR/docs/std/main.js" docs_to_upload/documentation/master/std/main.js
121		# cp "$LANGREF" docs_to_upload/documentation/master/index.html
122		# $S3CMD put -P --no-mime-magic --recursive --add-header="Cache-Control: max-age=0, must-revalidate" "docs_to_upload/" s3://ziglang.org/
	104	# Update autodocs and langref directly to S3 in order to prevent the
	105	# www.ziglang.org git repo from growing too big.
	106
	107	# Please do not edit this script to pre-compress the artifacts before they hit
	108	# S3. This prevents the website from working on browsers that do not support gzip
	109	# encoding. Cloudfront will automatically compress files if they are less than
	110	# 9.5 MiB, and the client advertises itself as capable of decompressing.
	111	# The data.js file is currently 16 MiB. In order to fix this problem, we need to do
	112	# one of the following things:
	113	# * Reduce the size of data.js to less than 9.5 MiB.
	114	# * Figure out how to adjust the Cloudfront settings to increase the max size for
	115	# auto-compressed objects.
	116	# * Migrate to fastly.
	117	$S3CMD put -P --no-mime-magic \
	118	--add-header="Cache-Control: max-age=0, must-revalidate" \
	119	"$LANGREF" s3://ziglang.org/documentation/master/index.html
	120
	121	$S3CMD put -P --no-mime-magic \
	122	--add-header="Cache-Control: max-age=0, must-revalidate" \
	123	"$ZIGDIR/docs/std/index.html" s3://ziglang.org/documentation/master/std/index.html
	124
	125	$S3CMD put -P --no-mime-magic \
	126	--add-header="Cache-Control: max-age=0, must-revalidate" \
	127	"$ZIGDIR/docs/std/main.js" s3://ziglang.org/documentation/master/std/main.js
	128
	129	$S3CMD put -P --no-mime-magic \
	130	--add-header="Cache-Control: max-age=0, must-revalidate" \
	131	"$ZIGDIR/docs/std/data.js" s3://ziglang.org/documentation/master/std/data.js
	132
	133	$S3CMD put -P --no-mime-magic \
	134	--add-header="cache-control: public, max-age=31536000, immutable" \
	135	"$HOME/$SRC_TARBALL" s3://ziglang.org/builds/
123	136
124	137	git clone --depth 1 git@github.com:ziglang/www.ziglang.org.git
125	138	cd www.ziglang.org
126	139	WWWDIR="$(pwd)"
127	140
128		$S3CMD put -P --no-mime-magic --add-header="cache-control: public, max-age=31536000, immutable" "$HOME/$SRC_TARBALL" s3://ziglang.org/builds/
129
130	141	cd "$WWWDIR"
131	142	cp "$CIDIR/out/index.json" data/releases.json
132	143	git add data/releases.json