authorgravatar for squeek502@hotmail.comRyan Liptak <squeek502@hotmail.com> 2023-10-03 01:19:38-07:00
committergravatar for squeek502@hotmail.comRyan Liptak <squeek502@hotmail.com> 2023-10-03 01:21:51-07:00
logcf3572a66bb956eff319ebcb123784322e2553de
treea3780d9e808a2853b629e7cace22232e658ba98e
parentda7ecfb2dec3869d3367fc730b677ae404ecac60

GeneralPurposeAllocator: Considerably improve worst case performance

Before this commit, GeneralPurposeAllocator could run into incredibly degraded performance in scenarios where the bucket count for a particular size class grew to be large. For example, if exactly `slot_count` allocations of a single size class were performed and then all of them were freed except one, then the bucket for those allocations would have to be kept around indefinitely. If that pattern of allocation were done over and over, then the bucket list for that size class could grow incredibly large. This allocation pattern has been seen in the wild: https://github.com/Vexu/arocc/issues/508#issuecomment-1738275688 In that case, the length of the bucket list for the `128` size class would grow to tens of thousands of buckets and cause Debug runtime to balloon to ~8 minutes whereas with the c_allocator the Debug runtime would be ~3 seconds. To address this, there are three different changes happening here: 1. std.Treap is used instead of a doubly linked list for the lists of buckets. This takes the time complexity of searchBucket [used in resize and free] from O(n) to O(log n), but increases the time complexity of insert from O(1) to O(log n) [before, all new buckets would get added to the head of the list]. Note: Any data structure with O(log n) or better search/insert/delete would also work for this use-case. 2. If the 'current' bucket for a size class is full, the list of buckets is never traversed and instead a new bucket is allocated. Previously, traversing the bucket list could only find a non-full bucket in specific circumstances, and only because of a separate optimization that is no longer needed (before, after any resize/free, the affected bucket would be moved to the head of the bucket list to allow searchBucket to perform better on average). Now, the current_bucket for each size class only changes when either (1) the current bucket is emptied/freed, or (2) a new bucket is allocated (due to the current bucket being full or null). Because each bucket's alloc_cursor only moves forward (i.e. slots within a bucket are never re-used), we can therefore always know that any bucket besides the current_bucket will be full, so traversing the list in the hopes of finding an existing non-full bucket is entirely pointless. 3. Size + alignment information for small allocations has been moved into the Bucket data instead of keeping it in a separate HashMap. This offers an improvement over the HashMap since whenever we need to get/modify the length/alignment of an allocation it's extremely likely we will already have calculated any bucket-related information necessary to get the data. The first change is the most relevant and accounts for most of the benefit here. Also note that the overall functionality of GeneralPurposeAllocator is unchanged. In the degraded `arocc` case, these changes bring Debug performance from ~8 minutes to ~20 seconds. Benchmark 1: test-master.bat Time (mean ± σ): 481.263 s ± 5.440 s [User: 479.159 s, System: 1.937 s] Range (min … max): 477.416 s … 485.109 s 2 runs Benchmark 2: test-optim-treap.bat Time (mean ± σ): 19.639 s ± 0.037 s [User: 18.183 s, System: 1.452 s] Range (min … max): 19.613 s … 19.665 s 2 runs Summary 'test-optim-treap.bat' ran 24.51 ± 0.28 times faster than 'test-master.bat' Note: Much of the time taken on Windows in this particular case is related to gathering stack traces. With `.stack_trace_frames = 0` the runtime goes down to 6.7 seconds, which is a little more than 2.5x slower compared to when the c_allocator is used. These changes may or mat not introduce a slight performance regression in the average case: Here's the standard library tests on Windows in Debug mode: Benchmark 1 (10 runs): std-tests-master.exe measurement mean ± σ min … max outliers delta wall_time 16.0s ± 30.8ms 15.9s … 16.1s 1 (10%) 0% peak_rss 42.8MB ± 8.24KB 42.8MB … 42.8MB 0 ( 0%) 0% Benchmark 2 (10 runs): std-tests-optim-treap.exe measurement mean ± σ min … max outliers delta wall_time 16.2s ± 37.6ms 16.1s … 16.3s 0 ( 0%) 💩+ 1.3% ± 0.2% peak_rss 42.8MB ± 5.18KB 42.8MB … 42.8MB 0 ( 0%) + 0.1% ± 0.0% And on Linux: Benchmark 1: ./test-master Time (mean ± σ): 16.091 s ± 0.088 s [User: 15.856 s, System: 0.453 s] Range (min … max): 15.870 s … 16.166 s 10 runs Benchmark 2: ./test-optim-treap Time (mean ± σ): 16.028 s ± 0.325 s [User: 15.755 s, System: 0.492 s] Range (min … max): 15.735 s … 16.709 s 10 runs Summary './test-optim-treap' ran 1.00 ± 0.02 times faster than './test-master'

1 files changed, 147 insertions(+), 133 deletions(-)

lib/std/heap/general_purpose_allocator.zig+147-133
......@@ -160,11 +160,12 @@ pub const Check = enum { ok, leak };
160160pub fn GeneralPurposeAllocator(comptime config: Config) type {
161161 return struct {
162162 backing_allocator: Allocator = std.heap.page_allocator,
163 buckets: [small_bucket_count]?*BucketHeader = [1]?*BucketHeader{null} ** small_bucket_count,
163 buckets: [small_bucket_count]Buckets = [1]Buckets{Buckets{}} ** small_bucket_count,
164 cur_buckets: [small_bucket_count]?*BucketHeader = [1]?*BucketHeader{null} ** small_bucket_count,
164165 large_allocations: LargeAllocTable = .{},
165 small_allocations: if (config.safety) SmallAllocTable else void = if (config.safety) .{} else {},
166 empty_buckets: if (config.retain_metadata) ?*BucketHeader else void =
167 if (config.retain_metadata) null else {},
166 empty_buckets: if (config.retain_metadata) Buckets else void =
167 if (config.retain_metadata) Buckets{} else {},
168 bucket_node_pool: std.heap.MemoryPool(Buckets.Node) = std.heap.MemoryPool(Buckets.Node).init(std.heap.page_allocator),
168169
169170 total_requested_bytes: @TypeOf(total_requested_bytes_init) = total_requested_bytes_init,
170171 requested_memory_limit: @TypeOf(requested_memory_limit_init) = requested_memory_limit_init,
......@@ -196,11 +197,14 @@ pub fn GeneralPurposeAllocator(comptime config: Config) type {
196197
197198 const small_bucket_count = math.log2(page_size);
198199 const largest_bucket_object_size = 1 << (small_bucket_count - 1);
200 const LargestSizeClassInt = std.math.IntFittingRange(0, largest_bucket_object_size);
199201
200 const SmallAlloc = struct {
201 requested_size: usize,
202 log2_ptr_align: u8,
203 };
202 const bucketCompare = struct {
203 fn compare(a: *BucketHeader, b: *BucketHeader) std.math.Order {
204 return std.math.order(@intFromPtr(a.page), @intFromPtr(b.page));
205 }
206 }.compare;
207 const Buckets = std.Treap(*BucketHeader, bucketCompare);
204208
205209 const LargeAlloc = struct {
206210 bytes: []u8,
......@@ -235,16 +239,17 @@ pub fn GeneralPurposeAllocator(comptime config: Config) type {
235239 }
236240 };
237241 const LargeAllocTable = std.AutoHashMapUnmanaged(usize, LargeAlloc);
238 const SmallAllocTable = std.AutoHashMapUnmanaged(usize, SmallAlloc);
239242
240243 // Bucket: In memory, in order:
241244 // * BucketHeader
242245 // * bucket_used_bits: [N]u8, // 1 bit for every slot; 1 byte for every 8 slots
246 // -- below only exists when config.safety is true --
247 // * requested_sizes: [N]LargestSizeClassInt // 1 int for every slot
248 // * log2_ptr_aligns: [N]u8 // 1 byte for every slot
249 // -- above only exists when config.safety is true --
243250 // * stack_trace_addresses: [N]usize, // traces_per_slot for every allocation
244251
245252 const BucketHeader = struct {
246 prev: *BucketHeader,
247 next: *BucketHeader,
248253 page: [*]align(page_size) u8,
249254 alloc_cursor: SlotIndex,
250255 used_count: SlotIndex,
......@@ -253,6 +258,21 @@ pub fn GeneralPurposeAllocator(comptime config: Config) type {
253258 return @as(*u8, @ptrFromInt(@intFromPtr(bucket) + @sizeOf(BucketHeader) + index));
254259 }
255260
261 fn requestedSizes(bucket: *BucketHeader, size_class: usize) []LargestSizeClassInt {
262 if (!config.safety) @compileError("requested size is only stored when safety is enabled");
263 const start_ptr = @as([*]u8, @ptrCast(bucket)) + bucketRequestedSizesStart(size_class);
264 const sizes = @as([*]LargestSizeClassInt, @ptrCast(@alignCast(start_ptr)));
265 const slot_count = @divExact(page_size, size_class);
266 return sizes[0..slot_count];
267 }
268
269 fn log2PtrAligns(bucket: *BucketHeader, size_class: usize) []u8 {
270 if (!config.safety) @compileError("requested size is only stored when safety is enabled");
271 const aligns_ptr = @as([*]u8, @ptrCast(bucket)) + bucketAlignsStart(size_class);
272 const slot_count = @divExact(page_size, size_class);
273 return aligns_ptr[0..slot_count];
274 }
275
256276 fn stackTracePtr(
257277 bucket: *BucketHeader,
258278 size_class: usize,
......@@ -307,10 +327,29 @@ pub fn GeneralPurposeAllocator(comptime config: Config) type {
307327 };
308328 }
309329
310 fn bucketStackFramesStart(size_class: usize) usize {
330 fn bucketRequestedSizesStart(size_class: usize) usize {
331 if (!config.safety) @compileError("requested sizes are not stored unless safety is enabled");
311332 return mem.alignForward(
312333 usize,
313334 @sizeOf(BucketHeader) + usedBitsCount(size_class),
335 @alignOf(LargestSizeClassInt),
336 );
337 }
338
339 fn bucketAlignsStart(size_class: usize) usize {
340 if (!config.safety) @compileError("requested sizes are not stored unless safety is enabled");
341 const slot_count = @divExact(page_size, size_class);
342 return bucketRequestedSizesStart(size_class) + (@sizeOf(LargestSizeClassInt) * slot_count);
343 }
344
345 fn bucketStackFramesStart(size_class: usize) usize {
346 const unaligned_start = if (config.safety) blk: {
347 const slot_count = @divExact(page_size, size_class);
348 break :blk bucketAlignsStart(size_class) + slot_count;
349 } else @sizeOf(BucketHeader) + usedBitsCount(size_class);
350 return mem.alignForward(
351 usize,
352 unaligned_start,
314353 @alignOf(usize),
315354 );
316355 }
......@@ -359,16 +398,15 @@ pub fn GeneralPurposeAllocator(comptime config: Config) type {
359398 /// Emits log messages for leaks and then returns whether there were any leaks.
360399 pub fn detectLeaks(self: *Self) bool {
361400 var leaks = false;
362 for (self.buckets, 0..) |optional_bucket, bucket_i| {
363 const first_bucket = optional_bucket orelse continue;
401
402 for (&self.buckets, 0..) |*buckets, bucket_i| {
403 if (buckets.root == null) continue;
364404 const size_class = @as(usize, 1) << @as(math.Log2Int(usize), @intCast(bucket_i));
365405 const used_bits_count = usedBitsCount(size_class);
366 var bucket = first_bucket;
367 while (true) {
406 var it = buckets.inorderIterator();
407 while (it.next()) |node| {
408 const bucket = node.key;
368409 leaks = detectLeaksInBucket(bucket, size_class, used_bits_count) or leaks;
369 bucket = bucket.next;
370 if (bucket == first_bucket)
371 break;
372410 }
373411 }
374412 var it = self.large_allocations.valueIterator();
......@@ -401,22 +439,18 @@ pub fn GeneralPurposeAllocator(comptime config: Config) type {
401439 }
402440 }
403441 // free retained metadata for small allocations
404 if (self.empty_buckets) |first_bucket| {
405 var bucket = first_bucket;
406 while (true) {
407 const prev = bucket.prev;
408 if (config.never_unmap) {
409 // free page that was intentionally leaked by never_unmap
410 self.backing_allocator.free(bucket.page[0..page_size]);
411 }
412 // alloc_cursor was set to slot count when bucket added to empty_buckets
413 self.freeBucket(bucket, @divExact(page_size, bucket.alloc_cursor));
414 bucket = prev;
415 if (bucket == first_bucket)
416 break;
442 var empty_it = self.empty_buckets.inorderIterator();
443 while (empty_it.next()) |node| {
444 var bucket = node.key;
445 if (config.never_unmap) {
446 // free page that was intentionally leaked by never_unmap
447 self.backing_allocator.free(bucket.page[0..page_size]);
417448 }
418 self.empty_buckets = null;
449 // alloc_cursor was set to slot count when bucket added to empty_buckets
450 self.freeBucket(bucket, @divExact(page_size, bucket.alloc_cursor));
451 self.bucket_node_pool.destroy(node);
419452 }
453 self.empty_buckets.root = null;
420454 }
421455 }
422456
......@@ -440,9 +474,7 @@ pub fn GeneralPurposeAllocator(comptime config: Config) type {
440474 self.freeRetainedMetadata();
441475 }
442476 self.large_allocations.deinit(self.backing_allocator);
443 if (config.safety) {
444 self.small_allocations.deinit(self.backing_allocator);
445 }
477 self.bucket_node_pool.deinit();
446478 self.* = undefined;
447479 return @as(Check, @enumFromInt(@intFromBool(leaks)));
448480 }
......@@ -469,28 +501,27 @@ pub fn GeneralPurposeAllocator(comptime config: Config) type {
469501 });
470502 }
471503
472 fn allocSlot(self: *Self, size_class: usize, trace_addr: usize) Error![*]u8 {
504 const Slot = struct {
505 bucket: *BucketHeader,
506 slot_index: usize,
507 ptr: [*]u8,
508 };
509
510 fn allocSlot(self: *Self, size_class: usize, trace_addr: usize) Error!Slot {
473511 const bucket_index = math.log2(size_class);
474 const first_bucket = self.buckets[bucket_index] orelse try self.createBucket(
475 size_class,
476 bucket_index,
477 );
478 var bucket = first_bucket;
512 var buckets = &self.buckets[bucket_index];
479513 const slot_count = @divExact(page_size, size_class);
480 while (bucket.alloc_cursor == slot_count) {
481 const prev_bucket = bucket;
482 bucket = prev_bucket.next;
483 if (bucket == first_bucket) {
484 // make a new one
485 bucket = try self.createBucket(size_class, bucket_index);
486 bucket.prev = prev_bucket;
487 bucket.next = prev_bucket.next;
488 prev_bucket.next = bucket;
489 bucket.next.prev = bucket;
490 }
514 if (self.cur_buckets[bucket_index] == null or self.cur_buckets[bucket_index].?.alloc_cursor == slot_count) {
515 var new_bucket = try self.createBucket(size_class);
516 errdefer self.freeBucket(new_bucket, size_class);
517 const node = try self.bucket_node_pool.create();
518 node.key = new_bucket;
519 var entry = buckets.getEntryFor(new_bucket);
520 std.debug.assert(entry.node == null);
521 entry.set(node);
522 self.cur_buckets[bucket_index] = node.key;
491523 }
492 // change the allocator's current bucket to be this one
493 self.buckets[bucket_index] = bucket;
524 const bucket = self.cur_buckets[bucket_index].?;
494525
495526 const slot_index = bucket.alloc_cursor;
496527 bucket.alloc_cursor += 1;
......@@ -500,24 +531,22 @@ pub fn GeneralPurposeAllocator(comptime config: Config) type {
500531 used_bits_byte.* |= (@as(u8, 1) << used_bit_index);
501532 bucket.used_count += 1;
502533 bucket.captureStackTrace(trace_addr, size_class, slot_index, .alloc);
503 return bucket.page + slot_index * size_class;
534 return .{
535 .bucket = bucket,
536 .slot_index = slot_index,
537 .ptr = bucket.page + slot_index * size_class,
538 };
504539 }
505540
506541 fn searchBucket(
507 bucket_list: ?*BucketHeader,
542 buckets: *Buckets,
508543 addr: usize,
509544 ) ?*BucketHeader {
510 const first_bucket = bucket_list orelse return null;
511 var bucket = first_bucket;
512 while (true) {
513 const in_bucket_range = (addr >= @intFromPtr(bucket.page) and
514 addr < @intFromPtr(bucket.page) + page_size);
515 if (in_bucket_range) return bucket;
516 bucket = bucket.prev;
517 if (bucket == first_bucket) {
518 return null;
519 }
520 }
545 const search_page = mem.alignBackward(usize, addr, page_size);
546 var search_header: BucketHeader = undefined;
547 search_header.page = @ptrFromInt(search_page);
548 const entry = buckets.getEntryFor(&search_header);
549 return if (entry.node) |node| node.key else null;
521550 }
522551
523552 /// This function assumes the object is in the large object storage regardless
......@@ -683,9 +712,7 @@ pub fn GeneralPurposeAllocator(comptime config: Config) type {
683712 var bucket_index = math.log2(size_class_hint);
684713 var size_class: usize = size_class_hint;
685714 const bucket = while (bucket_index < small_bucket_count) : (bucket_index += 1) {
686 if (searchBucket(self.buckets[bucket_index], @intFromPtr(old_mem.ptr))) |bucket| {
687 // move bucket to head of list to optimize search for nearby allocations
688 self.buckets[bucket_index] = bucket;
715 if (searchBucket(&self.buckets[bucket_index], @intFromPtr(old_mem.ptr))) |bucket| {
689716 break bucket;
690717 }
691718 size_class *= 2;
......@@ -693,7 +720,7 @@ pub fn GeneralPurposeAllocator(comptime config: Config) type {
693720 if (config.retain_metadata) {
694721 if (!self.large_allocations.contains(@intFromPtr(old_mem.ptr))) {
695722 // object not in active buckets or a large allocation, so search empty buckets
696 if (searchBucket(self.empty_buckets, @intFromPtr(old_mem.ptr))) |bucket| {
723 if (searchBucket(&self.empty_buckets, @intFromPtr(old_mem.ptr))) |bucket| {
697724 // bucket is empty so is_used below will always be false and we exit there
698725 break :blk bucket;
699726 } else {
......@@ -720,26 +747,27 @@ pub fn GeneralPurposeAllocator(comptime config: Config) type {
720747
721748 // Definitely an in-use small alloc now.
722749 if (config.safety) {
723 const entry = self.small_allocations.getEntry(@intFromPtr(old_mem.ptr)) orelse
724 @panic("Invalid free");
725 if (old_mem.len != entry.value_ptr.requested_size or log2_old_align != entry.value_ptr.log2_ptr_align) {
750 const requested_size = bucket.requestedSizes(size_class)[slot_index];
751 if (requested_size == 0) @panic("Invalid free");
752 const log2_ptr_align = bucket.log2PtrAligns(size_class)[slot_index];
753 if (old_mem.len != requested_size or log2_old_align != log2_ptr_align) {
726754 var addresses: [stack_n]usize = [1]usize{0} ** stack_n;
727755 var free_stack_trace = StackTrace{
728756 .instruction_addresses = &addresses,
729757 .index = 0,
730758 };
731759 std.debug.captureStackTrace(ret_addr, &free_stack_trace);
732 if (old_mem.len != entry.value_ptr.requested_size) {
760 if (old_mem.len != requested_size) {
733761 log.err("Allocation size {d} bytes does not match resize size {d}. Allocation: {} Resize: {}", .{
734 entry.value_ptr.requested_size,
762 requested_size,
735763 old_mem.len,
736764 bucketStackTrace(bucket, size_class, slot_index, .alloc),
737765 free_stack_trace,
738766 });
739767 }
740 if (log2_old_align != entry.value_ptr.log2_ptr_align) {
768 if (log2_old_align != log2_ptr_align) {
741769 log.err("Allocation alignment {d} does not match resize alignment {d}. Allocation: {} Resize: {}", .{
742 @as(usize, 1) << @as(math.Log2Int(usize), @intCast(entry.value_ptr.log2_ptr_align)),
770 @as(usize, 1) << @as(math.Log2Int(usize), @intCast(log2_ptr_align)),
743771 @as(usize, 1) << @as(math.Log2Int(usize), @intCast(log2_old_align)),
744772 bucketStackTrace(bucket, size_class, slot_index, .alloc),
745773 free_stack_trace,
......@@ -768,8 +796,7 @@ pub fn GeneralPurposeAllocator(comptime config: Config) type {
768796 });
769797 }
770798 if (config.safety) {
771 const entry = self.small_allocations.getEntry(@intFromPtr(old_mem.ptr)).?;
772 entry.value_ptr.requested_size = new_size;
799 bucket.requestedSizes(size_class)[slot_index] = @intCast(new_size);
773800 }
774801 return true;
775802 }
......@@ -803,9 +830,7 @@ pub fn GeneralPurposeAllocator(comptime config: Config) type {
803830 var bucket_index = math.log2(size_class_hint);
804831 var size_class: usize = size_class_hint;
805832 const bucket = while (bucket_index < small_bucket_count) : (bucket_index += 1) {
806 if (searchBucket(self.buckets[bucket_index], @intFromPtr(old_mem.ptr))) |bucket| {
807 // move bucket to head of list to optimize search for nearby allocations
808 self.buckets[bucket_index] = bucket;
833 if (searchBucket(&self.buckets[bucket_index], @intFromPtr(old_mem.ptr))) |bucket| {
809834 break bucket;
810835 }
811836 size_class *= 2;
......@@ -813,7 +838,7 @@ pub fn GeneralPurposeAllocator(comptime config: Config) type {
813838 if (config.retain_metadata) {
814839 if (!self.large_allocations.contains(@intFromPtr(old_mem.ptr))) {
815840 // object not in active buckets or a large allocation, so search empty buckets
816 if (searchBucket(self.empty_buckets, @intFromPtr(old_mem.ptr))) |bucket| {
841 if (searchBucket(&self.empty_buckets, @intFromPtr(old_mem.ptr))) |bucket| {
817842 // bucket is empty so is_used below will always be false and we exit there
818843 break :blk bucket;
819844 } else {
......@@ -842,26 +867,27 @@ pub fn GeneralPurposeAllocator(comptime config: Config) type {
842867
843868 // Definitely an in-use small alloc now.
844869 if (config.safety) {
845 const entry = self.small_allocations.getEntry(@intFromPtr(old_mem.ptr)) orelse
846 @panic("Invalid free");
847 if (old_mem.len != entry.value_ptr.requested_size or log2_old_align != entry.value_ptr.log2_ptr_align) {
870 const requested_size = bucket.requestedSizes(size_class)[slot_index];
871 if (requested_size == 0) @panic("Invalid free");
872 const log2_ptr_align = bucket.log2PtrAligns(size_class)[slot_index];
873 if (old_mem.len != requested_size or log2_old_align != log2_ptr_align) {
848874 var addresses: [stack_n]usize = [1]usize{0} ** stack_n;
849875 var free_stack_trace = StackTrace{
850876 .instruction_addresses = &addresses,
851877 .index = 0,
852878 };
853879 std.debug.captureStackTrace(ret_addr, &free_stack_trace);
854 if (old_mem.len != entry.value_ptr.requested_size) {
880 if (old_mem.len != requested_size) {
855881 log.err("Allocation size {d} bytes does not match free size {d}. Allocation: {} Free: {}", .{
856 entry.value_ptr.requested_size,
882 requested_size,
857883 old_mem.len,
858884 bucketStackTrace(bucket, size_class, slot_index, .alloc),
859885 free_stack_trace,
860886 });
861887 }
862 if (log2_old_align != entry.value_ptr.log2_ptr_align) {
888 if (log2_old_align != log2_ptr_align) {
863889 log.err("Allocation alignment {d} does not match free alignment {d}. Allocation: {} Free: {}", .{
864 @as(usize, 1) << @as(math.Log2Int(usize), @intCast(entry.value_ptr.log2_ptr_align)),
890 @as(usize, 1) << @as(math.Log2Int(usize), @intCast(log2_ptr_align)),
865891 @as(usize, 1) << @as(math.Log2Int(usize), @intCast(log2_old_align)),
866892 bucketStackTrace(bucket, size_class, slot_index, .alloc),
867893 free_stack_trace,
......@@ -879,44 +905,35 @@ pub fn GeneralPurposeAllocator(comptime config: Config) type {
879905
880906 used_byte.* &= ~(@as(u8, 1) << used_bit_index);
881907 bucket.used_count -= 1;
908 if (config.safety) {
909 bucket.requestedSizes(size_class)[slot_index] = 0;
910 }
882911 if (bucket.used_count == 0) {
883 if (bucket.next == bucket) {
884 // it's the only bucket and therefore the current one
885 self.buckets[bucket_index] = null;
886 } else {
887 bucket.next.prev = bucket.prev;
888 bucket.prev.next = bucket.next;
889 self.buckets[bucket_index] = bucket.prev;
912 var entry = self.buckets[bucket_index].getEntryFor(bucket);
913 // save the node for destruction/insertion into in empty_buckets
914 var node = entry.node.?;
915 entry.set(null);
916 // restore the node's key since Treap.remove sets it to undefined
917 node.key = bucket;
918 if (self.cur_buckets[bucket_index] == bucket) {
919 self.cur_buckets[bucket_index] = null;
890920 }
891921 if (!config.never_unmap) {
892922 self.backing_allocator.free(bucket.page[0..page_size]);
893923 }
894924 if (!config.retain_metadata) {
895925 self.freeBucket(bucket, size_class);
926 self.bucket_node_pool.destroy(node);
896927 } else {
897928 // move alloc_cursor to end so we can tell size_class later
898929 const slot_count = @divExact(page_size, size_class);
899930 bucket.alloc_cursor = @as(SlotIndex, @truncate(slot_count));
900 if (self.empty_buckets) |prev_bucket| {
901 // empty_buckets is ordered newest to oldest through prev so that if
902 // config.never_unmap is false and backing_allocator reuses freed memory
903 // then searchBuckets will always return the newer, relevant bucket
904 bucket.prev = prev_bucket;
905 bucket.next = prev_bucket.next;
906 prev_bucket.next = bucket;
907 bucket.next.prev = bucket;
908 } else {
909 bucket.prev = bucket;
910 bucket.next = bucket;
911 }
912 self.empty_buckets = bucket;
931 var empty_entry = self.empty_buckets.getEntryFor(node.key);
932 empty_entry.set(node);
913933 }
914934 } else {
915935 @memset(old_mem, undefined);
916936 }
917 if (config.safety) {
918 assert(self.small_allocations.remove(@intFromPtr(old_mem.ptr)));
919 }
920937 if (config.verbose_log) {
921938 log.info("small free {d} bytes at {*}", .{ old_mem.len, old_mem.ptr });
922939 }
......@@ -980,23 +997,19 @@ pub fn GeneralPurposeAllocator(comptime config: Config) type {
980997 return slice.ptr;
981998 }
982999
983 if (config.safety) {
984 try self.small_allocations.ensureUnusedCapacity(self.backing_allocator, 1);
985 }
9861000 const new_size_class = math.ceilPowerOfTwoAssert(usize, new_aligned_size);
987 const ptr = try self.allocSlot(new_size_class, ret_addr);
1001 const slot = try self.allocSlot(new_size_class, ret_addr);
9881002 if (config.safety) {
989 const gop = self.small_allocations.getOrPutAssumeCapacity(@intFromPtr(ptr));
990 gop.value_ptr.requested_size = len;
991 gop.value_ptr.log2_ptr_align = log2_ptr_align;
1003 slot.bucket.requestedSizes(new_size_class)[slot.slot_index] = @intCast(len);
1004 slot.bucket.log2PtrAligns(new_size_class)[slot.slot_index] = log2_ptr_align;
9921005 }
9931006 if (config.verbose_log) {
994 log.info("small alloc {d} bytes at {*}", .{ len, ptr });
1007 log.info("small alloc {d} bytes at {*}", .{ len, slot.ptr });
9951008 }
996 return ptr;
1009 return slot.ptr;
9971010 }
9981011
999 fn createBucket(self: *Self, size_class: usize, bucket_index: usize) Error!*BucketHeader {
1012 fn createBucket(self: *Self, size_class: usize) Error!*BucketHeader {
10001013 const page = try self.backing_allocator.alignedAlloc(u8, page_size, page_size);
10011014 errdefer self.backing_allocator.free(page);
10021015
......@@ -1004,15 +1017,16 @@ pub fn GeneralPurposeAllocator(comptime config: Config) type {
10041017 const bucket_bytes = try self.backing_allocator.alignedAlloc(u8, @alignOf(BucketHeader), bucket_size);
10051018 const ptr = @as(*BucketHeader, @ptrCast(bucket_bytes.ptr));
10061019 ptr.* = BucketHeader{
1007 .prev = ptr,
1008 .next = ptr,
10091020 .page = page.ptr,
10101021 .alloc_cursor = 0,
10111022 .used_count = 0,
10121023 };
1013 self.buckets[bucket_index] = ptr;
10141024 // Set the used bits to all zeroes
10151025 @memset(@as([*]u8, @as(*[1]u8, ptr.usedBits(0)))[0..usedBitsCount(size_class)], 0);
1026 if (config.safety) {
1027 // Set the requested sizes to zeroes
1028 @memset(mem.sliceAsBytes(ptr.requestedSizes(size_class)), 0);
1029 }
10161030 return ptr;
10171031 }
10181032 };
......@@ -1375,10 +1389,10 @@ test "double frees" {
13751389 const index: usize = 6;
13761390 const size_class: usize = @as(usize, 1) << 6;
13771391 const small = try allocator.alloc(u8, size_class);
1378 try std.testing.expect(GPA.searchBucket(gpa.buckets[index], @intFromPtr(small.ptr)) != null);
1392 try std.testing.expect(GPA.searchBucket(&gpa.buckets[index], @intFromPtr(small.ptr)) != null);
13791393 allocator.free(small);
1380 try std.testing.expect(GPA.searchBucket(gpa.buckets[index], @intFromPtr(small.ptr)) == null);
1381 try std.testing.expect(GPA.searchBucket(gpa.empty_buckets, @intFromPtr(small.ptr)) != null);
1394 try std.testing.expect(GPA.searchBucket(&gpa.buckets[index], @intFromPtr(small.ptr)) == null);
1395 try std.testing.expect(GPA.searchBucket(&gpa.empty_buckets, @intFromPtr(small.ptr)) != null);
13821396
13831397 // detect a large allocation double free
13841398 const large = try allocator.alloc(u8, 2 * page_size);
......@@ -1395,8 +1409,8 @@ test "double frees" {
13951409
13961410 // check that flushing retained metadata doesn't disturb live allocations
13971411 gpa.flushRetainedMetadata();
1398 try std.testing.expect(gpa.empty_buckets == null);
1399 try std.testing.expect(GPA.searchBucket(gpa.buckets[index], @intFromPtr(normal_small.ptr)) != null);
1412 try std.testing.expect(gpa.empty_buckets.root == null);
1413 try std.testing.expect(GPA.searchBucket(&gpa.buckets[index], @intFromPtr(normal_small.ptr)) != null);
14001414 try std.testing.expect(gpa.large_allocations.contains(@intFromPtr(normal_large.ptr)));
14011415 try std.testing.expect(!gpa.large_allocations.contains(@intFromPtr(large.ptr)));
14021416}