replace member strlen with getgr_max and getpw_max

This way we can use buffers in both Zig and C, without the performance
or size penalty.
This commit is contained in:
2022-04-18 19:35:26 +03:00
committed by Motiejus Jakštys
parent c6bd3f5cb8
commit 64bd5df633
4 changed files with 60 additions and 88 deletions

View File

@@ -111,7 +111,7 @@ A known implementation runs id(1) at ~250 rps sequentially on ~20k users and
To better reason about the trade-offs, it is useful to understand how `id(1)`
is implemented, in rough terms:
- lookup user by name ([`getpwent_r(3)`][getpwent_r]).
- lookup user by name ([`getpwent_r(3)`][getpwent]).
- get all gids for the user ([`getgrouplist(3)`][getgrouplist]). Note: it is
actually using `initgroups_dyn`, accepts a uid, and is very poorly
documented.
@@ -154,6 +154,9 @@ OFFSET TYPE NAME DESCRIPTION
40 u64 nblocks_users
48 u64 nblocks_groupmembers
56 u64 nblocks_additional_gids
64 u64 getgr_max
72 u64 getpw_max
80 [48]u8 padding
```
`magic` is 0xf09fa4b7, and `version` must be `0`. All integers are
@@ -165,6 +168,10 @@ number of bytes. However, interpreting `[2]u6` with `xxd(1)` is harder than
interpreting `[2]u8`. Therefore we are using the space we have to make these
integers byte-wide.
`getgr_max` and `getpw_max` is a hint for the caller of `getgr*` and
`getpw*`-family calls. This is the recommended size of the buffer, so the
caller does not receive `ENOMEM`.
Primitive types
---------------
@@ -279,20 +286,17 @@ respectively. Membership IDs are packed — not necessitating random access, thu
suitable for compression.
- `groupmembers` consists of a number X followed by a list of offsets to User
records, because `getgr*_r` returns pointers to membernames, thus a name has
to be immediately resolvable.
records, because `getgr*` returns pointers to membernames, thus a name has to
be immediately resolvable.
- `additional_gids` is a list of gids, because `initgroups_dyn` (and friends)
returns an array of gids.
Each entry of `groupmembers` and `additional_gids` starts with a varint N,
which is the number of upcoming elements. Then depending on the type:
which is the number of upcoming elements. Then N delta-compressed varints,
which are:
- **additional_gids** stores N delta-compressed varints. These varints
correspond to a list of gids.
- **groupmembers** stores a total length of the member names, followed by N,
followed by N offsets. These are byte-offsets to the User records in the
`users` section. Having the length of membernames saves some CPU cycles when
decompressing the members in the hot path.
- **additional_gids** a list of gids.
- **groupmembers** byte-offsets to the User records in the `users` section.
Indices
-------
@@ -339,7 +343,7 @@ Each section is padded to 64 bytes.
```
SECTION SIZE DESCRIPTION
header 64 see "Turbonss header" section
header 128 see "Turbonss header" section
bdz_gid ? bdz(gid)
bdz_groupname ? bdz(groupname)
bdz_uid ? bdz(uid)
@@ -365,7 +369,7 @@ Section creation order:
1. ✅ `groupmembers` requires `users`.
1. ✅ `groups` requires `groupmembers`.
1. ✅ `idx_*`. requires offsets to `groups` and `users`.
1. Header.
1. Header.
[git-subtrac]: https://apenwarr.ca/log/20191109
[cmph]: http://cmph.sourceforge.net/
@@ -374,6 +378,6 @@ Section creation order:
[data-oriented-design]: https://media.handmade-seattle.com/practical-data-oriented-design/
[getpwnam_r]: https://linux.die.net/man/3/getpwnam_r
[varint]: https://developers.google.com/protocol-buffers/docs/encoding#varints
[getpwent_r]: https://www.man7.org/linux/man-pages/man3/getpwent_r.3.html
[getpwent]: https://www.man7.org/linux/man-pages/man3/getpwent_r.3.html
[getgrouplist]: https://www.man7.org/linux/man-pages/man3/getgrouplist.3.html
[getgrid_r]: https://www.man7.org/linux/man-pages/man3/getgrid_r.3.html
[getgrid]: https://www.man7.org/linux/man-pages/man3/getgrid_r.3.html