motiejus/turbonss
1
Fork 0
Code Releases Activity

start with a full file structure

Browse Source
This commit is contained in:
Motiejus Jakštys 2022-02-13 18:01:44 +02:00 committed by Motiejus Jakštys
parent b3446ef307
commit 1327587838
1 changed files with 66 additions and 37 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

103
README.md
View File

@ -1,7 +1,7 @@
Turbo NSS Turbo NSS
--------- ---------
glibc nss library for passwd and group. Glibc nss library for passwd and group.
Checking out and building Checking out and building
------------------------- -------------------------
@ -22,27 +22,33 @@ And run tests:
$ zig build test $ zig build test
``` ```
... the other commands will be documented as they are implemented. Other commands will be documented as they are implemented.
This project uses [git subtrac][git-subtrac] for managing dependencies. This project uses [git subtrac][git-subtrac] for managing dependencies.
Steps remarks on `id(1)`
----- ------------------
A known implementation runs id(1) at ~250 rps sequentially. Our goal is 10k A known implementation runs id(1) at ~250 rps sequentially on ~20k users and
ID/s. ~10k groups. Our target is 10k id/s.
id(1) works as follows: `id(1)` works as follows:
- lookup user by name. - lookup user by name.
- get all additional gids (an array attached to a member). - get all additional gids (an array attached to a member).
- for each additional gid, get the group name. - for each additional gid, get the group name.
Assuming a member is in ~100 groups on average, that's 1M group lookups per Assuming a member is in ~100 groups on average, that's 1M group lookups per
second (cmph can do 1M in <200ms). We need to convert gid to a group index second. We need to convert gid to a group index, and group index to a group
quickly. gid/name quickly.
API Caveat: `struct group` contains an array of pointers to names of group members
--- (`char **gr_mem`). However, `id` does not use that information, resulting in a
significant read amplification. Therefore, if `argv[0] == "id"`, `getgrid(3)`
will return group without the members. This speeds up `id` by about 10x on a
known NSS implementation.
Indices
-------
The following operations need to be fast, in order of importance: The following operations need to be fast, in order of importance:
@ -54,50 +60,54 @@ The following operations need to be fast, in order of importance:
5. (optional) iterate users using a defined order (`getent passwd`). 5. (optional) iterate users using a defined order (`getent passwd`).
6. (optional) iterate groups using a defined order (`getent group`). 6. (optional) iterate groups using a defined order (`getent group`).
Indices First 4 can use perfect hashing like [cmph][cmph]: it hashes a list of bytes to
------- a sequential list of integers. Perfect hashing algorithms require some space,
and take some time to calculate ("hashing duration"). I've tested BDZ, which
Preliminary results of playing with [cmph][cmph]: hashes [][]u8 to a sequential list of integers (not preserving order) and CHM, which
does the same, but preserves order. BDZ accepts an argument 3 <= b <= 10.
BDZ: tried b=3, b=7 (default), and b=10. BDZ: tried b=3, b=7 (default), and b=10.
* BDZ algorithm stores 1M values in (900KB, 338KB, 306KB) respectively. * BDZ algorithm requires (900KB, 338KB, 306KB, respectively) for 1M values.
* Latency for 1M keys: (170ms, 180ms, 230ms). * Latency to resolve 1M keys: (170ms, 180ms, 230ms).
* Packed vs non-packed latency differences are not meaningful. * Packed vs non-packed latency differences are not meaningful.
CHM retains order, however, 1M keys weigh 8MB. 10k keys are ~20x larger with CHM retains order, however, 1M keys weigh 8MB. 10k keys are ~20x larger with
CHM than with BDZ, eliminating the benefit of preserved ordering. CHM than with BDZ, eliminating the benefit of preserved ordering.
Full file structure Turbonss header
------------------- ---------------
The turbonss header looks like this: The turbonss header looks like this:
``` ```
OFFSET TYPE NAME DESCRIPTION OFFSET TYPE NAME DESCRIPTION
0 [4]u8 magic always 0xf09fa4b7 0 [4]u8 magic always 0xf09fa4b7
4 u8 version now `0` 4 u8 version now `0`
5 u2 padding 5 u16 bom 0x1234
7 u2 padding
u6 num_shells see "SHELLS" section. u6 num_shells see "SHELLS" section.
6 u32 num_users number of passwd entries 8 u32 num_users number of passwd entries
10 u32 num_groups number of group entries 12 u32 num_groups number of group entries
14 u32 offset_cmph_gid2group 16 u32 offset_cmph_gid2group
18 u32 offset_cmph_uid2user 20 u32 offset_cmph_uid2user
22 u32 offset_cmph_groupname2group 24 u32 offset_cmph_groupname2group
26 u32 offset_cmph_username2user 28 u32 offset_cmph_username2user
30 u32 offset_sorted_groups 32 u32 offset_groupmembers
34 u32 offset_sorted_users 36 u32 offset_additional_gids
38 u32 offset_groupmembers
42 u32 offset_additional_gids
``` ```
`magic` is 0xf09fa4b7, and `version` must be `0`. All integers are big-endian. `magic` is 0xf09fa4b7, and `version` must be `0`. All integers are
native-endian. `bom` is a byte-order-mark. It must resolve to `0x1234` (4460).
If that's not true, the file is consumed in a different endianness than it was
created at. Turbonss files cannot be moved across different-endianness
computers. If that happens, turbonss will refuse to read the file.
Offsets are indices to further sections of the file, with zero being the first Offsets are indices to further sections of the file, with zero being the first
block (the magic number). As all blobs are 64-byte aligned, the offsets are block (pointing to the `magic` field). As all blobs are 64-byte aligned, the
always pointing to the beginning of an 64-byte "block". Therefore, all offsets are always pointing to the beginning of an 64-byte "block". Therefore,
`offset_*` values could be `u26`. As `u32` is easier to visualize with xxd, and all `offset_*` values could be `u26`. As `u32` is easier to visualize with xxd,
the File block fits to 64 bytes anyway, we are keeping them as u32 now. and the header block fits to 64 bytes anyway, we are keeping them as u32 now.
Primitive types: Primitive types:
@ -130,10 +140,29 @@ const User = struct {
} }
``` ```
Complete file structure
-----------------------
```
OFFSET Section SIZE DESCRIPTION
0<<6 Header 1<<6 documented above
*<<6 []Group num_groups * sizeof(Group)
*<<6 []User num_users * sizeof(User)
*<<6 []u32 num_groups * sizeof(u32)
*<<6 []u32 num_users * sizeof(u32)
*<<6 Shells unknown documented in "SHELLS"
*<<6 cmph_gid2group unknown offset by offset_cmph_gid2group
*<<6 cmph_uid2user unknown offset by offset_cmph_gid2group
*<<6 cmph_groupname2group unknown offset by offset_cmph_groupname2group
*<<6 cmph_username2user unknown offset by offset_cmph_username2user
*<<6 groupmembers unknown list of group members for each group
*<<6 additional_gids unknown list of gids (group membership) for each member
```
TODO explain: TODO explain:
- shells - shells
- `additional_gids` - `additional_gids`
- `members` - `groupmembers`
[git-subtrac]: https://github.com/apenwarr/git-subtrac/ [git-subtrac]: https://github.com/apenwarr/git-subtrac/
[cmph]: http://cmph.sourceforge.net/ [cmph]: http://cmph.sourceforge.net/