112 lines
3.0 KiB
Markdown
112 lines
3.0 KiB
Markdown
Turbo NSS
|
|
---------
|
|
|
|
glibc nss library for passwd and group.
|
|
|
|
Checking out and building
|
|
-------------------------
|
|
|
|
```
|
|
$ git clone --recursive https://git.sr.ht/~motiejus/turbonss
|
|
```
|
|
|
|
Alternatively, if you forgot `--recursive`:
|
|
|
|
```
|
|
$ git submodule update --init
|
|
```
|
|
|
|
And run tests:
|
|
|
|
```
|
|
$ zig build test
|
|
```
|
|
|
|
... the other commands will be documented as they are implemented.
|
|
|
|
This project uses [git subtrac][git-subtrac] for managing dependencies.
|
|
|
|
Steps
|
|
-----
|
|
|
|
A known implementation runs id(1) at ~250 rps sequentially. Our goal is 10k
|
|
ID/s.
|
|
|
|
id(1) works as follows:
|
|
- lookup user by name.
|
|
- get all additional gids (an array attached to a member).
|
|
- for each additional gid, return the group name.
|
|
|
|
Assuming a member is in ~100 groups on average, that's 1M group lookups per
|
|
second. We need to convert gid to a group index quickly.
|
|
|
|
Data structures
|
|
---------------
|
|
|
|
Basic data structures that allow efficient storage:
|
|
|
|
```lang=c
|
|
// reminder:
|
|
typedef uid_t uint32;
|
|
typedef gid_t uint32;
|
|
|
|
// 6*32b = 6*4B = 24B/user
|
|
typedef struct {
|
|
uid_t uid;
|
|
gid_t gid;
|
|
name_offset uint32; // offset into *usernames
|
|
gecos_offset uint32; // offset into *gecos
|
|
shell_offset uint32; // offset into *shells
|
|
additional_groups_offset uint32; // offset into additional_groups
|
|
} user;
|
|
|
|
const char* usernames; // all concatenated usernames, fsst-compressed
|
|
const char* gecoss; // all concatenated gecos, fsst-compressed
|
|
const char* shells; // all concatenated home directories, fsst-compressed
|
|
const uint8_t additional_groups; // all additional_groups, turbo compressed
|
|
|
|
typedef struct {
|
|
gid_t gid;
|
|
name_offset uint32; // offset into *groupnames
|
|
members_offset uint32; // offset into members
|
|
}
|
|
|
|
const char* groupnames; // all concatenated group names, fsst-compressed
|
|
const uint8_8 members; // all concatenated members, turbo compressed
|
|
```
|
|
|
|
"turbo compression" encodes a list of uids/gids with this algorithm:
|
|
1. sort ascending.
|
|
2. extract deltas and subtract 1: `awk '{diff=$0-prev; prev=$0; print
|
|
diff-1}'`.
|
|
3. varint-encode these deltas into an uint32, like protobuf or utf8.
|
|
|
|
With typical group memberships (as of writing) this requires ~1.3-1.5 byte per
|
|
entry.
|
|
|
|
Indexes
|
|
-------
|
|
|
|
The following operations need to be fast, in order of importance:
|
|
|
|
1. lookup gid -> group (this is on hot path in id).
|
|
2. lookup uid -> user.
|
|
3. lookup username -> user.
|
|
4. lookup groupname -> group.
|
|
5. (optional) iterate users using a defined order (`getent passwd`).
|
|
6. (optional) iterate groups using a defined order (`getent group`).
|
|
|
|
Preliminary results of playing with [cmph][cmph]:
|
|
|
|
BDZ: tried b=3, b=7 (default), and b=10.
|
|
|
|
* BDZ algorithm stores 1M values in (900KB, 338KB, 306KB) respectively.
|
|
* Latency for 1M keys: (170ms, 180ms, 230ms).
|
|
* Packed vs non-packed latency differences are not meaningful.
|
|
|
|
CHM retains order, however, 1M keys weigh 8MB. 10k keys are ~20x larger with
|
|
CHM than with BDZ, eliminating the benefit of preserved ordering.
|
|
|
|
[git-subtrac]: https://github.com/apenwarr/git-subtrac/
|
|
[cmph]: http://cmph.sourceforge.net/
|