198 lines
7.0 KiB
Markdown
198 lines
7.0 KiB
Markdown
Turbo NSS
|
|
---------
|
|
|
|
Turbonss is a plugin for GNU Name Service Switch ([NSS][nsswitch])
|
|
functionality of GNU C Library (glibc). Turbonss implements lookup for `user`
|
|
and `passwd` database entries (i.e. system users, groups, and group
|
|
memberships). It's main goal is to run [`id(1)`][id] as fast as possible.
|
|
|
|
Turbonss is optimized for reading. If the data changes in any way, the whole
|
|
file will need to be regenerated. Therefore, it was created, and best suited,
|
|
for environments that have a central user & group database which then needs to
|
|
be distributed to many servers/services, and the data does not change very
|
|
often.
|
|
|
|
This is the fastest known NSS passwd/group implementation for *reads*. On my
|
|
2018-era laptop a corpus with 10k users, 10k groups and 500 average members per
|
|
group, `id` takes 17 seconds with the glibc default implementation, 10-17
|
|
milliseconds with a pre-cached `nscd`, ~8 milliseconds with uncached
|
|
`turbonss`.
|
|
|
|
Due to the nature of being built with Zig, this will work on glibc versions as
|
|
old as 2.16 (may work with even older ones, I did not test beyond that).
|
|
|
|
Project goals
|
|
-------------
|
|
|
|
- Make it as fast as possible. Especially optimize for the `id` command.
|
|
- Small database size (helps making it fast).
|
|
- No runtime, no GC, as little as possible overhead.
|
|
- Easy to compile for ancient glibc versions (comes out of the box with Zig).
|
|
|
|
|
|
Dependencies
|
|
------------
|
|
|
|
1. zig around `0.11.0-dev.3132+465272921`.
|
|
2. [cmph][cmph]: bundled with this repository.
|
|
|
|
Building
|
|
--------
|
|
|
|
Clone, compile and run tests first:
|
|
|
|
$ git clone --recursive https://git.jakstys.lt/motiejus/turbonss
|
|
$ zig build test
|
|
$ zig build -Dtarget=x86_64-linux-gnu.2.16 -Doptimize=ReleaseSafe
|
|
|
|
One may choose different options, depending on requirements. Here are some
|
|
hints:
|
|
|
|
1. `-Dcpu=<...>` for the CPU [microarchitecture][mcpu].
|
|
2. `-Dstrip=true` to strip debug symbols.
|
|
|
|
For reference, size of the shared library and helper binaries when compiled
|
|
with `zig build -Dstrip=true -Doptimize=ReleaseSmall`:
|
|
|
|
$ ls -h1s zig-out/{bin/*,lib/libnss_turbo.so.2.0.0}
|
|
24K zig-out/bin/turbonss-analyze
|
|
20K zig-out/bin/turbonss-getent
|
|
24K zig-out/bin/turbonss-makecorpus
|
|
136K zig-out/bin/turbonss-unix2db
|
|
20K zig-out/lib/libnss_turbo.so.2.0.0
|
|
|
|
Many thanks to Ulrich Drepper for [teaching how to link it properly][dso].
|
|
|
|
Demo
|
|
----
|
|
|
|
turbonss is best tested, of course, with many users and groups. The guide below
|
|
will show how to synthesize 10k users, 10k groups with an avereage membership
|
|
of 1k users per group, and test ubernss with such corpus.
|
|
|
|
1. Synthesize some users and groups to `passwd` and `group` in the current directory:
|
|
|
|
```
|
|
$ zig-out/bin/turbonss-makecorpus
|
|
wrote users=10000 groups=10000 max-members=1000 to .
|
|
$ ls -1hs passwd group
|
|
48M group
|
|
668K passwd
|
|
```
|
|
|
|
2. Convert the generated `passwd` and `group` to the turbonss database. Note
|
|
the `db.turbo` database is more than 4 times smaller than the textual one:
|
|
|
|
```
|
|
$ zig-out/bin/turbonss-unix2db --group group --passwd passwd
|
|
total 10968064 bytes. groups=10000 users=10000
|
|
$ ls -1hs db.turbo
|
|
11M db.turbo
|
|
```
|
|
|
|
3. Optional: inspect the freshly created database:
|
|
|
|
```
|
|
$ zig-out/bin/turbonss-analyze db.turbo
|
|
File: db.turbo
|
|
Size: 10,968,064 bytes
|
|
Version: 0
|
|
Endian: little
|
|
Pointer size: 8 bytes
|
|
getgr buffer size: 18000
|
|
getpw buffer size: 57
|
|
Users: 10000
|
|
Groups: 10000
|
|
Shells: 4
|
|
Most memberships: u_1000000 (501)
|
|
Sections:
|
|
Name Begin End Size bytes
|
|
header 00000000 00000080 128
|
|
bdz_gid 00000080 00000e40 3,520
|
|
bdz_groupname 00000e40 00001c00 3,520
|
|
bdz_uid 00001c00 000029c0 3,520
|
|
bdz_username 000029c0 00003780 3,520
|
|
idx_gid2group 00003780 0000d3c0 40,000
|
|
idx_groupname2group 0000d3c0 00017000 40,000
|
|
idx_uid2user 00017000 00020c40 40,000
|
|
idx_name2user 00020c40 0002a880 40,000
|
|
shell_index 0002a880 0002a8c0 64
|
|
shell_blob 0002a8c0 0002a900 64
|
|
groups 0002a900 00065280 240,000
|
|
users 00065280 000da580 480,000
|
|
groupmembers 000da580 005a69c0 5,030,976
|
|
additional_gids 005a69c0 00a75c00 5,042,752
|
|
$ zig-out/bin/turbonss-getent --db db.turbo passwd u_1000000
|
|
u_1000000:x:1000000:1000000:User 1000000:/home/u_1000000:/bin/bash
|
|
$ zig-out/bin/turbonss-getent --db db.turbo group g_1000003
|
|
g_1000003:x:1000003:u_1000002,u_1000003,u_1000004
|
|
```
|
|
|
|
4. Now since we will be messing with the system, run all following commands in
|
|
a container:
|
|
|
|
```
|
|
$ docker run -ti --rm -v `pwd`:/etc/turbonss -w /etc/turbonss debian:bullseye
|
|
# cp zig-out/lib/libnss_turbo.so.2 /lib/x86_64-linux-gnu/
|
|
```
|
|
|
|
5. Instruct `nsswitch.conf` to use both turbonss and the standard resolver:
|
|
|
|
```
|
|
# sed -i '/passwd\|group/ s/files/turbo files/' /etc/nsswitch.conf
|
|
# time id u_1000000
|
|
<...>
|
|
real 0m0.006s
|
|
user 0m0.000s
|
|
sys 0m0.008s
|
|
```
|
|
|
|
The `id` call resolved `u_1000000` from `db.turbo`.
|
|
|
|
6. Compare the performance to plain `files` (that is, without turbonss):
|
|
|
|
```
|
|
# sed -i '/passwd\|group/ s/turbo files/files/' /etc/nsswitch.conf
|
|
# cat passwd >> /etc/passwd
|
|
# cat group >> /etc/group
|
|
# time id u_1000000
|
|
<...>
|
|
real 0m17.164s
|
|
user 0m13.288s
|
|
sys 0m3.876s
|
|
```
|
|
|
|
Over 2500x difference.
|
|
|
|
More Documentation
|
|
------------------
|
|
|
|
- Architecture is detailed in `docs/architecture.md`
|
|
- Development notes are in `docs/development.md`
|
|
|
|
Project status and known deficiencies
|
|
-------------------------------------
|
|
|
|
Turbonss works, but, to the author's knowledge, was not deployed to production.
|
|
If you want to use turbonss instead of a battle-tested, albeit slower nscd,
|
|
keep the following in mind:
|
|
- turbonss has not been fuzz-tested, so it will crash a program on invalid
|
|
database file. Please compile with `ReleaseSafe`. It is plenty fast with this
|
|
mode, but an invalid database will lead to defined behavior (i.e. crash with
|
|
a stack trace) instead of overwriting memory wherever.
|
|
- if the database file was replaced while the program has been running,
|
|
turbonss will not re-read the file (it holds to the previous file
|
|
descriptor).
|
|
- requires a nightly version of zig (that will change with 0.11).
|
|
|
|
The license is permissive, so feel free to fork and implement the above (I
|
|
would appreciate if you told me, but surely you don't have to). I am also
|
|
available for [consulting][consulting] if that's your preference instead.
|
|
|
|
[nsswitch]: https://linux.die.net/man/5/nsswitch.conf
|
|
[id]: https://linux.die.net/man/1/id
|
|
[cmph]: http://cmph.sourceforge.net/
|
|
[dso]: https://akkadia.org/drepper/dsohowto.pdf
|
|
[mcpu]: https://en.wikipedia.org/wiki/X86-64#Microarchitecture_levels
|
|
[consulting]: https://jakstys.lt/contact
|