178 lines
6.3 KiB
Markdown
178 lines
6.3 KiB
Markdown
Turbo NSS
|
|
---------
|
|
|
|
Turbonss is a plugin for GNU Name Service Switch ([NSS][nsswitch])
|
|
functionality of GNU C Library (glibc). Turbonss implements lookup for `user`
|
|
and `passwd` database entries (i.e. system users, groups, and group
|
|
memberships). It's main goal is to run [`id(1)`][id] as fast as possible.
|
|
|
|
Turbonss is optimized for reading. If the data changes in any way, the whole
|
|
file will need to be regenerated. Therefore, it was created, and best suited,
|
|
for environments that have a central user & group database which then needs to
|
|
be distributed to many servers/services, and the data does not change very
|
|
often (e.g. hourly).
|
|
|
|
This is the fastest known NSS passwd/group implementation for *reads*. On my
|
|
2018-era laptop a corpus with 10k users, 10k groups and 500 average members per
|
|
group, `id` takes 17 seconds with the glibc default implementation, 10-17
|
|
milliseconds with a pre-cached `nscd`, ~8 milliseconds with `turbonss`.
|
|
|
|
Project status
|
|
--------------
|
|
|
|
The project is finished and was never used recommended for production. If you
|
|
are considering using turbonss, try nscd first. Turbonss is only 2-5 times
|
|
faster than pre-warmed nscd, which usually does not matter enough to go through
|
|
the hoops of using a nonstandard nss library in the first place.
|
|
|
|
Yours truly worked on this for about 7 months. This was also my first zig
|
|
project which I never went to (nor really needed to) come back and clean up.
|
|
|
|
Update 2022-02: I am reviving it:
|
|
- updated to stage2, so it works on nightly again.
|
|
- I learned some zig over the last year, will be cleaning it up.
|
|
|
|
Currently it has not been fuzz-tested, so it will crash on invalid data. Please
|
|
use `ReleaseSafe` until it is fuzzed.
|
|
|
|
Dependencies
|
|
------------
|
|
|
|
1. zig 0.11.0-dev.1782+b52be973d or higher.
|
|
2. [cmph][cmph]: bundled with this repository.
|
|
|
|
Trying it out
|
|
-------------
|
|
|
|
Clone, compile and test first:
|
|
|
|
$ git clone --recursive https://git.sr.ht/~motiejus/turbonss
|
|
$ zig build test
|
|
$ zig build -Dtarget=x86_64-linux-gnu.2.16 -Dcpu=baseline -Doptimize=ReleaseSafe
|
|
|
|
One may choose different options, depending on requirements. Here are some
|
|
hints:
|
|
|
|
1. `-Dcpu=<...>` for the CPU
|
|
[microarchitecture](https://en.wikipedia.org/wiki/X86-64#Microarchitecture_levels).
|
|
2. `-Dstrip=true` to strip debug symbols.
|
|
|
|
For reference, size of the shared library and helper binaries when compiled
|
|
with `-Dstrip=true -Drelease-small=true`:
|
|
|
|
28K zig-out/bin/turbonss-analyze
|
|
20K zig-out/bin/turbonss-getent
|
|
24K zig-out/bin/turbonss-makecorpus
|
|
140K zig-out/bin/turbonss-unix2db
|
|
24K zig-out/lib/libnss_turbo.so.2.0.0
|
|
|
|
Many thanks to Ulrich Drepper for [teaching how to link it properly][dso].
|
|
|
|
Test turobnss on a real system
|
|
------------------------------
|
|
|
|
`db.turbo` is the TurboNSS database file. To create one from `/etc/group` and
|
|
`/etc/passwd`, use `turbonss-unix2db`:
|
|
|
|
$ zig-out/bin/turbonss-unix2db --passwd /etc/passwd --group /etc/group
|
|
$ zig-out/bin/turbonss-analyze db.turbo
|
|
File: /etc/turbonss/db.turbo
|
|
Size: 2,624 bytes
|
|
Version: 0
|
|
Endian: little
|
|
Pointer size: 8 bytes
|
|
getgr buffer size: 17
|
|
getpw buffer size: 74
|
|
Users: 19
|
|
Groups: 39
|
|
Shells: 1
|
|
Most memberships: _apt (1)
|
|
Sections:
|
|
Name Begin End Size bytes
|
|
header 00000000 00000080 128
|
|
bdz_gid 00000080 000000c0 64
|
|
bdz_groupname 000000c0 00000100 64
|
|
bdz_uid 00000100 00000140 64
|
|
bdz_username 00000140 00000180 64
|
|
idx_gid2group 00000180 00000240 192
|
|
idx_groupname2group 00000240 00000300 192
|
|
idx_uid2user 00000300 00000380 128
|
|
idx_name2user 00000380 00000400 128
|
|
shell_index 00000400 00000440 64
|
|
shell_blob 00000440 00000480 64
|
|
groups 00000480 00000700 640
|
|
users 00000700 000009c0 704
|
|
groupmembers 000009c0 00000a00 64
|
|
additional_gids 00000a00 00000a40 64
|
|
|
|
Run and configure a test container that uses `turbonss` instead of the default
|
|
`files`:
|
|
|
|
$ docker run -ti --rm -v `pwd`:/etc/turbonss -w /etc/turbonss debian:bullseye
|
|
# cp zig-out/lib/libnss_turbo.so.2 /lib/x86_64-linux-gnu/
|
|
# sed -i '/passwd\|group/ s/files/turbo/' /etc/nsswitch.conf
|
|
|
|
And run the commands:
|
|
|
|
$ getent passwd
|
|
$ getent group
|
|
$ id root
|
|
|
|
More users and groups
|
|
---------------------
|
|
|
|
`turbonss-makecorpus` can synthesize more `users` and `groups`:
|
|
|
|
# ./zig-out/bin/turbonss-makecorpus
|
|
wrote users=10000 groups=10000 avg-members=1000 to .
|
|
# cat group >> /etc/group
|
|
# cat passwd >> /etc/passwd
|
|
# time id u_1000000
|
|
<...>
|
|
real 0m17.380s
|
|
user 0m13.117s
|
|
sys 0m4.263s
|
|
|
|
17 seconds for an `id` command! Well, there are indeed many users and groups.
|
|
Let's see how turbonss fares with it:
|
|
|
|
# zig-out/bin/turbonss-unix2db --group /etc/group --passwd /etc/passwd
|
|
total 10968512 bytes. groups=10019 users=10039
|
|
# ls -hs /etc/group /etc/passwd db.turbo
|
|
48M /etc/group 668K /etc/passwd 11M db.turbo
|
|
# sed -i '/passwd\|group/ s/files/turbo/' /etc/nsswitch.conf
|
|
# time id u_1000000
|
|
real 0m0.008s
|
|
user 0m0.000s
|
|
sys 0m0.008s
|
|
|
|
That's ~1500x improvement for the `id` command (and notice about 4X compression
|
|
ratio compared to plain files). If the number of users and groups is increased
|
|
by 10x (to 100k each), the difference becomes even crazier:
|
|
|
|
# time id u_1000000
|
|
<...>
|
|
real 3m42.281s
|
|
user 2m30.482s
|
|
sys 0m55.840s
|
|
# sed -i '/passwd\|group/ s/files/turbo/' /etc/nsswitch.conf
|
|
# time id u_1000000
|
|
<...>
|
|
real 0m0.008s
|
|
user 0m0.000s
|
|
sys 0m0.008s
|
|
|
|
Note that to author's knowledge this has not been used on any real production
|
|
nor a development machine.
|
|
|
|
Documentation
|
|
-------------
|
|
|
|
- Architecture is detailed in `docs/architecture.md`
|
|
- Development notes are in `docs/development.md`
|
|
|
|
[nsswitch]: https://linux.die.net/man/5/nsswitch.conf
|
|
[id]: https://linux.die.net/man/1/id
|
|
[cmph]: http://cmph.sourceforge.net/
|
|
[dso]: https://akkadia.org/drepper/dsohowto.pdf
|