Turbo NSS --------- Turbonss is a plugin for GNU Name Service Switch ([NSS][nsswitch]) functionality of GNU C Library (glibc). Turbonss implements lookup for `user` and `passwd` database entries (i.e. system users, groups, and group memberships). It's main goal is to run [`id(1)`][id] as fast as possible. Turbonss is optimized for reading. If the data changes in any way, the whole file will need to be regenerated. Therefore, it was created, and best suited, for environments that have a central user & group database which then needs to be distributed to many servers/services, and the data does not change very often (e.g. hourly). This is the fastest known NSS passwd/group implementation for *reads*. On my 2018-era laptop a corpus with 10k users, 10k groups and 500 average members per group, `id` takes 17 seconds with the glibc default implementation, 10-17 milliseconds with a pre-cached `nscd`, ~8 milliseconds with `turbonss`. Project status -------------- The project is finished and was never used recommended for production. If you are considering using turbonss, try nscd first. Turbonss is only 2-5 times faster than pre-warmed nscd, which usually does not matter enough to go through the hoops of using a nonstandard nss library in the first place. Yours truly worked on this for about 7 months. This was also my first zig project which I never went to (nor really needed to) come back and clean up. Update 2022-02: I am reviving it: - updated to stage2, so it works on nightly again. - I learned some zig over the last year, will be cleaning it up. Currently it has not been fuzz-tested, so it will crash on invalid data. Please use `ReleaseSafe` until it is fuzzed. Dependencies ------------ 1. zig 0.11.0-dev.1580+a5b34a61a or higher. 2. [cmph][cmph]: bundled with this repository. Trying it out ------------- Clone, compile and test first: $ git clone --recursive https://git.sr.ht/~motiejus/turbonss $ zig build test $ zig build -Dtarget=x86_64-linux-gnu.2.16 -Dcpu=baseline -Doptimize=ReleaseSafe One may choose different options, depending on requirements. Here are some hints: 1. `-Dcpu=<...>` for the CPU [microarchitecture](https://en.wikipedia.org/wiki/X86-64#Microarchitecture_levels). 2. `-Dstrip=true` to strip debug symbols. For reference, size of the shared library and helper binaries when compiled with `-Dstrip=true -Drelease-small=true`: 28K zig-out/bin/turbonss-analyze 20K zig-out/bin/turbonss-getent 24K zig-out/bin/turbonss-makecorpus 140K zig-out/bin/turbonss-unix2db 24K zig-out/lib/libnss_turbo.so.2.0.0 Many thanks to Ulrich Drepper for [teaching how to link it properly][dso]. Test turobnss on a real system ------------------------------ `db.turbo` is the TurboNSS database file. To create one from `/etc/group` and `/etc/passwd`, use `turbonss-unix2db`: $ zig-out/bin/turbonss-unix2db --passwd /etc/passwd --group /etc/group $ zig-out/bin/turbonss-analyze db.turbo File: /etc/turbonss/db.turbo Size: 2,624 bytes Version: 0 Endian: little Pointer size: 8 bytes getgr buffer size: 17 getpw buffer size: 74 Users: 19 Groups: 39 Shells: 1 Most memberships: _apt (1) Sections: Name Begin End Size bytes header 00000000 00000080 128 bdz_gid 00000080 000000c0 64 bdz_groupname 000000c0 00000100 64 bdz_uid 00000100 00000140 64 bdz_username 00000140 00000180 64 idx_gid2group 00000180 00000240 192 idx_groupname2group 00000240 00000300 192 idx_uid2user 00000300 00000380 128 idx_name2user 00000380 00000400 128 shell_index 00000400 00000440 64 shell_blob 00000440 00000480 64 groups 00000480 00000700 640 users 00000700 000009c0 704 groupmembers 000009c0 00000a00 64 additional_gids 00000a00 00000a40 64 Run and configure a test container that uses `turbonss` instead of the default `files`: $ docker run -ti --rm -v `pwd`:/etc/turbonss -w /etc/turbonss debian:bullseye # cp zig-out/lib/libnss_turbo.so.2 /lib/x86_64-linux-gnu/ # sed -i '/passwd\|group/ s/files/turbo/' /etc/nsswitch.conf And run the commands: $ getent passwd $ getent group $ id root More users and groups --------------------- `turbonss-makecorpus` can synthesize more `users` and `groups`: # ./zig-out/bin/turbonss-makecorpus wrote users=10000 groups=10000 avg-members=1000 to . # cat group >> /etc/group # cat passwd >> /etc/passwd # time id u_1000000 <...> real 0m17.380s user 0m13.117s sys 0m4.263s 17 seconds for an `id` command! Well, there are indeed many users and groups. Let's see how turbonss fares with it: # zig-out/bin/turbonss-unix2db --group /etc/group --passwd /etc/passwd total 10968512 bytes. groups=10019 users=10039 # ls -hs /etc/group /etc/passwd db.turbo 48M /etc/group 668K /etc/passwd 11M db.turbo # sed -i '/passwd\|group/ s/files/turbo/' /etc/nsswitch.conf # time id u_1000000 real 0m0.008s user 0m0.000s sys 0m0.008s That's ~1500x improvement for the `id` command (and notice about 4X compression ratio compared to plain files). If the number of users and groups is increased by 10x (to 100k each), the difference becomes even crazier: # time id u_1000000 <...> real 3m42.281s user 2m30.482s sys 0m55.840s # sed -i '/passwd\|group/ s/files/turbo/' /etc/nsswitch.conf # time id u_1000000 <...> real 0m0.008s user 0m0.000s sys 0m0.008s Note that to author's knowledge this has not been used on any real production nor a development machine. Documentation ------------- - Architecture is detailed in `docs/architecture.md` - Development notes are in `docs/development.md` [nsswitch]: https://linux.die.net/man/5/nsswitch.conf [id]: https://linux.die.net/man/1/id [cmph]: http://cmph.sourceforge.net/ [dso]: https://akkadia.org/drepper/dsohowto.pdf