1
Fork 0
NSS plugin for passwd and groups databases
 
 
Go to file
Motiejus Jakštys fbd449b21f Move docs around; finish it 2022-08-21 06:08:21 +03:00
contrib remove contrib/passwd-assign0.awk 2022-07-27 07:48:14 -07:00
deps use bdz_read.c 2022-07-09 18:15:04 +03:00
docs Move docs around; finish it 2022-08-21 06:08:21 +03:00
include/deps/cmph compile cmph from source 2022-02-10 06:07:52 +02:00
src Move docs around; finish it 2022-08-21 06:08:21 +03:00
.gitignore gitignore db.turbo 2022-07-11 15:36:00 +03:00
.gitmodules add deps/cmph 2022-02-09 13:08:25 +02:00
COPYING add a license 2022-02-18 07:16:02 +02:00
README.md Move docs around; finish it 2022-08-21 06:08:21 +03:00
build.zig add turbonss-makecorpus 2022-07-29 10:29:27 -07:00

README.md

Turbo NSS

Turbonss is a plugin for GNU Name Service Switch (NSS) functionality of GNU C Library (glibc). Turbonss implements lookup for user and passwd database entries (i.e. system users, groups, and group memberships). It's main goal is to run id(1) as fast as possible.

Turbonss is optimized for reading. If the data changes in any way, the whole file will need to be regenerated. Therefore, it was created, and best suited, for environments that have a central user & group database which then needs to be distributed to many servers/services, and the data does not change very often (e.g. hourly).

This is the fastest known NSS passwd/group implementation for reads. On a corpus with 10k users, 10k groups and 500 average members per group, id takes 17 seconds with the glibc default implementation, 10-17 milliseconds with a pre-cached nscd, ~8 milliseconds with turbonss.

Project status

The project is finished and is not recommended for production; just use nscd. Turbonss duly implements the full user/group API in src/libnss.zig: feel free to copy that.

Yours truly (the author) worked on this for about 7 months. And when this was finished it turned out that just slapping nscd on top of the existing NSS implementation is almost as fast as this.

Dependencies

  1. Stage1 of the nightly zig compiler.
  2. cmph: bundled with this repository.

Trying it out

Clone, compile and test first:

$ git clone --recursive https://git.sr.ht/~motiejus/turbonss
$ zig build -fstage1 test
$ zig build -fstage1 -Dtarget=x86_64-linux-gnu.2.31 -Dcpu=x86_64_v3 -Drelease-safe=true

One may choose different options, depending on requirements. Here are some hints:

  1. -Dcpu=<...> for the CPU microarchitecture.
  2. -Drelease-fast=true for max speed
  3. -Drelease-small=true for smallest binary sizes.
  4. -Dstrip=true to strip debug symbols.

Test it on a real system

db.turbo is the TurboNSS database file. To create one from /etc/group and /etc/passwd, use turbonss-unix2db:

$ zig-out/bin/turbonss-unix2db --passwd /etc/passwd --group /etc/group
$ zig-out/bin/turbonss-analyze db.turbo
<...>

Run and configure a test container that uses turbonss instead of the default files:

$ docker run -ti --rm -v `pwd`:/etc/turbonss -w /etc/turbonss debian:bullseye
# cp zig-out/lib/libnss_turbo.so.2 /lib/x86_64-linux-gnu/
# sed -i '/passwd\|group/ s/files/turbo/' /etc/nsswitch.conf

And run the commands:

$ getent passwd
$ getent group
$ id root

More users and groups

turbonss-makecorpus can synthesize more users and groups:

# ./zig-out/bin/turbonss-makecorpus 
wrote users=10000 groups=10000 avg-members=1000 to .
# cat group >> /etc/group
# cat passwd >> /etc/passwd
# time id u_1000000
<...>
real    0m17.380s
user    0m13.117s
sys     0m4.263s

17 seconds for an id command! Well, there are indeed many users and groups. Let's see how turbonss fares with it:

# zig-out/bin/turbonss-unix2db --group /etc/group --passwd /etc/passwd
total 10968512 bytes. groups=10019 users=10039
# ls -hs /etc/group /etc/passwd db.turbo
48M /etc/group  668K /etc/passwd   11M db.turbo
# sed -i '/passwd\|group/ s/files/turbo/' /etc/nsswitch.conf
# time id u_1000000
real    0m0.008s
user    0m0.000s
sys     0m0.008s

That's ~1500x improvement for the id command (and notice about 4X compression ratio compared to plain files). If the number of users and groups is increased by 10x (to 100k each), the difference becomes even crazier:

# time id u_1000000
<...>
real    3m42.281s
user    2m30.482s
sys     0m55.840s
# sed -i '/passwd\|group/ s/files/turbo/' /etc/nsswitch.conf
# time id u_1000000
<...>
real    0m0.008s
user    0m0.000s
sys     0m0.008s

Documentation

Architecture is detailed in docs/architecture.md Development notes are in docs/development.md