Turbo NSS

Turbonss is a plugin for GNU Name Service Switch (NSS) functionality of GNU C Library (glibc). Turbonss implements lookup for user and passwd database entries (i.e. system users, groups, and group memberships). It's main goal is to run id(1) as fast as possible.

Turbonss is optimized for reading. If the data changes in any way, the whole file will need to be regenerated. Therefore, it was created, and best suited, for environments that have a central user & group database which then needs to be distributed to many servers/services, and the data does not change very often.

This is the fastest known NSS passwd/group implementation for reads. On my 2018-era laptop a corpus with 10k users, 10k groups and 500 average members per group, id takes 17 seconds with the glibc default implementation, 10-17 milliseconds with a pre-cached nscd, ~8 milliseconds with uncached turbonss.

Due to the nature of being built with Zig, this will work on glibc versions as old as 2.16 (may work with even older ones, I did not test beyond that).

Project status

This project works, but has never seen real production use. If you want to use turbonss instead of a battle-tested, albeit slower nscd, keep the following in mind:

• turbonss has not been fuzzed, so it will crash a program on invalid database file.
• requires a nightly version of zig (that will change with 0.11).

If you insist on using turbonss in prod, compile with ReleaseSafe. It is plenty as fast with this mode, but an invalid database will lead to defined behavior (i.e. crash with a stack trace) instead of overwriting memory wherever.

Dependencies

1. zig around 0.11.0-dev.2560+602029bb2.
2. cmph: bundled with this repository.

Trying it out

Clone, compile and test first:

$ git clone --recursive https://git.sr.ht/~motiejus/turbonss
$ zig build test
$ zig build -Dtarget=x86_64-linux-gnu.2.16 -Dcpu=baseline -Doptimize=ReleaseSafe

One may choose different options, depending on requirements. Here are some hints:

1. -Dcpu=<...> for the CPU microarchitecture.
2. -Dstrip=true to strip debug symbols.

For reference, size of the shared library and helper binaries when compiled with -Dstrip=true -Drelease-small=true:

  28K zig-out/bin/turbonss-analyze
  20K zig-out/bin/turbonss-getent
  24K zig-out/bin/turbonss-makecorpus
 140K zig-out/bin/turbonss-unix2db
  24K zig-out/lib/libnss_turbo.so.2.0.0

Many thanks to Ulrich Drepper for teaching how to link it properly.

Test turobnss on a real system

db.turbo is the TurboNSS database file. To create one from /etc/group and /etc/passwd, use turbonss-unix2db:

$ zig-out/bin/turbonss-unix2db --passwd /etc/passwd --group /etc/group
$ zig-out/bin/turbonss-analyze db.turbo
File:               /etc/turbonss/db.turbo
Size:               2,624 bytes
Version:            0
Endian:             little
Pointer size:       8 bytes
getgr buffer size:  17
getpw buffer size:  74
Users:              19
Groups:             39
Shells:             1
Most memberships:   _apt (1)
Sections:
    Name                 Begin    End          Size bytes
    header               00000000 00000080            128
    bdz_gid              00000080 000000c0             64
    bdz_groupname        000000c0 00000100             64
    bdz_uid              00000100 00000140             64
    bdz_username         00000140 00000180             64
    idx_gid2group        00000180 00000240            192
    idx_groupname2group  00000240 00000300            192
    idx_uid2user         00000300 00000380            128
    idx_name2user        00000380 00000400            128
    shell_index          00000400 00000440             64
    shell_blob           00000440 00000480             64
    groups               00000480 00000700            640
    users                00000700 000009c0            704
    groupmembers         000009c0 00000a00             64
    additional_gids      00000a00 00000a40             64

Run and configure a test container that uses turbonss instead of the default files:

$ docker run -ti --rm -v `pwd`:/etc/turbonss -w /etc/turbonss debian:bullseye
# cp zig-out/lib/libnss_turbo.so.2 /lib/x86_64-linux-gnu/
# sed -i '/passwd\|group/ s/files/turbo/' /etc/nsswitch.conf

And run the commands:

$ getent passwd
$ getent group
$ id root

More users and groups

turbonss-makecorpus can synthesize more users and groups:

# ./zig-out/bin/turbonss-makecorpus 
wrote users=10000 groups=10000 avg-members=1000 to .
# cat group >> /etc/group
# cat passwd >> /etc/passwd
# time id u_1000000
<...>
real    0m17.380s
user    0m13.117s
sys     0m4.263s

17 seconds for an id command! Well, there are indeed many users and groups. Let's see how turbonss fares with it:

# zig-out/bin/turbonss-unix2db --group /etc/group --passwd /etc/passwd
total 10968512 bytes. groups=10019 users=10039
# ls -hs /etc/group /etc/passwd db.turbo
48M /etc/group  668K /etc/passwd   11M db.turbo
# sed -i '/passwd\|group/ s/files/turbo/' /etc/nsswitch.conf
# time id u_1000000
real    0m0.008s
user    0m0.000s
sys     0m0.008s

That's ~1500x improvement for the id command (and notice about 4X compression ratio compared to plain files). If the number of users and groups is increased by 10x (to 100k each), the difference becomes even crazier:

# time id u_1000000
<...>
real    3m42.281s
user    2m30.482s
sys     0m55.840s
# sed -i '/passwd\|group/ s/files/turbo/' /etc/nsswitch.conf
# time id u_1000000
<...>
real    0m0.008s
user    0m0.000s
sys     0m0.008s

Note that to author's knowledge this has not been used on any real production nor a development machine.

Documentation

• Architecture is detailed in docs/architecture.md
• Development notes are in docs/development.md