From 6027508e60a66631f4b066dd838b2571eeb9a2f2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Motiejus=20Jak=C5=A1tys?= Date: Thu, 13 Apr 2023 11:10:47 +0300 Subject: [PATCH] update README --- README.md | 217 +++++++++++++++++++++++++++++------------------------- 1 file changed, 117 insertions(+), 100 deletions(-) diff --git a/README.md b/README.md index fbd0f9e..19f170e 100644 --- a/README.md +++ b/README.md @@ -21,41 +21,52 @@ milliseconds with a pre-cached `nscd`, ~8 milliseconds with uncached Due to the nature of being built with Zig, this will work on glibc versions as old as 2.16 (may work with even older ones, I did not test beyond that). -Project status --------------- +Project goals +------------- -This project works, but has never seen real production use. If you want to use -turbonss instead of a battle-tested, albeit slower nscd, keep the following in -mind: -- turbonss has not been fuzzed, so it will crash a program on invalid database - file. +- Make it as fast as possible. Especially optimize for the `id` command. +- Small database size (helps making it fast). +- No runtime, no GC, as little as possible overhead. +- Easy to compile for ancient glibc versions (comes out of the box with Zig). + +Project status and known deficiencies +------------------------------------- + +Turbonss works, but, to the author's knowledge, was not deployed to production. +If you want to use turbonss instead of a battle-tested, albeit slower nscd, +keep the following in mind: +- turbonss has not been fuzz-tested, so it will crash a program on invalid + database file. Please compile with `ReleaseSafe`. It is plenty fast with this + mode, but an invalid database will lead to defined behavior (i.e. crash with + a stack trace) instead of overwriting memory wherever. +- if the database file was replaced while the program has been running, + turbonss will not re-read the file (it holds to the previous file + descriptor). - requires a nightly version of zig (that will change with 0.11). -If you insist on using turbonss in prod, compile with `ReleaseSafe`. It is -plenty as fast with this mode, but an invalid database will lead to defined -behavior (i.e. crash with a stack trace) instead of overwriting memory -wherever. +The license is permissive, so feel free. I am also available for +[consulting][consulting] to fix all or part of those above, if that's your +preference. Dependencies ------------ -1. zig around 0.11.0-dev.2560+602029bb2. +1. zig around `0.11.0-dev.2560+602029bb2`. 2. [cmph][cmph]: bundled with this repository. -Trying it out -------------- +Demo +---- Clone, compile and test first: - $ git clone --recursive https://git.sr.ht/~motiejus/turbonss + $ git clone --recursive https://git.jakstys.lt/motiejus/turbonss $ zig build test - $ zig build -Dtarget=x86_64-linux-gnu.2.16 -Dcpu=baseline -Doptimize=ReleaseSafe + $ zig build -Dtarget=x86_64-linux-gnu.2.16 -Doptimize=ReleaseSafe One may choose different options, depending on requirements. Here are some hints: -1. `-Dcpu=<...>` for the CPU - [microarchitecture](https://en.wikipedia.org/wiki/X86-64#Microarchitecture_levels). +1. `-Dcpu=<...>` for the CPU [microarchitecture][mcpu]. 2. `-Dstrip=true` to strip debug symbols. For reference, size of the shared library and helper binaries when compiled @@ -69,102 +80,106 @@ with `-Dstrip=true -Drelease-small=true`: Many thanks to Ulrich Drepper for [teaching how to link it properly][dso]. -Test turobnss on a real system ------------------------------- +Quick test turbonss on a real system +------------------------------------ -`db.turbo` is the TurboNSS database file. To create one from `/etc/group` and -`/etc/passwd`, use `turbonss-unix2db`: +turbonss is best tested, of course, with many users and groups. The guide below +will show how to synthesize 10k users, 10k groups with an avereage membership +of 1k users per group, and test ubernss with such corpus. - $ zig-out/bin/turbonss-unix2db --passwd /etc/passwd --group /etc/group - $ zig-out/bin/turbonss-analyze db.turbo - File: /etc/turbonss/db.turbo - Size: 2,624 bytes - Version: 0 - Endian: little - Pointer size: 8 bytes - getgr buffer size: 17 - getpw buffer size: 74 - Users: 19 - Groups: 39 - Shells: 1 - Most memberships: _apt (1) - Sections: - Name Begin End Size bytes - header 00000000 00000080 128 - bdz_gid 00000080 000000c0 64 - bdz_groupname 000000c0 00000100 64 - bdz_uid 00000100 00000140 64 - bdz_username 00000140 00000180 64 - idx_gid2group 00000180 00000240 192 - idx_groupname2group 00000240 00000300 192 - idx_uid2user 00000300 00000380 128 - idx_name2user 00000380 00000400 128 - shell_index 00000400 00000440 64 - shell_blob 00000440 00000480 64 - groups 00000480 00000700 640 - users 00000700 000009c0 704 - groupmembers 000009c0 00000a00 64 - additional_gids 00000a00 00000a40 64 +1. Synthesize some users and groups to `passwd` and `group` in the current directory: -Run and configure a test container that uses `turbonss` instead of the default -`files`: + ``` + $ zig-out/bin/turbonss-makecorpus + wrote users=10000 groups=10000 avg-members=1000 to . + $ ls -1hs passwd group + 48M group + 668K passwd + ``` - $ docker run -ti --rm -v `pwd`:/etc/turbonss -w /etc/turbonss debian:bullseye - # cp zig-out/lib/libnss_turbo.so.2 /lib/x86_64-linux-gnu/ - # sed -i '/passwd\|group/ s/files/turbo/' /etc/nsswitch.conf +2. Convert the generated `passwd` and `group` to the turbonss database. Note + the `db.turbo` database is more than 4 times smaller than the textual one: -And run the commands: + ``` + $ zig-out/bin/turbonss-unix2db --group group --passwd passwd + total 10968064 bytes. groups=10000 users=10000 + $ ls -1hs db.turbo + 11M db.turbo + ``` - $ getent passwd - $ getent group - $ id root +3. Optional: inspect the freshly created database: -More users and groups ---------------------- + ``` + $ zig-out/bin/turbonss-analyze db.turbo + File: db.turbo + Size: 10,968,064 bytes + Version: 0 + Endian: little + Pointer size: 8 bytes + getgr buffer size: 18000 + getpw buffer size: 57 + Users: 10000 + Groups: 10000 + Shells: 4 + Most memberships: u_1000000 (501) + Sections: + Name Begin End Size bytes + header 00000000 00000080 128 + bdz_gid 00000080 00000e40 3,520 + bdz_groupname 00000e40 00001c00 3,520 + bdz_uid 00001c00 000029c0 3,520 + bdz_username 000029c0 00003780 3,520 + idx_gid2group 00003780 0000d3c0 40,000 + idx_groupname2group 0000d3c0 00017000 40,000 + idx_uid2user 00017000 00020c40 40,000 + idx_name2user 00020c40 0002a880 40,000 + shell_index 0002a880 0002a8c0 64 + shell_blob 0002a8c0 0002a900 64 + groups 0002a900 00065280 240,000 + users 00065280 000da580 480,000 + groupmembers 000da580 005a69c0 5,030,976 + additional_gids 005a69c0 00a75c00 5,042,752 + $ zig-out/bin/turbonss-getent --db db.turbo passwd u_1000000 + u_1000000:x:1000000:1000000:User 1000000:/home/u_1000000:/bin/bash + $ zig-out/bin/turbonss-getent --db db.turbo group g_1000003 + g_1000003:x:1000003:u_1000002,u_1000003,u_1000004 + ``` -`turbonss-makecorpus` can synthesize more `users` and `groups`: +4. Now since we will be messing with the system, run all following commands in + a container: - # ./zig-out/bin/turbonss-makecorpus - wrote users=10000 groups=10000 avg-members=1000 to . - # cat group >> /etc/group - # cat passwd >> /etc/passwd - # time id u_1000000 - <...> - real 0m17.380s - user 0m13.117s - sys 0m4.263s + ``` + $ docker run -ti --rm -v `pwd`:/etc/turbonss -w /etc/turbonss debian:bullseye + # cp zig-out/lib/libnss_turbo.so.2 /lib/x86_64-linux-gnu/ + ``` -17 seconds for an `id` command! Well, there are indeed many users and groups. -Let's see how turbonss fares with it: +5. Instruct `nsswitch.conf` to use both turbonss and the standard resolver: - # zig-out/bin/turbonss-unix2db --group /etc/group --passwd /etc/passwd - total 10968512 bytes. groups=10019 users=10039 - # ls -hs /etc/group /etc/passwd db.turbo - 48M /etc/group 668K /etc/passwd 11M db.turbo - # sed -i '/passwd\|group/ s/files/turbo/' /etc/nsswitch.conf - # time id u_1000000 - real 0m0.008s - user 0m0.000s - sys 0m0.008s + ``` + # sed -i '/passwd\|group/ s/files/turbo files/' /etc/nsswitch.conf + # time id u_1000000 + <...> + real 0m0.006s + user 0m0.000s + sys 0m0.008s + ``` -That's ~1500x improvement for the `id` command (and notice about 4X compression -ratio compared to plain files). If the number of users and groups is increased -by 10x (to 100k each), the difference becomes even crazier: + The `id` call resolved `u_1000000` from `db.turbo`. - # time id u_1000000 - <...> - real 3m42.281s - user 2m30.482s - sys 0m55.840s - # sed -i '/passwd\|group/ s/files/turbo/' /etc/nsswitch.conf - # time id u_1000000 - <...> - real 0m0.008s - user 0m0.000s - sys 0m0.008s +6. Compare the performance to plain `files` (that is, without turbonss): -Note that to author's knowledge this has not been used on any real production -nor a development machine. + ``` + # sed -i '/passwd\|group/ s/turbo files/files/' /etc/nsswitch.conf + # cat passwd >> /etc/passwd + # cat group >> /etc/group + # time id u_1000000 + <...> + real 0m17.164s + user 0m13.288s + sys 0m3.876s + ``` + + Over 2500x difference. Documentation ------------- @@ -176,3 +191,5 @@ Documentation [id]: https://linux.die.net/man/1/id [cmph]: http://cmph.sourceforge.net/ [dso]: https://akkadia.org/drepper/dsohowto.pdf +[mcpu]: https://en.wikipedia.org/wiki/X86-64#Microarchitecture_levels +[consulting]: https://jakstys.lt/contact