i think i just came up with a new compact data structure for variable sized integers that supports efficient search. let's say you want to encode 1000 people you follow into the social graph index using martti's encoding. you simply bucket the sorted user-id integers contiguously based on the size of its encoding. for example: u16_offset: 6 UID(u8) * 6 -> 1,2,42,53,128,255 UID(u16) * 1000 -> 1024, 4086, 65532, ... the header of the array simply stores offsets into the 2/4-byte array segments. if you want to check if a user id of varint encoding size 1 exists in the person follow list, you can linear scan through the 1 byte entries (probably faster than binary search for ~255 max entries) if the userid encodes to 2 bytes, you can jump to the 2-byte offset (stored in the array header) and then binary search through that, assuming this would be the larger part of the array. this gives you the smallest possible encoding and direct indexing... not sure if anyone has thought of this before. I love compact and fast data structures 🤓 martti already added follow/WoT indices to #nostrdb, now I'm just brainstorming space optimizations!

ok this took way too long... but #nostrdb inside damus ios/notedeck/notecrumbs now automatically counts: - reactions - quote reposts - reposts - direct replies - thread replies no complex work needs to be done in the client to display these. you can just pull them from the database in realtime. yay!