so, i am progressing in my understanding of how to program key value stores
i had a scheme set up, where for a second level caching scheme, there was an access counter record with also the size of the event itself, and i was zeroing that size as well as flipping the event record for its ID
this turned out to be a problem somehow, i can't quite figure out how, but eventually the mark step of the garbage collection code was marking already deleted events and for several days i have been fighting with this and got one small victory yesterday when i slimmed it down a bit
i just was deleting the counter record, then it wasn't found in the mark step and voila, no more sweeps of already swept items
so, i've learned something here, that is a reflection of other learning about simplicity
every key in a key value store should be simple, one key, one value
it is a code smell when your value is composite, but divisible, ie, in my example, the access timestamp AND the storage size of the event
so, tomorrow i'm revising the code so the access counter is just a timestamp, and the event itself signifies having been pushed out to a second level store by flipping the whole event for the event id (hash)
just as previously, removing the counter event solved the problem, i expect this will solve the problem, as now the access counter only does one thing: update it every time it comes up in a query
and the event itself is the primary source of "is this now pushed out to the level 2 store?"
some time in the future i am going to write a go-native style key value store wrapper around badger that lets you abstract out all the codec parts of the code so you just store keys that are based on structs and define them with simple encoders that turn things like hashes into 8 byte prefix codes and such
don't have the time to build out the nice clean interface for now but i've learned something in the last week of pain, and i will definitely be using this hard won knowledge in the future
#devstr #golang #keyvalue #deardiary #programming #nostr
#golang only
fuck python, c++, java, c#, vlang, C, D, F, pascal, javascript, all the rest there's too many crimes to mention
vlang has a stupid mutability thing like rust, which is a total waste of time because most of the time you need mutability and making that necessary to specify all the time is retarded
the thing i liked about it was they made it a simpler syntax and so it compiles a lot faster than rust
i just don't see any reason to bother with any other language... in 2018 i became a #golang maxi and like how i became a #bitcoin maxi in 2021 it's a one way change, i chafe at even having to include other languages in my project at all
there just is no reason for it, and the irritation at all the other bullshit of other languages, the compile times, the syntax ambiguity, the ugly appearance, retarded idiom, whatever else... i'm impatient and i just don't see why i should wait when i don't have to
when i compile Go code it runs under 2 seconds later, every time except the very first for a big codebase, then maybe i wait 10 seconds
no other language has optimized for this kind of ergonomics
i was just checking out the configuration options for #golang #badger key/value store, and hadn't actually looked that close before, but here's some useful facts to understand about how it is implemented:
https://github.com/dgraph-io/badger/blob/main/options.go#L180
it uses mmaped files, the default allocates a 2Gb value log and a ~130Mb key log file
this is different to any other k/v store because these logs are kept separated
also by default it uses 4kb blocks and snappy compression, but this can be cranked up to ZSTD - by the comments though it says that this is now a native Go implementation rather than CGO which it used to be, i presume that the benchmarks are probably somewhere in range even still, and that these figures are per CPU thread
probably can increase those figures, larger blocks and higher compression rates to squeeze a lot more events
been idly thinking a bit about the idea of creating a simple event store that keeps them in a simple table with the ID as key, and then the canonical form, which generates the hash, and the signature bytes afterwards, for implementing a #blossom data store
anyhow, just random thoughts, back to work!
there is soooo much work yet to be done, and so much polish on what work has been done so far.... i've been focusing on what matters to me, the #golang libraries and relay implementations from the fiatjaf
because they are the most complete of all of any language and because they are in my preferred language, and the language you will also prefer eventually (partly because pretty soon they are gonna be the hottest, fastest, and sleekest implementations of nostr protocol that exists, and you're welcome...
i am funded via a company that is working on making decentralised competitors for matrix, discord, telegram, gdocs, proton, and so on and we get our funds from whoever we can see a strategic relationship that links to the work we want to deploy devs, managers of various kinds and artists on working on... my current project is being funded by dfinity, the builders of internet computer protocol, and maybe we will get funding from arweave soon to build out advanced full text search capability in clients and relays (and arweave smart contracts of course)
we are nearly 15 months in since inception and maybe in 3 months time we will finally have a decent reference implementation for full text search of text in events
i'm sure that there will be more outside funding going forward but it may take some dirty work
for me, to build in #golang but to be funded by shitcoins is a zero sum, so it's all about the benefit of being able to help advance the tech behind #nostr
there will be more people coming into this in a similar way as me but i think our fearless leader, who is a fan of nostr, has been very early on this and sees the benefit of an interoperable protocol for competitors for the incumbent business collaboration systems, and a means to unifying these systems to become a seamless internet of collab systems, which hopefully his enterprise will pay off well enough to keep him moving forwards towards such worthwhile goals
looking forward to first release, won't be long, another 3 weeks we'll kick this beast out the door
had a really hard problem with the database interface i have been building from (fiatjaf's eventstore/badger) to add storage management capabilities but pretty soon it's gonna have full capability to both manage a badger store alone, or to combine it with an external distributed database (ICP in this case but it shouldn't take much to be blossom or ipfs or mongo or redis)
Showing page 1 of
5 pages
