#realy now has a tested working garbage collector functionality, previously it was scratched in from previous work but i wrote a revised tester that fits in with the new schema of everything in realy
it is the afternoon and i have a little work to do now to update an Internet Computer #layer2 eventstore implementation, that allows relay operators who somehow like ICP to run a relay that uses a sharded blockchain database to store events
it is of course of interest to said shitcoiners, and good for them, and you know what i mean when i say good for you
my life would be misery without my bad jokes so also, good for you, directly
but it's also nice for people who want to run a plain and simple realy instance, now you can contain the event storage with typical low/high water mark cache management options and your specified gigabyteage with just one little .env file
and did i mention that it's also got the fastest JSON encoder and event store database out there?
some day in the future, when i have more time on my hands, i will build a test rig that lets me really see realy head to head with #strfry and i'm almost certain that realy is not gonna be far behind it, and probably, on a parity hardware basis, better
good for hoyte and his C++ lunacy
my main paid gig at the moment most of my time on the clock i'm learning a new language and runtime API (for sui and aptos blockchains) ... which is really boring but i gotta get up to speed because i'm sure i can do this stuff, it's simple, andd tedious, like i say - "smart contracts" are just database scripts, so i just gotta grit my teeth and do this boring shit that will involve no high performance, no significant opportunities for optimization or anything actually fun really
i'm assigned to maintaining a port to a second chain with the same language, and right off the bat this stupid thing is getting in my face because when i have the plugins for both "languages" enabled it breaks my goland, as i discovered just now... so much yawn... they invent these new languages and then make nowhere near the commensurate effort to make it easy to work with their bullshit
anyhow
so, what am i doing?
yes, working on #realy right now - and currently i've modified my codec tester to just print out the events that fail somewhere in the process of encoding/decoding to thin out the field... rn ~10k events failing to encode/decode to binary or possibly just decode from json, out of ~250k..., and about 300mb so my codec is failing on what the json linter in the IDE is happy with, so my code is wrong
there's about 4 or 5 different categories of errors popping out of this, and anyway, the point is if i just let it run and clean out 180gb of events , will thin it down to probably 200-300mb of raw events, which will speed up subsequent work to debug what is causing these decode/encode failures
trying to debug the current first bug that it hits is a complicated tag encoding bug and all the noise in the logs makes it harder to focus on it
so, i figure, collect all the ones that are not working, then i can thin those out, try to find the lowest hanging fruit and thin it out even further sooner... i want to get this codec working 100% correct and to eventually have it so that anything it doesn't like is invalid JSON... and it's already just rejecting and not logging anything that has linebreaks anyway, they all fail with EOFs
i wish i could focus on getting this right, but i understand that nostr is still so new it's extremely speculative that it will ever have a market footprint and that's fine, but it's what i love working with and what i love to do with what i love working with is making it as correct, and as efficient as possible
when i finally get this codec all fixed, and it only rejects garbage, i can finally start tinkering with other things i want to do... i'm very experienced with binary encoding and network transports and error correction so it will be fundamental stuff, and it will make a big difference but it isn't gonna win me any fame
i'm working in the bowels of the factory, i'm not the courier delivering it to the customer
#realy installation is simple
install go 1.23 and git
git clone https://github.com/mleku/realy.git
cd realy
for the simplest but not as performant
export CGO_ENABLED=0
cd cmd/realy
go build .
this produces the binary in the current directory, then move the binary `realy` where you want, eg /usr/local/bin
realy env
spits out the default configuration (and when customised, the current configuration)
mkdir ~/.realy
realy env > ~/.realy/.env
these are the expected default locations for the data directory and configuration file
use your editor (nano i guess would be most people, maybe vi/vim for some) to edit, it's just a set of key/value pairs in standard env style
you can move it to another directory if you must, but you have to invoke it with the data directory environment variable set in the command, eg:
ROOT_DIR=/path/to/where/you/want/it PROFILE=.profileFolderNameCanBeAnyValidDirectoryName realy
this will then make realy look in that place to load the .env file (it's always called .env though, sorry #notsorry
for more information, see https://realy.lol it will explain how to disable CGO as well as, from the main readme.md, direct you to the instructions on how to build the bitcoin/secp256k1 signature library to get much better event signature verification performance
#realy uses an event store based on fiatjaf's eventstore/badger but much less resource hungry, and using the fastest #golang binary event encoder i am aware of, there is no extra configuration, no docker, no environment, #nobullshit
it is, in my 8 years experience at Go development a reasonably well written, maybe could be better documented codebase, and it should not be difficult, if you have the will, to learn Go and to modify this relay to do whatever you like, and please do ping me if you create featurse for it, i am happy to integrate them into the mainline with a suitable, consistent interface for configuring to use it (such as how it currently is a no-auth open public relay by default currently, but if you set AUTH_REQUIRED it requires nip-42 auth, and if you put your npub in OWNERS it will allow everyone on your follow list, once it's published to the relay, to publish notes to your relay and use it as an outbox
#devstr #progressreport w00t, i've now got the queries bumping their last access time counters in #realy
now i can add back the garbage collector... first the counter, which collects all of the events, their data size (including indexes), and last access time, this list then has a sorter, which sorts them by their access time
then the mark function, which then goes in ascending order (oldest) and selects the list of events that exceed the low water mark target (the GC triggers when the count exceeds the high water mark, and we mark the ones that would bring it to the first event under the low water mark)
then finally with all of the events selected that we are going to delete, we run the "sweep" function which deletes the events that are the least recently accessed that give us our low water mark target, and voila
this is IMO an essential feature to put on a database that can potentially grow very large, because there really is no other way to deal with containing the event database size to the available storage on your server without this, IMHO this is a mandatory feature for a relay, but afaik no other relay (except my buggy previous version, replicatr) actually has this...
i could be wrong...
probably many other database drivers have access time marking but idk if anyone has bothered to do this because the most popular relay, strfry, is used by hipster damus boi who resolves this issue by his periodic nukenings
realy will have its own periodic nukening automatically so you don't have this issue of hot data going dark like a nukening causes
#realy #progressreport
it appears that i have finally squashed all the most salient causes of forever loops in my event query code
i have also implemented a separate count function that avoids decoding events if it can, so it performs better, it returns an approximate flag if it finds replaceable events because it is planned to not actually delete them, but i haven't written the post-processing in the query to enable this or removed the delete from replacement on event save (coming soon, on request from my shadowy super-nostr-dev friend who needs this for his work)
amethyst's endless stream of replaceable events helped expose more of the issues in the query and save for replaceable events, which was an edge case that neither nostrudel nor coracle triggered, this also now does use a bit of memory when it happens but it's quickly freed within the following minute (on more constrained hardware with less memory - eg 1gb - this might cause a brief spike of CPU usage to manage the more aggressive garbage collection - nothing i can really do about that...)
realy now uses 1-2gb of memory most of the time now, closer to 1gb
i think it's getting very close to release candidate, which will be a minor version bump to v1.1.x
there has been quite a few breaking changes, but i don't think anyone is importing it yet so 😅 whew
when it's bumped to v1.1.x i'm going to probably deploy it remotely on my VPS server
ah yes, i have other work to do today but the other major features that will be part of the v1.1.x will be a working garbage collector, a framework for shared/remote second layer event stores (even algaefs or blossom potentially could be implemented) and i also want to get around to making a badger database driver for btcd... too many things to do
better check my calendar and make sure i'm not slacking on someone haha
Showing page 1 of
2 pages
