Nostr View

David @mleku - 1y

yay, everything is back to being groovy #replicatr now uses bugger all memory, bugger all CPU, and even when hammering it as hard as it can cop events streamed from my new upload/testing tool `blower` barely consuming one core worth of processing, and swallowing 5+Mb/s of new events without raising a sweat well, for a while, until it starts to have to do garbage collection it's purring like a cat at this point, very pleased, yes, v1.2.10 is now the tag on it nostr:npub1ky4kxtyg0uxgw8g5p5mmedh8c8s6sqny6zmaaqj44gv4rk0plaus3m4fd2 this tag should run nice i am now poring over the heap profiling to figure out if there is any way for me to eliminate some of that garbage collection work so it can suck up events over the wire without blinking #devstr #devdiary #replicatr #relaydev #deardiary

1
0
3

David @mleku - 1y

yeah, it really does look like the only way is to write a freelist, the heap profile literally everything is from badger's memory allocator, so the only possible way to make it better is to batch the event writes, maybe event reads, idk, i think that is probably fairly fast and sleek already, so, meh, whatever gonna be looking closely anyhow, there must be some ways to trim it down better i think partly it's just the database compacting records whtile its idle

1
0
3

David @mleku - 1y

gonna open it back up to all and sundry see what y'alls clients does to it, and profile that as well

1
0
3

David @mleku - 1y

so, restricting the number of threads does a treat on the amount of memory and CPU usage, with max 2 threads it's under 200Mb and 16% CPU (which is one core of my 6 core ryzen 7) so far... running blower to hammer it... oh yeah now we see some memory usage but still popping out to no more than 500Mb this definitely will fit cosy into a time4vps or contabo storage VPS and store all the events and when that gets too expensive, it is not difficult to extend it to use second level databases and automatically maintain disk usage and not blow up

1
0
3

David @mleku - 1y

it just occurs to me also that the second level thing could become an interface to forward queries to other relays, and a #layer2 could be written that even polls at L2 multiple other relays and caches the requests well, more things for later work, but nostr:npub180cvv07tjdrrgpa0j7j7tmnyl2yr6yr7l8j4s3evf6u64th6gkwsyjh6w6 you probably didn't think of this but you could write an eventstore that is just a proxy for other relays

1
0
3

David @mleku - 1y

with #replicatr it can be a caching proxy 😛

1
0
3

David @mleku - 1y

in fact, a cloud of #replicatr relays could cache quite a large amount of events but only cache the ones that are relevant to the clients, and randomly select a slow shifting set of peer relays similar to how #bitcoin p2p network maintains connectivity, it would just require auth so the relays can control the amount of traffic within the bounds of their budget but it does suggest a very cool way to distribute events across the network... and if you have some archivists in your cluster the whole network will maintain availability to the whole history at 1 degree of separation most of the time the databases are slower when they are bigger so dividing up the work would really be a serious optimization for the protocol

0
0
3
Showing page 1 of 1 pages