#realy #devstr #progressreport
i have got a new trick now for when i make an app that rarely does very memory/disk intensive operations like importing events, or my fiat mine job, doing an (N-1)N number of operations involving comparing all of a collection of JSON to each other (this blew up the memory on a heroku instance to just over 1gb and their server kills the process and calls it crashed
there is a manual trigger function in the "debug" library called debug.FreeOSMemory() which does this
so i just run that frequently through import and export, and voila, i can disable the swap on my VPS and realy doesn't blow up the memory.
in slightly unrelated matters, i learned about zigzag matrix traversal, and just now found this:
https://algocademy.com/blog/matrix-traversal-mastering-spiral-diagonal-and-zigzag-patterns/
spiral, diagonal and zigzag are three common methods that are used to optimize iterations on specific types of operation
in my fiat mine comparison matrix, the optimization is about reducing the number of decode operations on the JSON of the entries at each point in the matrix. the optimization that helps here is where the the path is such that you can cache some number of pre-decoded values from each of the elements of the array (which is being compared exhaustively)
spiral wouldn't do the trick, because it only saves you decoding one again for each circuit
diagonal would be no different from left-right-top-bottom path (scanline pattern)
zigzag is like diagonal except instead of starting again from the the same side, you move to the next non-visited node and then take a reverse diagonal
with zigzag iteration i will be able to tune the number of decoded elements in each comparison to reduce the number of times the data has to be marshaled again to be compared
i will likely make it a divide and conquer multi level zigzag too depending on the number of entries that need to have this comparison, that will traverse in a scanline pattern because that means that all of one of the segments in the grid i cut the matrix into will stay hot the whole time it passes left-right-top-bottom simple iteration
i am aiming to have a comparison algorithm that can efficiently and quickly render the results of such a comparison of all elements with each other in an array, to 100k and beyond, without needing tens of gigabytes of memory to do the job
it will also use a sync.Pool for this purpose, that keeps a limited size cache of the decoded elements, and when the cache reaches that size it will discard some percentage of the least recently used elements
probably i can concurrently run it too, so like zigzag of sub-matrixes left to right, if i can use 4 cores then i could run a grid in 4 sections and zigzag each section in parallel and then move down to the next row and so on until the computation is complete
Showing page 1 of
1 pages
