#realy #devstr #progressreport
so, now i am up to the stage of actually writing a search function, i realise that the fulltext indexes need a lot more data in them
the fulltext index will have all this :
[ 15 ][ word ][ 32 bytes eventid.T ][ 32 bytes pubkey ][ 8 bytes timestamp.T ][ 2 bytes kind ][ 4 bytes sequence number of word in text ][ 8 bytes Serial ]
with this as the result from the word query, i can now then filter out by timestamps, pubkeys and kind immediately upon getting them, and with the sequence number in the text i can quickly determine sequence similarity by the words being in query order and then compared by the number of them that are in the same sequence
with all of these extra fields in the keys, almost all of the necessary filtering required for a fulltext search can be done effectively, and the amount of data in the index is not so great but the main thing is that the word matches can be found immediately, filtered by their timestamp and kind, grouped by their common presence in an event, and then sorted in their sequence order within the event.
it is a lot of data but all of these fields are needed to evaluate and without them being in one index i would make a lot more queries and decoding to find them, and this trades off some extra size in the database for speed at finding, filtering and sorting matches
gonna need to make a specific data structure for it, well, probably a set of functions that pick out the field and decode it, that would be more efficient since i can then quickly jump to the field that needs to be filtered according to the other, non-fulltext part of the search, eg, the kinds, the pubkey, the timestamp range... not really exactly sure which of those the process should be ordered in... probably pubkey, kind, timestamp, in that order, depending on the presence of those in the filter
this may take longer than i hoped but i honestly didn't really have any idea what would be involved... at day 3 working on this now, and i only have the index writing done, but with this scheme designed i should progress quickly to a working search function
Showing page 1 of
1 pages
