From kstars-devel Tue Oct 16 00:25:40 2007 From: Jason Harris Date: Tue, 16 Oct 2007 00:25:40 +0000 To: kstars-devel Subject: Re: [Kstars-devel] strategies for expanding the size of catalogs Message-Id: <200710151725.40587.jharris () 30doradus ! org> X-MARC-Message: https://marc.info/?l=kstars-devel&m=119249450028016 Hi James, On Monday 15 October 2007 16:01, James Bowlin wrote: > I think we would get the best disk performance using flat binary > files sorted by brightness as you suggest. Unfortunately binary > format will not be portable so we would probably want to generate > the binary files at install time or have separate data sets for > different architectures. Also, I don't know if C++ is designed > to let us initialize objects by just loading in memory. In C > and assembly, I am accustom to reading from disk directly into > essentially an array of structs. > I hadn't considered the non-portability issue. Probably creating the binary data files at compile time is the way to go. Someone in KDE-land must be doing this already... It would be nice if we knew ahead of time how much of the data file to read to reach magnitude X. However, I think the variable-length string names make this difficult, unless we pad each object's name data to be the same number of characters. Anyway, maybe this is not important, if we will just be reading records sequentially from the beginning of each file. > Another idea would be to use a MySQL backend but I think we can > actually get better performance out of flat binary files. > > Instead of (or in addition to) giving users a faintness magnitude > slider we could give them a slider for how much memory to use for > holding stars. The idea is that we can usually keep all of the > stars we display in RAM and only hit the disk when the user zooms > or pans (or the time changes significantly). We should avoid > hitting the disk on every draw cycle if at all possible, maybe we > should just refuse to do it except in very special situations (such > as with automatic updating turned off). > Good idea. We'll need some way to keep track of which stars are currently loaded into memory. Maybe something as simple as a bool for each HTM trixel? > A crude form of progressive drawing would be to first draw > the visible stars we have in memory then hit the disks and > trigger another draw after the new stars have been loaded. > I tried doing this when the faint magnitude was increased but > I had to give it up because the splash screen caused the main > window to gray out so the first draw with just the stars already > in memory was immediately erased. We might want to solve this > existing problem before enlarging the catalog. > > One potential show stopper would be proper motion. We currently > re-index parts (or all of) our catalog whenever the user makes > a very large change in the time. I don't know what to do about > this. Can we reindex in a separate thread to keep the UI responsive, and rely on the old index in the meantime? For most stars, the index is not going to change even if they have moved significantly. It's only those objects whose motion happens to carry them across a trixel boundary that we have to worry about. Also, we have the additional buffer that the area covered by the "visible" trixels extends beyond the rectangular boundary of the SkyMap widget. Imagine the case where the display is zoomed in on some region of the sky, and the user makes a big timestep. There is a star that was in a non-visible trixel with the old index, but after re-indexing it will be found to have moved into a visible trixel. I don't think it's terrible if that star appears some seconds after the time change is made (as might happen if we reindex in a background thread). Again, most of the stars drawn on screen won't be affected by the reindexing, so the user probably won't even notice the reindexing. This is a classic "corner case"; it's kind of frustrating to spend so much effort on something that no one will notice :) But kind of satisfying to come up with the right way to do it. regards, Jason -- Jason Harris jharris@30doradus.org _______________________________________________ Kstars-devel mailing list Kstars-devel@kde.org https://mail.kde.org/mailman/listinfo/kstars-devel