[prev in list] [next in list] [prev in thread] [next in thread]
List: kstars-devel
Subject: Re: question about kstars catalogs
From: Jasem Mutlaq <mutlaqja () ikarustech ! com>
Date: 2022-06-08 7:34:50
Message-ID: CAE0bU5kFwd4Np0DG7V8j4OPQ4LAcWNyJqHjWjVX22JhR1QS95Q () mail ! gmail ! com
[Download RAW message or body]
Thank you for the detailed response Akarsh. We documented stars handling in
the API here: https://api.kde.org/kstars/html/Stars.html
--
Best Regards,
Jasem Mutlaq
On Wed, Jun 8, 2022 at 9:28 AM Akarsh Simha <akarshsimha@gmail.com> wrote:
> Hi Vincent
>
> Thank you for your contributions to Siril, it's an excellent piece of
> software and fills a crucial need in the open-source software ecosystems.
>
>>
>
>
> Hello, I'm a developer of Siril, I'm looking at options to integrate a
>> photometric star catalog for offline operation ; we currently use NOMAD
>> and APASS from the vizier web service.
>
>
> I'm not sure exactly what your use-case here is, but please be warned that
> NOMAD was generated through algorithms running on images without human
> curation, so at least in the version used in KStars there are several
> artifacts, especially around milky way patches, diffraction spikes of
> bright stars, and cores of galaxies. Perhaps there is a better version
> released, but I haven't kept up.
>
>
>> I'm looking for more information about the kstars catalog binary format,
>> I was hoping you could give me pointers.
>
>
> The KStars star catalog binary format is documented here:
>
> https://invent.kde.org/education/kstars/-/blob/master/kstars/skycomponents/stars.dox
> (for some reason I can't find the doxygen-generated HTML on the web)
>
> I believe that the same file also has details of how the Hierarchical
> Triangular Mesh index is structured. If you search "Hierarchical Triangular
> Mesh" on the internet, you will find some papers, but a useful page is on
> the Sloan DSS survey:
> http://www.skyserver.org/htm/
>
> These days HEALPix is more popular. It has more complicated mesh
> structure, but they tend to prefer it due to the avoidance of
> compute-intensive trigonomeric functions. I believe KStars implements both
> HTMesh (for stars and deep-sky objects) and HEALPix (for progressive
> surveys). The HTMesh implementation in KStars, as a C++ library, lies here:
> https://invent.kde.org/education/kstars/-/tree/master/kstars/htmesh
> I believe the above is a standalone library. There is also a Perl wrapper
> around it:
>
> https://invent.kde.org/education/kstars/-/tree/master/kstars/data/tools/HTMesh-0.01
>
> When we built this system back in 2008, the approach we took was an
> optimization-first approach. This resulted in many non-portable decisions,
> like using a custom-designed binary file format, restricting to 16- and
> 32-byte data structures, and C-style memory allocations. If I were to
> redesign this system today, the first thing I'd try is to use an existing
> low-footprint, fast, serverless-database. The binary file is essentially a
> key-value database indexed by trixel as the key, mapping to an array that
> has entries sorted by magnitude within each trixel. A separate index file
> is provided for lookup by Henry-Draper catalog number.
>
> There are several other subtleties in the system, including replicating
> some high-proper motion stars in multiple trixels that they may appear in
> over the course of 10000 years. Most of these are explained in the doxygen
> file.
>
>>
> I believe I have the raw text data from the USNO on my hard drive, if it
> is easier to start from there rather than from the binary file. But
> obviously, this does not have a HTM index associated with it.
>
> The code in kstars/data/tools unfortunately does a weird rigmarole of
> loading this data into a MySQL database, doing some intermediate processing
> there, and then writing out the binary file from the MySQL. At that time,
> it made sense to load the data into MySQL, but today, SQLite would probably
> be a better choice.
>
> Unfortunately, this has made debugging or extending the catalog hard.
> Please be aware that we have some issues reported on our Gitlab, I believe
> they mostly concern Tycho2-stars. Hipparcos and Tycho-2 (I believe?)
> stars have been removed from the NOMAD catalog binary file supplied in
> KStars so as to avoid duplication, and they are shipped in separate binary
> files (There's four files in total: namedstars.dat, unnamedstars.dat,
> deepstars.dat and the NOMAD-1e8.dat -- the first two I believe are
> Hipparcos, third Tycho-2, and last USNO NOMAD, although I could be wrong).
>
> I am going to propose that if you are going to take the effort to re-work
> this pipeline, we might as well collaborate and do it in such a manner that
> would be beneficial to both projects. This way, our users need to install
> the catalog only once. I am very open to porting away from the binary file
> format, although I may not be able to commit the time to do the work today.
>
> I've been looking at the code and binary format as shown in
>> nomadbinfiletester.c, I think it would not be difficult to use this to
>> create a simple search of stars in NOMAD, not using a database but
>> simply the file organized in "trixels".
>
>
> Yes, this should be possible. In fact our binary file format already has
> this solution. The header contains the 64-bit offsets into the file that
> correspond to each trixel, and at each of those offsets, one finds a small
> header followed by the stars organized by magnitude, brightest first.
>
>>
>>
>> I didn't find how the trixels were distributed in the sky, is there a
>> code you can point me to that converts RA,DEC to a trixel number?
>> In Siril we only need to get a list of stars within a radius or a
>> square around a target's coordinates, with a filter on magnitude too.
>
>
> See the SDSS documentation for a visualization of how trixels are
> distributed in the sky:
> http://www.skyserver.org/htm/
> An important parameter is the "level" of the HTM, which is the number of
> nested levels up to some constant. Each binary file catalog specifies the
> level of HTM that it's indexed on in the header.
>
> The trixel mesh operates, as you can expect, in J2000 coordinates.
>
> As for finding a trixel corresponding to an (RA, Dec): you are looking for
> HTMesh::index
>
> https://invent.kde.org/education/kstars/-/blob/master/kstars/htmesh/HTMesh.cpp#L72
>
> For the latter two usecases, finding the trixels covered by a region, you
> are looking for the various overloads of HTMesh::intersect:
>
> https://invent.kde.org/education/kstars/-/blob/master/kstars/htmesh/HTMesh.cpp#L104
> I believe you can then iterate over the returned trixels through some
> iterator structure.
>
> You may also find the wrapper in SkyMesh.cpp worth looking at, although it
> has several KStars-specific patterns:
>
> https://invent.kde.org/education/kstars/-/blob/master/kstars/skycomponents/skymesh.cpp
>
> For an example use of the mesh, see:
>
> https://invent.kde.org/education/kstars/-/blob/master/kstars/skycomponents/deepstarcomponent.cpp#L283
> (I believe aperture is a wrapper around intersect)
>
> The intersect call is on line 283, which is followed by getting the
> iterator on the next line, and the iteration loop is seen on line 310.
>
> The deep-sky object implementation also uses HTMesh, but the data is
> instead stored in a SQLite database:
>
> https://invent.kde.org/education/kstars/-/blob/master/kstars/skycomponents/catalogscomponent.cpp#L92
>
>>
>
>> Another thing I was wondering is how to convert object names to
>> coordinates (coordinates to name could be useful too), I guess you have
>> a list in kstars, but would it easy to extract from whichever catalog
>> contains them?
>>
>
> In KStars, each class of objects is managed differently when it comes to
> name.
>
> I believe all stars that have names are always held in memory and are
> never loaded. For this, there is a supplemental binary file called
> starnames.dat that contains the names of the stars. Each star's data in the
> binary file may contain some catalog numbers (notably Henry-Draper). The
> reverse mapping is stored and distributed as Henry-Draper.idx (apparently,
> this is broken right now; I filed a number of issues on invent.kde.org
> related to this recently).
>
> For deep-sky objects, we store them in the SQLite database. The object
> names are dynamically queried against the database via the
> CatalogsComponent::findByName() wrapper, and the results are loaded into
> memory (not sure of the details) perhaps upon centering the object, simply
> because we load the trixel's contents.
>
> For all other objects (eg: planets / comets ...), the corresponding
> subclass of SkyComponent supplies the names (look for something like
> objectNames).
>
> Finally, if a name is not known to KStars, we submit a query to CDS Sesame
> (which is an API front-end to SIMBAD and VizieR, and ostensibly NED
> although it's never worked for me), parse the result, store it in the
> SQLite deep-sky object database and then render it on screen for the user.
>
> So you could pick our catalogs workflow created by Valentin Boettcher to
> generate SQLite tables with information from VizieR:
> https://protagon.space/catalogs/
> I am unable to locate the repo which contains the code, and I'm not even
> sure if it is publicly accessible. Perhaps Jasem / Valentin can comment on
> that.
>
> Depending on how you intend to proceed, we may be interested in finding
> ways to modernize our implementation of star catalogs in KStars by
> leveraging any portable implementations you plan to make.
>
> Regards
> Akarsh
>
>
>
>>
>> Thanks a lot for your work and your help!
>>
>> Vincent
>>
>
[Attachment #3 (text/html)]
<div dir="ltr">Thank you for the detailed response Akarsh. We documented stars handling in the API here: \
<a href="https://api.kde.org/kstars/html/Stars.html">https://api.kde.org/kstars/html/Stars.html</a><div><br \
clear="all"><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div \
dir="ltr"><div><div dir="ltr"><div>--</div><div>Best Regards,<br>Jasem \
Mutlaq<br></div><div><br></div></div></div></div></div></div><br></div></div><br><div \
class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jun 8, 2022 at 9:28 AM Akarsh Simha <<a \
href="mailto:akarshsimha@gmail.com">akarshsimha@gmail.com</a>> wrote:<br></div><blockquote \
class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex"><div><div dir="auto">Hi Vincent</div><div><div \
class="gmail_quote"><div dir="ltr" class="gmail_attr"><br></div><div dir="ltr" class="gmail_attr">Thank \
you for your contributions to Siril, it's an excellent piece of software and fills a crucial need in \
the open-source software ecosystems.</div><blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" dir="auto"></blockquote><div \
dir="auto"><br></div><div dir="auto"><br></div><div dir="auto"></div></div></div></div><div><div \
dir="ltr" class="gmail_attr" style="border-color:rgb(0,0,0);color:rgb(0,0,0)"><br></div><blockquote \
class="gmail_quote" style="width:592px;margin:0px 0px 0px 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex;border-top-color:rgb(0,0,0);border-right-color:rgb(0,0,0);border-bottom-color:rgb(0,0,0);color:rgb(0,0,0)">Hello, \
I'm a developer of Siril, I'm looking at options to integrate a<br>photometric star catalog for \
offline operation ; we currently use NOMAD<br>and APASS from the vizier web service.</blockquote><div \
dir="auto"><br></div></div><div><div dir="auto">I'm not sure exactly what your use-case here is, but \
please be warned that NOMAD was generated through algorithms running on images without human curation, so \
at least in the version used in KStars there are several artifacts, especially around milky way patches, \
diffraction spikes of bright stars, and cores of galaxies. Perhaps there is a better version released, \
but I haven't kept up.</div><div dir="auto"><br></div></div><div><blockquote class="gmail_quote" \
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" dir="auto"><br> \
I'm looking for more information about the kstars catalog binary format,<br> I was hoping you could \
give me pointers.</blockquote><div dir="auto"><br></div></div><div><div dir="auto">The KStars star \
catalog binary format is documented here:</div><div dir="auto"><div><a \
href="https://invent.kde.org/education/kstars/-/blob/master/kstars/skycomponents/stars.dox" \
target="_blank">https://invent.kde.org/education/kstars/-/blob/master/kstars/skycomponents/stars.dox</a></div><div \
dir="auto">(for some reason I can't find the doxygen-generated HTML on the web)</div><div \
dir="auto"><br></div><div dir="auto">I believe that the same file also has details of how the \
Hierarchical Triangular Mesh index is structured. If you search "Hierarchical Triangular Mesh" \
on the internet, you will find some papers, but a useful page is on the Sloan DSS survey:<div><a \
href="http://www.skyserver.org/htm/" target="_blank">http://www.skyserver.org/htm/</a></div><div \
dir="auto"><br></div><div dir="auto">These days HEALPix is more popular. It has more complicated mesh \
structure, but they tend to prefer it due to the avoidance of compute-intensive trigonomeric functions. I \
believe KStars implements both HTMesh (for stars and deep-sky objects) and HEALPix (for progressive \
surveys). The HTMesh implementation in KStars, as a C++ library, lies here:</div><div dir="auto"><div><a \
href="https://invent.kde.org/education/kstars/-/tree/master/kstars/htmesh" \
target="_blank">https://invent.kde.org/education/kstars/-/tree/master/kstars/htmesh</a></div></div><div \
dir="auto">I believe the above is a standalone library. There is also a Perl wrapper around it:</div><div \
dir="auto"><div><a href="https://invent.kde.org/education/kstars/-/tree/master/kstars/data/tools/HTMesh-0.01" \
target="_blank">https://invent.kde.org/education/kstars/-/tree/master/kstars/data/tools/HTMesh-0.01</a></div></div><div \
dir="auto"><br></div><div dir="auto">When we built this system back in 2008, the approach we took was an \
optimization-first approach. This resulted in many non-portable decisions, like using a custom-designed \
binary file format, restricting to 16- and 32-byte data structures, and C-style memory allocations. If I \
were to redesign this system today, the first thing I'd try is to use an existing low-footprint, \
fast, serverless-database. The binary file is essentially a key-value database indexed by trixel as the \
key, mapping to an array that has entries sorted by magnitude within each trixel. A separate index file \
is provided for lookup by Henry-Draper catalog number.</div><div dir="auto"><br></div><div \
dir="auto">There are several other subtleties in the system, including replicating some high-proper \
motion stars in multiple trixels that they may appear in over the course of 10000 years. Most of these \
are explained in the doxygen file.</div></div></div><blockquote class="gmail_quote" style="margin:0px 0px \
0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" dir="auto"></blockquote><blockquote \
class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex" dir="auto"></blockquote><div dir="auto"><br></div><div dir="auto">I \
believe I have the raw text data from the USNO on my hard drive, if it is easier to start from there \
rather than from the binary file. But obviously, this does not have a HTM index associated with \
it.</div><div dir="auto"><br></div><div dir="auto">The code in kstars/data/tools unfortunately does a \
weird rigmarole of loading this data into a MySQL database, doing some intermediate processing there, and \
then writing out the binary file from the MySQL. At that time, it made sense to load the data into MySQL, \
but today, SQLite would probably be a better choice.</div><div dir="auto"><br></div><div \
dir="auto">Unfortunately, this has made debugging or extending the catalog hard. Please be aware that we \
have some issues reported on our Gitlab, I believe they mostly concern Tycho2-stars. Hipparcos and<span \
style="color:rgb(0,0,0)"> Tycho-2 (I believe?) stars have been removed from the NOMAD catalog binary \
file supplied in KStars so as to avoid duplication, and they are shipped in separate binary files \
(There's four files in total: namedstars.dat, unnamedstars.dat, deepstars.dat and the NOMAD-1e8.dat \
-- the first two I believe are Hipparcos, third Tycho-2, and last USNO NOMAD, although I could be \
wrong).</span></div><div dir="auto"><span style="color:rgb(0,0,0)"><br></span></div><div dir="auto"><span \
style="color:rgb(0,0,0)">I am going to propose that if you are going to take the effort to re-work this \
pipeline, we might as well collaborate and do it in such a manner that would be beneficial to both \
projects. This way, our users need to install the catalog only once. I am very open to porting away from \
the binary file format, although I may not be able to commit the time to do the work \
today.</span></div></div><div><div dir="auto"><br></div><blockquote class="gmail_quote" style="margin:0px \
0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" dir="auto">I've been looking \
at the code and binary format as shown in<br> nomadbinfiletester.c, I think it would not be difficult to \
use this to<br> create a simple search of stars in NOMAD, not using a database but<br>
simply the file organized in "trixels".</blockquote><div dir="auto"><br></div></div><div><div \
dir="auto">Yes, this should be possible. In fact our binary file format already has this solution. The \
header contains the 64-bit offsets into the file that correspond to each trixel, and at each of those \
offsets, one finds a small header followed by the stars organized by magnitude, brightest \
first.</div></div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px \
solid rgb(204,204,204);padding-left:1ex" dir="auto"><br> <br>
I didn't find how the trixels were distributed in the sky, is there a<br>
code you can point me to that converts RA,DEC to a trixel number?<br>
In Siril we only need to get a list of stars within a radius or a<br>
square around a target's coordinates, with a filter on magnitude too.</blockquote><div \
dir="auto"><br></div></div><div><div dir="auto">See the SDSS documentation for a visualization of how \
trixels are distributed in the sky: <div><a href="http://www.skyserver.org/htm/" \
target="_blank">http://www.skyserver.org/htm/</a></div><div dir="auto">An important parameter is the \
"level" of the HTM, which is the number of nested levels up to some constant. Each binary file \
catalog specifies the level of HTM that it's indexed on in the header.</div><div \
dir="auto"><br></div><div dir="auto">The trixel mesh operates, as you can expect, in J2000 \
coordinates.</div><div dir="auto"><br></div><div dir="auto">As for finding a trixel corresponding to an \
(RA, Dec): you are looking for HTMesh::index</div><div dir="auto"><div><a \
href="https://invent.kde.org/education/kstars/-/blob/master/kstars/htmesh/HTMesh.cpp#L72" \
target="_blank">https://invent.kde.org/education/kstars/-/blob/master/kstars/htmesh/HTMesh.cpp#L72</a></div><br></div><div \
dir="auto">For the latter two usecases, finding the trixels covered by a region, you are looking for the \
various overloads of HTMesh::intersect:</div><div dir="auto"><div dir="auto"><a \
href="https://invent.kde.org/education/kstars/-/blob/master/kstars/htmesh/HTMesh.cpp#L104" \
target="_blank">https://invent.kde.org/education/kstars/-/blob/master/kstars/htmesh/HTMesh.cpp#L104</a></div><div \
dir="auto">I believe you can then iterate over the returned trixels through some iterator \
structure.</div></div><div dir="auto"><br></div><div dir="auto">You may also find the wrapper in \
SkyMesh.cpp worth looking at, although it has several KStars-specific patterns:</div><div \
dir="auto"><div><a href="https://invent.kde.org/education/kstars/-/blob/master/kstars/skycomponents/skymesh.cpp" \
target="_blank">https://invent.kde.org/education/kstars/-/blob/master/kstars/skycomponents/skymesh.cpp</a></div><br></div><div \
dir="auto">For an example use of the mesh, see:</div><div dir="auto"><div><a \
href="https://invent.kde.org/education/kstars/-/blob/master/kstars/skycomponents/deepstarcomponent.cpp#L283" \
target="_blank">https://invent.kde.org/education/kstars/-/blob/master/kstars/skycomponents/deepstarcomponent.cpp#L283</a></div></div><div \
dir="auto">(I believe aperture is a wrapper around intersect)</div><div dir="auto"><br></div><div \
dir="auto">The intersect call is on line 283, which is followed by getting the iterator on the next line, \
and the iteration loop is seen on line 310.</div><div dir="auto"><br></div><div dir="auto">The deep-sky \
object implementation also uses HTMesh, but the data is instead stored in a SQLite database:<div><a \
href="https://invent.kde.org/education/kstars/-/blob/master/kstars/skycomponents/catalogscomponent.cpp#L92" \
target="_blank">https://invent.kde.org/education/kstars/-/blob/master/kstars/skycomponents/catalogscomponent.cpp#L92</a></div></div></div></div><div><div><div \
class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex" dir="auto"></blockquote><div dir="auto"><br></div><blockquote \
class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex" dir="auto"><br> Another thing I was wondering is how to convert object \
names to<br> coordinates (coordinates to name could be useful too), I guess you have<br>
a list in kstars, but would it easy to extract from whichever catalog<br>
contains them?<br>
</blockquote><div dir="auto"><br></div><div dir="auto">In KStars, each class of objects is managed \
differently when it comes to name.</div><div dir="auto"><br></div><div dir="auto">I believe all stars \
that have names are always held in memory and are never loaded. For this, there is a supplemental binary \
file called starnames.dat that contains the names of the stars. Each star's data in the binary file \
may contain some catalog numbers (notably Henry-Draper). The reverse mapping is stored and distributed as \
Henry-Draper.idx (apparently, this is broken right now; I filed a number of issues on <a \
href="http://invent.kde.org" target="_blank">invent.kde.org</a> related to this recently).</div><div \
dir="auto"><br></div><div dir="auto">For deep-sky objects, we store them in the SQLite database. The \
object names are dynamically queried against the database via the CatalogsComponent::findByName() \
wrapper, and the results are loaded into memory (not sure of the details) perhaps upon centering the \
object, simply because we load the trixel's contents.</div><div dir="auto"><br></div><div \
dir="auto">For all other objects (eg: planets / comets ...), the corresponding subclass of SkyComponent \
supplies the names (look for something like objectNames).</div><div dir="auto"><br></div><div \
dir="auto">Finally, if a name is not known to KStars, we submit a query to CDS Sesame (which is an API \
front-end to SIMBAD and VizieR, and ostensibly NED although it's never worked for me), parse the \
result, store it in the SQLite deep-sky object database and then render it on screen for the \
user.</div><div dir="auto"><br></div><div dir="auto">So you could pick our catalogs workflow created by \
Valentin Boettcher to generate SQLite tables with information from VizieR:</div><div dir="auto"><div><a \
href="https://protagon.space/catalogs/" target="_blank">https://protagon.space/catalogs/</a></div><div \
dir="auto">I am unable to locate the repo which contains the code, and I'm not even sure if it is \
publicly accessible. Perhaps Jasem / Valentin can comment on that.</div><div dir="auto"><br></div><div \
dir="auto">Depending on how you intend to proceed, we may be interested in finding ways to modernize our \
implementation of star catalogs in KStars by leveraging any portable implementations you plan to \
make.</div><div dir="auto"><br></div><div dir="auto">Regards</div><div \
dir="auto">Akarsh</div><br></div><div dir="auto"><br></div><blockquote class="gmail_quote" \
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" dir="auto"><br> \
<br> Thanks a lot for your work and your help!<br>
<br>
Vincent<br>
</blockquote></div></div>
</div>
</blockquote></div>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic