[prev in list] [next in list] [prev in thread] [next in thread]
List: kstars-devel
Subject: Re: question about kstars catalogs
From: Jasem Mutlaq <mutlaqja () ikarustech ! com>
Date: 2022-06-08 7:34:50
Message-ID: CAE0bU5kFwd4Np0DG7V8j4OPQ4LAcWNyJqHjWjVX22JhR1QS95Q () mail ! gmail ! com
[Download RAW message or body]
Thank you for the detailed response Akarsh. We documented stars handling in
the API here: https://api.kde.org/kstars/html/Stars.html
--
Best Regards,
Jasem Mutlaq
On Wed, Jun 8, 2022 at 9:28 AM Akarsh Simha <akarshsimha@gmail.com> wrote:
> Hi Vincent
>
> Thank you for your contributions to Siril, it's an excellent piece of
> software and fills a crucial need in the open-source software ecosystems.
>
>>
>
>
> Hello, I'm a developer of Siril, I'm looking at options to integrate a
>> photometric star catalog for offline operation ; we currently use NOMAD
>> and APASS from the vizier web service.
>
>
> I'm not sure exactly what your use-case here is, but please be warned that
> NOMAD was generated through algorithms running on images without human
> curation, so at least in the version used in KStars there are several
> artifacts, especially around milky way patches, diffraction spikes of
> bright stars, and cores of galaxies. Perhaps there is a better version
> released, but I haven't kept up.
>
>
>> I'm looking for more information about the kstars catalog binary format,
>> I was hoping you could give me pointers.
>
>
> The KStars star catalog binary format is documented here:
>
> https://invent.kde.org/education/kstars/-/blob/master/kstars/skycomponents/stars.dox
> (for some reason I can't find the doxygen-generated HTML on the web)
>
> I believe that the same file also has details of how the Hierarchical
> Triangular Mesh index is structured. If you search "Hierarchical Triangular
> Mesh" on the internet, you will find some papers, but a useful page is on
> the Sloan DSS survey:
> http://www.skyserver.org/htm/
>
> These days HEALPix is more popular. It has more complicated mesh
> structure, but they tend to prefer it due to the avoidance of
> compute-intensive trigonomeric functions. I believe KStars implements both
> HTMesh (for stars and deep-sky objects) and HEALPix (for progressive
> surveys). The HTMesh implementation in KStars, as a C++ library, lies here:
> https://invent.kde.org/education/kstars/-/tree/master/kstars/htmesh
> I believe the above is a standalone library. There is also a Perl wrapper
> around it:
>
> https://invent.kde.org/education/kstars/-/tree/master/kstars/data/tools/HTMesh-0.01
>
> When we built this system back in 2008, the approach we took was an
> optimization-first approach. This resulted in many non-portable decisions,
> like using a custom-designed binary file format, restricting to 16- and
> 32-byte data structures, and C-style memory allocations. If I were to
> redesign this system today, the first thing I'd try is to use an existing
> low-footprint, fast, serverless-database. The binary file is essentially a
> key-value database indexed by trixel as the key, mapping to an array that
> has entries sorted by magnitude within each trixel. A separate index file
> is provided for lookup by Henry-Draper catalog number.
>
> There are several other subtleties in the system, including replicating
> some high-proper motion stars in multiple trixels that they may appear in
> over the course of 10000 years. Most of these are explained in the doxygen
> file.
>
>>
> I believe I have the raw text data from the USNO on my hard drive, if it
> is easier to start from there rather than from the binary file. But
> obviously, this does not have a HTM index associated with it.
>
> The code in kstars/data/tools unfortunately does a weird rigmarole of
> loading this data into a MySQL database, doing some intermediate processing
> there, and then writing out the binary file from the MySQL. At that time,
> it made sense to load the data into MySQL, but today, SQLite would probably
> be a better choice.
>
> Unfortunately, this has made debugging or extending the catalog hard.
> Please be aware that we have some issues reported on our Gitlab, I believe
> they mostly concern Tycho2-stars. Hipparcos and Tycho-2 (I believe?)
> stars have been removed from the NOMAD catalog binary file supplied in
> KStars so as to avoid duplication, and they are shipped in separate binary
> files (There's four files in total: namedstars.dat, unnamedstars.dat,
> deepstars.dat and the NOMAD-1e8.dat -- the first two I believe are
> Hipparcos, third Tycho-2, and last USNO NOMAD, although I could be wrong).
>
> I am going to propose that if you are going to take the effort to re-work
> this pipeline, we might as well collaborate and do it in such a manner that
> would be beneficial to both projects. This way, our users need to install
> the catalog only once. I am very open to porting away from the binary file
> format, although I may not be able to commit the time to do the work today.
>
> I've been looking at the code and binary format as shown in
>> nomadbinfiletester.c, I think it would not be difficult to use this to
>> create a simple search of stars in NOMAD, not using a database but
>> simply the file organized in "trixels".
>
>
> Yes, this should be possible. In fact our binary file format already has
> this solution. The header contains the 64-bit offsets into the file that
> correspond to each trixel, and at each of those offsets, one finds a small
> header followed by the stars organized by magnitude, brightest first.
>
>>
>>
>> I didn't find how the trixels were distributed in the sky, is there a
>> code you can point me to that converts RA,DEC to a trixel number?
>> In Siril we only need to get a list of stars within a radius or a
>> square around a target's coordinates, with a filter on magnitude too.
>
>
> See the SDSS documentation for a visualization of how trixels are
> distributed in the sky:
> http://www.skyserver.org/htm/
> An important parameter is the "level" of the HTM, which is the number of
> nested levels up to some constant. Each binary file catalog specifies the
> level of HTM that it's indexed on in the header.
>
> The trixel mesh operates, as you can expect, in J2000 coordinates.
>
> As for finding a trixel corresponding to an (RA, Dec): you are looking for
> HTMesh::index
>
> https://invent.kde.org/education/kstars/-/blob/master/kstars/htmesh/HTMesh.cpp#L72
>
> For the latter two usecases, finding the trixels covered by a region, you
> are looking for the various overloads of HTMesh::intersect:
>
> https://invent.kde.org/education/kstars/-/blob/master/kstars/htmesh/HTMesh.cpp#L104
> I believe you can then iterate over the returned trixels through some
> iterator structure.
>
> You may also find the wrapper in SkyMesh.cpp worth looking at, although it
> has several KStars-specific patterns:
>
> https://invent.kde.org/education/kstars/-/blob/master/kstars/skycomponents/skymesh.cpp
>
> For an example use of the mesh, see:
>
> https://invent.kde.org/education/kstars/-/blob/master/kstars/skycomponents/deepstarcomponent.cpp#L283
> (I believe aperture is a wrapper around intersect)
>
> The intersect call is on line 283, which is followed by getting the
> iterator on the next line, and the iteration loop is seen on line 310.
>
> The deep-sky object implementation also uses HTMesh, but the data is
> instead stored in a SQLite database:
>
> https://invent.kde.org/education/kstars/-/blob/master/kstars/skycomponents/catalogscomponent.cpp#L92
>
>>
>
>> Another thing I was wondering is how to convert object names to
>> coordinates (coordinates to name could be useful too), I guess you have
>> a list in kstars, but would it easy to extract from whichever catalog
>> contains them?
>>
>
> In KStars, each class of objects is managed differently when it comes to
> name.
>
> I believe all stars that have names are always held in memory and are
> never loaded. For this, there is a supplemental binary file called
> starnames.dat that contains the names of the stars. Each star's data in the
> binary file may contain some catalog numbers (notably Henry-Draper). The
> reverse mapping is stored and distributed as Henry-Draper.idx (apparently,
> this is broken right now; I filed a number of issues on invent.kde.org
> related to this recently).
>
> For deep-sky objects, we store them in the SQLite database. The object
> names are dynamically queried against the database via the
> CatalogsComponent::findByName() wrapper, and the results are loaded into
> memory (not sure of the details) perhaps upon centering the object, simply
> because we load the trixel's contents.
>
> For all other objects (eg: planets / comets ...), the corresponding
> subclass of SkyComponent supplies the names (look for something like
> objectNames).
>
> Finally, if a name is not known to KStars, we submit a query to CDS Sesame
> (which is an API front-end to SIMBAD and VizieR, and ostensibly NED
> although it's never worked for me), parse the result, store it in the
> SQLite deep-sky object database and then render it on screen for the user.
>
> So you could pick our catalogs workflow created by Valentin Boettcher to
> generate SQLite tables with information from VizieR:
> https://protagon.space/catalogs/
> I am unable to locate the repo which contains the code, and I'm not even
> sure if it is publicly accessible. Perhaps Jasem / Valentin can comment on
> that.
>
> Depending on how you intend to proceed, we may be interested in finding
> ways to modernize our implementation of star catalogs in KStars by
> leveraging any portable implementations you plan to make.
>
> Regards
> Akarsh
>
>
>
>>
>> Thanks a lot for your work and your help!
>>
>> Vincent
>>
>
[Attachment #3 (text/html)]
<div dir="ltr">Thank you for the detailed response Akarsh. We documented stars \
handling in the API here: <a \
href="https://api.kde.org/kstars/html/Stars.html">https://api.kde.org/kstars/html/Stars.html</a><div><br \
clear="all"><div><div dir="ltr" class="gmail_signature" \
data-smartmail="gmail_signature"><div dir="ltr"><div><div \
dir="ltr"><div>--</div><div>Best Regards,<br>Jasem \
Mutlaq<br></div><div><br></div></div></div></div></div></div><br></div></div><br><div \
class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jun 8, 2022 at 9:28 AM \
Akarsh Simha <<a href="mailto:akarshsimha@gmail.com">akarshsimha@gmail.com</a>> \
wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div \
dir="auto">Hi Vincent</div><div><div class="gmail_quote"><div dir="ltr" \
class="gmail_attr"><br></div><div dir="ltr" class="gmail_attr">Thank you for your \
contributions to Siril, it's an excellent piece of software and fills a crucial \
need in the open-source software ecosystems.</div><blockquote class="gmail_quote" \
style="margin:0px 0px 0px 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex" dir="auto"></blockquote><div \
dir="auto"><br></div><div dir="auto"><br></div><div \
dir="auto"></div></div></div></div><div><div dir="ltr" class="gmail_attr" \
style="border-color:rgb(0,0,0);color:rgb(0,0,0)"><br></div><blockquote \
class="gmail_quote" style="width:592px;margin:0px 0px 0px 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex;border-top-color:rgb(0,0,0);border-right-color:rgb(0,0,0);border-bottom-color:rgb(0,0,0);color:rgb(0,0,0)">Hello, \
I'm a developer of Siril, I'm looking at options to integrate \
a<br>photometric star catalog for offline operation ; we currently use NOMAD<br>and \
APASS from the vizier web service.</blockquote><div \
dir="auto"><br></div></div><div><div dir="auto">I'm not sure exactly what your \
use-case here is, but please be warned that NOMAD was generated through algorithms \
running on images without human curation, so at least in the version used in KStars \
there are several artifacts, especially around milky way patches, diffraction spikes \
of bright stars, and cores of galaxies. Perhaps there is a better version released, \
but I haven't kept up.</div><div dir="auto"><br></div></div><div><blockquote \
class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex" dir="auto"><br> I'm looking for more \
information about the kstars catalog binary format,<br> I was hoping you could give \
me pointers.</blockquote><div dir="auto"><br></div></div><div><div dir="auto">The \
KStars star catalog binary format is documented here:</div><div dir="auto"><div><a \
href="https://invent.kde.org/education/kstars/-/blob/master/kstars/skycomponents/stars.dox" \
target="_blank">https://invent.kde.org/education/kstars/-/blob/master/kstars/skycomponents/stars.dox</a></div><div \
dir="auto">(for some reason I can't find the doxygen-generated HTML on the \
web)</div><div dir="auto"><br></div><div dir="auto">I believe that the same file also \
has details of how the Hierarchical Triangular Mesh index is structured. If you \
search "Hierarchical Triangular Mesh" on the internet, you will find some \
papers, but a useful page is on the Sloan DSS survey:<div><a \
href="http://www.skyserver.org/htm/" \
target="_blank">http://www.skyserver.org/htm/</a></div><div dir="auto"><br></div><div \
dir="auto">These days HEALPix is more popular. It has more complicated mesh \
structure, but they tend to prefer it due to the avoidance of compute-intensive \
trigonomeric functions. I believe KStars implements both HTMesh (for stars and \
deep-sky objects) and HEALPix (for progressive surveys). The HTMesh implementation in \
KStars, as a C++ library, lies here:</div><div dir="auto"><div><a \
href="https://invent.kde.org/education/kstars/-/tree/master/kstars/htmesh" \
target="_blank">https://invent.kde.org/education/kstars/-/tree/master/kstars/htmesh</a></div></div><div \
dir="auto">I believe the above is a standalone library. There is also a Perl wrapper \
around it:</div><div dir="auto"><div><a \
href="https://invent.kde.org/education/kstars/-/tree/master/kstars/data/tools/HTMesh-0.01" \
target="_blank">https://invent.kde.org/education/kstars/-/tree/master/kstars/data/tools/HTMesh-0.01</a></div></div><div \
dir="auto"><br></div><div dir="auto">When we built this system back in 2008, the \
approach we took was an optimization-first approach. This resulted in many \
non-portable decisions, like using a custom-designed binary file format, restricting \
to 16- and 32-byte data structures, and C-style memory allocations. If I were to \
redesign this system today, the first thing I'd try is to use an existing \
low-footprint, fast, serverless-database. The binary file is essentially a key-value \
database indexed by trixel as the key, mapping to an array that has entries sorted by \
magnitude within each trixel. A separate index file is provided for lookup by \
Henry-Draper catalog number.</div><div dir="auto"><br></div><div dir="auto">There are \
several other subtleties in the system, including replicating some high-proper motion \
stars in multiple trixels that they may appear in over the course of 10000 years. \
Most of these are explained in the doxygen file.</div></div></div><blockquote \
class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex" dir="auto"></blockquote><blockquote \
class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex" dir="auto"></blockquote><div \
dir="auto"><br></div><div dir="auto">I believe I have the raw text data from the USNO \
on my hard drive, if it is easier to start from there rather than from the binary \
file. But obviously, this does not have a HTM index associated with it.</div><div \
dir="auto"><br></div><div dir="auto">The code in kstars/data/tools unfortunately does \
a weird rigmarole of loading this data into a MySQL database, doing some intermediate \
processing there, and then writing out the binary file from the MySQL. At that time, \
it made sense to load the data into MySQL, but today, SQLite would probably be a \
better choice.</div><div dir="auto"><br></div><div dir="auto">Unfortunately, this has \
made debugging or extending the catalog hard. Please be aware that we have some \
issues reported on our Gitlab, I believe they mostly concern Tycho2-stars. \
Hipparcos and<span style="color:rgb(0,0,0)"> Tycho-2 (I believe?) stars have been \
removed from the NOMAD catalog binary file supplied in KStars so as to avoid \
duplication, and they are shipped in separate binary files (There's four files in \
total: namedstars.dat, unnamedstars.dat, deepstars.dat and the NOMAD-1e8.dat -- the \
first two I believe are Hipparcos, third Tycho-2, and last USNO NOMAD, although I \
could be wrong).</span></div><div dir="auto"><span \
style="color:rgb(0,0,0)"><br></span></div><div dir="auto"><span \
style="color:rgb(0,0,0)">I am going to propose that if you are going to take the \
effort to re-work this pipeline, we might as well collaborate and do it in such a \
manner that would be beneficial to both projects. This way, our users need to install \
the catalog only once. I am very open to porting away from the binary file format, \
although I may not be able to commit the time to do the work \
today.</span></div></div><div><div dir="auto"><br></div><blockquote \
class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex" dir="auto">I've been looking at the code and \
binary format as shown in<br> nomadbinfiletester.c, I think it would not be difficult \
to use this to<br> create a simple search of stars in NOMAD, not using a database \
but<br> simply the file organized in "trixels".</blockquote><div \
dir="auto"><br></div></div><div><div dir="auto">Yes, this should be possible. In fact \
our binary file format already has this solution. The header contains the 64-bit \
offsets into the file that correspond to each trixel, and at each of those offsets, \
one finds a small header followed by the stars organized by magnitude, brightest \
first.</div></div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" dir="auto"><br> <br>
I didn't find how the trixels were distributed in the sky, is there a<br>
code you can point me to that converts RA,DEC to a trixel number?<br>
In Siril we only need to get a list of stars within a radius or a<br>
square around a target's coordinates, with a filter on magnitude \
too.</blockquote><div dir="auto"><br></div></div><div><div dir="auto">See the SDSS \
documentation for a visualization of how trixels are distributed in the sky: <div><a \
href="http://www.skyserver.org/htm/" \
target="_blank">http://www.skyserver.org/htm/</a></div><div dir="auto">An important \
parameter is the "level" of the HTM, which is the number of nested levels \
up to some constant. Each binary file catalog specifies the level of HTM that \
it's indexed on in the header.</div><div dir="auto"><br></div><div dir="auto">The \
trixel mesh operates, as you can expect, in J2000 coordinates.</div><div \
dir="auto"><br></div><div dir="auto">As for finding a trixel corresponding to an (RA, \
Dec): you are looking for HTMesh::index</div><div dir="auto"><div><a \
href="https://invent.kde.org/education/kstars/-/blob/master/kstars/htmesh/HTMesh.cpp#L72" \
target="_blank">https://invent.kde.org/education/kstars/-/blob/master/kstars/htmesh/HTMesh.cpp#L72</a></div><br></div><div \
dir="auto">For the latter two usecases, finding the trixels covered by a region, you \
are looking for the various overloads of HTMesh::intersect:</div><div dir="auto"><div \
dir="auto"><a href="https://invent.kde.org/education/kstars/-/blob/master/kstars/htmesh/HTMesh.cpp#L104" \
target="_blank">https://invent.kde.org/education/kstars/-/blob/master/kstars/htmesh/HTMesh.cpp#L104</a></div><div \
dir="auto">I believe you can then iterate over the returned trixels through some \
iterator structure.</div></div><div dir="auto"><br></div><div dir="auto">You may also \
find the wrapper in SkyMesh.cpp worth looking at, although it has several \
KStars-specific patterns:</div><div dir="auto"><div><a \
href="https://invent.kde.org/education/kstars/-/blob/master/kstars/skycomponents/skymesh.cpp" \
target="_blank">https://invent.kde.org/education/kstars/-/blob/master/kstars/skycomponents/skymesh.cpp</a></div><br></div><div \
dir="auto">For an example use of the mesh, see:</div><div dir="auto"><div><a \
href="https://invent.kde.org/education/kstars/-/blob/master/kstars/skycomponents/deepstarcomponent.cpp#L283" \
target="_blank">https://invent.kde.org/education/kstars/-/blob/master/kstars/skycomponents/deepstarcomponent.cpp#L283</a></div></div><div \
dir="auto">(I believe aperture is a wrapper around intersect)</div><div \
dir="auto"><br></div><div dir="auto">The intersect call is on line 283, which is \
followed by getting the iterator on the next line, and the iteration loop is seen on \
line 310.</div><div dir="auto"><br></div><div dir="auto">The deep-sky object \
implementation also uses HTMesh, but the data is instead stored in a SQLite \
database:<div><a href="https://invent.kde.org/education/kstars/-/blob/master/kstars/skycomponents/catalogscomponent.cpp#L92" \
target="_blank">https://invent.kde.org/education/kstars/-/blob/master/kstars/skycomponents/catalogscomponent.cpp#L92</a></div></div></div></div><div><div><div \
class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" \
dir="auto"></blockquote><div dir="auto"><br></div><blockquote class="gmail_quote" \
style="margin:0px 0px 0px 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex" dir="auto"><br> Another thing I was wondering is \
how to convert object names to<br> coordinates (coordinates to name could be useful \
too), I guess you have<br> a list in kstars, but would it easy to extract from \
whichever catalog<br> contains them?<br>
</blockquote><div dir="auto"><br></div><div dir="auto">In KStars, each class of \
objects is managed differently when it comes to name.</div><div \
dir="auto"><br></div><div dir="auto">I believe all stars that have names are always \
held in memory and are never loaded. For this, there is a supplemental binary file \
called starnames.dat that contains the names of the stars. Each star's data in \
the binary file may contain some catalog numbers (notably Henry-Draper). The reverse \
mapping is stored and distributed as Henry-Draper.idx (apparently, this is broken \
right now; I filed a number of issues on <a href="http://invent.kde.org" \
target="_blank">invent.kde.org</a> related to this recently).</div><div \
dir="auto"><br></div><div dir="auto">For deep-sky objects, we store them in the \
SQLite database. The object names are dynamically queried against the database via \
the CatalogsComponent::findByName() wrapper, and the results are loaded into memory \
(not sure of the details) perhaps upon centering the object, simply because we load \
the trixel's contents.</div><div dir="auto"><br></div><div dir="auto">For all \
other objects (eg: planets / comets ...), the corresponding subclass of SkyComponent \
supplies the names (look for something like objectNames).</div><div \
dir="auto"><br></div><div dir="auto">Finally, if a name is not known to KStars, we \
submit a query to CDS Sesame (which is an API front-end to SIMBAD and VizieR, and \
ostensibly NED although it's never worked for me), parse the result, store it in \
the SQLite deep-sky object database and then render it on screen for the \
user.</div><div dir="auto"><br></div><div dir="auto">So you could pick our catalogs \
workflow created by Valentin Boettcher to generate SQLite tables with information \
from VizieR:</div><div dir="auto"><div><a href="https://protagon.space/catalogs/" \
target="_blank">https://protagon.space/catalogs/</a></div><div dir="auto">I am unable \
to locate the repo which contains the code, and I'm not even sure if it is \
publicly accessible. Perhaps Jasem / Valentin can comment on that.</div><div \
dir="auto"><br></div><div dir="auto">Depending on how you intend to proceed, we may \
be interested in finding ways to modernize our implementation of star catalogs in \
KStars by leveraging any portable implementations you plan to make.</div><div \
dir="auto"><br></div><div dir="auto">Regards</div><div \
dir="auto">Akarsh</div><br></div><div dir="auto"><br></div><blockquote \
class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex" dir="auto"><br> <br>
Thanks a lot for your work and your help!<br>
<br>
Vincent<br>
</blockquote></div></div>
</div>
</blockquote></div>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic