Hi Wolfgang,
So I just want to clarify something you said here, there are a couple of parallel things and that can be a little confusing, so I just want to make sure we are talking about the same things. The cause of the confusion is the terminology that
astrometry.net uses
1.
Load all Indexes in Memory / Load all indexes in Parallel. This is the inParallel option for
astrometry.net. In the options I tried to call this “Load all Indexes in Memory” to attempt to avoid the confusion with the Parallel Algorithm. This has nothing to do with parallelization in different threads or processors. It has to do with memory management. The
astrometry.net solver can load the indexes and search them one after the other, or it can try to load all the indexes at once and then solve. The second option is much much faster, but comes with risk.
astrometry.net does NOT check to see if it has enough RAM before it tries to solve, They have big warnings in the documentation about using this option. If you don’t have enough RAM, it could use all the RAM and crash.
I programmed StellarSolver to check the available RAM prior to starting the solve. If there is not enough RAM, it is supposed to turn off the option. The user can also disable the option entirely, so that there is never a problem. But you really do want the option turned on if your system can handle it. We had some issues earlier about the RAM calculation. I think the “inParallel” option causes the greatest crash risk. I would really like it if somebody could look over the code for determining enough RAM and see if it is good now. One thought that I have is that we can make the calculation more conservative and we could change the option to have 3 choices, Auto, on, or off. So that if a user is really brave, or convinced they have enough RAM for sure, they could turn the option on regardless of the risk, If they are risk averse, they could turn it off, but most users could just leave it on auto. What do you think?
2. Parallelization Algorithm for solving. I am assuming this second option is what you meant in your email. This one is entirely of my creation and is what makes StellarSolver stellar. Modern computers really have great capacity for computing in parallel and it causes a HUGE performance boost to use this capability, even on a Pi, since the PI has 4 processors.
I programmed StellarSolver to have 2 different parallel algorithms, one that solves simultaneously at multiple “depths” and one that solves simultaneously at different scales. If you set it to Auto, it will select the appropriate one based on whether you specified the scale or position (or neither). If the image has both scale AND position, it does NOT solve in parallel and goes back to solving with a single thread.
When Jasem wanted to me to de-thread the StellarSolver and make it so that just the solvers are threads, I had to make a bunch of changes and one change I forgot was to make the star extraction before parallel solving asynchronous. That does mean that when doing a parallel solve, it might look like things have frozen for a moment during the star extraction before the threads start up. I have already fixed this, but it is in the releaseExperiment branch of StellarSolver, not in Master. I would like to get this fix integrated before we release, but I will need to test this thoroughly first as I mentioned in a previous email. I am wondering if this freezing behavior was what caused the “crash” you observed?
Thanks,
Rob
OK, I did a quick check on my RPi4 with Parallel Algorithm set to „Auto“ - and it works super fast! But since it is daytime, I can only test the „Load and Slew“ option. So maybe the WCS info in the file gave hints that are not present for normal capture and slew or sync.
I need to check it under real conditions, which might be tricky due to the fog hanging around here…
Wolfgang
Alright, let's look at this:
1. Parallel algorithm: This is related to SOLVER, not image partitioning. It should work fine on Rpi4 and the checks are more reliable now as Robert worked on that.
2. WCS Polar Align: Can this be reproduced with simulators?
--
Best Regards,
Jasem Mutlaq
It wasn’t that bad. The problem was that KStars went to 100% CPU usage and died (or I killed it, do not exactly remember). I’ll try to reproduce it...
OK, well I believe it was fixed a week ago, so if you can still recreate it, you should report it.
It should be fixed before release if it is still freezing the Pi.
Hy
OK, I have to check it. The problem occurred only a few days ago and I think I’m always on bleeding edge...
Wolfgang: I believe Rob and/or Jasem fixed the issue with parallel algorithm bringing down the RPi4 a while back.
I have the solver on auto parallelism and load all indexes in memory, and it seems to work fine (and in parallel).
Similarly, for star extraction, Jasem implemented a threaded extraction that also automatically determines how many threads to use and seems fine on the RPi4.
Eric: I believe these parallel options are the defaults. Hopefully users won't need to configure things like this.
For star detection, I don't believe you can turn it off.
For star detection Jasem split the frame before detection (into at most num-threads parts--4 for the RPi4).
For align, I'm not sure how Rob divided things.
Hy
Hi all,
I think we are close to finishing the release. I personally would opt to wait for another week and keep an eye stability.
Maybe we should take another look if the default settings in the StellarSolver profiles work a) for typical camera/scope combinations and b) for all platforms.
For example with my RPi, I needed to change the Parallel Algorithm to „None“ because parallelity brought KStars down. Is the default setting „None“ and I changed it somewhen? With all the new parameters I would prefer having a robust setup and leave it to the user to optimize speed.
@Jasem: please take a closer look to MR!122, since it fixed 4(!) regressions I introduced with my capture counting fix MR!114. Hopefully now we have at least a proper coverage with automated tests...
Wolfgang
Hello Folks,
So back to this topic, any major blockers to the KStars 3.5.0 release now?
1. Remote Solver should be fixed now.
2. StellarSolver Profiles are more optimized now.
3. Handbook not updated yet, but we can probably work on this shortly.
4. Couple of pending MRs to take care of.
How about Friday the 13th?
--
Best Regards,
Jasem Mutlaq
Hi Eric,
Ok so then we would be changing the way we do version numbering with this, right?
I believe now we typically add features in each new iteration 3.4.1, 3.4.2, etc etc
and when it is really big like StellarSolver, then we make it a big release like 3.5.0
With this new paradigm, we wouldn’t put new features into the master of the main 3.5 branch
But instead we would work on a new 3.6 branch, and then bug fixes would go into the 3.5 branch
to make each new minor release, like 3.5.1, 3.5.2 etc.
Do I have this correct?
If this is right, then it would be longer before users see new features in the main branch, but the
tradeoff is that the main branch would have a LOT more stability. I see this as a big positive.
Thanks,
Rob
> On Nov 4, 2020, at 5:54 PM, Eric Dejouhanet <eric.dejouhanet@gmail.com> wrote:
>
> Hello Hy,
>
> Version 3.5.0 is only the beginning of the 3.5.x series, with more
> bugfixes on each iteration (and possibly, only bugfixes).
> So I have no problem leaving unresolved issues in 3.5.0.
>
> For instance, the Focus module now has a slight and unforeseeable
> delay after the capture completes.
> The UI reflects the end of the capture only, not the end of the detection.
> This makes the UI Focus test quite difficult to tweak, as running an
> average of the HFR over multiple frames now has an unknown duration.
> Right now, the test is trying to click the capture button too soon 2
> out of 10 attempts.
> But this won't block 3.5 in my opinion (and now that I understood the
> problem, I won't work on it immediately).
>
> In terms of reporting problems, the official way is stil bugs.kde.org,
> but there's quite a cleanup/followup to do there.
> I'd say we can use issues in invent.kde.org to discuss planned
> development around a forum/bugzilla issue or invent proposal (like
> agile stories).
> There are milestones associated with several issues (although I think
> they should be reviewed and postponed).
> And we can certainly write a punchlist: check the board at
> https://invent.kde.org/education/kstars/-/milestones/3
>
> Le mer. 4 nov. 2020 à 22:38, Hy Murveit <murveit@gmail.com> a écrit :
>>
>> Eric,
>>
>> I would add to your list:
>>
>> - KStars Handbook (review update sections to reflect 3.5.0) and finally (perhaps manually if necessary) put the latest handbook online.
>>
>> - Review the extraction settings. I spent a bit of time looking at the default HFR settings, and based on some experimentation (truth be told, with a limited amount of data) adjust things a little differently than my first guess (which was basically focus' settings).
>> Rob: My intuition is that I should adjust the default StellarSolver star-extraction settings for Focus and Guide as well in stellarsolverprofile.cpp. I don't know whether you've already verified them, and want to release them as they are, or whether they are a first shot and you'd welcome adjustment?
>>
>> Also, Eric, I suppose I should be adding these things here: https://invent.kde.org/education/kstars/-/issues
>> Is that right? Sorry about that--ok, after this thread ;) But seriously, your email is a good summary, and from that link
>> it doesn't seem as easy to see which are "must do by 3.5.0" and which are "nice to have someday".
>> A 3.5.0 punchlist would be a nice thing to have.
>>
>> Hy
>>
>> On Wed, Nov 4, 2020 at 12:58 PM Eric Dejouhanet <eric.dejouhanet@gmail.com> wrote:
>>>
>>> Hello,
>>>
>>> Where do we stand now in terms of bugfixing towards 3.5.0?
>>>
>>> - StellarSolver has all features in, and 1.5 is finally out at Jasem's PPA.
>>> - However Gitlab CI still complains about that lib package (see
>>> https://invent.kde.org/education/kstars/-/jobs/75941)
>>> - Unitary tests are being fixed progressively, mount tests are down to
>>> ~20 minutes (yeees!)
>>> - From my tests, the remote Astrometry INDI driver is not usable
>>> anymore from Ekos.
>>> - The issue raised with flat frames is confirmed fixed (at least by me).
>>> - Meridian flip is OK (but I had not enough time to test TWO flips in a row).
>>> - Memory leaks are still being researched in Ekos.
>>> - There is an issue when duplicating an entry in a scheduler job,
>>> where the sequence associated is copied from the next job.
>>>
>>> Could we get a 3.6 branch where we will merge development of new features?
>>> And master for bugfixing 3.5.x until we merge 3.6 new features in?
>>> (we'd still have to port bugfixes from master to 3.6)
>>> I don't think the opposite, master for 3.6 and a separate living
>>> 3.5.x, is doable in the current configuration (build, ppas, MRs...).
>>>
>>> --
>>> -- eric.dejouhanet@gmail.com - https://astronomy.dejouha.net
>
>
>
> --
> -- eric.dejouhanet@gmail.com - https://astronomy.dejouha.net