[prev in list] [next in list] [prev in thread] [next in thread] 

List:       ispman-users
Subject:    Re: [Ispman-users] Efficiency within the ispman-agent
From:       Tarjei Huse <tarjei () nu ! no>
Date:       2004-02-12 15:13:29
Message-ID: 1076598809.3270.575.camel () elprinsessekaja ! mail ! nu ! no
[Download RAW message or body]

On Tue, 2004-02-10 at 22:18, Jeff Meden wrote:
> Well, at present we're running ISPman on exactly 0 production domains.  
> I have spend the past few weeks extensively testing its scalability in 
> every direction, in preparation for starting to use it in production.  
> Ive been testing its upward scaling on a pair of dual p3 freebsd 
> machines, and the biggest problem I noticed was Perl's tendency to be a 
> total resource hog.  As the number of domains scale up, the DNS 
> manipulation running time scales directly.  At 500 domains, the total 
> time per change to implement was 35 seconds, due to a rather lengthy 
> named.conf generation process.  This means that (rough math) Only 3000 
> changes/day can be posted, or 6 changes per domain, supposing it was 
> okay to have the script running nonstop.  While this isn't a problem day 
> to day, since its unlikely that there will be that many changes, the 
> process of migrating a system to ISPman would be rather intensive. 

Hmm, how about using bind-ldap at the master server and then use
something else on the slaves if you think it is too slow?

Also, if you're going to migrate 500 domains, you might consider
modifying the code so adding the domains doesn't add a process an then
run ispman.ldap2named after adding all domains.

Tarjei




> I am currently gearing up a test network for production simulation, of 
> over 250 domains.  This will probably cost 24 hours or more of time to 
> run the initial migrate, assuming everything runs as planned.  I haven't 
> spent much time looking at the actual code behind the system, but I can 
> point out one clear potential enhancement.  As processes are added to 
> the queue, only allow one 'named.conf update' per dns host, if one 
> exists purge it and add the new one at the end of the process queue.  
> This would cut down a lot of time spend reworking the file needlessly.  
> The ultimate solution would be for the agent to parallelize every task 
> in the queue, making it aware that x number of zone files must be 
> created, and then the conf file rebuilt, then only spending one instance 
> of each 'expensive' process on the whole group of updates.  This would 
> allow for more optimization in other areas as well, but would probably 
> take a lot of reworking of the code.  Hopefully, if/when our company can 
> make the migration, I can spend more time focusing on it (as opposed to 
> our old system) and actually get into the code to improve everyone's 
> experience.  If anyone has any code primers on the structure of ispman, 
> I would appreciate it, I am eager to get more involved in this.
> 
> Jeff Meden
> 
> 
> Wim Kerkhoff wrote:
> 
> > Jeff Meden wrote:
> >
> >> Has anyone tested the scaled-up efficiency of the ispman-agent and 
> >> process system?  I did a test run with 520 domains and the result was 
> >> very depressing.  While the domains were being submitted rapidly by 
> >> the CLI, the agent was powerless to keep up with the rate at which 
> >> the new processes were being entered.  After the entire test run of 
> >> domains loaded into LDAP, at a rate of about 3 seconds per domain, 
> >> the agent is up to about 35 seconds to process each new domain, which 
> >> is the result of rebuilding the dns index and zones, and postfix 
> >> hashes for every set of processes.  This has effectively killed the 
> >> front end, the time before any remotely new processes get handled is 
> >> several hours, and the server's load is pegged.  There are a few 
> >> possible solutions, but I haven't looked at the code in depth enough 
> >> to say with authority what they might be.  Is this too unlikely of a 
> >> situation to consider worth improvement?
> >>
> > What kind of system specs? SMP? SCSI?
> >
> > While I haven't seen the process queue process be a problem yet (only 
> > have 50 domains so far), I have noticed that both the CLI tools and 
> > the web-based tools aren't very responsive.
> >
> > About 4-5 years ago I worked on a similar platform, that is, a Linux 
> > blackbox that had a pretty web-based GUI, user management, email, 
> > firewalling, local file server, and so on, using an SQL database for 
> > things like settings and firewall rules. PostgreSQL itself was fast, 
> > but because the way the Perl modules were designed, it could be a real 
> > dog when the minutely cronjob ran.
> >
> > A process queuing mechanism sounds like a great idea, and in fact it 
> > does work pretty good in the distributed environment that ISPMan 
> > supports. Processes that are spun off as a result of clicking around 
> > in the GUI are processed fairly quickly.
> >
> > The way I see it, the big bottleneck is when there are many actions in 
> > the queue that are extremely similar. Yet, they are being processed 
> > one at a time. For example, let's say that when adding 500 domains, a 
> > new process will be created for each domain to add a line to a file in 
> > /etc. So, that's 500 inserts into LDAP. The agent then finds his 
> > processes, and for each one, does a functional call to the appropriate 
> > library. The library's sub routine then opens /etc/foo, seeks to the 
> > end, writes a line, closes the file. The "smarter" way is for the 
> > agent to send the list of 500 domains to that sub routine, so that 
> > they can be all added at once. That's just one example. I've seen code 
> > (not in ISPMan) where the sub routine would lookup into the database 
> > for some more info, then rewrite the file. Having to rewrite a file 
> > 500 times requires lots of disk i/o and SQL/LDAP crunching.
> >
> > What is the biggest installation of ISPMan out there? That is, in 
> > domains per server/cluster?
> >
> > Overall, I really like the architecture/model of ISPMan. However, 
> > sometimes I think that the capabilities of LDAP are being pushed to 
> > the limit. Data that is of a relational nature should really be in an 
> > RDBMS, or not? My experience so far has been DBI/SQL is much easier to 
> > work with then LDAP. (Yes, I'm probably biased :-) )
> >
> > It could be interesting to tweak the ispman-agent to have sub agents. 
> > That is, separate perl processes dedicated to processing Apache config 
> > changes, BIND changes, manipulating Cyrus users, and so on. My mind 
> > has a better job designing that with SQL table schemas in mind, but 
> > perhaps it can be done with pure LDAP as well.
> >
> > Over the next month or two, I'm hoping to be able to spend a lot of 
> > time with my hands in the back/front end code. I said that a month 
> > ago, but things have changed in my day job since then, so we'll see 
> > what happens... I'd like to spend time on database integration, 
> > user-configurable mail options for spam/virii, billing integration, 
> > revise JavaScript/DHTML, and so on.
> >
> > Wim
> 
> 
> 
> -------------------------------------------------------
> The SF.Net email is sponsored by EclipseCon 2004
> Premiere Conference on Open Tools Development and Integration
> See the breadth of Eclipse activity. February 3-5 in Anaheim, CA.
> http://www.eclipsecon.org/osdn
> _______________________________________________
> Ispman-users mailing list
> Ispman-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/ispman-users



-------------------------------------------------------
SF.Net is sponsored by: Speed Start Your Linux Apps Now.
Build and deploy apps & Web services for Linux with
a free DVD software kit from IBM. Click Now!
http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click
_______________________________________________
Ispman-users mailing list
Ispman-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ispman-users
[prev in list] [next in list] [prev in thread] [next in thread]