[prev in list] [next in list] [prev in thread] [next in thread] 

List:       cobbler
Subject:    cobbler API
From:       mdehaan () redhat ! com (Michael DeHaan)
Date:       2009-01-28 16:56:24
Message-ID: 49808E38.2050700 () redhat ! com
[Download RAW message or body]

Michael DeHaan wrote:
> Shuichi Ihara wrote:
>   
>>   
>>     
>>> The cobbler command line is written in the Python API and can do 100% of 
>>> everything.
>>>     
>>>       
>> Sounds great!
>> But, why is 'cobbler xxx' slow even though cobbler command line is written
>> in the Python API? In my testing for 1000 system registration into cobbler, it
>> was 2411[sec] ('cobbler add xxxx') VS 4[sec] (python based script with API).
>> So, what's most overheads in cobbler commands?
>>   
>>     
>
> One aspect is the startup cost is largely in starting a new Python 
> instance.   As your system count grows, there is also more cost for 
> loading files off disk and into memory.  Each time you start up a new 
> CLI session it does this.    With an open API handle you are doing these 
> things in batch.  So the total number of file reads is Sigma N...1000 of 
> N (sum the series in your heads, math students) with additions, or 1000 
> * 1000 for edits, which is not super efficient as you can imagine.   The 
> API however allows you to only load the configuration once, rather than 
> 1000 times.
>   

In the above I mean Sigma 1... 1000 of N, aka 500500 reads for 1000 
inserts.   Obviously sub-optimal, though we will address this :)
> There have been a number of recent improvements in cobbler to make the 
> XMLRPC API also not have this penalty.    Right now on the development 
> branch
> the Cobbler API loads the configuration only once -- when the service 
> starts and is notified of changes that occur outside of cobblerd so it 
> does not have to do reloads.  Also it does not suffer from having to 
> start up a new interpreter.
>
> Long term, the answer to this long term is to make the command line tool 
> also speak XMLRPC (locally) and never have to reload
> the configuration.  (A sql rewrite would be so intrusive to not make it 
> worth it, and risks being equally slow).    The cobbler CLI could also, 
> if we really wanted, be done in C, but I don't think we care that 
> much.   (time python /tmp/blank.py takes 1/3 of a second on my machine).   
>
> For most things this is already pretty easy to do, though it eventually 
> requires that we make the remaining cobbler functions that are not
> callable over XMLRPC usable over XMLRPC (like cobbler reposync and 
> cobbler import)
>
> I think it makes sense to persue this route sooner than later.  In fact, 
> we could make simple edits speedy before doing the "import-over-XMLRPC" 
> if we really wanted.   
>
> We could do this regardless of the cobbler auth system employed since 
> the cobbler CLI could have access to a common shared secret that 
> cobblerd could also read, and we could use SSL from Apache to provide an 
> encryption against local sniffing.
>
>   
>>   
>>     
>>> It looks like you're using the Python API from what you have below 
>>> (excellent!), in which case you have numerous examples.  The best thing 
>>> you can do is look in the "modules/" directory of a source checkout.  
>>> The cobbler CLI makes every call for every command line argument there, 
>>> and you can see what calls are made for each CLI getopt argument.
>>>     
>>>       
>> Thanks! I had a look at thees codes in modules directory and could make a python
>> scripts with API to add system information (includes mac_address, ip_address
>> whatever I want it) into cobbler.
>>
>>
>>   
>>     
>>> Pretty close.  The main thing is that I've exposed most of the internals 
>>> directly in API.py, so other than calling methods on the actual cobbler 
>>> objects like 'system', you can access everything through the API 
>>> module.    Anything lower level down is subject to change and isn't a 
>>> stable API.  api.py, however, is stable.
>>>
>>> import cobbler.api as capi               # avoid namespace issue as you 
>>> use 'api' later.
>>> api = capi.BootAPI()                     
>>> s = api.find_system(name='abc')    # abc is a string, plus find is a top 
>>> level API call, it's best not to go any deeper and just use the api.py 
>>> implementation
>>> s.set_hostname('xyz')                     # use the set_ methods to 
>>> ensure validation is always run on the input (very important!)
>>> api.add_system(s)                          # again, use the top level 
>>> API function, which sets reasonable defaults for all the arguments
>>>     
>>>       
>> This is very useful to me also. Thanks again.
>>
>>   
>>     
>
> No problem, glad to help!
>
> --Michael
> _______________________________________________
> cobbler mailing list
> cobbler at lists.fedorahosted.org
> https://fedorahosted.org/mailman/listinfo/cobbler
>   



[prev in list] [next in list] [prev in thread] [next in thread]