[prev in list] [next in list] [prev in thread] [next in thread]
List: calendarserver-users
Subject: Re: [CalendarServer-users] DCS Production Deployment
From: Gaurav Jain <monkeyfdude () gmail ! com>
Date: 2015-02-19 1:33:40
Message-ID: CAJ9azkD0bPXZUHVCwOV0QDpGA5J_taO0zHyQWUzR=RTKMGQ2HA () mail ! gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
Thank you very much for all your support!
On Wed, Feb 18, 2015 at 4:49 PM, Andre LaBranche <dre@apple.com> wrote:
> Hi Rhodey,
>
> Comments inline.
>
> > On Feb 18, 2015, at 4:18 PM, rhodey <rhodey@anhonesteffort.org> wrote:
> >
> > I've been running DCS in production for the past ~year, at the moment my
> > deployment supports ~800 users. It took *a lot* of reading through doc
> > and code to feel even remotely comfortable with storing real data for
> > real people and I still don't feel all that great about it.
>
> We are slowly compiling actual documentation. I'll take this as feedback
> in support of accelerating this process.
>
> > ...
> >
> > 2. DCS uses the python twisted framework but at the same time it
> > doesn't-- calls to DirectoryService (regardless of the implementation)
> > are blocking 0.o
>
> I believe this is addressed in 6.0+ with the new twext.who stuff that is
> all async, iirc.
>
> > 3. DCS has this weird multi-process, master-slave concept where multiple
> > DCS instances run on the same box-- I just think this is weird and
> > dealing with dead processes has been strange.
>
> The Python GIL means we need a multi process architecture to saturate CPU
> on multi core systems. Technically these processes are all part of the same
> service instance because they all cooperate on doing the work.
>
> > 4. If a user uploads a contact containing a photo and you're using a
> > database for storage (as you should), DCS will store that photo in your
> > database-- yuck! Even with your 1000 users, each with maybe 100 contact
> > photos, things get kinda bloated. So, I modified DCS to strip contact
> > photos from contacts and store them in Amazon S3, then piece them back
> > together on demand.
>
> Cool hack :)
>
> > 5. Someone could DOS my DCS instances via network resource exhaustion
> > very easily and there is very little that I could do to stop it other
> > than launch a crazy number of EC2 instances and burn through money. DCS
> > is not efficient (blocking calls to DirectoryService, etc) and I feel
> > very uncomfortable having any less than than two m3.medium instances
> > running at all times (costly!).
>
> Understood, however this is true of any network service that limits its
> own concurrency (imo this is required of any sane service). Part of the
> challenge is the extremely large range of request cost for legitimate
> CalDAV requests. We are moving to a model where we do more work a
> synchronously in a database work queue, and we now have a notion of cost in
> that implementation, so we are becoming better equipped to address resource
> utilization problems.
>
> > All that being said, DCS is still the best free and open source option
> > out there but I wish I had written my own WebDAV server from scratch
> > using 'dropwizard' instead. The world of high availability services
> > changes so rapidly and many of the ideas in this project are dated--
> > like, even table joins really shouldn't exist anymore.
>
> We're not yet ready for NoSQL but maybe someday :)
>
> Thanks for the feedback. I'm interested to hear anything else you'd like
> to share about your experiences hosting DCS.
>
> -dre
>
> >
> > -- rhodey orbits
> >
> >> On 02/18/2015 01:04 PM, Gaurav Jain wrote:
> >> Hi,
> >>
> >> Thank you very much for CalendarServer
> >>
> >> I have few questions:
> >>
> >> * I used CalendarServer-5.2.2 during development phase.
> >> * Now I want to use it for some real (users) use cases.
> >> * Right now, I do not have any scaling issues.
> >> * I run it on Ubuntu EC2 instance.
> >>
> >> My question is:
> >>
> >> * Has CalendarSever-5.2.2 been used in PROD deployments?
> >> * Since I don't have scaling issues, is it ok to use XMLDirectoryService
> >> for like 1000 accounts?
> >>
> >> Any tip would be useful for PROD deployments. Even for 1000 users, we
> would
> >> like it not to go down at all.
> >>
> >>
> >> Best Regards,
> >> Gaurav Jain
> >>
> >>
> >>
> >> _______________________________________________
> >> calendarserver-users mailing list
> >> calendarserver-users@lists.macosforge.org
> >> https://lists.macosforge.org/mailman/listinfo/calendarserver-users
> > _______________________________________________
> > calendarserver-users mailing list
> > calendarserver-users@lists.macosforge.org
> > https://lists.macosforge.org/mailman/listinfo/calendarserver-users
> _______________________________________________
> calendarserver-users mailing list
> calendarserver-users@lists.macosforge.org
> https://lists.macosforge.org/mailman/listinfo/calendarserver-users
>
[Attachment #5 (text/html)]
<div dir="ltr">Thank you very much for all your support!</div><div \
class="gmail_extra"><br><div class="gmail_quote">On Wed, Feb 18, 2015 at 4:49 PM, \
Andre LaBranche <span dir="ltr"><<a href="mailto:dre@apple.com" \
target="_blank">dre@apple.com</a>></span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex">Hi Rhodey,<br> <br>
Comments inline.<br>
<span class=""><br>
> On Feb 18, 2015, at 4:18 PM, rhodey <<a \
href="mailto:rhodey@anhonesteffort.org">rhodey@anhonesteffort.org</a>> wrote:<br> \
><br> > I've been running DCS in production for the past ~year, at the \
moment my<br> > deployment supports ~800 users. It took *a lot* of reading through \
doc<br> > and code to feel even remotely comfortable with storing real data \
for<br> > real people and I still don't feel all that great about it.<br>
<br>
</span>We are slowly compiling actual documentation. I'll take this as feedback \
in support of accelerating this process.<br> <br>
> ...<br>
<span class="">><br>
> 2. DCS uses the python twisted framework but at the same time it<br>
> doesn't-- calls to DirectoryService (regardless of the implementation)<br>
> are blocking 0.o<br>
<br>
</span>I believe this is addressed in 6.0+ with the new twext.who stuff that is all \
async, iirc.<br> <span class=""><br>
> 3. DCS has this weird multi-process, master-slave concept where multiple<br>
> DCS instances run on the same box-- I just think this is weird and<br>
> dealing with dead processes has been strange.<br>
<br>
</span>The Python GIL means we need a multi process architecture to saturate CPU on \
multi core systems. Technically these processes are all part of the same service \
instance because they all cooperate on doing the work.<br> <span class=""><br>
> 4. If a user uploads a contact containing a photo and you're using a<br>
> database for storage (as you should), DCS will store that photo in your<br>
> database-- yuck! Even with your 1000 users, each with maybe 100 contact<br>
> photos, things get kinda bloated. So, I modified DCS to strip contact<br>
> photos from contacts and store them in Amazon S3, then piece them back<br>
> together on demand.<br>
<br>
</span>Cool hack :)<br>
<span class=""><br>
> 5. Someone could DOS my DCS instances via network resource exhaustion<br>
> very easily and there is very little that I could do to stop it other<br>
> than launch a crazy number of EC2 instances and burn through money. DCS<br>
> is not efficient (blocking calls to DirectoryService, etc) and I feel<br>
> very uncomfortable having any less than than two m3.medium instances<br>
> running at all times (costly!).<br>
<br>
</span>Understood, however this is true of any network service that limits its own \
concurrency (imo this is required of any sane service). Part of the challenge is the \
extremely large range of request cost for legitimate CalDAV requests. We are moving \
to a model where we do more work a synchronously in a database work queue, and we now \
have a notion of cost in that implementation, so we are becoming better equipped to \
address resource utilization problems.<br> <span class=""><br>
> All that being said, DCS is still the best free and open source option<br>
> out there but I wish I had written my own WebDAV server from scratch<br>
> using 'dropwizard' instead. The world of high availability services<br>
> changes so rapidly and many of the ideas in this project are dated--<br>
> like, even table joins really shouldn't exist anymore.<br>
<br>
</span>We're not yet ready for NoSQL but maybe someday :)<br>
<br>
Thanks for the feedback. I'm interested to hear anything else you'd like to \
share about your experiences hosting DCS.<br> <br>
-dre<br>
<div class="HOEnZb"><div class="h5"><br>
><br>
> -- rhodey orbits<br>
><br>
>> On 02/18/2015 01:04 PM, Gaurav Jain wrote:<br>
>> Hi,<br>
>><br>
>> Thank you very much for CalendarServer<br>
>><br>
>> I have few questions:<br>
>><br>
>> * I used CalendarServer-5.2.2 during development phase.<br>
>> * Now I want to use it for some real (users) use cases.<br>
>> * Right now, I do not have any scaling issues.<br>
>> * I run it on Ubuntu EC2 instance.<br>
>><br>
>> My question is:<br>
>><br>
>> * Has CalendarSever-5.2.2 been used in PROD deployments?<br>
>> * Since I don't have scaling issues, is it ok to use \
XMLDirectoryService<br> >> for like 1000 accounts?<br>
>><br>
>> Any tip would be useful for PROD deployments. Even for 1000 users, we \
would<br> >> like it not to go down at all.<br>
>><br>
>><br>
>> Best Regards,<br>
>> Gaurav Jain<br>
>><br>
>><br>
>><br>
>> _______________________________________________<br>
>> calendarserver-users mailing list<br>
>> <a href="mailto:calendarserver-users@lists.macosforge.org">calendarserver-users@lists.macosforge.org</a><br>
>> <a href="https://lists.macosforge.org/mailman/listinfo/calendarserver-users" \
target="_blank">https://lists.macosforge.org/mailman/listinfo/calendarserver-users</a><br>
> _______________________________________________<br>
> calendarserver-users mailing list<br>
> <a href="mailto:calendarserver-users@lists.macosforge.org">calendarserver-users@lists.macosforge.org</a><br>
> <a href="https://lists.macosforge.org/mailman/listinfo/calendarserver-users" \
target="_blank">https://lists.macosforge.org/mailman/listinfo/calendarserver-users</a><br>
_______________________________________________<br>
calendarserver-users mailing list<br>
<a href="mailto:calendarserver-users@lists.macosforge.org">calendarserver-users@lists.macosforge.org</a><br>
<a href="https://lists.macosforge.org/mailman/listinfo/calendarserver-users" \
target="_blank">https://lists.macosforge.org/mailman/listinfo/calendarserver-users</a><br>
</div></div></blockquote></div><br></div>
_______________________________________________
calendarserver-users mailing list
calendarserver-users@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/calendarserver-users
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic