'Re: [Filesystem-dev] Re: [Common-dev] Re: [Client-dev] CR: shared'

[prev in list] [next in list] [prev in thread] [next in thread] 

List:       helix-filesystem-dev
Subject:    Re: [Filesystem-dev] Re: [Common-dev] Re: [Client-dev] CR: shared
From:       Qiang Luo <qluo () real ! com>
Date:       2010-10-25 19:39:16
Message-ID: 4CC5DCE4.8010208 () real ! com
[Download RAW message or body]

OK, I will revoke this CR request and take the discussion offline.  I 
will do some resolver performance timing measurement for android and 
other platforms to see the best way to address the resolver performance 
issues if any.

Thanks,
Qiang

On 10/25/2010 12:20 PM, Sheldon Fu wrote:
> It's probably still per media platform. Since network service is a
> shared lib. Static data structure inside it can only live as long as the
> shared lib is kept in memory. This means that on Android, the cache
> effectively can live at most as long as the media platform instance and
> probably shorter if it is unloaded early (like when the client engine is
> unloaded).
>
> fxd
>
> Jamie Gordon wrote:
>> On 10/25/2010 11:39 AM, Qiang Luo wrote:
>>
>>> Jamie and Sheldon,
>>>
>>> Thanks for the review and comments.
>>>
>>> I was using the 310 branch, and I couldn't see any caching occurring
>>> when I traced the code.  Could you be more specific about the client
>>> resolver caching usage/code location Jamie?
>>>
>>>
>> Yeah the one I saw in the client is that HXResolver (in
>> client/hxnetwksvc) uses conn::dns_find_ip_addr (common/netio), which
>> maintains a static cache. It is just static though, so it would be per
>> process instead of per media platform.
>>
>> -j
>>
>>
>>> Anyways, the address info cache object's life spane must be persistent
>>> throughout the media platforms lifetime to be useful.  The The address
>>> info cache as part of the network services or resolver object with the
>>> same lifetime is useless.  My address info cache implementation has a
>>> map with a lifetime the same as that of media platform.
>>>
>>> See some response inline...
>>>
>>> On 10/25/2010 10:03 AM, Jamie Gordon wrote:
>>>
>>>> hmmm it also looks like there is an older cache implementation that
>>>> is used by the client resolver, so maybe it is just a bug that it is
>>>> not caching results or not finding result in the cache?
>>>>
>>>> On 10/25/2010 9:15 AM, Jamie Gordon wrote:
>>>>
>>>>> We actually do already have DNS caching implemented which has long been
>>>>> used by the common and server resolvers, but the client resolver
>>>>> implementation doesn't use it! You should probably be able to just that
>>>>> cache directly within the client resolver. One thing that might be a
>>>>> little funky is that the client's resolver is still implemented via
>>>>> the old net API (with a shim bridging to the newer API) and the cache
>>>>> uses the new APIs. I think the only thing that matters as far as that
>>>>> goes is for address passing.
>>>>>
>>>>> Resolver cache is in common/netio, resolvecache.[h|cpp]. Resolvers using
>>>>> it are in common/netio (iresolv.[h|cpp]) and server/engine/netio
>>>>> (servresolvimp.[h|cpp] for reference.
>>>>>
>>>>> -Jamie
>>>>>
>>>>>
>>> It could have saved me a lot of time reinventing the cache code...  I
>>> will take a look and see how I can use the existing code.
>>>
>>>>> On 10/25/2010 8:27 AM, Sheldon Fu wrote:
>>>>>
>>>>>> DNS record has a TTL field to specify how long the DNS query result can
>>>>>> be cached. We need to use that to determine how long we can keep the
>>>>>> result, instead of an arbitrary constant, to be conforming to the DNS
>>>>>> spec. DNS load balance is widely deployed now. We need to respect what
>>>>>> the server wants.
>>>>>>
>>>>>>
>>> I initially considered the TTL stuff.  Normally the TTL are in hours if
>>> not in days.  The client is not a dns server and the cache has the same
>>> time span as that of the mediaplatform.  I will see if I can get TTL
>>> from the current client code.  If not, using a hard coded exp value
>>> maybe reasonable here since all we are interested in is to cut the DNS
>>> query from 3 every 10s to 3 every 30m.
>>>
>>>>>> And I think this caching logic should be implemented in the resolver
>>>>>> itself, not http filesystem. All the changes here are generic to the
>>>>>> resolving logic, not related to http at all.
>>>>>>
>>>>>>
>>> Yes you are correct, logically the cache should belong to/used by the
>>> resolver.  Here I was trying to skip the resolver setup all together to
>>> be more efficient.
>>>
>>> Qiang
>>>
>>>
>>>>>> Also according to some online discussion, Android's resolver (in bionic)
>>>>>> already has a caching mechanism. Not sure how Helix resolver is
>>>>>> implemented. It might be that we are not calling the correct API.
>>>>>>
>>>>>>
>>> The caching code can be used for all platforms.  This feature maybe put
>>> under a new helix feature defines to be included only in platform builds
>>> that can benefit from it.
>>>
>>>>>> For this particular httplive use case though, instead of having this DNS
>>>>>> caching enhancement, it might be more beneficial to implement the logic
>>>>>> to re-use the socket connection (http persistent connection) because
>>>>>> even with dns caching, you'll have to do that still.
>>>>>>
>>>>>> fxd
>>>>>>
>>>>>> Qiang Luo wrote:
>>>>>>
>>>>>>> Overview:
>>>>>>> Helix file system creates resolver to do dns lookup for each
>>>>>>> connection.  It doesn't cache the address info data.  This haven't been
>>>>>>> any issue until recently where we need to create httpfileobj frequently
>>>>>>> to handle httplive streaming's usage.  For httplive streaming, there can
>>>>>>> be 3 httpfsys connections, playlist update, ts segment file downloading,
>>>>>>> decryption key downloading for each segment.  The typical segment
>>>>>>> duration is 6 to10 seconds.  In this case, a resolver-addressInfo cache
>>>>>>> would be beneficial in improve the httpfilesystem performance.
>>>>>>>
>>>>>>> In this implementation, we create a sheared map to store the address
>>>>>>> info data/objects.  Instead of having a global map for a process, we
>>>>>>> create a single map for the singleton media platform.
>>>>>>>
>>>>>>> Branches:
>>>>>>> head, hxclient_3_1_0_atlas
>>>>>>>
>>>>>>> files added:
>>>>>>> common/include/ihxaddressinfomap.h
>>>>>>> common/util/hxaddrinfomap.cpp
>>>>>>> common/util/pub/hxaddrinfomap.h
>>>>>>>
>>>>>>> files modified:
>>>>>>> common/util/Umakefil
>>>>>>> client/medpltfm/chxmedpltfm.cpp
>>>>>>> client/medpltfm/dlliids.cpp
>>>>>>> client/medpltfm/pub/chxmedpltfm.h
>>>>>>> filesystem/http/factory.cpp
>>>>>>> filesystem/http/httpfsys.cpp
>>>>>>> filesystem/http/httpfsys.h
>>>>>>>
>>>>>>> Please review the implementation
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Qiang
>>>>>>>
>>>>>>>
>>>>>> _______________________________________________
>>>>>> Common-dev mailing list
>>>>>> Common-dev@helixcommunity.org
>>>>>> http://lists.helixcommunity.org/mailman/listinfo/common-dev
>>>>>>
>>>>> _______________________________________________
>>>>> Filesystem-dev mailing list
>>>>> Filesystem-dev@helixcommunity.org
>>>>> http://lists.helixcommunity.org/mailman/listinfo/filesystem-dev
>>>>>
>>>> _______________________________________________
>>>> Filesystem-dev mailing list
>>>> Filesystem-dev@helixcommunity.org
>>>> http://lists.helixcommunity.org/mailman/listinfo/filesystem-dev
>>>> .
>>>>
>>>>
>> .
>>
>>
> .
>

_______________________________________________
Filesystem-dev mailing list
Filesystem-dev@helixcommunity.org
http://lists.helixcommunity.org/mailman/listinfo/filesystem-dev
[prev in list] [next in list] [prev in thread] [next in thread] 