'[Opensim-users] Intermittent crashing - System.OutofMemoryException'

[prev in list] [next in list] [prev in thread] [next in thread] 

List:       opensim-users
Subject:    [Opensim-users] Intermittent crashing -	System.OutofMemoryException
From:       jjustincc () googlemail ! com (Justin Clark-Casey)
Date:       2012-02-21 4:28:46
Message-ID: 4F431D7E.2040901 () googlemail ! com
[Download RAW message or body]

In theory, running each simulator in it's own process should have a higher memory \
overhead but some aspects of  performance may be better.  For instance, with ODE only \
one thread can call the collider at a time in right now in any  particular OpenSim \
process.  How much this impacts performance I can't yet say but not enforcing this \
gets you random  hard ODE crashes.

It also just so happens that as of half an hour ago, in git master a27e5a9, \
OpenSim.exe can be run as a 64-bit Windows  process (I got round to getting the \
necessary 64-bit native Windows libraries and finding out how to get Windows to  \
automatically load the correct ones).

So you should be able to run OpenSim.exe now (rather than OpenSim.32BitLaunch.exe) \
and access upto 8TB.  This is still  experimental but if it works for most people \
then I'll almost certainly pop it into the forthcoming 0.7.3.

On 20/02/12 12:43, Fleep Tuque wrote:
> I may be crazy, but I swear I'm seeing an immediate and very noticeable improvement \
> in performance from the end user perspective.  Avatar movement not as sluggish, \
> almost snappy even, textures rezzing more quickly.. 
> The CPU usage is about the same as before the split (57-59% on average with no \
> activity) but the Page File usage is higher.  I'll be interested to see how things \
> go when the HGAC visits tomorrow to give a good test with higher concurrency. 
> Thanks again!
> 
> - Chris/Fleep
> 
> 
> Chris M. Collins (SL/OS: Fleep Tuque)
> Center for Simulations & Virtual Environments Research (UCSIM)
> UCIT Instructional & Research Computing
> University of Cincinnati
> 406A Zimmer Hall
> 315 College Drive
> PO BOX 210088
> Cincinnati, OH 45221-0088
> chris.collins at uc.edu <mailto:chris.collins at uc.edu>
> (513) 556-3018
> 
> http://ucsim.uc.edu
> 
> 
> 
> On Mon, Feb 20, 2012 at 7:28 AM, Gary Banham <banham1 at telus.net <mailto:banham1 \
> at telus.net>> wrote: 
> Your more than welcome____
> 
> TC____
> 
> Gary____
> 
> __ __
> 
> *From:*opensim-users-bounces at lists.berlios.de <mailto:opensim-users-bounces at \
> lists.berlios.de> [mailto:opensim-users-bounces at lists.berlios.de \
> <mailto:opensim-users-bounces at lists.berlios.de>] *On Behalf Of *Fleep Tuque
> *Sent:* Monday, February 20, 2012 5:26 AM
> 
> 
> *To:* opensim-users at lists.berlios.de <mailto:opensim-users at lists.berlios.de>
> *Subject:* Re: [Opensim-users] Intermittent crashing - \
> System.OutofMemoryException____ 
> __ __
> 
> Wow, it really was that simple.  I successfully split off a few regions into a \
> separate instance on FleepGrid with no trouble at all. ____
> 
> __ __
> 
> The key, as Gary mentioned, was uncommenting the http_listener_port line (267 in my \
> opensim.ini file) and incrementing the TCP port (in my case to 9001) for the new \
> instance.____ 
> __ __
> 
> Thanks Gary!  I'll have to work out how to split the regions up logically for our \
> UCSIM grid but I'm surprised it's that simple!  Yay!  :)____
> 
> __ __
> 
> - Chris/Fleep____
> 
> __ __
> 
> __ __
> 
> Chris M. Collins (SL/OS: Fleep Tuque)____
> 
> Center for Simulations & Virtual Environments Research (UCSIM)____
> 
> UCIT Instructional & Research Computing____
> 
> University of Cincinnati____
> 
> 406A Zimmer Hall____
> 
> 315 College Drive____
> 
> PO BOX 210088____
> 
> Cincinnati, OH 45221-0088____
> 
> chris.collins at uc.edu <mailto:chris.collins at uc.edu>____
> 
> (513) 556-3018 <tel:%28513%29%20556-3018>____
> 
> __ __
> 
> http://ucsim.uc.edu____
> 
> On Mon, Feb 20, 2012 at 6:48 AM, Gary Banham <banham1 at telus.net <mailto:banham1 \
> at telus.net>> wrote:____ 
> PS____
> 
> just an additional note I think the problem is the single instance of Opensim will \
> only access 2 gigs of memory ____ 
> with a large number of regions and lots of building and scripts the single instance \
> can't count that high it hits a wall____
> 
> by placing regions over a number of instances each can use 2 gigs____
> 
> so yes you use more memory EG 5 instances could potentially use 10 gigs of memory \
> but one instance only address 2 Gigs____
> 
> ____
> 
> TC____
> 
> Gary____
> 
> ____
> 
> *From:*opensim-users-bounces at lists.berlios.de <mailto:opensim-users-bounces at \
> lists.berlios.de> [mailto:opensim-users-bounces at lists.berlios.de \
> <mailto:opensim-users-bounces at lists.berlios.de>] *On Behalf Of *Fleep Tuque____
> 
> 
> *Sent:* Monday, February 20, 2012 4:34 AM
> *To:* opensim-users at lists.berlios.de <mailto:opensim-users at \
> lists.berlios.de>____ 
> *Subject:* Re: [Opensim-users] Intermittent crashing - \
> System.OutofMemoryException____ 
> ____
> 
> I'm worried that you may be right about that and it's just a matter of time before \
> adding more content becomes a problem, in which case it sounds like perhaps instead \
> of running all the regions on a single opensim instance I should split them up \
> across several as Gary mentioned above?____ 
> ____
> 
> I'm not sure exactly how that works, I guess, and I'm trying to understand how that \
> would use less memory instead of more..  Would I simply copy the contents of the \
> opensim directory to multiple locations and break the regions up in the regions.ini \
> file or..? ____ 
> ____
> 
> I'd like to experiment with it on FleepGrid first before I hose our campus install, \
> but the HGAC group is coming for a visit in a couple days so if anyone has pointers \
> about how to do that splitting up process before I break anything, I'm all ears.  \
> :)____ 
> ____
> 
> - Chris/Fleep____
> 
> ____
> 
> ____
> 
> Chris M. Collins (SL/OS: Fleep Tuque)____
> 
> Center for Simulations & Virtual Environments Research (UCSIM)____
> 
> UCIT Instructional & Research Computing____
> 
> University of Cincinnati____
> 
> 406A Zimmer Hall____
> 
> 315 College Drive____
> 
> PO BOX 210088____
> 
> Cincinnati, OH 45221-0088____
> 
> chris.collins at uc.edu <mailto:chris.collins at uc.edu>____
> 
> (513) 556-3018 <tel:%28513%29%20556-3018>____
> 
> ____
> 
> http://ucsim.uc.edu____
> 
> ____
> 
> ____
> 
> On Thu, Feb 16, 2012 at 10:19 PM, Justin Clark-Casey <jjustincc at googlemail.com \
> <mailto:jjustincc at googlemail.com>> wrote:____
> 
> I would actually be rather surprised if errors during the OAR import were related \
> to your memory problems.  I think that it's more likely that the content of the OAR \
> in some way pushed you beyond memory boundaries and wiping it has reset things.
> 
> Still, if the problem is solved then that's what matters :)____
> 
> 
> 
> On 16/02/12 22:50, Fleep Tuque wrote:____
> 
> Hi all,
> 
> After going through the log files and trying to figure out what changed in the time \
> period when the out of memory errors
> began, we traced it back to an OAR file import. (Thanks to whoever suggested that \
> might be an issue earlier in this thread!)
> 
> I'd ported over a copy of the FleepGrid Plaza region so my students could easily \
> grab some of the freebies I'd created,
> and it looks like something got horked during the import to the UCSIM grid:
> 
> 012-02-07 09:12:05,991 ERROR - Flotsam.RegionModules.AssetCache.FlotsamAssetCache \
> [FLOTSAM ASSET CACHE]: Failed to update cache for asset \
> 3d0b3a52-40e5-48fe-8595-b72e163917fa.  Exception The process cannot access the file \
> 'c:\Opensim\bin\assetcache\3d0\3d0b3a52-40e5-48fe-8595-b72e163917fa' because it is \
> being used by another process.    at
> System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
> at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 \
> rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, \
> SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy)
> at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, \
> FileShare share, Int32 bufferSize) at System.IO.File.OpenFile(String path, \
> FileAccess access, SafeFileHandle& handle) at \
> System.IO.File.SetLastAccessTimeUtc(String path, DateTime lastAccessTimeUtc) at \
> Flotsam.RegionModules.AssetCache.FlotsamAssetCache.UpdateFileCache(String key, \
> AssetBase asset) 
> Shortly after that, we begin to see the first out of memory errors..
> 
> 2012-02-07 09:15:17,453 DEBUG - OpenSim.Region.ScriptEngine.XEngine.XEngine \
> [XEngine] Loaded script LK-Stool-Sit #2.poseball-zero-lag, script UUID \
> e73e1114-297a-43a7-b4b8-dd423cd83b95, prim UUID \
> 2ae59628-943d-49ec-9673-ee3465bd0242 @ <168.1464, 156.7953, 26.90887>.UCSIM Staff 1
> 2012-02-07 09:15:17,984 ERROR - OpenSim.Region.ScriptEngine.XEngine.XEngine \
>                 [XEngine] Exception creating app domain:
> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was \
> thrown. at System.AppDomain.nCreateDomain(String friendlyName, AppDomainSetup \
> setup, Evidence providedSecurityInfo, Evidence
> creatorsSecurityInfo, IntPtr parentSecurityDescriptor)
> at System.AppDomain.CreateDomain(String friendlyName, Evidence securityInfo, \
> AppDomainSetup info) at \
> OpenSim.Region.ScriptEngine.XEngine.XEngine.DoOnRezScript(Object[] parms) 
> And we see those out of memory errors stack up until eventually the application \
> crashed.. 
> When we realized it was probably the OAR file that was the source of the issue, I \
> wiped the region by importing a fresh
> "blank" OAR and we haven't had the out of memory exception since.
> 
> Looks like the problem is resolved and super thanks again to everyone who helped \
> with pointers and info in this thread -
> much appreciated!
> 
> - Chris/Fleep
> 
> 
> Chris M. Collins (SL/OS: Fleep Tuque)
> Center for Simulations & Virtual Environments Research (UCSIM)
> UCIT Instructional & Research Computing
> University of Cincinnati
> 406A Zimmer Hall
> 315 College Drive
> PO BOX 210088
> Cincinnati, OH 45221-0088____
> 
> chris.collins at uc.edu <mailto:chris.collins at uc.edu> <mailto:chris.collins at \
> uc.edu <mailto:chris.collins at uc.edu>> (513) 556-3018 <tel:%28513%29%20556-3018>
> 
> http://ucsim.uc.edu____
> 
> 
> 
> 
> On Thu, Feb 16, 2012 at 2:16 PM, R.Gunther <rigun at rigutech.nl <mailto:rigun at \
> rigutech.nl> <mailto:rigun at rigutech.nl <mailto:rigun at rigutech.nl>>> wrote:
> 
> There's a 3e option. Munin.
> http://munin-monitoring.org/
> 
> 
> On 2012-02-16 16:49, BlueWall wrote:
> 
> The rrdtools is very good for storing and organizing the data for statistics and it \
> will also produce graphs from that data. It relies on external scripts to collect \
> the data - that's where Cacti comes in. It has the scripts and templates to graph \
> your system (and snmp devices) in the distribution. I have done it on Linux and
> they claim to work on Windows too.
> 
> You could also write scripts and templates to graph OpenSim. I haven't tried that, \
> though. There is documentation included in the distro, and scattered across the \
> Cacti user forums. 
> -BlueWall
> 
> On 02/16/2012 09:55 AM, Rick Anderson wrote:
> 
> One question about Cacti, and RDDTools. The sites describe them as
> general purpose graphing tools. Is there any documentation or tutorial
> for configuring them to monitor system status. Maybe, it's really
> simple, and I'm over complicating it.
> 
> -_Rick____
> 
> _____________________________________________________
> 
> 
> Opensim-users mailing list
> Opensim-users at lists.berlios.de <mailto:Opensim-users at lists.berlios.de> \
> <mailto:Opensim-users at lists.berlios.de <mailto:Opensim-users at \
> lists.berlios.de>>____ 
> https://lists.berlios.de/__mailman/listinfo/opensim-users
> <https://lists.berlios.de/mailman/listinfo/opensim-users>____
> 
> 
> 
> 
> 
> 
> _______________________________________________
> Opensim-users mailing list
> Opensim-users at lists.berlios.de <mailto:Opensim-users at lists.berlios.de>
> https://lists.berlios.de/mailman/listinfo/opensim-users____
> 
> ____
> 
> 
> --
> Justin Clark-Casey (justincc)
> http://justincc.org/blog
> http://twitter.com/justincc____
> 
> _______________________________________________
> Opensim-users mailing list
> Opensim-users at lists.berlios.de <mailto:Opensim-users at lists.berlios.de>
> https://lists.berlios.de/mailman/listinfo/opensim-users____
> 
> ____
> 
> 
> _______________________________________________
> Opensim-users mailing list
> Opensim-users at lists.berlios.de <mailto:Opensim-users at lists.berlios.de>
> https://lists.berlios.de/mailman/listinfo/opensim-users____
> 
> __ __
> 
> 
> _______________________________________________
> Opensim-users mailing list
> Opensim-users at lists.berlios.de <mailto:Opensim-users at lists.berlios.de>
> https://lists.berlios.de/mailman/listinfo/opensim-users
> 
> 
> 
> 
> _______________________________________________
> Opensim-users mailing list
> Opensim-users at lists.berlios.de
> https://lists.berlios.de/mailman/listinfo/opensim-users

-- 
Justin Clark-Casey (justincc)
http://justincc.org/blog
http://twitter.com/justincc

[prev in list] [next in list] [prev in thread] [next in thread] 