[prev in list] [next in list] [prev in thread] [next in thread]
List: grinder-use
Subject: Re: [Grinder-use] Creating a multi-threaded TestRunner class
From: "Noel O'Brien" <nobrien () newbay ! com>
Date: 2008-10-28 9:21:50
Message-ID: 200810281019.31921.nobrien () newbay ! com
[Download RAW message or body]
Hi Pol,
If you could provide an example of the way you did it, that would be great!
I'm always looking for better / alternative ways to solve problems.
Thanks,
Noel
On Friday 24 October 2008 19:45:24 Pol van Aubel wrote:
> Hi Noel,
>
> Last time I worked with The Grinder is over half a year ago, so I'd have
> to look up exactly how I did it, but for a company I worked for I
> created a test suite which did exactly that: Call an external class and
> manipulate test data programmatically. Of course, that way, you wouldn't
> actually get statistics for each upload thread seperately, you'd have to
> aggregate them and store them in the custom fields (average, total,
> standard deviation, stuff like that).
>
> I can see if I can give you a code example if you want. I looked at how
> the HTTPClient did it and just incorporated that in my own classes.
>
> Regarding your own methods, like I said, my own work with The Grinder
> isn't recent, so I don't think I can comment on it.
>
> Regards,
>
> Pol
>
> Noel O'Brien wrote:
> > Hi Pol,
> >
> > I need to measure stats from each upload call so i don't think this
> > approach works for me. Thanks for your suggestion though :)
> >
> > I have put together something using a combination of some of the examples
> > in the script gallery (serial and parrallel tests), which does what I
> > need; multi-user, multi-threaded, and repeatable, but it does have some
> > limitations
> >
> > from net.grinder.script.Grinder import grinder
> > from java.util.concurrent.atomic import AtomicInteger
> > from util.filehandling import removeFile
> >
> > testNum = AtomicInteger(1)
> > snooze = grinder.properties.getInt("sng.thinktime", 1000)
> >
> > ## run pre scripts
> > prescripts =
> > dict(grinder.properties.getPropertySubset("sng.pretestscript.")).items()
> > prescripts.sort()
> > for key, module in prescripts:
> > exec("import %s" % module)
> >
> > ## run multi-threaded upload scripts
> > scripts =
> > dict(grinder.properties.getPropertySubset("sng.testscript.")).items()
> > scripts.sort()
> > for key, module in scripts:
> > exec("import %s" % module)
> >
> > ## run post scripts
> > postscripts =
> > dict(grinder.properties.getPropertySubset("sng.posttestscript.")).items()
> > postscripts.sort()
> > for key, module in postscripts:
> > exec("import %s" % module)
> >
> >
> > def createTestRunner(key, module, snID):
> > exec("x = %s.TestRunner(%i, '%s', '%s')" % (module,
> > testNum.getAndIncrement(), key, snID))
> > return x
> >
> >
> > class TestRunner:
> > def __init__(self):
> > self.pid = grinder.getProcessNumber()
> > self.tid = grinder.getThreadNumber()
> >
> > # This method is called for every run.
> > def __call__(self):
> > try:
> >
> > if self.tid == 0:
> > self.__run(prescripts, "SNG")
> > grinder.properties.setBoolean("sng.
> > %i.media.resource.created" % (self.pid), True)
> >
> > grinder.properties.setBoolean("sng.%i.media.run.completed" % (self.pid),
> > False)
> >
> > self.__run(scripts, "SNG")
> > grinder.properties.setBoolean("sng.
> > %i.media.resource.part.uploaded.%i" % (self.pid, self.tid), True)
> >
> > ## Wait for the other threads to finish and notify thread
> > 0 looping = True
> > numThreads = grinder.properties.getInt("grinder.threads",
> > -1) while looping:
> > threadsNotified =
> > dict(grinder.properties.getPropertySubset("sng.
> > %i.media.resource.part.uploaded." % (self.pid))).values()
> > itemList = threadsNotified.count("true")
> > if itemList == numThreads: ## we're done
> > looping = False
> > grinder.properties.setBoolean("sng.
> > %i.media.resource.created" % (self.pid), False)
> > else:
> > grinder.sleep(100)
> >
> > ## reset the notifying flag back to false
> > for i in range(numThreads):
> > grinder.properties.setBoolean("sng.
> > %i.media.resource.part.uploaded.%i" % (self.pid, i), False)
> >
> > self.__run(postscripts, "SNG")
> >
> > ## signal to other threads that run is finished
> >
> > grinder.properties.setBoolean("sng.%i.media.run.completed" % (self.pid),
> > True)
> >
> > else:
> > while not grinder.properties.getBoolean("sng.
> > %i.media.resource.created" % (self.pid), False):
> > grinder.sleep(100)
> >
> > self.__run(scripts, "SNG")
> > grinder.properties.setBoolean("sng.
> > %i.media.resource.part.uploaded.%i" % (self.pid, self.tid), True)
> >
> > while not grinder.properties.getBoolean("sng.
> > %i.media.run.completed" % (self.pid), False):
> > grinder.sleep(100)
> >
> >
> > finally:
> > removeFile()
> >
> >
> > def __run(self, scriptList, sn):
> > self.testRunners = [createTestRunner(k, m, sn) for k, m in
> > scriptList] for testRunner in self.testRunners:
> > testRunner()
> > grinder.sleep(snooze)
> >
> > I'm using grinder.properties the number of threads in total to use (equal
> > to the number of file chunks). Thread 0 executes all 'prescripts' (auth
> > and resource creation) while all other threads wait. When resource
> > creation is done thread 0 flags to the other threads by means of a
> > grinder property that it's time to upload their file parts. Once parts
> > are uploaded, all other threads wait for thread 0 to run 'postscripts'
> > (confirmation that the complete file is uploaded and then de-auth) before
> > commencing the next run.
> >
> > This seems kind of hackish but it does the job. If anyone has any
> > feedback on how this could be improved or any other comments, I'd be
> > greatful.. Otherwise, hope it helps someone else with the same problem.
> >
> > Regards,
> > Noel
> >
> >
> >
> >
> >
> > Date: Fri, 24 Oct 2008 01:37:02 +0200
> > From: Pol van Aubel <macgyver@polvanaubel.com>
> > Subject: Re: [Grinder-use] Creating a multi-threaded TestRunner class
> > To: grinder-use <grinder-use@lists.sourceforge.net>
> > Message-ID: <49010A9E.8040706@polvanaubel.com>
> > Content-Type: text/plain; charset="iso-8859-1"
> >
> > Hi Philip, Noel,
> >
> > I'm wondering... Since The Grinder can run any Java code, couldn't you
> > in theory, for now, create a Java class which handles the multithreaded
> > upload? I'm thinking, perhaps a function call in the class that takes
> > the data to upload as an argument, the function spawns the correct
> > number of Threads, uploads the data, waits for all Threads to finish and
> > returns.
> > Then from within your test, create an instance of that class (or re-use
> > an existing one) and call that function.
> >
> > Philip, would that be possible?
> > Noel, would that suit your needs? You'd still only have one test per
> > upload, but you can do manipulation of the test statistics to reflect
> > number of failed threads in a test or something like that.
> >
> > Regards,
> >
> > Pol
> >
> > -------------------------------------------------------------------------
> > This SF.Net email is sponsored by the Moblin Your Move Developer's
> > challenge Build the coolest Linux based applications with Moblin SDK &
> > win great prizes Grand prize is a trip for two to an Open Source event
> > anywhere in the world
> > http://moblin-contest.org/redirect.php?banner_id=100&url=/
> > _______________________________________________
> > grinder-use mailing list
> > grinder-use@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/grinder-use
-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
grinder-use mailing list
grinder-use@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/grinder-use
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic