[prev in list] [next in list] [prev in thread] [next in thread] 

List:       python-list
Subject:    Re: multiprocess passing arguments double asterisks
From:       ricemom () gmail ! com
Date:       2016-10-27 15:14:36
Message-ID: 3b52cc7c-cc99-4efd-8413-19d9cc8f1a3a () googlegroups ! com
[Download RAW message or body]

On Wednesday, October 26, 2016 at 5:31:18 PM UTC-5, MRAB wrote:
> On 2016-10-26 21:44, pic8690@gmail.com wrote:
> > On Monday, October 24, 2016 at 12:39:47 PM UTC-5, Thomas Nyberg wrote:
> > > On 10/24/2016 12:45 PM, pic8690@gmail.com wrote:
> > > > Thanks for the reply.
> > > > 
> > > > The code snippet given by Peter is not very clear
> > > > 
> > > > I would like to multiprocess a function which is written in python of the \
> > > > form bar(**kwargs) which returns a value.  This example does not return \
> > > > anything 
> > > > Would you please use my example for the map function?
> > > > 
> > > > I appreciate your help,
> > > > 
> > > I'm honestly not totally sure what you want to do. However, say you want
> > > to do the following (btw this is basically what Dennis said i nhis last
> > > email, but maybe I can help clarify):
> > > 
> > > 	kwargs = {'param1': val1, 'param2': val2})
> > > 
> > > Then you'd like to have the following two operations performed in
> > > separate processes:
> > > 
> > > 	bar(param1=val1)
> > > 	bar(param2=val2)
> > > 
> > > In that case, I guess I would do something like the following. First
> > > define bar_wrapper as follows (*I haven't tested any code here!):
> > > 
> > > def bar_wrapper(pair):
> > > 	key, val = pair
> > > 	return bar(**{key: val})
> > > 
> > > Then I would probably do something like
> > > 
> > > map(bar_wrapper, kwargs.items())
> > > 
> > > I.e. basically what I'm doing is taking the key-val pairs and producing
> > > a list of them (as tuples). This is something that you can apply map
> > > too, but not with the original function. So then the wrapper function
> > > converts the tuple back to what you want originally.
> > > 
> > > Hopefully I'm understanding correctly and hopefully this helps.
> > > 
> > > Cheers,
> > > Thomas
> > 
> > Thomas,
> > 
> > I have strings & numpy.ndarray as arguments.  The wrapper function works great \
> > for strings. 
> > Here's what I'm trying to do...
> > **************************
> > 
> > I have a serial python fn of the form
> > 
> > def bar(**kwargs):
> > a=kwargs.get("name")
> > print a
> > self.a1d=ma.asanyarray(kwargs.get('a1d'), dtype=float)
> > (****more calculations--takes a long time to compute
> > returns an object)
> > 
> > I am trying to run this function in parallel.
> > 
> > Here's my Main program
> > 
> > import numpy as np
> > import numpy.ma as ma
> > from delegate import parallelize
> > from hashlib import sha1
> > a1d=np.zeros((32))
> > b1d=np.zeros((32))
> > p=open("/tmp/pdata","rb")
> > pdata=np.load(p)
> > for i in range(0,10):
> > a1d=pdata['t1d']
> > b1d=pdata['gz1d']
> > print a1d,b1d
> > kwargs={'name':'special','a':a1d,'b':b1d}
> > val=parallelize(bar,kwargs)
> > ***************************************
> > Error:
> > line 30, in <module>
> > val=parallelize(bar,kwargs)
> > delegate.py", line 311, in process
> > item, items = items[0], items[1:]
> > KeyError: 0
> > **************************************
> > Error: (with the wrapper function)
> > val=parallelize(bar_wrapper,kwargs.items())
> > TypeError: unhashable type: 'numpy.ndarray'
> > ***************************************
> > 
> > How do I pass the string & numpy.ndarray to the function bar(**kwargs)?
> > 
> > Thank you for suggestions & help,
> > 
> > 
> > a1d=np.zeros((32))
> > b1d=np.zeros((32))
> > p=open("/tmp/pdata","rb")
> > pdata=np.load(p)
> > for i in range(0,100):
> > a1d=pdata['t1d']
> > b1d=pdata['gz1d']
> > print a1d,b1d
> > kwargs={'name':'special','a':a1d,'b':b1d}
> > val=parallelize(bar,kwargs)
> > 
> 'parallelize' expects a function and a list of items. It calls the 
> function with each item in the list in parallel, passing the item as the 
> argument of the function.
> 
> That _single_ argument item can be a tuple or a dict.

--if the dict has numpy arrays it fails..

In the above example if
   kwargs={'name':'special','a':2}
   val=parallelize(bar_wrapper,kwargs.items()) it works fine

But if
     kwargs={'name':'special','a':a1d,'b':b1d}
it fails  (a1d & b1d are numpy arrays)
-- 
https://mail.python.org/mailman/listinfo/python-list


[prev in list] [next in list] [prev in thread] [next in thread]