i have list of elements processing in multiprocessing apply_async
task , updating elements processed 1 one key in manager dict on want map whole list.
i tried following code:
#!/usr/bin/python multiprocessing import pool, manager def spammer_task(d, my_list): #initialize manager dict d['task'] = { 'processed_list': [] } ele in my_list: #process here d['task']['processed_list'].append(ele) return p = pool() m = manager() d = m.dict() my_list = ["one", "two", "three"] p.apply_async(spammer_task (d, my_list)) print d
at end posts empty list in dict. output:
{'task': {'processed_list': []}}
now after researching bit, got know elements inside manager dict become immutable have re-initialize whole dict new data in order update it. tried following code , gives weird error.
#!/usr/bin/python multiprocessing import pool, manager def spammer_task(d, my_list): #initialize manager dict d['task'] = { 'processed_list': [] } ele in my_list: #process here old_list = d['task']['processed_list'] new_list = old_list.append(ele) #have way since elements inside manager dict become #immutable d['task'] = { 'processed_list': new_list } return p = pool() m = manager() d = m.dict() my_list = ["one", "two", "three"] p.apply_async(spammer_task (d, my_list)) print d
output:
traceback (most recent call last): file "./a.py", line 29, in p.apply_async(spammer_task (d, my_list)) file "./a.py", line 14, in spammer_task new_list = old_list.append(ele) attributeerror: 'nonetype' object has no attribute 'append'
somehow seems appending none
list cant figure out why.
accoridng solution @ https://bugs.python.org/issue6766
following code fixes it, copying whole task dict , modifying , recopying it
#!/usr/bin/python multiprocessing import pool, manager def spammer_task(d, my_list): #initialize manager dict d['task'] = { 'processed_list': [] } ele in my_list: #process here foo = d['task'] foo['processed_list'].append(ele) d['task'] = foo return p = pool() m = manager() d = m.dict() my_list = ["one", "two", "three"] p.apply_async(spammer_task (d, my_list)) print d
output:
{'task': {'processed_list': ['one', 'two', 'three']}}
Comments
Post a Comment