python - Modify a list in Multiprocessing pool's manager dict -


i have list of elements processing in multiprocessing apply_async task , updating elements processed 1 one key in manager dict on want map whole list.

i tried following code:

#!/usr/bin/python  multiprocessing import pool, manager  def spammer_task(d, my_list):     #initialize manager dict     d['task'] = {         'processed_list': []     }      ele in my_list:         #process here         d['task']['processed_list'].append(ele)      return  p = pool() m = manager() d = m.dict()  my_list = ["one", "two", "three"]  p.apply_async(spammer_task (d, my_list)) print d 

at end posts empty list in dict. output:

{'task': {'processed_list': []}}

now after researching bit, got know elements inside manager dict become immutable have re-initialize whole dict new data in order update it. tried following code , gives weird error.

#!/usr/bin/python  multiprocessing import pool, manager  def spammer_task(d, my_list):     #initialize manager dict     d['task'] = {         'processed_list': []     }      ele in my_list:         #process here         old_list = d['task']['processed_list']         new_list = old_list.append(ele)         #have way since elements inside manager dict become         #immutable         d['task'] = {             'processed_list': new_list         }      return  p = pool() m = manager() d = m.dict()  my_list = ["one", "two", "three"]  p.apply_async(spammer_task (d, my_list)) print d 

output:

traceback (most recent call last): file "./a.py", line 29, in p.apply_async(spammer_task (d, my_list)) file "./a.py", line 14, in spammer_task new_list = old_list.append(ele) attributeerror: 'nonetype' object has no attribute 'append'

somehow seems appending none list cant figure out why.

accoridng solution @ https://bugs.python.org/issue6766

following code fixes it, copying whole task dict , modifying , recopying it

#!/usr/bin/python  multiprocessing import pool, manager  def spammer_task(d, my_list):     #initialize manager dict     d['task'] = {         'processed_list': []     }      ele in my_list:         #process here         foo = d['task']         foo['processed_list'].append(ele)         d['task'] = foo     return  p = pool() m = manager() d = m.dict()  my_list = ["one", "two", "three"]  p.apply_async(spammer_task (d, my_list)) print d 

output:

{'task': {'processed_list': ['one', 'two', 'three']}}


Comments