I received this error after switching to starmap from map, from the multiprocessing package. I went back to map and the error went away.
Specs: Ubuntu 16.04, nvidia docker, jupyter notebook 5.2.2, python 3.6.4, anaconda, tqdm 4.19.4, tqdm_notebook
Starmap code:
from multiprocessing import Pool
def get_and_save_patch(out_path, img, i):
img = random_crop(img, PATCH_SIZE)
out_path_png = out_path.split('.')[0] + "_" + str(i) + "_.png"
cv2.imwrite(out_path_png, img)
cameras = sorted(os.listdir(TRN_DIR))
for camera in tqdm_notebook(cameras):
in_dir = osp.join(TRN_DIR, camera)
out_dir = osp.join(TRN_64_DIR, camera)
os.makedirs(out_dir)
in_filenames = sort_files_by_int(os.listdir(in_dir))
for in_filename in tqdm_notebook(in_filenames):
in_path = osp.join(in_dir, in_filename)
out_path = osp.join(out_dir, in_filename)
img = np.array(cv2.imread(in_path))
args = [(out_path, img, i) for i in range(NUM_PATCHES_PER_IMG)] # Shallow copies.
with Pool(NPROC) as pool: # Multiprocessing.
pool.starmap(get_and_save_patch, args)
Map code:
from multiprocessing import Pool
def get_and_save_patch(args): ######### Diff.
out_path, img, i = args ######### Diff.
img = random_crop(img, PATCH_SIZE)
out_path_png = out_path.split('.')[0] + "_" + str(i) + "_.png"
cv2.imwrite(out_path_png, img)
cameras = sorted(os.listdir(TRN_DIR))
for camera in tqdm_notebook(cameras):
in_dir = osp.join(TRN_DIR, camera)
out_dir = osp.join(TRN_64_DIR, camera)
os.makedirs(out_dir)
in_filenames = sort_files_by_int(os.listdir(in_dir))
for in_filename in tqdm_notebook(in_filenames):
in_path = osp.join(in_dir, in_filename)
out_path = osp.join(out_dir, in_filename)
img = np.array(cv2.imread(in_path))
args = [(out_path, img, i) for i in range(NUM_PATCHES_PER_IMG)] # Shallow copies.
with Pool(NPROC) as pool: # Multiprocessing.
pool.map(get_and_save_patch, args) ######### Diff.
- Creating a Shallow Copy of the Dictionary
- Casting Dictionary Items to a List
- Appending Keys to an Empty List
This Runtime error occurs when we remove, modify, or add new entries in a dictionary object during iteration. This error occurs when iterating through a dictionary but virtually all iterable objects regardless of the programming language used.
The code snippet below illustrates how this error comes about when iterating through a dictionary and making changes simultaneously.
cars = {
"brand": "Tesla",
"model": "Model S Plaid",
"year": 2021
}
for x in cars.keys():
cars["color"] = "white"
print(x)
In the code block above, we add a new item to the original dictionary while iterating. This will return a Runtime Error, letting us know that the dictionary size changed during iteration, implying that we cannot modify the dictionary while iterating simultaneously.
Sample Code:
Traceback (most recent call last):
File "<string>", line 8, in <module>
RuntimeError: dictionary changed size during iteration
While performing any iteration to an object, both deletion, addition, or modification are considered an alteration and cannot be performed while iterating. The code example below demonstrates that this error will also persist if we modify the dictionary while iterating. Therefore, we will still get the same error if we remove an existing item from a dictionary while iterating.
Sample Code:
cars = {
"brand": "Tesla",
"model": "Model S Plaid",
"year": 2021
}
for x in cars.keys():
del cars["model"]
print(cars)
Output:
Traceback (most recent call last):
File "<string>", line 8, in <module>
RuntimeError: dictionary changed size during iteration
In Python 3, iterating over a changing object is considered a bad style of writing code and unsafe. Generally, in Programming, we cannot mutate an object while iterating over it simultaneously; this rule extends to iterables such as lists and even arrays.
However, if a function mutates an object, we must make sure that the function only mutates a copy of the original object, leaving the original object intact. This is one of the widely used approaches of making alterations to objects while iterating through them simultaneously.
This is a good practice and the best way of avoiding instances of creating an infinite loop that may eventually lead to memory exhaustion. Several solutions can be used to handle this error, and we will discuss each one here.
Creating a Shallow Copy of the Dictionary
Python provides us with the copy()
module that allows us to create a copy of an object with no binding to the original object. This lets us freely modify the copy of the object, leaving the original object intact.
Note that the same cannot be realized by using the assignment operator in Python. Using the assignment operator does not create a copy of the original object but rather a variable that refers to the original object.
Therefore any modifications made to the new object will also affect the original object. New developers often misuse this operator.
Sample Code:
import copy
cars = {
"brand": "Tesla",
"model": "Model S Plaid",
"year": 2021,
}
#creating a shallow copy
cars_copy = copy.copy(cars)
for x in cars_copy.keys():
cars["color"] = "black"
print(cars)
print(cars_copy)
Output:
{'brand': 'Tesla', 'model': 'Model S Plaid', 'year': 2021, 'color': 'black'}
{'brand': 'Tesla', 'model': 'Model S Plaid', 'year': 2021}
In the sample code provided, we have used the copy module’s copy
function to create a dictionary copy that we can freely iterate without affecting the original dictionary. Making changes to a dictionary copy allows us to iterate over the dictionary without encountering an error.
Alternatively, we can use the **
operator that is often referred to as the two asterisk operators to rewrite the code above, as shown below.
Sample Code:
cars = {
"brand": "Tesla",
"model": "Model S Plaid",
"year": 2021
}
#creating a shallow copy
cars_copy = {**cars}
for x in cars_copy.keys():
cars["color"] = "black"
print(cars)
The **
operator can take key-value pairs from one dictionary and dump them into another dictionary.
Although the operator is widely used to pass in keyword arguments in Python, we have used the operator to unpack the dictionary and obtain the key-value pairs in the code above. We then create a copy of the dictionary and dump the unpacked values in this new dictionary.
Output:
'brand': 'Tesla', 'model': 'Model S Plaid', 'year': 2021, 'color': 'black'}
Deleting a key-value pair from a dictionary is no exception either when performing an iteration and thus should follow a similar approach. Therefore using the same procedure, we will delete the key named model
and its value Model S Plaid
as shown below.
Sample Code:
import copy
cars = {
"brand": "Tesla",
"model": "Model S Plaid",
"year": 2021,
"color": "black"
}
cars_copy = copy.copy(cars)
for x in cars_copy.keys():
if x == "model":
del cars["model"]
print(cars)
Output:
{'brand': 'Tesla', 'year': 2021, 'color': 'black'}
Another solution would be to create a copy of the keys that we can then iterate over while modifying the dictionary. However, this can only work in Python 2 and not Python 3 because when done in Python 3, the keys do not return the iterable.
Sample Code:
cars = {
"brand": "Tesla",
"model": "Model S Plaid",
"year": 2021,
"color": "black"
}
key_copys = list(cars.keys())
print(key_copys)
for key in list(key_copys):
if cars[key] == "model":
cars.pop("model")
print(cars)
Sample Output:
['brand', 'model', 'year', 'color']
{'brand': 'Tesla', 'model': 'Model S Plaid', 'year': 2021, 'color': 'black'}
Casting Dictionary Items to a List
Since we cannot iterate over the dictionary while making changes, we can instead create a casting list and iterate over the list while making changes to the dictionary. Iterating over the casting list instead of the original dictionary does not return a Runtime error.
Sample Code:
cars = {
"brand": "Tesla",
"model": "Model S Plaid",
"year": 2021
}
for i in list(cars):
cars["color"] = "black"
print(cars)
Output:
{'brand': 'Tesla', 'model': 'Model S Plaid', 'year': 2021, 'color': 'black'}
Appending Keys to an Empty List
To avoid changing the dictionary while iterating, we can create an empty list containing the dictionary keys while we perform the iteration. Using this empty list, we can append all the keys that we want to remove or change and then use the pop()
function to remove the keys or the append
function to add new key-value pairs.
This can be executed as shown in the code below.
Sample Code:
cars = {
"brand": "Tesla",
"model": "Model S Plaid",
"year": 2021
}
list = []
for i in cars:
list.append(i)
for x in list:
if x == "model":
cars.pop(x)
print(cars)
Output:
{'brand': 'Tesla', 'year': 2021}
As shown below, we can add a new key-value pair to the dictionary using the same procedure while iterating using the for loop.
Sample Code:
cars = {
"brand": "Tesla",
"model": "Model S Plaid",
"year": 2021
}
list = []
for i in cars:
list.append(i)
for x in list:
cars["color"] = "black"
print(cars)
Output:
{'brand': 'Tesla', 'model': 'Model S Plaid', 'year': 2021, 'color': 'black'}
Уведомления
- Начало
- » Python для новичков
- » Удаление частей словаря в цикле
#1 Май 4, 2013 19:46:30
Удаление частей словаря в цикле
Всем привет. Возникла необходимость в цикле обойти словарь и удалить некоторые его элементы. Кандидаты на удаление обнаружаются с помощью условия внутри этого цикла. Вот банальный пример подобного кода:
myDict = {'a': 1, 'b': 2} for k in myDict: if k == 'a': del myDict[k]
Третий питон, при запуске, ругается на то, что во время обхода словаря меняется его размер
RuntimeError: dictionary changed size during iteration
Можно ли как-то этого избежать? Со списками такой фокус проходит (удаляю элементы через .remove()). Хотелось бы и со словарями научиться это делать.
Офлайн
- Пожаловаться
#2 Май 4, 2013 20:04:50
Удаление частей словаря в цикле
myDict = {'a': 1, 'b': 2} for k in myDict.keys(): if k == 'a': del myDict[k] print myDict
Офлайн
- Пожаловаться
#3 Май 4, 2013 21:56:06
Удаление частей словаря в цикле
Romiss , ваш фрагмент кода вызывает такой же протест
Офлайн
- Пожаловаться
#4 Май 4, 2013 22:19:28
Удаление частей словаря в цикле
Yott
Romiss , ваш фрагмент кода вызывает такой же протест
Да, вы правы в python 2.7 работает,а вот 3 выдает ошибку.
Офлайн
- Пожаловаться
#5 Май 4, 2013 22:22:16
Удаление частей словаря в цикле
Выше написанный код сработает во втором питоне. В третьем немного по-другому. Один из способов — через копию:
#-*- coding: utf-8 -*- myDict = {'a': 1, 'b': 2} Dict = myDict.copy() for k in Dict.keys(): if k == 'a': del myDict[k] print("myDict: ", myDict) # myDict: {'b': 2} print("Dict: ", Dict) # Dict: {'a': 1, 'b': 2}
Офлайн
- Пожаловаться
#6 Май 4, 2013 22:24:14
Удаление частей словаря в цикле
for k in list(myDict.keys()): ...
И если проверки в коде так выглядят именно так
то логичнее было бы пробежаться по списку удаляемых ключей:
keys_to_delete = ['a', 'b', 'c'] for key in keys_to_delete: dct.pop(key, None)
Офлайн
- Пожаловаться
#7 Май 4, 2013 22:44:21
Удаление частей словаря в цикле
reclosedev
for k in list(myDict.keys()): ...
Скорей всего использование списков наилучший способ:
myDict = {'a': 1, 'b': 2} for k in list(myDict.keys()): if k == 'a': myDict.pop(k) print("myDict: ", myDict) # myDict: {'b': 2}
Отредактировано elisk (Май 4, 2013 22:45:52)
Офлайн
- Пожаловаться
#8 Май 5, 2013 17:45:49
Удаление частей словаря в цикле
Всем спасибо! Решал через создание копии. Думал что использовал костыль, а оказывается нет
Офлайн
- Пожаловаться
#9 Май 6, 2013 05:35:36
Удаление частей словаря в цикле
>>> myDict = {'a': 1, 'b': 2} >>> for k in set(myDict): ... if k == 'a': ... del myDict[k] ... >>> myDict {'b': 2} >>>
Офлайн
- Пожаловаться
#10 Май 6, 2013 10:42:41
Удаление частей словаря в цикле
py.user.next
а в чем здесь выгода множеств в отличии от списка, кортежа?
_________________________________________________________________________________
полезный блог о python john16blog.blogspot.com
Офлайн
- Пожаловаться
- Начало
- » Python для новичков
- » Удаление частей словаря в цикле
Background and Problem Description:
I have some code that solves the graph coloring problem (broadly defined as the problem of assigning «colors» to an undirected graph, making sure that no two vertices connected by an edge have the same color). I’m trying to implement a solution using constraint propagation to improve on the efficiency of a standard recursive backtracking algorithm, but am running into the following error:
File "C:UsersdanisgDesktopcoloringSolver.py",
line 99, in solve
for color in self.domains[var]:
RuntimeError: Set changed size during iteration
Here, for each vertex, I keep a set
of possible particular values for that particular vertex:
self.domains = { var: set(self.colors) for var in self.vars }
After I make an assignment, I propagate this constraint to the neighboring domains, to limit the search space:
for key in node.neighbors: # list of keys corresponding to adjacent vertices
if color in self.domains[key]: # remove now to prune possible choices
self.domains[key].remove(color)
This isn’t where the actual error is thrown (in my code, I indicate where the problem is in a try-except
block), but may be the source of the problem.
My Question:
Do I have the right idea, here, if not the right implementation? More to the point, how can I fix this? Also, is it necessary to keep a separate domains
dictionary? Or could we make domain
a property of each node in the graph?
My Code:
Here’s the solve
function where this code is called:
def solve(self):
uncolored = [var for var in self.vars if self.map[var].color == None]
if len(uncolored) == 0:
return True
var = min(uncolored, key = lambda x: len(self.domains[var]))
node = self.map[var]
old = { var: set(self.domains[var]) for var in self.vars }
for color in self.domains[var]:
if not self._valid(var, color):
continue
self.map[var].color = color
for key in node.neighbors:
if color in self.domains[key]:
self.domains[key].remove(color)
try:
if self.solve():
return True
except:
print('happening now')
self.map[var].color = None
self.domains = old
return False
My implementation uses a Node
object:
class Solver:
class Node:
def __init__(self, var, neighbors, color = None, domain = set()):
self.var = var
self.neighbors = neighbors
self.color = color
self.domain = domain
def __str__(self):
return str((self.var, self.color))
def __init__(self, graph, K):
self.vars = sorted( graph.keys(), key = lambda x: len(graph[x]), reverse = True ) # sort by number of links; start with most constrained
self.colors = range(K)
self.map = { var: self.Node(var, graph[var]) for var in self.vars }
self.domains = { var: set(self.colors) for var in self.vars }
Here are two other functions that are used/are helpful:
def validate(self):
for var in self.vars:
node = self.map[var]
for key in node.neighbors:
if node.color == self.map[key].color:
return False
return True
def _valid(self, var, color):
node = self.map[var]
for key in node.neighbors:
if self.map[key].color == None:
continue
if self.map[key].color == color:
return False
return True
Data and Example for which the Code is Failing:
The example graph I’m using can be found here.
The function for reading the data:
def read_and_make_graph(input_data):
lines = input_data.split('n')
first_line = lines[0].split()
node_count = int(first_line[0])
edge_count = int(first_line[1])
graph = {}
for i in range(1, edge_count + 1):
line = lines[i]
parts = line.split()
node, edge = int(parts[0]), int(parts[1])
if node in graph:
graph[node].add(edge)
if edge in graph:
graph[edge].add(node)
if node not in graph:
graph[node] = {edge}
if edge not in graph:
graph[edge] = {node}
return graph
It should be called as follows:
file_location = 'C:\Users\danisg\Desktop\coloring\data\gc_50_3'
input_data_file = open(file_location, 'r')
input_data = ''.join(input_data_file.readlines())
input_data_file.close()
graph = read_and_make_graph(input_data)
solver = Solver(graph, 6) # a 6 coloring IS possible
print(solver.solve()) # True if we solved; False if we didn't