Write a subclass of dict, override __setitem__ such that it throws an error when replacing an existing key; rewrite the file to use your new subclass’s constructor instead of the default dict built-ins.
import collections
class Dict(dict):
def __init__(self, inp=None):
if isinstance(inp,dict):
super(Dict,self).__init__(inp)
else:
super(Dict,self).__init__()
if isinstance(inp, (collections.Mapping, collections.Iterable)):
si = self.__setitem__
for k,v in inp:
si(k,v)
def __setitem__(self, k, v):
try:
self.__getitem__(k)
raise ValueError("duplicate key '{0}' found".format(k))
except KeyError:
super(Dict,self).__setitem__(k,v)
then your file will have to be written as
dico = Dict(
('root', Dict(
('a', Dict(
('some_key', 'value'),
('another_key', 'another_value')
),
('b', Dict(
('some_key', 'value')
),
('c', Dict(
('some_key', 'value'),
('another_key', 'another_value')
),
....
)
)
using tuples instead of dicts for the file import (written using the {} notation, it would use the default dict constructor, and the duplicates would disappear before the Dict constructor ever gets them!).
Could anybody please tell me how to handle DuplicateKeyError in MongoDB?
I am writing a python script, where I move several docs from two different collections into a third one. There is a small overlap between the two collections due to having a few identical documents (with identical ObjectId). This results in the following:
DuplicateKeyError: E11000 duplicate key error collection: admin.collection_test index: id dup key: { : ObjectId(‘593a920b529e170d4b8fbf72’) }
In order to get rid of the error I use:
try:
do something
except pymongo.errors.DuplicateKeyError:
pass
I expect by using of the «try-except» to move all non-crossing documents to the third collection, but instead the script just peacefully stops running once a first overlap (an already existing document in the collection) appears.
Would appreciate any help a lot!
asked Jun 9, 2017 at 16:12
1
If you’re iterating over the documents, try using continue instead of pass.
for doc in documents:
try:
# insert into new collection
except pymongo.errors.DuplicateKeyError:
# skip document because it already exists in new collection
continue
answered Jun 12, 2017 at 16:14
Johnny MetzJohnny Metz
5,14715 gold badges73 silver badges138 bronze badges
1
for doc in documents:
client.update_one({'_id': doc['_id']}, doc, upsert=True)
You can use update_one with upsert=True. This updates doc with new doc if doc exists already otherwise it creates new doc.
answered Jun 17, 2017 at 17:39
daemon24daemon24
1,3881 gold badge11 silver badges23 bronze badges
CONNECT TO DATABASE
connection = pymongo.MongoClient("192.168.2.202", 27017)
CREATE DATABASE
database = connection['my_database']
CREATE COLLECTION
collection = database['my_collection']
INSERT DOCUMENTS IN COLLECTION
url="http://some/api/url/path/?format=json"
data = {
'_id': url,
'timestamp': datetime.datetime.now(),
'data': {
'XX': 1,
'YY': 2,
'ZZ': 3
}
}
TO AVOID DUPLICATES — THIS WILL CREATE NEW DOCUMENT IF SAME ID NOT EXIST
collection.update_one({'_id': url}, {"$set": data}, upsert=True)
answered Dec 5, 2018 at 3:46
Rajiv SharmaRajiv Sharma
6,46050 silver badges53 bronze badges
In this Python tutorial, we will study about Python dictionary duplicate keys using some examples in python. Moreover, we will also cover these topics.
- Python dictionary duplicate keys
- Python dictionary copy keys
- Python dictionary remove duplicate keys
- Python copy dict remove key
- Python dictionary update duplicate keys
- Python dict copy specific keys
- Python dictionary copy keys and values
- In this section, we will learn how to create a dictionary by using duplicate keys in list Python.
- To do this task we are going to use the concept of dict() and zip() method. These methods are available in Python and the zip() method takes iterable as a parameter where iterables can be a list of tuples.
- To create a dictionary we will use the dict() method. In Python the dict() method will check the condition if there are no values in arguments then it declares an empty dictionary.
- In this example, we have created two lists named ‘dup_lis_key’ and ‘list2’ in which we have assigned the string and integer value.
- Next, we will declare a variable ‘new_result’ and use the dict() function for creating a dictionary. Now we will consider the ‘dup_lis_key’ list as a dictionary key and ‘list2’ as a dictionary value.
Example:
dup_lis_key = ['U.S.A', 'U.S.A', 'China']
list2 = [78,16,256]
new_result = dict(zip(dup_lis_key, list2))
print("Dictionary created:",new_result)In the above example, we have created a list in which we have assigned the duplicate country name ‘U.S.A’. Now we will check the duplicate key ‘U.S.A’ will contain in the dictionary or not.
Here is the Screenshot of the following given code.
As you can see in the Screenshot, the output displays the dictionary but it does not have duplicate keys in a dictionary because in Python dictionary does not allow duplicate keys.
If you want to get all those values from a list and store them in the dictionary then you have to use the unique key with every value.
Let’s take an example and check how to create a Python dictionary with unique keys
dup_lis_key = ['U.S.A', 'Japan', 'China']
list2 = [78,16,256]
new_result = dict(zip(dup_lis_key, list2))
print("Dictionary created:",new_result)In the above code, we have just updated the key element from ‘U.S.A’ to ‘Japan’. Once you will execute this code it will display all the key elements along with values.
Here is the implementation of the following given code
Also, check: If not condition in Python
Python dictionary copy keys
- In this Program, we will learn how to copy keys from the Python dictionary.
- To perform this particular task, we are going to use the dict.keys() method. This is an inbuilt() method in the Python package and it is used to get the key elements from the dictionary and store them into a new list.
- In this example, we have to copy only key elements from the given dictionary. To do this task first we will create a dictionary in which we will assign the key-value pair elements. Next, we will use the dict.keys() method where dict is the dictionary name.
Syntax:
Let’s have a look at the syntax and understand the working of dict.keys() method
dict.keys()Example:
Let’s take an example and check how to copy the key elements from a dictionary by using the dict.keys() method
Source Code:
As you can see in the Screenshot, the output displays the keys from the dictionary.
Read: Python Return Function
Python dictionary remove duplicate keys
We had already covered this topic on the Python dictionary remove in an article. You get all the information regarding how to remove duplicate keys from the Python dictionary.
Python copy dict remove key
- Here we are going to see how to remove key from copy dictionary in Python.
- By using the dict.copy() method we can easily copy the orginal dictionaary to another one and this method is a built-in function in python and it does not take any parameter. As per the program if you print the ‘new_result’ variable the output displays the shallow copy of the original dictionary.
- Now we are going to use the del keyword to remove the key from the copied dictionary and this keyword is basically used to delete items in Python.
Syntax:
Here is the Syntax of Python dict.copy() method
dict.copy()Source Code:
my_dict = {'U.S.A':567,'China':976,'Japan':178,'Polland':289}
new_result=my_dict.copy()
print("Dictionary copy:",new_result)
del new_result['Japan']
print("Copy dictionary after removing key:",new_result)In the above code, we have created a simple dictionary named ‘my_dict’ and assigned key-value pairs. After that, we have used the del keyword and specified the key name [‘japan’] which we want to remove from the dictionary.
Here is the implementation of the following given code
As you can see in the Screenshot the output displays that the ‘japan’ key has been removed from the dictionary.
Read: Get First Key in dictionary Python
Python dictionary update duplicate keys
- In this section, we will learn how to update duplicate keys in Python dictionary.
- To do this task we are going to use the dictionary.update() function. This function is available in Python package and it is used to update the dictionary elements(key-value) pair.
- To get detail information regarding dictionary update() function. You can refer our article “Python dictionary update“.
Syntax:
Let’s have a look at the Syntax and understand the working of the Python dictionary.update() function.
dict.update([other])Example:
Let’s take an example and check how to update duplicate keys in the Python dictionary
Source Code:
new_dict = {'George':167, 'John':456, 'Micheal':657, 'Oliva':456, 'John':978 }
print("Duplicates keys dictionary:",new_dict)
new_dict.update({'john':891})
print("updated duplicate country value",new_dict)In the following given code, we have created a dictionary and assigned key-value pair elements. After that, we have used the dict.update() method and pass the duplicate key as an argument.
You can refer to the below Screenshot
As you can see in the Screenshot the output displays the ‘John’ key element has been updated.
Read: Python dictionary increment value
Python dict copy specific keys
- In this example, we will discuss how to get the specific keys from a copied dictionary in Python.
- To perform this particular task first we will create a dictionary named ‘my_dictionary’ and then we are going to use the dict.copy() function for creating a shallow copy of the original dictionary.
- Next to get the specific keys from the dictionary we are going to use the dictionary comprehension method in which we have specified the key elements in a list.
Example:
my_dictionary = {'U.S.A' : 78, "United kingdom" : 56, 'Australia' : 145, 'NewZealand' : 567}
new_result=my_dictionary.copy()
print("Dictionary copy:",new_result)
new_output = dict((i, new_result[i]) for i in ['U.S.A', 'Australia']
if i in new_result)
print("Specific keys:",new_output)
Here is the execution of the following given code.
As you can see in the Screenshot the output displays the specific keys along with values.
Also, read: Python dictionary of lists
Python dictionary copy keys and values
- In this section, we will learn how to copy keys and values in Python dictionary.
- By using the dictionary.copy() method, we can easily copy the key-value pair elements from one dictionary to another.
- To get detail information regarding dictionary update() function. You can refer our article “Python dictionary copy“.
Example:
Let’s take an example and check how to use the dictionary.copy() function in Python
Source Code:
new_dict = {'U.S.A':67, 'NewZealand':567, 'United Kingdom':156, 'China':2456, 'France':897 }
print("Original dictionary:",new_dict)
new_result = new_dict.copy()
print("Copied keys and values:",new_result)
In the above code, first, we have created a dictionary named ‘new_dict’ with some key-value pair elements. And then, we have applied the dict.copy() function and it will return a shallow copy of the given dictionary.
Here is the execution of the following given code.
You may also like to read the following Python tutorials.
- Python dictionary multiple keys
- Python dictionary extend
- Python Dictionary to CSV
- Python dictionary of tuples
- Python dictionary multiple values
- Check if two dictionaries are equal in Python
In this tutorial, we have discussed about Python dictionary duplicate keys. Moreover, we have covered these topics.
- Python dictionary duplicate keys
- Python dictionary copy keys
- Python dictionary remove duplicate keys
- Python copy dict remove key
- Python dictionary update duplicate keys
- Python dict copy specific keys
- Python dictionary copy keys and values
Python is one of the most popular languages in the United States of America. I have been working with Python for a long time and I have expertise in working with various libraries on Tkinter, Pandas, NumPy, Turtle, Django, Matplotlib, Tensorflow, Scipy, Scikit-Learn, etc… I have experience in working with various clients in countries like United States, Canada, United Kingdom, Australia, New Zealand, etc. Check out my profile.
In this article, we will find out whether a dictionary has duplicate keys or not in Python.
The straight answer is NO. You can not have duplicate keys in a dictionary in Python. But we can have a similar effect as keeping duplicate keys in dictionary. We need to understand the reason behind this question and how we can achieve duplicate keys in Python.
You can not have duplicate keys in Python, but you can have multiple values associated with a key in Python. If you want to keep duplicate keys in a dictionary, you have two or more different values that you want to associate with same key in dictionary. The dictionary can not have the same keys, but we can achieve a similar effect by keeping multiple values for a key in the dictionary.
Let’s understand with an example,
Suppose we have a dictionary of names and phone numbers,
Advertisements
# Dictionary of names and phone numbers
phone_details = { 'Mathew': 212323,
'Ritika': 334455,
'John' : 345323 }
As of now, each name (key) has a phone number associated with it. But suppose “John” has two more phone numbers, and we want to add those too in the dictionary. As the Key ‘John’ already exist in the dictionary, so if we try adding two more key-value pair with the same key like this,
phone_details['John'] = 111223 phone_details['John'] = 333444
OR
phone_details.update({ 'John' : 111223})
phone_details.update({ 'John' : 333444})
It will update the value of the existing key ‘John’ i.e.
Mathew - 212323 Ritika - 334455 John - 333444
To avoid this kind of problem, we can assign multiple values to a single key. Like this,
Mathew - 212323 Ritika - 334455 John - [345323, 111223, 333444]
Adding multiple values for a key in dictionary in Python
Instead of inserting a duplicate key, we can change the type of value to the list and assign more values to it.
Let’s see an example,
def add_value(dict_obj, key, value):
''' Adds a key-value pair to the dictionary.
If the key already exists in the dictionary,
it will associate multiple values with that
key instead of overwritting its value'''
if key not in dict_obj:
dict_obj[key] = value
elif isinstance(dict_obj[key], list):
dict_obj[key].append(value)
else:
dict_obj[key] = [dict_obj[key], value]
# Dictionary of names and phone numbers
phone_details = { 'Mathew': 212323,
'Ritika': 334455,
'John' : 345323 }
# Append a value to the existing key
add_value(phone_details, 'John', 111223)
# Append a value to the existing key
add_value(phone_details, 'John', 333444)
for key, value in phone_details.items():
print(key, ' - ', value)
Output
Mathew - 212323 Ritika - 334455 John - [345323, 111223, 333444]
Here we created a function add_value() that adds a key-value pair to the dictionary. If the key already exists in the dictionary, it will associate multiple values with that key instead of overwritting its value. It follows this logic.
- If the key does not exist, then add the key-value pair.
- If key exists in dictionary and type of value is not list. Then create a temporary list and add old and new values to it. Then assign the list object as the value for the key in dictionary.
- If key exists in dictionary and type of value is a list. Then add new value to the list.
So, this is how we can have duplicate keys in a dictionary, i.e., by adding multiple values for the same key in a dictionary in Python.
Advertisements
Thanks for reading.
Problem
I want to edit a JSON file by hand but I’m afraid that somewhere I introduce a duplicate key by accident. If it happens, then the second key silently overwrites the first one. Example:
$ cat input.json
{
"content": {
"a": 1,
"a": 2
}
}
Naive approach:
import json
with open("input.json") as f:
d = json.load(f)
print(d)
# {'content': {'a': 2}}
If there is a duplicate key, it should fail! But it remains silent and you have no idea that you just lost some data.
Solution
I found the solution here.
import json
def dict_raise_on_duplicates(ordered_pairs):
"""Reject duplicate keys."""
d = {}
for k, v in ordered_pairs:
if k in d:
raise ValueError("duplicate key: %r" % (k,))
else:
d[k] = v
return d
def main():
with open("input.json") as f:
d = json.load(f, object_pairs_hook=dict_raise_on_duplicates)
print(d)
Now you get a nice error message:
Traceback (most recent call last):
File "./check_duplicates.py", line 28, in <module>
main()
File "./check_duplicates.py", line 21, in main
d = json.load(f, object_pairs_hook=dict_raise_on_duplicates)
File "/usr/lib64/python3.5/json/__init__.py", line 268, in load
parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
File "/usr/lib64/python3.5/json/__init__.py", line 332, in loads
return cls(**kw).decode(s)
File "/usr/lib64/python3.5/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib64/python3.5/json/decoder.py", line 355, in raw_decode
obj, end = self.scan_once(s, idx)
File "./check_duplicates.py", line 13, in dict_raise_on_duplicates
raise ValueError("duplicate key: %r" % (k,))
ValueError: duplicate key: 'a'
If your json file has no duplicates, then the code aboce nicely prints its content.
Here are protocol messages that are sending(=>) and receiving(<=) during those two INSERT executions:
=> Query: First "INSERT INTO demo SELECT 1,'RT0132',1,100,10023,now();"
<= RowDescription
<= ParameterStatus
<= ParameterStatus
<= ParameterStatus
<= DataRow: the number of accepted rows = 1 NOTE: cursor.execute() returned at here
<= CommandComplete
<= ReadyForQuery
=> Query: Second "INSERT INTO demo SELECT 1,'RT0132',1,100,10023,now();"
<= RowDescription
<= ParameterStatus
<= ParameterStatus
<= ParameterStatus
<= DataRow: the number of accepted rows = 1 NOTE: cursor.execute() returned at here
<= ErrorResponse: "Duplicate key values: 'demo_id=1' -- violates constraint 'public.demo.C_PRIMARY'"
<= ReadyForQuery
The logic of cursor.execute() makes the function stop reading remaining protocol messages if it receives a DataRow message. But since the client doesn’t receive a CommandComplete message, that means the server is still running the query and errors can be thrown afterward.
When you run cursor.execute(), the client will read and ignore all remaining protocol messages of the previous command cycle, even an ErrorResponse. If you run cursor.fetchall(), you can iterate through all remaining protocol messages and catch those ErrorResponse.
So your expected behavior need a change of cursor.execute() to read a few more messages after a DataRow message is received. But this would also affect the logic of cursor.fetch*(). We’ll see if it is easy to fix.