Mongoerror e11000 duplicate key error collection

Apr 29, 2019

MongoDB’s E11000 error is a common source of confusion. This error occurs when two documents have the
same value for a field that’s defined as unique in your Mongoose schema.

Mongoose models have an _id field that’s always unique. If you try to insert two documents with the same _id, you get the below error message.

MongoError: E11000 duplicate key error collection: test.customers index: _id_
dup key: { : ObjectId('5cc5ea092dca872442916cf5') }

The test.customers part represents the MongoDB collection that the error occurred in. The _id_ string is the name of the unique index, and the ObjectId() is the duplicate value.

The below code is one way you might get the above error message. MongoDB collections always have a unique index on _id, so trying to insert a document
with a duplicate id will cause a duplicate key error.

const CharacterModel = mongoose.model('Character',
  new Schema({ name: String }));

const doc = await CharacterModel.create({ name: 'Jon Snow' });

doc._id; // Something like "5cc5e9be172acd237a893610"

try {
  // Try to create a document with the same `_id`. This will always fail
  // because MongoDB collections always have a unique index on `_id`.
  await CharacterModel.create(Object.assign({}, doc.toObject()));
} catch (error) {
  // MongoError: E11000 duplicate key error collection: test.characters
  // index: _id_ dup key: { : ObjectId('5cc5ea092dca872442916cf5') }
  error.message;
}

This error is often caused by null or undefined field values. null and undefined count as distinct values, so if you declare a field email as unique, two documents cannot have email = undefined. The below example creates two documents without an email property, which causes a duplicate key error.

const UserModel = mongoose.model('User', new Schema({
  name: String,
  email: {
    type: String,
    unique: true
  }
}));

// Wait for the index to build. The index name will be `email_1`
await UserModel.init();

// Create a document with no `email` set
await UserModel.create({ name: 'user 1' });

try {
  await UserModel.create({ name: 'user 2' });
} catch (error) {
  // E11000 duplicate key error collection: test.users index: email_1
  // dup key: { : null }
  error.message;
}

To make MongoDB E11000 error messages user-friendly, you should use the mongoose-beautiful-unique-validation plugin.

const schema = new Schema({ name: String });
schema.plugin(require('mongoose-beautiful-unique-validation'));

const CharacterModel = mongoose.model('Character', schema);

const doc = await CharacterModel.create({ name: 'Jon Snow' });

try {
  // Try to create a document with the same `_id`. This will always fail
  // because MongoDB collections always have a unique index on `_id`.
  await CharacterModel.create(Object.assign({}, doc.toObject()));
} catch (error) {
  // Path `_id` (5cc60c5603a95a15cfb9204d) is not unique.
  error.errors['_id'].message;
}

More Mongoose Tutorials

• Implementing Soft Delete in Mongoose
• Using limit() with Mongoose Queries
• How to Fix «Buffering timed out after 10000ms» Error in Mongoose
• Using insertOne() in Mongoose
• Mongoose on(‘delete’)
• Enums in Mongoose
• Mongoose find() Certain Fields

MongoError: E11000 duplicate key error collection ??

I’ve encountered the issue sometime in my developer path. So I decided to write down my experience and to firstly note to myself and secondly to help other developers/engineers who are a newbie in this topic.

If you only need the fix, you can skip to the last part of this article!

One of the most advantages of using NoSQL databases is the flexible feature that allows us to update the number of fields and their data types any time we want.

I’m using Nodejs and Mongoose driver to connect to MongoDB. And in the very beginning phase of a project development life and this helps us to able to update collections and their fields.

However, I encountered an error.

MongoError: E11000 duplicate key error collection: companies index: code_1 dup key:

What happened here?

To understand, I would go further a bit with Indexing and unique keys in Databases.

Database Indexes

A database index is a data structure that improves the speed of data retrieval operations on a database table at the cost of additional writes and storage space to maintain the index data structure. Indexes are used to quickly locate data without having to search every row in a database table every time a database table is accessed. Indexes can be created using one or more columns of a database table, providing the basis for both rapid random lookups and efficient access of ordered records.

If you want to query a field or set of fields without iterating all the entries in a table/collection, you can create indexes for these fields.

To understand more how indexes are created & organized to support efficient queries, you can check at this

However, the cost for indexes is not cheap, especially in the world of big data nowadays. Because to make queries on indexed fields efficient, these indexes need to be stored in a fast query memory (for example RAM). So, be careful when you want to add an index for a field, some factors should be put on the table to have good enough decisions: are data queried frequently? the user behaviors? regions that data are stored? etc

And last but not least, an indexed field may be a non-unique value for each entry in the column. For example, you can have an index on the field “Region” where we can have multiple users in the same region.

Unique keys

In the real-life, there are use cases that we want to limit the appearance of one or a set of values of factors. For example, you want there is only 1 email that is used to register per user, no more.

So, unique keys help you to achieve this constrain by defining the rule in the schema. For example in mongoose:

You can see that the email attribute is unique. With this, you cannot add more than one user with the same email. If developers violate the rule, Mongodb will throw errors. This helps us preventing developer mistakes.

Again, you can check more at Wikipedia

When Mongo DB schema become out of date with Mongoose schema

Come back to the beginning error, let’s me show you the mongoose schema that was defined by code:

Look good! right?

But what makes the error?

MongoError: E11000 duplicate key error collection: companies index: code_1 dup key:

Let’s check a bit:

I used mongo client by querying with the command line to check the existing keys for the `companies` collection. And the result was:

Yeah! There is an established index field for `code` and it is set to unique. And once an index is set, it is there until you remove it, and the rule unique is still there also.

And the reason is the schema was modified due to the product business has changed. The previous schema was:

So, this is a case when the Mongo schema becomes out date with the Mongoose schema that you defined in code.

To fix this, I need to remove manually the unnecessary index key. Mongo query provides some methods to remove indexes manually:

** *Note: You cannot drop the default index on the _id field.* **

This method will drop all non-_id indexes

You can verify again by using the command line:

Your comments & discussion are warmly welcomed!

Источник

Handling Mongoose Dublication Errors

Introduction

If you ever wrote Node.js code and decided to have a document based DB, your main goto will be MongoDB ofc and you will be using mongoose as your ODM then you have met this error before.

MongoError: E11000 duplicate key error collection: testDB.users index: name_1 dup key: < : «some random name» >.

the problem is there are multiple ways to handle it. one of them is using a library called mongoose-unique-validator. but we are not going to use an external library which I don’t know how it works under the hood.

Before we keep going

there is some stuff that needs to be clarified
1- name < type :string , unqiue: true>unique param in mongoose is not a validator meaning doing const myUser = new User(data) will not throw an error in case of duplication.
it will only throw and error when doing myUser.save()

2- when trying to add a user I suggest using either insertOne or create function. and keep using it through your whole application because We are about to overwrite one of them.

Note: I will be using create here.

why we wanna handle duplication error globally anyway?
because you might have 10 or 20 collections where each one has 2 or 3 unique keys and you are not going to check for every one manually.

Implmentation

you can easily overwrite mongoose function by doing

Exit fullscreen mode

my logic here is when I am using create function I will insert a new option which is some keys to check if they are duplicated or no.

Exit fullscreen mode

I am going for this format where checkForDublication is a new option I created and will be sending the keys as array format.

Logic

check if options has a checkForDublication param.

check if its values exist in the schema and are unique.

the last step (checking if the key is unique) is very important, Because we are going to use findOne(<$or: searchQuery>) . and as you know searchQuery is going to be an array, If one Element in this array is not unique or index it’s going to perform collectionScan instead of indexScan which is very slow.

filter the checkForDublication array meaning remove every key that doesn’t exist in the schema or is not unique.

generating the search query

checking if the result of the search query exist.

Источник

MongoError: E11000 duplicate key error collection: userDB.users index: username_1 dup key: < username: null >#2

when I try to add my second email I got eroor

The text was updated successfully, but these errors were encountered:

when I try to add my second email I got eroor

did your error got fixed

he reason behind this error is that. The index is not present in your collection, in which you are trying insert. So Solution is to drop that collection and run your program again

getspooky How to drop a collection?

getspooky How to drop a collection?

I was also facing the same problem. I fixed it by just changing 2 line of code. By adding the email field in :

and also in :

Actually, the package mongoose-findorcreate implicitly passes two arguments(ID and Username) to the mongo server, When you tap into the first user, You store only the «ID» of the user, So the findorcreate package assigns null value to the username field. Again when you try to tap into second or multiple users, Since you already set a null value to the username field. MongoDB server allows only one entry of the null valued field. So It will throw an error.

Here is the solution. Replace the existing package function of findorcreate with the below code

Finally by adding this code may solve your issue. Also, remove the findorcreate plugin to your userSchema.

That happened to me when i changed a collection’s schema which already had documents and tried adding new documents with the new type of schema.
The solutions was to delete the collection because it had different types of schema.

you have to add to your schema googled
const userSchema = new mongoose.Schema( <
email: String,
password: String,
googleId: String
>)
drop the collection and try again

i had the same problem and i fixed it by adding a username to my entries, here’s my code
passport.use(new GoogleStrategy( <
clientID: process.env.CLIENT_ID,
clientSecret: process.env.CLIENT_SECRET,
callbackURL: «http://localhost:3000/auth/google/home»
>,
function(accessToken, refreshToken, profile, cb) <
console.log(profile);
User.findOrCreate(< username: profile.displayName, googleId: profile.id >, function (err, user) <
return cb(err, user);
>);
>
));
see the username:profile.displayName in findorcreate, now all your entrues have a username so it wont show username=null

I was also facing the same problem. I fixed it by just changing 2 line of code. By adding the email field in :

and also in :

Thanks for the solution, but can you explain or share a link to understand why this works?

I had the same issue, I deleted the respective collection from database and recreated it.
Issue resolved.

I added ’email’ as an argument in the callback function and it worked fine

passport.use(new GoogleStrategy( <
clientID: process.env.CLIENT_ID,
clientSecret: process.env.CLIENT_SECRET,
callbackURL: «http://localhost:3000/auth/google/home»
>,
function(accessToken, refreshToken, profile, email , cb) <
console.log(profile);
User.findOrCreate(< username: profile.displayName, googleId: profile.id >, function (err, user) <
return cb(err, user);
>);
>
));

But on later steps you have to remove other changes to store googleId in schema for which one has to edit the schema ny adding a field later :

removing the above changes

I have experienced the same issue.
After some googling, I simply drop my «users» collection. Then everything just works fine.
I still don’t know what was the cause.

he reason behind this error is that. The index is not present in your collection, in which you are trying insert. So Solution is to drop that collection and run your program again

Thank god! This works to me!

That happened to me when i changed a collection’s schema which already had documents and tried adding new documents with the new type of schema. The solutions was to delete the collection because it had different types of schema.

I have experienced the same issue. After some googling, I simply drop my «users» collection. Then everything just works fine. I still don’t know what was the cause.

Hey, sorry but none of the solutions are working for me

just add another field in the mongoose schema along with googleId.
like this. <username: profile.displayName, googleId: profile.id >,
and it should work.
also try to drop the collection and rerun the application i it doesn’t work.

Just modify your findOrCreate inputs like this:
function(accessToken, refreshToken, profile, cb) <
User.findOrCreate(< username: profile.id, googleId: profile.id >, function (err, user) <
return cb(err, user);>);>
Set username: profile.id since it is unique. I had some issues when used «username: profile.displayName» an diferent social media strategies.

he reason behind this error is that. The index is not present in your collection, in which you are trying insert. So Solution is to drop that collection and run your program again

Still getting Same error.

I figured out that this problem may caused by the indexes which automatically set by the database. You can remove the username_1 index in the database, and eventually this problem will no longer exists. Hope it helps!

© 2023 GitHub, Inc.

You can’t perform that action at this time.

You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.

Источник

Application and websites generate too much data even in a single transaction.

How can we handle this huge amount of data efficiently?

Thanks to the NoSQL capabilities of MongoDB, it makes websites scalable and offers superior performance.

However, the MongoDB error code 11000 can happen due to duplicate entries or bad syntax.

At Bobcares, we often get requests from our customers to fix MongoDB error code 11000 as part of our Server Migration Services.

Today, let’s see how our Migration Engineers fix MongoDB error code 11000.

How we fixed MongoDB error code 11000

At Bobcares, where we have more than a decade of expertise in managing servers, we see many customers face problems while managing MongoDB database.

Now let’s see the major reasons for MongoDB errors and how our Support Engineers fix the top errors.

1. Wrong syntax

Recently, one of our customers wanted to recover data from a MySQL database and transform it into JSON for integration into the MongoDB database.

When it was a single insertion it worked fine. But while doing 45000 insertion, it did not work and resulted in the error:

(node:31032) UnhandledPromiseRejectionWarning: WriteError({"code":11000,"index":0,"errmsg":"E11000 duplicate key error collection: meteor.depart index: _id_ dup key: { : ObjectId('5b1527ee6161057938e0aef0') }","op":{"jourExpl
oitation":::::,"_id":"5b1527ee6161057938e0aef0"}})

On checking our MongoDB Experts found that the problem happened due to syntax error.

When using insertMany, we should use the forceServerObjectId flag. Therefore, we suggested the customer to use the following code to solve the problem.

manyInsert.insertMany(dataJsonInsert, { forceServerObjectId: true }); 

This fixed the problem.

2. Duplicate record

Similarly, another customer had an  E11000 duplicate key error index in MongoDB mongoose. The error said

Error creating new user: WriteError({"code":11000,"index":0,"errmsg":"insertDocument :: caused by :: 11000 E11000 duplicate key error index

On checking the schema, we found that the customer renamed the field to username, but didn’t remove the old index. By default, MongoDB will set the value of a non-existent field to null in that case.

If a document does not have a value for the indexed field in a unique index, the index for this document will store a null value.

Due to the unique condition, MongoDB will only allow one document lacking the indexed field. Also, If there is more than one document without a value for the indexed field or is missing the indexed field, the index execution will fail and results in a duplicate key error.

So, our Support Engineers removed the index for the renamed name field and solved the error.

[Need assistance to fix MongoDB error code 11000? We’ll help you.]

Conclusion

In short, the MongoDB error code 11000 may happen when a document does not have a value for the indexed field or due to the wrong syntax used. Today, we saw the various reasons for MongoDB errors and how our Support Engineers fix them.

var google_conversion_label = «owonCMyG5nEQ0aD71QM»;

Are you sure you’re dropping the right database? Make sure you do use dbName; db.dropDatabase();.

Yes, I switched to dbName before dropping the database

The insert code is straightforward:

db.collection('users', function (err, collection) {
  collection.insert(contacts, function(err, result) {
    if (err) {
      job.log('Failed to save contacts as user with error: ' + JSON.stringify(err));
      done('Failed to save contacts as user with error: ' + JSON.stringify(err));
    } else {
      console.log(result.length);
      done();
    }
  });
});

contacts is an array of json in accordance with the users' schema whose schema is defined such that no unique index exists.

I have about 760 records out of which about 93 are inserted without any issues, the processing stops at 94th entry. This is what the 94th entry looks like:

{
    "phone": [
        {
            "number": "91xxxxxxxxxxx",
            "numberType": "work",
            "contactReferrer": "546a20a7a8b0aaf0175f3ae1"
        },
        {
            "number": "91yyyyyyyyyyy",
            "numberType": "home",
            "contactReferrer": "546a20a7a8b0aaf0175f3ae1"
        },
        {
            "number": "91zzzzzzzzzzz",
            "numberType": "work",
            "contactReferrer": "546a20a7a8b0aaf0175f3ae1"
        }
    ],
    "organization": [
        {
            "organization": "the org",
            "title": null,
            "employmentType": "employment"
        }
    ],
    "email": [
        {
            "emailId": "xxx@gmail.com",
            "emailType": "other",
            "contactReferrer": "546a20a7a8b0aaf0175f3ae1"
        }
    ],
    "address": [
        {
            "state": "Karnataka",
            "postalCode": null,
            "addressType": "other",
            "poBox": null,
            "street": null,
            "city": "Bidar",
            "country": "India"
        },
        {
            "state": "Karnataka",
            "postalCode": null,
            "addressType": "other",
            "poBox": null,
            "street": null,
            "city": "Bidar",
            "country": "India"
        }
    ],
    "name": [
        {
            "firstName": "Rakesh",
            "lastName": "Raman",
            "contactReferrer": "546a20a7a8b0aaf0175f3ae1"
        }
    ],
    "_id": "546a2786044e2cd928698eb0"
}

The error:

{"code":11000,"index":109,"errmsg":"insertDocument :: caused by :: 11000 E11000 duplicate key error index: dbName.users.$_id_ dup key: { : ObjectId('546a2786044e2cd928698eb0') }