Mongo server selection error

The read preference tag_sets parameter is an ordered list of tag sets used to restrict the eligibility of servers, such as for data center awareness.

Clients MUST raise an error if a non-empty tag set is given in tag_sets and the mode field is ‘primary’.

A read preference tag set ( T ) matches a server tag set ( S ) – or equivalently a server tag set ( S ) matches a read preference tag set ( T ) — if T is a subset of S (i.e. T ⊆ S ).

For example, the read preference tag set «< dc: ‘ny’, rack: ‘2’ >» matches a secondary server with tag set «< dc: ‘ny’, rack: ‘2’, size: ‘large’ >«.

A tag set that is an empty document matches any server, because the empty tag set is a subset of any tag set. This means the default tag_sets parameter ( [<>] ) matches all servers.

Tag sets are applied after filtering servers by mode and maxStalenessSeconds , and before selecting one server within the latency window.

Eligibility MUST be determined from tag_sets as follows:

• If the tag_sets list is empty then all candidate servers are eligible servers. (Note, the default of [<>] means an empty list probably won’t often be seen, but if the client does not forbid an empty list, this rule MUST be implemented to handle that case.)
• If the tag_sets list is not empty, then tag sets are tried in order until a tag set matches at least one candidate server. All candidate servers matching that tag set are eligible servers. Subsequent tag sets in the list are ignored.
• If the tag_sets list is not empty and no tag set in the list matches any candidate server, no servers are eligible servers.

The read preference hedge parameter is a document that configures how the server will perform hedged reads. It consists of the following keys:

• enabled : Enables or disables hedging

Hedged reads are automatically enabled in MongoDB 4.4+ when using a nearest read preference. To explicitly enable hedging, the hedge document must be passed. An empty document uses server defaults to control hedging, but the enabled key may be set to true or false to explicitly enable or disable hedged reads.

Drivers MAY allow users to specify an empty hedge document if they accept documents for read preference options. Any driver that exposes a builder API for read preference objects MUST NOT allow an empty hedge document to be constructed. In this case, the user MUST specify a value for enabled , which MUST default to true . If the user does not call a hedge API method, drivers MUST NOT send a hedge option to the server.

Drivers MUST allow users to configure a default read preference on a MongoClient object. Drivers MAY allow users to configure a default read preference on a Database or Collection object.

A read preference MAY be specified as an object, document or individual mode , tag_sets , and maxStalenessSeconds parameters, depending on what is most idiomatic for the language.

If more than one object has a default read preference, the default of the most specific object takes precedence. I.e. Collection is preferred over Database , which is preferred over MongoClient .

Drivers MAY allow users to set a read preference on queries on a per-operation basis similar to how hint or batchSize are set. E.g., in Python:

If a server of type Mongos or LoadBalancer is selected for a read operation, the read preference is passed to the selected mongos through the use of $readPreference (as a Global Command Argument for OP_MSG or a query modifier for OP_QUERY) and, for OP_QUERY only, the SecondaryOk wire protocol flag, according to the following rules.

• For mode ‘primary’, drivers MUST NOT set $readPreference
• For all other read preference modes (i.e. ‘secondary’, ‘primaryPreferred’, . ), drivers MUST set $readPreference

If the read preference contains only a mode parameter and the mode is ‘primary’ or ‘secondaryPreferred’, for maximum backwards compatibility with older versions of mongos, drivers MUST only use the value of the SecondaryOk wire protocol flag (i.e. set or unset) to indicate the desired read preference and MUST NOT use a $readPreference query modifier.

Therefore, when sending queries to a mongos or load balancer, the following rules apply:

• For mode ‘primary’, drivers MUST NOT set the SecondaryOk wire protocol flag and MUST NOT use $readPreference
• For mode ‘secondary’, drivers MUST set the SecondaryOk wire protocol flag and MUST also use $readPreference
• For mode ‘primaryPreferred’, drivers MUST set the SecondaryOk wire protocol flag and MUST also use $readPreference
• For mode ‘secondaryPreferred’, drivers MUST set the SecondaryOk wire protocol flag. If the read preference contains a non-empty tag_sets parameter, maxStalenessSeconds is a positive integer, or the hedge parameter is non-empty, drivers MUST use $readPreference ; otherwise, drivers MUST NOT use $readPreference
• For mode ‘nearest’, drivers MUST set the SecondaryOk wire protocol flag and MUST also use $readPreference

The $readPreference query modifier sends the read preference as part of the query. The read preference fields tag_sets is represented in a $readPreference document using the field name tags .

When sending a read operation via OP_QUERY and any $ modifier is used, including the $readPreference modifier, the query MUST be provided using the $query modifier like so:

A valid $readPreference document for mongos or load balancer has the following requirements:

The mode field MUST be present exactly once with the mode represented in camel case:

• ‘primary’
• ‘secondary’
• ‘primaryPreferred’
• ‘secondaryPreferred’
• ‘nearest’

If the mode field is «primary», the tags , maxStalenessSeconds , and hedge fields MUST be absent.

Otherwise, for other mode values, the tags field MUST either be absent or be present exactly once and have an array value containing at least one document. It MUST contain only documents, no other type.

The maxStalenessSeconds field MUST be either be absent or be present exactly once with an integer value.

The hedge field MUST be either absent or be a document.

Mongos or service receiving a query with $readPreference SHOULD validate the mode , tags , maxStalenessSeconds , and hedge fields according to rules 1 and 2 above, but SHOULD ignore unrecognized fields for forward-compatibility rather than throwing an error.

Because some commands are used for writes, deployment-changes or other state-changing side-effects, the use of read preference by a driver depends on the command and how it is invoked:

Write commands: insert , update , delete , findAndModify

Write commands are considered write operations and MUST follow the corresponding Rules for server selection for each topology type.

Generic command method: typically command or runCommand

The generic command method MUST act as a read operation for the purposes of server selection.

The generic command method has a default read preference of mode ‘primary’. The generic command method MUST ignore any default read preference from client, database or collection configuration. The generic command method SHOULD allow an optional read preference argument.

If an explicit read preference argument is provided as part of the generic command method call, it MUST be used for server selection, regardless of the name of the command. It is up to the user to use an appropriate read preference, e.g. not calling renameCollection with a mode of ‘secondary’.

N.B.: «used for server selection» does not supercede rules for server selection on «Standalone» topologies, which ignore any requested read preference.

Command-specific helper: methods that wrap database commands, like count , distinct , listCollections or renameCollection .

Command-specific helpers MUST act as read operations for the purposes of server selection, with read preference rules defined by the following three categories of commands:

«must-use-primary»: these commands have state-modifying effects and will only succeed on a primary. An example is renameCollection .

These command-specific helpers MUST use a read preference mode of ‘primary’, MUST NOT take a read preference argument and MUST ignore any default read preference from client, database or collection configuration. Languages with dynamic argument lists MUST throw an error if a read preference is provided as an argument.

Clients SHOULD rely on the server to return a «not writable primary» or other error if the command is «must-use-primary». Clients MAY raise an exception before sending the command if the topology type is Single and the server type is not «Standalone», «RSPrimary» or «Mongos», but the identification of the set of ‘must-use-primary’ commands is out of scope for this specification.

«should-use-primary»: these commands are intended to be run on a primary, but would succeed — albeit with possibly stale data — when run against a secondary. An example is listCollections .

These command-specific helpers MUST use a read preference mode of ‘primary’, MUST NOT take a read preference argument and MUST ignore any default read preference from client, database or collection configuration. Languages with dynamic argument lists MUST throw an error if a read preference is provided as an argument.

Clients MUST NOT raise an exception if the topology type is Single.

«may-use-secondary»: these commands run against primaries or secondaries, according to users’ read preferences. They are sometimes called «query-like» commands.

The current list of «may-use-secondary» commands includes:

• aggregate without a write stage (e.g. $out , $merge )
• collStats
• count
• dbStats
• distinct
• find
• geoNear
• geoSearch
• group
• mapReduce where the out option is
• parallelCollectionScan

Associated command-specific helpers SHOULD take a read preference argument and otherwise MUST use the default read preference from client, database, or collection configuration.

For pre-5.0 servers, an aggregate command is «must-use-primary» if its pipeline contains a write stage (e.g. $out , $merge ); otherwise, it is «may-use-secondary». For 5.0+ servers, secondaries can execute an aggregate command with a write stage and all aggregate commands are «may-use-secondary». This is discussed in more detail in Read preferences and server selection in the CRUD spec.

If a client provides a specific helper for inline mapReduce, then it is «may-use-secondary» and the regular mapReduce helper is «must-use-primary». Otherwise, the mapReduce helper is «may-use-secondary» and it is the user’s responsibility to specify when running mapReduce on a secondary.

New command-specific helpers implemented in the future will be considered «must-use-primary», «should-use-primary» or «may-use-secondary» according to the specifications for those future commands. Command helper specifications SHOULD use those terms for clarity.

Server selection is a process which takes an operation type (read or write), a ClusterDescription, and optionally a read preference and, on success, returns a ServerDescription for an operation of the given type.

Server selection varies depending on whether a client is multi-threaded/asynchronous or single-threaded because a single-threaded client cannot rely on the topology state being updated in the background.

Multi-threaded drivers and single-threaded drivers with serverSelectionTryOnce set to false MUST enforce a timeout for the server selection process. The timeout MUST be computed as described in Client Side Operations Timeout: Server Selection.

A driver that uses multi-threaded or asynchronous monitoring MUST unblock waiting operations as soon as server selection completes, even if not all servers have been checked by a monitor. Put differently, the client MUST NOT block server selection while waiting for server discovery to finish.

For example, if the client is discovering a replica set and the application attempts a read operation with mode ‘primaryPreferred’, the operation MUST proceed immediately if a suitable secondary is found, rather than blocking until the client has checked all members and possibly discovered a primary.

The number of threads allowed to wait for server selection SHOULD be either (a) the same as the number of threads allowed to wait for a connection from a pool; or (b) governed by a global or client-wide limit on number of waiting threads, depending on how resource limits are implemented by a driver.

Multi-threaded or async drivers MUST keep track of the number of operations that a given server is currently executing (the server’s operationCount ). This value MUST be incremented once a server is selected for an operation and MUST be decremented once that operation has completed, regardless of its outcome. Where this value is stored is left as a implementation detail of the driver; some example locations include the Server type that also owns the connection pool for the server (if there exists such a type in the driver’s implementation) or on the pool itself. Incrementing or decrementing a server’s operationCount MUST NOT wake up any threads that are waiting for a topology update as part of server selection. See operationCount-based selection within the latency window (multi-threaded or async) for the rationale behind the way this value is used.

For multi-threaded clients, the server selection algorithm is as follows:

1. Record the server selection start time
2. If the topology wire version is invalid, raise an error
3. Find suitable servers by topology type and operation type
4. Filter the suitable servers by calling the optional, application-provided server selector.
5. If there are any suitable servers, filter them according to Filtering suitable servers based on the latency window and continue to the next step; otherwise, goto Step #9.
6. Choose two servers at random from the set of suitable servers in the latency window. If there is only 1 server in the latency window, just select that server and goto Step #8.
7. Of the two randomly chosen servers, select the one with the lower operationCount . If both servers have the same operationCount , select arbitrarily between the two of them.
8. Increment the operationCount of the selected server and return it. Do not go onto later steps.
9. Request an immediate topology check, then block the server selection thread until the topology changes or until the server selection timeout has elapsed
10. If server selection has timed out, raise a server selection error
11. Goto Step #2

Single-threaded drivers do not monitor the topology in the background. Instead, they MUST periodically update the topology during server selection as described below.

When serverSelectionTryOnce is true, server selection timeouts have no effect; a single immediate topology check will be done if the topology starts stale or if the first selection attempt fails.

When serverSelectionTryOnce is false, then the server selection loops until a server is successfully selected or until the selection timeout is exceeded.

Therefore, for single-threaded clients, the server selection algorithm is as follows:

1. Record the server selection start time
2. Record the maximum time as start time plus the computed timeout
3. If the topology has not been scanned in heartbeatFrequencyMS milliseconds, mark the topology stale
4. If the topology is stale, proceed as follows:
   • record the target scan time as last scan time plus minHeartBeatFrequencyMS
   • if serverSelectionTryOnce is false and the target scan time would exceed the maximum time, raise a server selection error
   • if the current time is less than the target scan time, sleep until the target scan time
   • do a blocking immediate topology check (which must also update the last scan time and mark the topology as no longer stale)
5. If the topology wire version is invalid, raise an error
6. Find suitable servers by topology type and operation type
7. Filter the suitable servers by calling the optional, application-provided server selector.
8. If there are any suitable servers, filter them according to Filtering suitable servers based on the latency window and return one at random from the filtered servers; otherwise, mark the topology stale and continue to step #9.
9. If serverSelectionTryOnce is true and the last scan time is newer than the selection start time, raise a server selection error; otherwise, goto Step #4
10. If the current time exceeds the maximum time, raise a server selection error
11. Goto Step #4

Before using a socket to the selected server, drivers MUST check whether the socket has been used in socketCheckIntervalMS milliseconds. If the socket has been idle for longer, the driver MUST update the ServerDescription for the selected server. After updating, if the server is no longer suitable, the driver MUST repeat the server selection algorithm and select a new server.

Because single-threaded selection can do a blocking immediate check, the server selection timeout is not a hard deadline. The actual maximum server selection time for any given request can vary from the timeout minus minHeartbeatFrequencyMS to the timeout plus the time required for a blocking scan.

Single-threaded drivers MUST document that when serverSelectionTryOne is true, selection may take up to the time required for a blocking scan, and when serverSelectionTryOne is false, selection may take up to the timeout plus the time required for a blocking scan.

When a deployment has topology type «Unknown», no servers are suitable for read or write operations.

A deployment of topology type Single contains only a single server of any type. Topology type Single signifies a direct connection intended to receive all read and write operations.

Therefore, read preference is ignored during server selection with topology type Single. The single server is always suitable for reads if it is available. Depending on server type, the read preference is communicated to the server differently:

• Type Mongos: the read preference is sent to the server using the rules for Passing read preference to mongos and load balancers.
• Type Standalone: clients MUST NOT send the read preference to the server
• For all other types, using OP_QUERY: clients MUST always set the SecondaryOk wire protocol flag on reads to ensure that any server type can handle the request.
• For all other types, using OP_MSG: If no read preference is configured by the application, or if the application read preference is Primary, then $readPreference MUST be set to < «mode»: «primaryPreferred» >to ensure that any server type can handle the request. If the application read preference is set otherwise, $readPreference MUST be set following Document structure.

The single server is always suitable for write operations if it is available.

During command construction, drivers MUST add a $readPreference field to the command when required by Passing read preference to mongos and load balancers; see the Load Balancer Specification for details.

A deployment with topology type ReplicaSetWithPrimary or ReplicaSetNoPrimary can have a mix of server types: RSPrimary (only in ReplicaSetWithPrimary), RSSecondary, RSArbiter, RSOther, RSGhost, Unknown or PossiblePrimary.

For the purpose of selecting a server for read operations, the same rules apply to both ReplicaSetWithPrimary and ReplicaSetNoPrimary.

To select from the topology a server that matches the user’s Read Preference:

If mode is ‘primary’, select the primary server.

If mode is ‘secondary’ or ‘nearest’:

1. Select all secondaries if mode is ‘secondary’, or all secondaries and the primary if mode is ‘nearest’.
2. From these, filter out servers staler than maxStalenessSeconds if it is a positive number.
3. From the remaining servers, select servers matching the tag_sets .
4. From these, select one server within the latency window.

If mode is ‘secondaryPreferred’, attempt the selection algorithm with mode ‘secondary’ and the user’s maxStalenessSeconds and tag_sets . If no server matches, select the primary.

If mode is ‘primaryPreferred’, select the primary if it is known, otherwise attempt the selection algorithm with mode ‘secondary’ and the user’s maxStalenessSeconds and tag_sets .

For all read preferences modes except ‘primary’, clients MUST set the SecondaryOk wire protocol flag (OP_QUERY) or $readPreference global command argument (OP_MSG) to ensure that any suitable server can handle the request. If the read preference mode is ‘primary’, clients MUST NOT set the SecondaryOk wire protocol flag (OP_QUERY) or $readPreference global command argument (OP_MSG).

If the topology type is ReplicaSetWithPrimary, only an available primary is suitable for write operations.

If the topology type is ReplicaSetNoPrimary, no servers are suitable for write operations.

A deployment of topology type Sharded contains one or more servers of type Mongos or Unknown.

For read operations, all servers of type Mongos are suitable; the mode , tag_sets , and maxStalenessSeconds read preference parameters are ignored for selecting a server, but are passed through to mongos. See Passing read preference to mongos and load balancers.

For write operations, all servers of type Mongos are suitable.

If more than one mongos is suitable, drivers MUST select a suitable server within the latency window (see Filtering suitable servers based on the latency window).

For every available server, clients MUST track the average RTT of server monitoring hello or legacy hello commands.

An Unknown server has no average RTT. When a server becomes unavailable, its average RTT MUST be cleared. Clients MAY implement this idiomatically (e.g nil, -1, etc.).

When there is no average RTT for a server, the average RTT MUST be set equal to the first RTT measurement (i.e. the first hello or legacy hello command after the server becomes available).

After the first measurement, average RTT MUST be computed using an exponentially-weighted moving average formula, with a weighting factor ( alpha ) of 0.2. If the prior average is denoted old_rtt , then the new average ( new_rtt ) is computed from a new RTT measurement ( x ) using the following formula:

A weighting factor of 0.2 was chosen to put about 85% of the weight of the average RTT on the 9 most recent observations.

Server selection results in a set of zero or more suitable servers. If more than one server is suitable, a server MUST be selected from among those within the latency window.

The localThresholdMS configuration parameter controls the size of the latency window used to select a suitable server.

The shortest average round trip time (RTT) from among suitable servers anchors one end of the latency window ( A ). The other end is determined by adding localThresholdMS ( B = A + localThresholdMS ).

A server MUST be selected from among suitable servers that have an average RTT ( RTT ) within the latency window (i.e. A ≤ RTT ≤ B ). In other words, the suitable server with the shortest average RTT is always a possible choice. Other servers could be chosen if their average RTTs are no more than localThresholdMS more than the shortest average RTT.

See either Single-threaded server selection or Multi-threaded or asynchronous server selection for information on how to select a server from among those within the latency window.

Only for single-threaded drivers.

If a server is selected that has an existing connection that has been idle for socketCheckIntervalMS, the driver MUST check the connection with the «ping» command. If the ping succeeds, use the selected connection. If not, set the server’s type to Unknown and update the Topology Description according to the Server Discovery and Monitoring Spec, and attempt once more to select a server.

The logic is expressed in this pseudocode. The algorithm for the «getServer» function is suggested below, in Single-threaded server selection implementation:

The prior read preference specification included the concept of a «request», which pinned a server to a thread for subsequent, related reads. Requests and pinning are now deprecated. See What happened to pinning? for the rationale for this change.

Drivers with an existing request API MAY continue to provide it for backwards compatibility, but MUST document that pinning for the request does not guarantee monotonic reads.

Drivers MUST NOT automatically pin the client or a thread to a particular server without an explicit start_request (or comparable) method call.

Outside a legacy «request» API, drivers MUST use server selection for each individual read operation.

The single-threaded reference implementation is the Perl master branch (work towards v1.0.0). The multi-threaded reference implementation is TBD.

These are suggestions. As always, driver authors should balance cross-language standardization with backwards compatibility and the idioms of their language.

Modes (‘primary’, ‘secondary’, . ) are constants declared in whatever way is idiomatic for the programming language. The constant values may be ints, strings, or whatever. However, when attaching modes to $readPreference camel case must be used as described above in Passing read preference to mongos and load balancers.

‘primaryPreferred’ is equivalent to selecting a server with read preference mode ‘primary’ (without tag_sets or maxStalenessSeconds ), or, if that fails, falling back to selecting with read preference mode ‘secondary’ (with tag_sets and maxStalenessSeconds , if provided).

‘secondaryPreferred’ is the inverse: selecting with mode ‘secondary’ (with tag_sets and maxStalenessSeconds ) and falling back to selecting with mode ‘primary’ (without tag_sets or maxStalenessSeconds ).

Depending on the implementation, this may result in cleaner code.

The term ‘nearest’ is unfortunate, as it implies a choice based on geographic locality or absolute lowest latency, neither of which are true.

Instead, and unlike the other read preference modes, ‘nearest’ does not favor either primaries or secondaries; instead all servers are candidates and are filtered by tag_sets and maxStalenessSeconds .

To always select the server with the lowest RTT, users should use mode ‘nearest’ without tag_sets or maxStalenessSeconds and set localThresholdMS to zero.

To distribute reads across all members evenly regardless of RTT, users should use mode ‘nearest’ without tag_sets or maxStalenessSeconds and set localThresholdMS very high so that all servers fall within the latency window.

In both cases, tag_sets and maxStalenessSeconds could be used to further restrict the set of eligible servers, if desired.

Tag set lists can be configured in the driver in whatever way is natural for the language.

The following example uses a single lock for clarity. Drivers are free to implement whatever concurrency model best suits their design.

Drivers should use server descriptions and their error attributes (if set) to return useful error messages.

For example, when there are no members matching the ReadPreference:

• «No server available for query with ReadPreference primary»
• «No server available for query with ReadPreference secondary»
• «No server available for query with ReadPreference » + mode + «, tag set list » + tag_sets + «, and maxStalenessSeconds » + maxStalenessSeconds

Or, if authentication failed:

• «Authentication failed: [specific error message]»

Here is a sketch of some pseudocode for handling error reporting when errors could be different across servers:

Cursor operations OP_GET_MORE and OP_KILL_CURSOR do not go through the server selection process. Cursor operations must be sent to the original server that received the query and sent the OP_REPLY. For exhaust cursors, the same socket must be used for OP_GET_MORE until the cursor is exhausted.

Operations that are part of a sharded transaction (after the initial command) do not go through the server selection process. Sharded transaction operations MUST be sent to the original mongos server on which the transaction was started.

Note: As of MongoDB 2.6, mongos doesn’t distribute the «text» command to secondaries, see SERVER-10947.

However, the «text» command is deprecated in 2.6, so this command-specific helper may become deprecated before this is fixed.

The server selection test plan is given in a separate document that describes the tests and supporting data files: Server Selection Tests

The prior version of the read preference spec had only a loose definition of server or topology types. The Server Discovery and Monitoring spec defines these terms explicitly and they are used here for consistency and clarity.

In order to ensure that behavior is consistent regardless of topology type, read preference behaviors are limited to those that mongos can proxy.

For example, mongos ignores read preference ‘secondary’ when a shard consists of a single server. Therefore, this spec calls for topology type Single to ignore read preferences for consistency.

The spec has been written with the intention that it can apply to both drivers and mongos and the term «client» has been used when behaviors should apply to both. Behaviors that are specific to drivers are largely limited to those for communicating with a mongos.

Because this does not apply only to secondaries and does not limit absolute latency, the name secondaryAcceptableLatencyMS is misleading.

The mongos name localThreshold misleads because it has nothing to do with locality. It also doesn’t include the MS units suffix for consistency with other time-related configuration options.

However, given a choice between the two, localThreshold is a more general term. For drivers, we add the MS suffix for clarity about units and consistency with other configuration options.

When more than one server is judged to be suitable, the spec calls for random selection to ensure a fair distribution of work among servers within the latency window.

It would be hard to ensure a fair round-robin approach given the potential for servers to come and go. Making newly available servers either first or last could lead to unbalanced work. Random selection has a better fairness guarantee and keeps the design simpler.

As operation execution slows down on a node (e.g. due to degraded server-side performance or increased network latency), checked-out pooled connections to that node will begin to remain checked out for longer periods of time. Assuming at least constant incoming operation load, more connections will then need to be opened against the node to service new operations that it gets selected for, further straining it and slowing it down. This can lead to runaway connection creation scenarios that can cripple a deployment («connection storms»). As part of DRIVERS-781, the random choice portion of multi-threaded server selection was changed to more evenly spread out the workload among suitable servers in order to prevent any single node from being overloaded. The new steps achieve this by approximating an individual server’s load via the number of concurrent operations that node is processing (operationCount) and then routing operations to servers with less load. This should reduce the number of new operations routed towards nodes that are busier and thus increase the number routed towards nodes that are servicing operations faster or are simply less busy. The previous random selection mechanism did not take load into account and could assign work to nodes that were under too much stress already.

As an added benefit, the new approach gives preference to nodes that have recently been discovered and are thus are more likely to be alive (e.g. during a rolling restart). The narrowing to two random choices first ensures new servers aren’t overly preferred however, preventing a «thundering herd» situation. Additionally, the maxConnecting provisions included in the CMAP specification prevent drivers from crippling new nodes with connection storms.

This approach is based on the «Power of Two Random Choices with Least Connections» load balancing algorithm.

An alternative approach to this would be to prefer selecting servers that already have available connections. While that approach could help reduce latency, it does not achieve the benefits of routing operations away from slow servers or of preferring newly introduced servers. Additionally, that approach could lead to the same node being selected repeatedly rather than spreading the load out among all suitable servers.

In server selection, there is a race condition that could exist between what a selected server type is believed to be and what it actually is.

The SecondaryOk wire protocol flag solves the race problem by communicating to the server whether a secondary is acceptable. The server knows its type and can return a «not writable primary» error if SecondaryOk is false and the server is a secondary.

However, because topology type Single is used for direct connections, we want read operations to succeed even against a secondary, so the SecondaryOk wire protocol flag must be sent to mongods with topology type Single.

(If the server type is Mongos, follow the rules for Passing read preference to mongos and load balancers, even for topology type Single.)

The list of commands that can go to secondaries changes over time and depends not just on the command but on parameters. For example, the mapReduce command may or may not be able to be run on secondaries depending on the value of the out parameter.

It significantly simplifies implementation for the general command method always to go to the primary unless a explicit read preference is set and rely on users of the general command method to provide a read preference appropriate to the command.

The command-specific helpers will need to implement a check of read preferences against the semantics of the command and its parameters, but keeping this logic close to the command rather than in a generic method is a better design than either delegating this check to the generic method, duplicating the logic in the generic method, or coupling both to another validation method.

Using an exponentially-weighted moving average avoids having to store and rotate an arbitrary number of RTT observations. All observations count towards the average. The weighting makes recent observations count more heavily while smoothing volatility.

Error messages should be sufficiently verbose to allow users and/or support engineers to determine the reasons for server selection failures from log or other error messages.

Single-threaded drivers in languages like PHP and Perl are typically deployed as many processes per application server. Each process must independently discover and monitor the MongoDB deployment.

When no suitable server is available (due to a partition or misconfiguration), it is better for each request to fail as soon as its process detects a problem, instead of waiting and retrying to see if the deployment recovers.

Minimizing response latency is important for maximizing request-handling capacity and for user experience (e.g. a quick fail message instead of a slow web page).

However, when a request arrives and the topology information is already stale, or no suitable server is known, making a single attempt to update the topology to service the request is acceptable.

A user of a single-threaded driver who prefers resilience in the face of topology problems, rather than short response times, can turn the «try once» mode off. Then driver rescans the topology every minHeartbeatFrequencyMS until a suitable server is found or the timeout expires.

Single-threaded clients need to make a compromise: if they check servers too frequently it slows down regular operations, but if they check too rarely they cannot proactively avoid errors.

Errors are more disruptive for single-threaded clients than for multi-threaded. If one thread in a multi-threaded process encounters an error, it warns the other threads not to use the disconnected server. But single-threaded clients are deployed as many independent processes per application server, and each process must throw an error until all have discovered that a server is down.

The compromise specified here balances the cost of frequent checks against the disruption of many errors. The client preemptively checks individual sockets that have not been used in the last socketCheckIntervalMS, which is more frequent by default than heartbeatFrequencyMS defined in the Server Discovery and Monitoring Spec.

The client checks the socket with a «ping» command, rather than «hello» or legacy hello, because it is not checking the server’s full state as in the Server Discovery and Monitoring Spec, it is only verifying that the connection is still open. We might also consider a select or poll call to check if the socket layer considers the socket closed, without requiring a round-trip to the server. However, this technique usually will not detect an uncleanly shutdown server or a network outage.

In general, backwards breaking changes have been made in the name of consistency with mongos and avoiding misleading users about monotonicity.

• Automatic pinning (see What happened to pinning?)
• Auto retry (replaced by the general server selection algorithm)
• mongos «high availability» mode (effectively, mongos pinning)

Other features and behaviors have changed explicitly

• Ignoring read preferences for topology type Single
• Default read preference for the generic command method

Changes with grandfather clauses

• Alternate names for localThresholdMS
• Pinning for legacy request APIs

Internal changes with little user-visibility

The prior read preference spec, which was implemented in the versions of the drivers and mongos released concomitantly with MongoDB 2.2, stated that a thread / client should remain pinned to an RS member as long as that member matched the current mode, tags, and acceptable latency. This increased the odds that reads would be monotonic (assuming no rollback), but had the following surprising consequence:

1. Thread / client reads with mode ‘secondary’ or ‘secondaryPreferred’, gets pinned to a secondary
2. Thread / client reads with mode ‘primaryPreferred’, driver / mongos sees that the pinned member (a secondary) matches the mode (which allows for a secondary) and reads from secondary, even though the primary is available and preferable

The old spec also had the swapped problem, reading from the primary with ‘secondaryPreferred’, except for mongos which was changed at the last minute before release with SERVER-6565.

This left application developers with two problems:

1. ‘primaryPreferred’ and ‘secondaryPreferred’ acted surprisingly and unpredictably within requests
2. There was no way to specify a common need: read from a secondary if possible with ‘secondaryPreferred’, then from primary if possible with ‘primaryPreferred’, all within a request. Instead an application developer would have to do the second read with ‘primary’, which would unpin the thread but risk unavailability if only secondaries were up.

Additionally, mongos 2.4 introduced the releaseConnectionsAfterResponse option (RCAR), mongos 2.6 made it the default and mongos 2.8 will remove the ability to turn it off. This means that pinning to a mongos offers no guarantee that connections to shards are pinned. Since we can’t provide the same guarantees for replica sets and sharded clusters, we removed automatic pinning entirely and deprecated «requests». See SERVER-11956 and SERVER-12273.

Regardless, even for replica sets, pinning offers no monotonicity because of the ever-present possibility of rollbacks. Through MongoDB 2.6, secondaries did not close sockets on rollback, so a rollback could happen between any two queries without any indication to the driver.

Therefore, an inconsistent feature that doesn’t actually do what people think it does has no place in the spec and has been removed. Should the server eventually implement some form of «sessions», this spec will need to be revised accordingly.

Mongos HA has similar problems with pinning, in that one can wind up pinned to a high-latency mongos even if a lower-latency mongos later becomes available.

Selection within the latency window avoids this problem and makes server selection exactly analogous to having multiple suitable servers from a replica set. This is easier to explain and implement.

The old auto-retry mechanism was closely connected to server pinning, which has been removed. It also mandated exactly three attempts to carry out a query on different servers, with no way to disable or adjust that value, and only for the first query within a request.

To the extent that auto-retry was trying to compensate for unavailable servers, the Server Discovery and Monitoring spec and new server selection algorithm provide a more robust and configurable way to direct all queries to available servers.

After a server is selected, several error conditions could still occur that make the selected server unsuitable for sending the operation, such as:

• the server could have shutdown the socket (e.g. a primary stepping down),
• a connection pool could be empty, requiring new connections; those connections could fail to connect or could fail the server handshake

Once an operation is sent over the wire, several additional error conditions could occur, such as:

• a socket timeout could occur before the server responds
• the server might send an RST packet, indicating the socket was already closed
• for write operations, the server might return a «not writable primary» error

This specification does not require nor prohibit drivers from attempting automatic recovery for various cases where it might be considered reasonable to do so, such as:

• repeating server selection if, after selection, a socket is determined to be unsuitable before a message is sent on it
• for a read operation, after a socket error, selecting a new server meeting the read preference and resending the query
• for a write operation, after a «not writable primary» error, selecting a new server (to locate the primary) and resending the write operation

Driver-common rules for retrying operations (and configuring such retries) could be the topic of a different, future specification.

The intention of read preference’s list of tag sets is to allow a user to prefer the first tag set but fall back to members matching later tag sets. In order to know whether to fall back or not, we must first filter by all other criteria.

Say you have two secondaries:

• Node 1, tagged <‘tag’: ‘value1’>, estimated staleness 5 minutes
• Node 2, tagged <‘tag’: ‘value2’>, estimated staleness 1 minute

And a read preference:

If tag sets were applied before maxStalenessSeconds, we would select Node 1 since it matches the first tag set, then filter it out because it is too stale, and be left with no eligible servers.

The user’s intent in specifying two tag sets was to fall back to the second set if needed, so we filter by maxStalenessSeconds first, then tag_sets, and select Node 2.

Источник

Adblock
detector

Error handling using the .NET driver is a requirement for a fault-tolerant system. While MongoDB supports automatic replica set failover and the driver supports multiple mongos’s, this doesn’t make a program immune to errors.

Almost all errors will occur while attempting to perform an operation on the server. In other words, you will not receive an error when constructing a MongoClient, getting a database, or getting a collection. This is because we are connecting to servers in the background and continually trying to reconnect when a problem occurs. Only when you attempt to perform an operation do those errors become apparent.

There are a few types of errors you will see.

Server Selection Errors

Even when some servers are available, it might not be possible to satisfy a request. For example, using tag sets in a read preference when no server exists with those tags or attempting to write to a replica set when the Primary is unavailable. Both of these would result in a TimeoutException. Below is an example exception (formatted for readability) when attempting to insert into a replica set without a primary.

System.TimeoutException: A timeout occured after 30000ms selecting a server using 
CompositeServerSelector{ 
    Selectors = 
        WritableServerSelector, 
        LatencyLimitingServerSelector{ AllowedLatencyRange = 00:00:00.0150000 } 
}. 

Client view of cluster state is 
{ 
    ClusterId : "1", 
    Type : "ReplicaSet", 
    State : "Connected", 
    Servers : [
        { 
            ServerId: "{ ClusterId : 1, EndPoint : "Unspecified/clover:30000" }", 
            EndPoint: "Unspecified/clover:30000", 
            State: "Disconnected", 
            Type: "Unknown" 
        }, 
        { 
            ServerId: "{ ClusterId : 1, EndPoint : "Unspecified/clover:30001" }", 
            EndPoint: "Unspecified/clover:30001", 
            State: "Connected", 
            Type: "ReplicaSetSecondary", 
            WireVersionRange: "[0, 3]" 
        }, 
        { 
            ServerId: "{ ClusterId : 1, EndPoint : "Unspecified/clover:30002" }", 
            EndPoint: "Unspecified/clover:30002", 
            State: "Disconnected", 
            Type: "Unknown"
        }, 
    ] 
}.

Inspecting this error will tell you that we were attempting to find a writable server. The servers that are known are clover:30000, clover:30001, and clover:30002. Of these, clover:30000 and clover:30002 are disconnected and clover:30001 is connected and a secondary. By default, we wait for 30 seconds to fulfill the request. During this time, we are trying to connect to clover:30000 and clover:30002 in hopes that one of them becomes available and and takes the role of primary. In this case, we timed out waiting.

Connection Errors

A server may be unavailable for a variety of reasons. Each of these reasons will manifest themselves as a TimeoutException. This is because, over time, it is entirely possible for the state of the servers to change in such a way
that a matching server becomes available.

Server is unavailable

When a server is not listening at the location specified, the driver can’t connect to it.

System.TimeoutException: A timeout occured after 30000ms selecting a server using
CompositeServerSelector{ 
    Selectors = 
        WritableServerSelector, 
        LatencyLimitingServerSelector{ AllowedLatencyRange = 00:00:00.0150000 } 
}. 

Client view of cluster state is 
{ 
    ClusterId : "1", 
    Type : "Unknown", 
    State : "Disconnected", 
    Servers : [
        { 
            ServerId: "{ ClusterId : 1, EndPoint : "Unspecified/localhost:27017" }", 
            EndPoint: "Unspecified/localhost:27017", 
            State: "Disconnected", 
            Type: "Unknown", 
            HeartbeatException: "MongoDB.Driver.MongoConnectionException: An exception occurred while opening a connection to the server. 
---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 127.0.0.1:27017
   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult).
   ... snip ..."
        }
    ]
}

We see that we are attempting to connect to localhost:27017, but “No connection could be made because the target machine actively refused it”.

Fixing this problem either involves starting a server at the specified location or restarting your application with the correct host and port.

DNS problems

When DNS is misconfigured, or the hostname provided is not registered, resolution from hostname to IP address may fail.

System.TimeoutException: A timeout occured after 30000ms selecting a server using 
CompositeServerSelector{ 
    Selectors = 
        WritableServerSelector, 
        LatencyLimitingServerSelector{ AllowedLatencyRange = 00:00:00.0150000 } 
}. 

Client view of cluster state is 
{ 
    ClusterId : "1", 
    Type : "Unknown", 
    State : "Disconnected", 
    Servers : [
        { 
            ServerId: "{ ClusterId : 1, EndPoint : "Unspecified/idonotexist:27017" }", 
            EndPoint: "Unspecified/idonotexist:27017", 
            State: "Disconnected", 
            Type: "Unknown", 
            HeartbeatException: "MongoDB.Driver.MongoConnectionException: An exception occurred while opening a connection to the server. 
---> System.Net.Sockets.SocketException: No such host is known
   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
   ... snip ..."
        }
    ]
}

We see that we are attempting to connect to idonotexist:27017, but “No such host is known”.

Fixing this problem either involves bringing up a server at the specified location, fixing DNS to resolve the host correctly, or modifing your hosts file.

Replica Set Misconfiguration

DNS problems might be seen when a replica set is misconfigured. It is imperative that the hosts in your replica set configuration be DNS resolvable from the client. Just because the replica set members can talk to one another does not mean that your application servers can also talk to the replica set members.

Taking too long to respond

When the latency between the driver and the server is too great, the driver may give up.

System.TimeoutException: A timeout occured after 30000ms selecting a server using
CompositeServerSelector{ 
    Selectors = 
        WritableServerSelector, 
        LatencyLimitingServerSelector{ AllowedLatencyRange = 00:00:00.0150000 } 
}.

Client view of cluster state is 
{ 
    ClusterId : "1", 
    Type : "Unknown", 
    State : "Disconnected", 
    Servers : [
        {
            ServerId: "{ ClusterId : 1, EndPoint : "Unspecified/somefaroffplace:27017" }", 
            EndPoint: "Unspecified/somefaroffplace:27017", 
            State: "Disconnected", 
            Type: "Unknown",
            HeartbeatException: "MongoDB.Driver.MongoConnectionException: An exception occurred while opening a connection to the server. 
---> System.TimeoutException: Timed out connecting to Unspecified/somefaroffplace:27017. Timeout was 00:00:30.
   at MongoDB.Driver.Core.Connections.TcpStreamFactory.<ConnectAsync>d__7.MoveNext()
   ... snip ...
        }
    ]
}

We see that attempting to connect to somefaroffplace:27017 failed because we “Timed out connecting to Unspecified/somefaroffplace:27017. Timeout was 00:00:30.”

The default connection timeout is 30 seconds and can be changed using the MongoClientSettings.ConnectTimeout property or the connectTimeout option on a connection string.

Authentication Errors

When the credentials or the authentication mechanism is incorrect, the application will fail to connect.

System.TimeoutException: A timeout occured after 30000ms selecting a server using
CompositeServerSelector{ 
    Selectors = 
        WritableServerSelector, 
        LatencyLimitingServerSelector{ AllowedLatencyRange = 00:00:00.0150000 } 
}. 
        
Client view of cluster state is 
{ 
    ClusterId : "1", 
    Type : "Unknown", 
    State : "Disconnected", 
    Servers : [
        {
            ServerId: "{ ClusterId : 1, EndPoint : "Unspecified/localhost:27017" }", 
            EndPoint: "Unspecified/localhost:27017",
            State: "Disconnected", 
            Type: "Unknown", 
            HeartbeatException: "MongoDB.Driver.MongoConnectionException: An exception occurred while opening a connection to the server. 
 ---> MongoDB.Driver.MongoAuthenticationException: Unable to authenticate using sasl protocol mechanism SCRAM-SHA-1. 
 ---> MongoDB.Driver.MongoCommandException: Command saslStart failed: Authentication failed..
   ... snip ...
        }
    ]
}

We see that attempting to connect to localhost:27017 failed because the driver was “Unable to authenticate using sasl protocol mechanism SCRAM-SHA-1.” In place of SCRAM-SHA-1 could be any other authentication protocol supported by MongoDB.

Fixing this problem either involves adding the specified user to the server or restarting the application with the correct credentials and mechanisms.

Operation Errors

After successfully selecting a server to run an operation against, errors are still possible. Unlike connection errors, it is sometimes possible to take action at runtime.

Connection Errors

A server may go down after it was selected, but before the operation was executed. These will always manifest as a MongoConnectionException. Inspecting the inner exception will provide the actual details of the error.

MongoDB.Driver.MongoConnectionException: An exception occurred while receiving a message from the server. 
---> System.IO.IOException: Unable to read data from the transport connection: Anestablished connection was aborted by the software in your host machine. 
---> System.Net.Sockets.SocketException: An established connection was aborted by the software in your host machine at System.Net.Sockets.Socket.BeginReceive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags, AsyncCallback callback, Object state)
   at System.Net.Sockets.NetworkStream.BeginRead(Byte[] buffer, Int32 offset, Int32 size, AsyncCallback callback, Object state)
   --- End of inner exception stack trace ---
... snip ...

We see from this exception that a transport connection that was successfully open at one point has been aborted.

There are too many forms of this type of exception to enumerate. In general, it is not safe to retry operations that threw a MongoConnectionException unless the operation was idempotent. Simply getting an exception of this type doesn’t give any insight into whether the operations was received by the server or what happened in the server if it was received.

Write Exceptions

When performing a write, it is possible to receive a MongoWriteException. This exception has two important properties, WriteError and WriteConcernError.

Write Error

A write error means that there was an error applying the write. The cause could be many different things. The WriteError contains a number of properties which may help in the diagnosis of the problem. The Code property will indicate specifically what went wrong. For general categories of errors, the driver also provides a helpful Category property which classifies certain codes.

MongoDB.Driver.MongoWriteException: A write operation resulted in an error. E11000 duplicate key error index: test.people.$_id_ dup key: { : 0 } 
---> MonoDB.Driver.MongoBulkWriteException`1[MongoDB.Bson.BsonDocument]: A bulk write oeration resulted in one or more errors. E11000 duplicate key error index: test.people.$_id_ dup key: { : 0 }
   at MongoDB.Driver.MongoCollectionImpl`1.<BulkWriteAsync>d__11.MoveNext() in :projectsmongo-csharp-driversrcMongoDB.DriverMongoCollectionImpl.cs:line 16

We see from this exception that we’ve attempted to insert a document with a duplicate _id field. In this case, the write error would contain the category DuplicateKeyError.

Write Concern Error

A write concern error indicates that the server was unable to guarantee the write operation to the level specified. See the server’s documentation for more information.

Bulk Write Exceptions

A MongoBulkWriteException will occur when using the InsertManyAsync or BulkWriteAsync. This exception is just a rollup of a bunch of individual write errors. It also includes a write concern error and a Result property.

MongoDB.Driver.MongoBulkWriteException`1[MongoDB.Bson.BsonDocument]: A bulk write operation resulted in one or more errors. 
  E11000 duplicate key error index: test.people.$_id_ dup key: { : 0 }
   at MongoDB.Driver.MongoCollectionImpl`1.<BulkWriteAsync>d__11.MoveNext() in c :projectsmongo-csharp-driversrcMongoDB.DriverMongoCollectionImpl.cs:line 166

Above, we see that a duplicate key exception occured. In this case, two writes existed in the batch. Inspected the WriteErrors property would allow the identification of which write failed.

Ordered Writes

Bulk writes are allowed to be processed either in order or without order. When processing in order, the first error that occurs will stop the processing of all subsequent writes. In this case, all of the unprocessed writes will be available via the UnprocessedRequests property.

Hi @vkarpov15,

I see this problem only there is a poller function constantly calling find/findAndUpdate function on a collection. It waits serverSelectionTimeoutMS milliseconds unresponsively with READYSTATE 0. Then throws error.

You can try this simple script to see that it stuck at readystate 0 when you bring the primary node down.

const mongoose = require('mongoose');
const mongoUri = 'mongodb+srv://username:password@hostname/failover?retryWrites=true&w=majority&readPreference=primaryPreferred';
const Test = require('./test');
mongoose.connect(mongoUri, {keepAlive: 1, useNewUrlParser: true,
    useUnifiedTopology: true, poolSize: 5});
mongoose.connection.on('error', (err) => {
    console.error('Mongoose connection error: ' + JSON.stringify(err));
});
mongoose.connection.on('reconnected', () => {
    console.log('Mongoose default connection reconnected: ' + new Date());
});
mongoose.connection.on('reconnectFailed', () => {
    console.log('Mongoose default connection reconnect failed: ' + new Date());
});
mongoose.connection.on('disconnected', function () {
    console.log('Mongoose default connection disconnected: ' + new Date());
});
mongoose.connection.on('connected', function () {
    console.log('Mongoose default connection connected: ' + new Date());
});
async function finder() {
    try  {
        // console.log('READYSTATE', mongoose.connection.readyState);
        await Test.find({});
    } catch(err) {
        console.error('error inside finder: ' + JSON.stringify(err));
    }
}
(async () => {
    try {
        await Test.deleteMany({});
        console.log('numbers are removed');
        setInterval(finder, 200);
        const numbers = Array.from(Array(10000).keys());
        for(const number of numbers) {
            console.log(number);
            const test = new Test({num: number});
            await test.save();
        }
    } catch (err) {
        console.error('error inside main: ' + JSON.stringify(err));
    }
})();

Test.js

const mongoose = require('mongoose');
const testSchema = new mongoose.Schema({
    num: Number
},{
    typeKey: '$type', timestamps: {createdAt: 'created_at', updatedAt: 'updated_at'}
});
module.exports = mongoose.model('test', testSchema);

This script prints out all the numbers until primary node is down. Additionally, the last number logged to the console doesn’t exists in the collection even if code continues successfully (happens time to time).

Sample output:
1803 not exists in DB.

1801
1802
1803
Mongoose default connection disconnected: Mon Apr 06 2020 20:55:09 GMT+0300 (GMT+03:00)
error inside main: {"message":"not master","name":"MongooseServerSelectionError","reason":{"type":"ReplicaSetNoPrimary","setName":"Cluster0-shard-0","maxSetVersion":2,"maxElectionId":"7fffffff0000000000000033","servers":{},"stale":false,"compatible":true,"compatibilityError":null,"logicalSessionTimeoutMinutes":30,"heartbeatFrequencyMS":10000,"localThresholdMS":15,"commonWireVersion":8}}
error inside finder: {"message":"not master","name":"MongooseServerSelectionError","reason":{"type":"ReplicaSetNoPrimary","setName":"Cluster0-shard-0","maxSetVersion":2,"maxElectionId":"7fffffff0000000000000033","servers":{},"stale":false,"compatible":true,"compatibilityError":null,"logicalSessionTimeoutMinutes":30,"heartbeatFrequencyMS":10000,"localThresholdMS":15,"commonWireVersion":8}}
error inside finder: {"message":"not master","name":"MongooseServerSelectionError","reason":{"type":"ReplicaSetNoPrimary","setName":"Cluster0-shard-0","maxSetVersion":2,"maxElectionId":"7fffffff0000000000000033","servers":{},"stale":false,"compatible":true,"compatibilityError":null,"logicalSessionTimeoutMinutes":30,"heartbeatFrequencyMS":10000,"localThresholdMS":15,"commonWireVersion":8}}

The one below using native mongo driver works fine. It continue printing logs a second after I bring the primary node down.

const MongoClient = require('mongodb').MongoClient;
let db, collection;


async function finder() {
    try  {
        await collection.find({});
    } catch(err) {
        console.error('error inside finder: ' + JSON.stringify(err));
    }
}
(async () => {
    try {
        const client  = await MongoClient.connect(mongoUri,
            {keepAlive: 1, useNewUrlParser: true, useUnifiedTopology: true, poolSize: 5});
        db = client.db('failover');
        collection = db.collection('tests');
        await collection.deleteMany({});
        console.log('numbers are removed');
        db.s.topology.on('close', () => console.log('Connection closed') );
        db.s.topology.on('reconnect', () => console.log('Reconnected') );
        setInterval(finder, 200);

        const numbers = Array.from(Array(10000).keys());
        for(const number of numbers) {
            console.log(number);
            await collection.insertOne({num: number});
        }


    } catch (err) {
        console.error('error inside main: ' + JSON.stringify(err));
    }
})();

Trying to insert several JSON files to MongoDB collections using shell script as following,

#!/bin/bash

NUM=50000
for ((i=o;i<NUM;i++))
do
    mongoimport --host localhost --port 27018 -u 'admin' -p 'password' --authenticationDatabase 'admin' -d random_test -c tri_${i} /home/test/json_files/json_${i}.csv --jsonArray
done  

after several successful adding, these errors were shown on terminal

Failed: connection(localhost:27017[-3]), incomplete read of message header: EOF
error connecting to host: could not connect to server: 
server selection error: server selection timeout, 
current topology: { Type: Single, Servers: 
[{ Addr: localhost:27017, Type: Unknown, 
State: Connected, Average RTT: 0, Last error: connection() : 
dial tcp [::1]:27017: connect: connection refused }, ] }

And below the eoor messages from mongo.log, that said too many open files, can I somehow limit the thread number? or what should I do to fix it?? Thanks a lot!

2020-07-21T11:13:33.613+0200 E  STORAGE  [conn971] WiredTiger error (24) [1595322813:613873][53971:0x7f7c8d228700], WT_SESSION.create: __posix_directory_sync, 151: /home/mongodb/bin/data/db/index-969--7295385362343345274.wt: directory-sync: Too many open files Raw: [1595322813:613873][53971:0x7f7c8d228700], WT_SESSION.create: __posix_directory_sync, 151: /home/mongodb/bin/data/db/index-969--7295385362343345274.wt: directory-sync: Too many open files
2020-07-21T11:13:33.613+0200 E  STORAGE  [conn971] WiredTiger error (-31804) [1595322813:613892][53971:0x7f7c8d228700], WT_SESSION.create: __wt_panic, 490: the process must exit and restart: WT_PANIC: WiredTiger library panic Raw: [1595322813:613892][53971:0x7f7c8d228700], WT_SESSION.create: __wt_panic, 490: the process must exit and restart: WT_PANIC: WiredTiger library panic
2020-07-21T11:13:33.613+0200 F  -        [conn971] Fatal Assertion 50853 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 414
2020-07-21T11:13:33.613+0200 F  -        [conn971]

***aborting after fassert() failure

Connections

You can connect to MongoDB with the mongoose.connect() method.

mongoose.connect('mongodb://127.0.0.1:27017/myapp');

This is the minimum needed to connect the myapp database running locally
on the default port (27017). If connecting fails on your machine, try using
127.0.0.1 instead of localhost.

You can also specify several more parameters in the uri:

mongoose.connect('mongodb://username:password@host:port/database?options...');

See the mongodb connection string spec for more details.

• Buffering
• Error Handling
• Options
• Connection String Options
• Connection Events
• A note about keepAlive
• Server Selection
• Replica Set Connections
• Replica Set Host Names
• Multi-mongos support
• Multiple connections
• Connection Pools

Operation Buffering

Mongoose lets you start using your models immediately, without waiting for
mongoose to establish a connection to MongoDB.

mongoose.connect('mongodb://127.0.0.1:27017/myapp');
const MyModel = mongoose.model('Test', new Schema({ name: String }));
// Works
MyModel.findOne(function(error, result) { /* ... */ });

That’s because mongoose buffers model function calls internally. This
buffering is convenient, but also a common source of confusion. Mongoose
will not throw any errors by default if you use a model without
connecting.

const MyModel = mongoose.model('Test', new Schema({ name: String }));
// Will just hang until mongoose successfully connects
MyModel.findOne(function(error, result) { /* ... */ });

setTimeout(function() {
  mongoose.connect('mongodb://127.0.0.1:27017/myapp');
}, 60000);

To disable buffering, turn off the bufferCommands option on your schema.
If you have bufferCommands on and your connection is hanging, try turning
bufferCommands off to see if you haven’t opened a connection properly.
You can also disable bufferCommands globally:

mongoose.set('bufferCommands', false);

Note that buffering is also responsible for waiting until Mongoose
creates collections if you use the autoCreate option.
If you disable buffering, you should also disable the autoCreate
option and use createCollection()
to create capped collections or
collections with collations.

const schema = new Schema({
  name: String
}, {
  capped: { size: 1024 },
  bufferCommands: false,
  autoCreate: false // disable `autoCreate` since `bufferCommands` is false
});

const Model = mongoose.model('Test', schema);
// Explicitly create the collection before using it
// so the collection is capped.
await Model.createCollection();

Error Handling

There are two classes of errors that can occur with a Mongoose connection.

• Error on initial connection. If initial connection fails, Mongoose will emit an ‘error’ event and the promise mongoose.connect() returns will reject. However, Mongoose will not automatically try to reconnect.
• Error after initial connection was established. Mongoose will attempt to reconnect, and it will emit an ‘error’ event.

To handle initial connection errors, you should use .catch() or try/catch with async/await.

mongoose.connect('mongodb://127.0.0.1:27017/test').
  catch(error => handleError(error));

// Or:
try {
  await mongoose.connect('mongodb://127.0.0.1:27017/test');
} catch (error) {
  handleError(error);
}

To handle errors after initial connection was established, you should
listen for error events on the connection. However, you still need to
handle initial connection errors as shown above.

mongoose.connection.on('error', err => {
  logError(err);
});

Note that Mongoose does not necessarily emit an ‘error’ event if it loses connectivity to MongoDB. You should
listen to the disconnected event to report when Mongoose is disconnected from MongoDB.

Options

The connect method also accepts an options object which will be passed
on to the underlying MongoDB driver.

mongoose.connect(uri, options);

A full list of options can be found on the MongoDB Node.js driver docs for MongoClientOptions.
Mongoose passes options to the driver without modification, modulo a few
exceptions that are explained below.

• bufferCommands — This is a mongoose-specific option (not passed to the MongoDB driver) that disables Mongoose’s buffering mechanism
• user/pass — The username and password for authentication. These options are Mongoose-specific, they are equivalent to the MongoDB driver’s auth.username and auth.password options.
• autoIndex — By default, mongoose will automatically build indexes defined in your schema when it connects. This is great for development, but not ideal for large production deployments, because index builds can cause performance degradation. If you set autoIndex to false, mongoose will not automatically build indexes for any model associated with this connection.
• dbName — Specifies which database to connect to and overrides any database specified in the connection string. This is useful if you are unable to specify a default database in the connection string like with some mongodb+srv syntax connections.

Below are some of the options that are important for tuning Mongoose.

• promiseLibrary — Sets the underlying driver’s promise library.
• maxPoolSize — The maximum number of sockets the MongoDB driver will keep open for this connection. By default, maxPoolSize is 100. Keep in mind that MongoDB only allows one operation per socket at a time, so you may want to increase this if you find you have a few slow queries that are blocking faster queries from proceeding. See Slow Trains in MongoDB and Node.js. You may want to decrease maxPoolSize if you are running into connection limits.
• minPoolSize — The minimum number of sockets the MongoDB driver will keep open for this connection. The MongoDB driver may close sockets that have been inactive for some time. You may want to increase minPoolSize if you expect your app to go through long idle times and want to make sure your sockets stay open to avoid slow trains when activity picks up.
• socketTimeoutMS — How long the MongoDB driver will wait before killing a socket due to inactivity after initial connection. A socket may be inactive because of either no activity or a long-running operation. This is set to 30000 by default, you should set this to 2-3x your longest running operation if you expect some of your database operations to run longer than 20 seconds. This option is passed to Node.js socket#setTimeout() function after the MongoDB driver successfully completes.
• family — Whether to connect using IPv4 or IPv6. This option passed to Node.js’ dns.lookup() function. If you don’t specify this option, the MongoDB driver will try IPv6 first and then IPv4 if IPv6 fails. If your mongoose.connect(uri) call takes a long time, try mongoose.connect(uri, { family: 4 })
• authSource — The database to use when authenticating with user and pass. In MongoDB, users are scoped to a database. If you are getting an unexpected login failure, you may need to set this option.
• serverSelectionTimeoutMS — The MongoDB driver will try to find a server to send any given operation to, and keep retrying for serverSelectionTimeoutMS milliseconds. If not set, the MongoDB driver defaults to using 30000 (30 seconds).
• heartbeatFrequencyMS — The MongoDB driver sends a heartbeat every heartbeatFrequencyMS to check on the status of the connection. A heartbeat is subject to serverSelectionTimeoutMS, so the MongoDB driver will retry failed heartbeats for up to 30 seconds by default. Mongoose only emits a 'disconnected' event after a heartbeat has failed, so you may want to decrease this setting to reduce the time between when your server goes down and when Mongoose emits 'disconnected'. We recommend you do not set this setting below 1000, too many heartbeats can lead to performance degradation.

The serverSelectionTimeoutMS option also handles how long mongoose.connect() will
retry initial connection before erroring out. mongoose.connect()
will retry for 30 seconds by default (default serverSelectionTimeoutMS) before
erroring out. To get faster feedback on failed operations, you can reduce serverSelectionTimeoutMS
to 5000 as shown below.

Example:

const options = {
  autoIndex: false, // Don't build indexes
  maxPoolSize: 10, // Maintain up to 10 socket connections
  serverSelectionTimeoutMS: 5000, // Keep trying to send operations for 5 seconds
  socketTimeoutMS: 45000, // Close sockets after 45 seconds of inactivity
  family: 4 // Use IPv4, skip trying IPv6
};
mongoose.connect(uri, options);

See this page for more information about connectTimeoutMS and socketTimeoutMS

Callback

The connect() function also accepts a callback parameter and returns a
promise.

mongoose.connect(uri, options, function(error) {
  // Check error in initial connection. There is no 2nd param to the callback.
});

// Or using promises
mongoose.connect(uri, options).then(
  () => { /** ready to use. The `mongoose.connect()` promise resolves to mongoose instance. */ },
  err => { /** handle initial connection error */ }
);

Connection String Options

You can also specify driver options in your connection string as
parameters in the query string
portion of the URI. This only applies to options passed to the MongoDB
driver. You can’t set Mongoose-specific options like bufferCommands
in the query string.

mongoose.connect('mongodb://127.0.0.1:27017/test?connectTimeoutMS=1000&bufferCommands=false&authSource=otherdb');
// The above is equivalent to:
mongoose.connect('mongodb://127.0.0.1:27017/test', {
  connectTimeoutMS: 1000
  // Note that mongoose will **not** pull `bufferCommands` from the query string
});

The disadvantage of putting options in the query string is that query
string options are harder to read. The advantage is that you only need a
single configuration option, the URI, rather than separate options for
socketTimeoutMS, connectTimeoutMS, etc. Best practice is to put options
that likely differ between development and production, like replicaSet
or ssl, in the connection string, and options that should remain constant,
like connectTimeoutMS or maxPoolSize, in the options object.

The MongoDB docs have a full list of
supported connection string options.
Below are some options that are often useful to set in the connection string because they
are closely associated with the hostname and authentication information.

• authSource — The database to use when authenticating with user and pass. In MongoDB, users are scoped to a database. If you are getting an unexpected login failure, you may need to set this option.
• family — Whether to connect using IPv4 or IPv6. This option passed to Node.js’ dns.lookup() function. If you don’t specify this option, the MongoDB driver will try IPv6 first and then IPv4 if IPv6 fails. If your mongoose.connect(uri) call takes a long time, try mongoose.connect(uri, { family: 4 })

Connection Events

Connections inherit from Node.js’ EventEmitter class,
and emit events when something happens to the connection, like losing
connectivity to the MongoDB server. Below is a list of events that a
connection may emit.

• connecting: Emitted when Mongoose starts making its initial connection to the MongoDB server
• connected: Emitted when Mongoose successfully makes its initial connection to the MongoDB server, or when Mongoose reconnects after losing connectivity. May be emitted multiple times if Mongoose loses connectivity.
• open: Emitted after 'connected' and onOpen is executed on all of this connection’s models.
• disconnecting: Your app called Connection#close() to disconnect from MongoDB
• disconnected: Emitted when Mongoose lost connection to the MongoDB server. This event may be due to your code explicitly closing the connection, the database server crashing, or network connectivity issues.
• close: Emitted after Connection#close() successfully closes the connection. If you call conn.close(), you’ll get both a ‘disconnected’ event and a ‘close’ event.
• reconnected: Emitted if Mongoose lost connectivity to MongoDB and successfully reconnected. Mongoose attempts to automatically reconnect when it loses connection to the database.
• error: Emitted if an error occurs on a connection, like a parseError due to malformed data or a payload larger than 16MB.
• fullsetup: Emitted when you’re connecting to a replica set and Mongoose has successfully connected to the primary and at least one secondary.
• all: Emitted when you’re connecting to a replica set and Mongoose has successfully connected to all servers specified in your connection string.

When you’re connecting to a single MongoDB server (a «standalone»), Mongoose will emit ‘disconnected’ if it gets
disconnected from the standalone server, and ‘connected’ if it successfully connects to the standalone. In a
replica set, Mongoose will emit ‘disconnected’ if it loses connectivity to the replica set primary, and ‘connected’ if it manages to reconnect to the replica set primary.

A note about keepAlive

For long running applications, it is often prudent to enable keepAlive
with a number of milliseconds. Without it, after some period of time
you may start to see "connection closed" errors for what seems like
no reason. If so, after
reading this,
you may decide to enable keepAlive:

mongoose.connect(uri, { keepAlive: true, keepAliveInitialDelay: 300000 });

keepAliveInitialDelay is the number of milliseconds to wait before initiating keepAlive on the socket.
keepAlive is true by default since mongoose 5.2.0.

Replica Set Connections

To connect to a replica set you pass a comma delimited list of hosts to
connect to rather than a single host.

mongoose.connect('mongodb://[username:password@]host1[:port1][,host2[:port2],...[,hostN[:portN]]][/[database][?options]]' [, options]);

For example:

mongoose.connect('mongodb://user:pw@host1.com:27017,host2.com:27017,host3.com:27017/testdb');

To connect to a single node replica set, specify the replicaSet option.

mongoose.connect('mongodb://host1:port1/?replicaSet=rsName');

Server Selection

The underlying MongoDB driver uses a process known as server selection to connect to MongoDB and send operations to MongoDB.
If the MongoDB driver can’t find a server to send an operation to after serverSelectionTimeoutMS,
you’ll get the below error:

MongoTimeoutError: Server selection timed out after 30000 ms

You can configure the timeout using the serverSelectionTimeoutMS option
to mongoose.connect():

mongoose.connect(uri, {
  serverSelectionTimeoutMS: 5000 // Timeout after 5s instead of 30s
});

A MongoTimeoutError has a reason property that explains why
server selection timed out. For example, if you’re connecting to
a standalone server with an incorrect password, reason
will contain an «Authentication failed» error.

const mongoose = require('mongoose');

const uri = 'mongodb+srv://username:badpw@cluster0-OMITTED.mongodb.net/' +
  'test?retryWrites=true&w=majority';
// Prints "MongoServerError: bad auth Authentication failed."
mongoose.connect(uri, {
  serverSelectionTimeoutMS: 5000
}).catch(err => console.log(err.reason));

Replica Set Host Names

MongoDB replica sets rely on being able to reliably figure out the domain name for each member.
On Linux and OSX, the MongoDB server uses the output of the hostname command to figure out the domain name to report to the replica set.
This can cause confusing errors if you’re connecting to a remote MongoDB replica set running on a machine that reports its hostname as localhost:

// Can get this error even if your connection string doesn't include
// `localhost` if `rs.conf()` reports that one replica set member has
// `localhost` as its host name.
MongooseServerSelectionError: connect ECONNREFUSED localhost:27017

If you’re experiencing a similar error, connect to the replica set using the mongo shell and run the rs.conf() command to check the host names of each replica set member.
Follow this page’s instructions to change a replica set member’s host name.

You can also check the reason.servers property of MongooseServerSelectionError to see what the MongoDB Node driver thinks the state of your replica set is.
The reason.servers property contains a map of server descriptions.

if (err.name === 'MongooseServerSelectionError') {
  // Contains a Map describing the state of your replica set. For example:
  // Map(1) {
  //   'localhost:27017' => ServerDescription {
  //     address: 'localhost:27017',
  //     type: 'Unknown',
  //     ...
  //   }
  // }
  console.log(err.reason.servers);
}

Multi-mongos support

You can also connect to multiple mongos instances
for high availability in a sharded cluster. You do
not need to pass any special options to connect to multiple mongos in mongoose 5.x.

// Connect to 2 mongos servers
mongoose.connect('mongodb://mongosA:27501,mongosB:27501', cb);

Multiple connections

So far we’ve seen how to connect to MongoDB using Mongoose’s default
connection. Mongoose creates a default connection when you call mongoose.connect().
You can access the default connection using mongoose.connection.

You may need multiple connections to MongoDB for several reasons.
One reason is if you have multiple databases or multiple MongoDB clusters.
Another reason is to work around slow trains.
The mongoose.createConnection() function takes the same arguments as
mongoose.connect() and returns a new connection.

const conn = mongoose.createConnection('mongodb://[username:password@]host1[:port1][,host2[:port2],...[,hostN[:portN]]][/[database][?options]]', options);

This connection object is then used to
create and retrieve models. Models are
always scoped to a single connection.

const UserModel = conn.model('User', userSchema);

If you use multiple connections, you should make sure you export schemas,
not models. Exporting a model from a file is called the export model pattern.
The export model pattern is limited because you can only use one connection.

const userSchema = new Schema({ name: String, email: String });

// The alternative to the export model pattern is the export schema pattern.
module.exports = userSchema;

// Because if you export a model as shown below, the model will be scoped
// to Mongoose's default connection.
// module.exports = mongoose.model('User', userSchema);

If you use the export schema pattern, you still need to create models
somewhere. There are two common patterns. First is to export a connection
and register the models on the connection in the file:

// connections/fast.js
const mongoose = require('mongoose');

const conn = mongoose.createConnection(process.env.MONGODB_URI);
conn.model('User', require('../schemas/user'));

module.exports = conn;

// connections/slow.js
const mongoose = require('mongoose');

const conn = mongoose.createConnection(process.env.MONGODB_URI);
conn.model('User', require('../schemas/user'));
conn.model('PageView', require('../schemas/pageView'));

module.exports = conn;

Another alternative is to register connections with a dependency injector
or another inversion of control (IOC) pattern.

const mongoose = require('mongoose');

module.exports = function connectionFactory() {
  const conn = mongoose.createConnection(process.env.MONGODB_URI);

  conn.model('User', require('../schemas/user'));
  conn.model('PageView', require('../schemas/pageView'));

  return conn; 
};

Connection Pools

Each connection, whether created with mongoose.connect or
mongoose.createConnection are all backed by an internal configurable
connection pool defaulting to a maximum size of 100. Adjust the pool size
using your connection options:

// With object options
mongoose.createConnection(uri, { maxPoolSize: 10 });

// With connection string options
const uri = 'mongodb://127.0.0.1:27017/test?maxPoolSize=10';
mongoose.createConnection(uri);

Next Up

Now that we’ve covered connections, let’s take a look at models.